perfSONAR User Q&A and Other Discussion

[Horst Severini <>]

Hi Philippe,

I just had a closer look at our production host, ps1 and ps2, and the 
latest fail2ban-0.8.14-1 from last August did create a jail.conf.rpmnew, 
which does indeed not comment out this line anymore:

----
99c77
<            sendmail-whois[name=SSH, , 
, sendername="Fail2Ban"]
---
> #           sendmail-whois[name=SSH, dest=root, 
, sendername="Fail2Ban"]
----

So this probably means that all hosts which were installed before last 
August weren't affected by that, since the .rpmnew file didn't take 
effect, but all freshly installed hosts since then are incorrectly 
configured to send unconfigured emails?

Then again, it looks like the old production installs also commented 
this out during the initial install, since we have all these files here:

----
-rw-r--r--    1 root 13769 Feb  4  2014 jail.conf
-rw-r--r--    1 root 13768 Dec  3  2013 jail.conf.orig
-rw-r--r--    1 root 19313 Jul 29  2014 jail.conf.rpmnew
-rw-r--r--    1 root 13769 Feb  4  2014 jail.conf.save
-rw-r--r--    1 root 13768 Dec  3  2013 jail.conf~
----

So some post-install script seems to have made a .orig backup, then 
commented this line out with a 'sed -i~' command (or equivalent), and 
then made another .save backup. Why is this no longer being done in new 
installs now? Did something in the rpmnew file break the 'sed' command 
maybe? Just guessing here ...

Cheers,

        Horst

On 02/18/2015 10:55 AM, Philippe Laurens wrote:
> Hi Andy,
>
> Ok, thanks.  I can certainly do that, but being sure it truly helped
> will take some time as, at least for psmsu05, fail2ban was not
> generating daily emails.  The mailq is still empty since I cleared
> it yesterday. This may be because this is a machine lesser known
> to attackers.
>
> Ah! now I see that psmsu01 and 02 are getting similar daily entries
> in the mailq.  I will install the fix on one of those and we will
> have a cleaner test.
>
> I am still not sure why the psumxx machines have the same jail.conf,
> no jail.local and seemingly no messages in mailq.
> What line is no longer commented out in the fail2ban RPM?
>
> Thanks,
>
> Philippe
>
>
> On 18-Feb-2015 10:53 AM, Andrew Lake wrote:
> > Hi,
> >
> > On the hosts giving you trouble if you copy the attached file to
> > /etc/fail2ban/jail.local (note that I said copy it to jail.local NOT
> > jail.conf) and run "/sbin/service fail2ban restart" do the mail issues
> > subside? It looks like the fail2ban RPM no longer comments out a line
> > that says to send mail by default. Copying the attached file to the
> > location specified should override this default. If that helps, we
> > will squeeze it into the next toolkit release.
> >
> > Thanks,
> > Andy
> >
> > On Feb 17, 2015, at 5:26 PM, Horst Severini 
> > <>
> >  wrote:
> >
> > > Hi Philippe,
> > >
> > > yes, so we're seeing the same problem with fail2ban.
> > > Not sure why only new installs are affected by that, though,
> > > and not our production instances.
> > >
> > > The other nightly cron email is a different issue, I think I just
> > > haven't completely confugred the mesh yet, since I wasn't sure
> > > whether I should leave that up to the pundit admins to configure.
> > >
> > > So I'm sure that will go away once the mesh is configured.
> > >
> > > Thanks,
> > >
> > >     Horst
> > >
> > > Philippe Laurens 
> > > <>
> > >  wrote:
> > >
> > > > Hi Horst et al,
> > > >
> > > > I found a similar status on psmsu05.aglt2.org,
> > > > but only fail2ban messages are in the queue.
> > > >
> > > > All 164 entries were from Feb 12, 13, 14
> > > > while psmsu05 had been installed Feb 6-7 (from SL6 and RPMs)
> > > >
> > > > I checked 4-5 of these messages and all those looked like
> > > >
> > > > > [root@psmsu05
> > > > >  ~]# postcat -q E62D720DC5
> > > > >     *** ENVELOPE RECORDS deferred/E/E62D720DC5 ***
> > > > >     message_size:            2420             213
> > > > > 1               0            2420
> > > > >     message_arrival_time: Sat Feb 14 11:05:16 2015
> > > > >     create_time: Sat Feb 14 11:05:16 2015
> > > > >     named_attribute: log_message_origin=local
> > > > >     named_attribute: trace_flags=0
> > > > >     sender:
> > > > >     original_recipient: 
> > > > >
> > > > >     recipient: 
> > > > >
> > > > >     *** MESSAGE CONTENTS deferred/E/E62D720DC5 ***
> > > > >     Received: by psmsu05.aglt2.org (Postfix)
> > > > >             id E62D720DC5; Sat, 14 Feb 2015 11:05:16 -0500 (EST)
> > > > >     Date: Sat, 14 Feb 2015 11:05:16 -0500 (EST)
> > > > >     From: 
> > > > >
> > > > >  (Mail Delivery System)
> > > > >     Subject: Undelivered Mail Returned to Sender
> > > > >     To: 
> > > > >
> > > > >     Auto-Submitted: auto-replied
> > > > >     MIME-Version: 1.0
> > > > >     Content-Type: multipart/report; report-type=delivery-status;
> > > > >             boundary="7754320DC3.1423929916/psmsu05.aglt2.org"
> > > > >     Message-Id: 
> > > > > <>
> > > > >
> > > > >     This is a MIME-encapsulated message.
> > > > >
> > > > >     --7754320DC3.1423929916/psmsu05.aglt2.org
> > > > >     Content-Description: Notification
> > > > >     Content-Type: text/plain; charset=us-ascii
> > > > >
> > > > >     This is the mail system at host psmsu05.aglt2.org.
> > > > >
> > > > >     I'm sorry to have to inform you that your message could not
> > > > >     be delivered to one or more recipients. It's attached below.
> > > > >
> > > > >     For further assistance, please send mail to postmaster.
> > > > >
> > > > >     If you do so, please include this problem report. You can
> > > > >     delete your own text from the attached returned message.
> > > > >
> > > > >                        The mail system
> > > > >
> > > > > <>:
> > > > >  delivery temporarily suspended: connect to
> > > > >         example.com[93.184.216.34]:25: Connection timed out
> > > > >
> > > > >     --7754320DC3.1423929916/psmsu05.aglt2.org
> > > > >     Content-Description: Delivery report
> > > > >     Content-Type: message/delivery-status
> > > > >
> > > > >     Reporting-MTA: dns; psmsu05.aglt2.org
> > > > >     X-Postfix-Queue-ID: 7754320DC3
> > > > >     X-Postfix-Sender: rfc822; 
> > > > >
> > > > >     Arrival-Date: Mon,  9 Feb 2015 10:20:46 -0500 (EST)
> > > > >
> > > > >     Final-Recipient: rfc822; 
> > > > >
> > > > >     Action: failed
> > > > >     Status: 4.4.1
> > > > >     Diagnostic-Code: X-Postfix; delivery temporarily suspended:
> > > > > connect to
> > > > >         example.com[93.184.216.34]:25: Connection timed out
> > > > >
> > > > >     --7754320DC3.1423929916/psmsu05.aglt2.org
> > > > >     Content-Description: Undelivered Message
> > > > >     Content-Type: message/rfc822
> > > > >
> > > > >     Return-Path: 
> > > > > <>
> > > > >     Received: by psmsu05.aglt2.org (Postfix, from userid 0)
> > > > >             id 7754320DC3; Mon,  9 Feb 2015 10:20:46 -0500 (EST)
> > > > >     Subject: [Fail2Ban] SSH: banned 103.41.124.13 from
> > > > > psmsu05.aglt2.org
> > > > >     Date: Mon, 09 Feb 2015 15:20:46 +0000
> > > > >     From: Fail2Ban 
> > > > > <>
> > > > >     To: 
> > > > >
> > > > >     Message-Id: 
> > > > > <>
> > > > >
> > > > >     Hi,
> > > > >
> > > > >     The IP 103.41.124.13 has just been banned by Fail2Ban after
> > > > >     5 attempts against SSH.
> > > > >
> > > > >
> > > > >     Here is more information about 103.41.124.13:
> > > > >
> > > > >     missing whois program
> > > > >
> > > > >     Regards,
> > > > >
> > > > >     Fail2Ban
> > > > >
> > > > >     --7754320DC3.1423929916/psmsu05.aglt2.org--
> > > > >     *** HEADER EXTRACTED deferred/E/E62D720DC5 ***
> > > > >     *** MESSAGE FILE END deferred/E/E62D720DC5 ***
> > > >
> > > > I know very little about fail2ban... I looked around the config
> > > > files, but could not find where it could be told to send emails.
> > > > I compared a couple likely candidate config files with another
> > > > instance (psum05) and could not find any difference either.
> > > >
> > > >
> > > > I restarted fail2ban (which put 2 more messages in the queue),
> > > > cleared the queue, and will be watching.
> > > >
> > > > Thanks,
> > > >
> > > > Philippe
> > > >
> > > >
> > > > On 17-Feb-2015 4:10 PM, Horst Severini wrote:
> > > > > Hi all,
> > > > >
> > > > > as we have just discussed in the throughput meeting, we are seeing
> > > > > some strange emails in our newly installed 3.4.1 test machine,
> > > > > ps3.ochep.ou.edu .
> > > > >
> > > > > One is that fail2ban is now trying to send email to 
> > > > > ,
> > > > > claiming to be from 
> > > > > ,
> > > > >  so there must be some
> > > > > incorrect
> > > > > initial configuration going on during the installation. There's not
> > > > > a new
> > > > > version of fail2ban out, though, it's from last August:
> > > > >
> > > > > fail2ban-0.8.14-1.el6.noarch
> > > > >
> > > > > But something else must've changed in the mail configuration
> > > > > somehow ...
> > > > >
> > > > > The other thing is that the nightly cron job cron-service_watcher,
> > > > > which runs
> > > > > /opt/perfsonar_ps/toolkit/scripts/watcher_log_archive_cleanup,
> > > > > apparently wasn't quite configured correctly, either, since it still
> > > > > contains
> > > > >
> > > > > STORE_LOCATION=/mnt/store/NPTools/debug
> > > > >
> > > > > which is causing a nightly error mail about that directory not
> > > > > being there.
> > > > >
> > > > > Did I miss a configuration step there somehow?
> > > > >
> > > > > I installed this machine about a week ago, on one of our old KOI boxes
> > > > > which used to be our production PS boxes, before we upgraded to the
> > > > > R310s..
> > > > >
> > > > > Oh, and the other strange thing is that I was trying to install the
> > > > > other
> > > > > KOI box, it wouldn't let me, since during the PS installation,
> > > > > when it comes time to choose where to install the OS on,
> > > > > anaconda can't find the 160 GB hard drive, so there's no way to
> > > > > continue
> > > > > the install.
> > > > >
> > > > > Which makes no sense at all, since I can still boot up the old PS-3.2
> > > > > instance from the hard drive just fine, so there's obviously
> > > > > nothing wrong
> > > > > with the hard drive at all.
> > > > >
> > > > > I also verified all the BIOS settings, they're identical between the
> > > > > two KOI boxes, so why would one of them allow me to install on the
> > > > > hard drive,
> > > > > and the other one won't find it at that time?
> > > > >
> > > > > How can I debug this? We don't immediately need it right now,
> > > > > but I'd like to be able to install a test bandwidth node there at
> > > > > some point,
> > > > > to have a full test set.
> > > > >
> > > > > Thanks,
> > > > >
> > > > >     Horst