perfSONAR User Q&A and Other Discussion

[Philippe Laurens <>]

Hi Andy,

Yes, I can confirm that the fail.local solution works
as intended.

On psmsu01 I installed jail.local, restarted fail2ban,
cleared the mail queue: zero message have been added
to the mail queue since.

psmsu02 was the control sample and just had fail2ban restarted,
the queue cleared, but the jail files were NOT touched:
more messages have appeared on its mail queue since this morning.

Cross-check:  the /var/log/messages file on both nodes
show ongoing fail2ban activity during that period.

Thanks,

Philippe


On 18-Feb-2015 11:55 AM, Philippe Laurens wrote:
> Hi Andy,
>
> Ok, thanks.  I can certainly do that, but being sure it truly helped
> will take some time as, at least for psmsu05, fail2ban was not
> generating daily emails.  The mailq is still empty since I cleared
> it yesterday. This may be because this is a machine lesser known
> to attackers.
>
> Ah! now I see that psmsu01 and 02 are getting similar daily entries
> in the mailq.  I will install the fix on one of those and we will
> have a cleaner test.
>
> I am still not sure why the psumxx machines have the same jail.conf,
> no jail.local and seemingly no messages in mailq.
> What line is no longer commented out in the fail2ban RPM?
>
> Thanks,
>
> Philippe
>
>
> On 18-Feb-2015 10:53 AM, Andrew Lake wrote:
> > Hi,
> >
> > On the hosts giving you trouble if you copy the attached file to
> > /etc/fail2ban/jail.local (note that I said copy it to jail.local NOT
> > jail.conf) and run "/sbin/service fail2ban restart" do the mail issues
> > subside? It looks like the fail2ban RPM no longer comments out a line
> > that says to send mail by default. Copying the attached file to the
> > location specified should override this default. If that helps, we
> > will squeeze it into the next toolkit release.
> >
> > Thanks,
> > Andy
> >
> > On Feb 17, 2015, at 5:26 PM, Horst Severini 
> > <>
> >  wrote:
> >
> > > Hi Philippe,
> > >
> > > yes, so we're seeing the same problem with fail2ban.
> > > Not sure why only new installs are affected by that, though,
> > > and not our production instances.
> > >
> > > The other nightly cron email is a different issue, I think I just
> > > haven't completely confugred the mesh yet, since I wasn't sure
> > > whether I should leave that up to the pundit admins to configure.
> > >
> > > So I'm sure that will go away once the mesh is configured.
> > >
> > > Thanks,
> > >
> > >     Horst
> > >
> > > Philippe Laurens 
> > > <>
> > >  wrote:
> > >
> > > > Hi Horst et al,
> > > >
> > > > I found a similar status on psmsu05.aglt2.org,
> > > > but only fail2ban messages are in the queue.
> > > >
> > > > All 164 entries were from Feb 12, 13, 14
> > > > while psmsu05 had been installed Feb 6-7 (from SL6 and RPMs)
> > > >
> > > > I checked 4-5 of these messages and all those looked like
> > > >
> > > > > [root@psmsu05
> > > > >  ~]# postcat -q E62D720DC5
> > > > >     *** ENVELOPE RECORDS deferred/E/E62D720DC5 ***
> > > > >     message_size:            2420             213
> > > > > 1               0            2420
> > > > >     message_arrival_time: Sat Feb 14 11:05:16 2015
> > > > >     create_time: Sat Feb 14 11:05:16 2015
> > > > >     named_attribute: log_message_origin=local
> > > > >     named_attribute: trace_flags=0
> > > > >     sender:
> > > > >     original_recipient: 
> > > > >
> > > > >     recipient: 
> > > > >
> > > > >     *** MESSAGE CONTENTS deferred/E/E62D720DC5 ***
> > > > >     Received: by psmsu05.aglt2.org (Postfix)
> > > > >             id E62D720DC5; Sat, 14 Feb 2015 11:05:16 -0500 (EST)
> > > > >     Date: Sat, 14 Feb 2015 11:05:16 -0500 (EST)
> > > > >     From: 
> > > > >
> > > > >  (Mail Delivery System)
> > > > >     Subject: Undelivered Mail Returned to Sender
> > > > >     To: 
> > > > >
> > > > >     Auto-Submitted: auto-replied
> > > > >     MIME-Version: 1.0
> > > > >     Content-Type: multipart/report; report-type=delivery-status;
> > > > >             boundary="7754320DC3.1423929916/psmsu05.aglt2.org"
> > > > >     Message-Id: 
> > > > > <>
> > > > >
> > > > >     This is a MIME-encapsulated message.
> > > > >
> > > > >     --7754320DC3.1423929916/psmsu05.aglt2.org
> > > > >     Content-Description: Notification
> > > > >     Content-Type: text/plain; charset=us-ascii
> > > > >
> > > > >     This is the mail system at host psmsu05.aglt2.org.
> > > > >
> > > > >     I'm sorry to have to inform you that your message could not
> > > > >     be delivered to one or more recipients. It's attached below.
> > > > >
> > > > >     For further assistance, please send mail to postmaster.
> > > > >
> > > > >     If you do so, please include this problem report. You can
> > > > >     delete your own text from the attached returned message.
> > > > >
> > > > >                        The mail system
> > > > >
> > > > >     
> > > > > <>:
> > > > >  delivery temporarily suspended: connect to
> > > > >         example.com[93.184.216.34]:25: Connection timed out
> > > > >
> > > > >     --7754320DC3.1423929916/psmsu05.aglt2.org
> > > > >     Content-Description: Delivery report
> > > > >     Content-Type: message/delivery-status
> > > > >
> > > > >     Reporting-MTA: dns; psmsu05.aglt2.org
> > > > >     X-Postfix-Queue-ID: 7754320DC3
> > > > >     X-Postfix-Sender: rfc822; 
> > > > >
> > > > >     Arrival-Date: Mon,  9 Feb 2015 10:20:46 -0500 (EST)
> > > > >
> > > > >     Final-Recipient: rfc822; 
> > > > >
> > > > >     Action: failed
> > > > >     Status: 4.4.1
> > > > >     Diagnostic-Code: X-Postfix; delivery temporarily suspended:
> > > > > connect to
> > > > >         example.com[93.184.216.34]:25: Connection timed out
> > > > >
> > > > >     --7754320DC3.1423929916/psmsu05.aglt2.org
> > > > >     Content-Description: Undelivered Message
> > > > >     Content-Type: message/rfc822
> > > > >
> > > > >     Return-Path: 
> > > > > <>
> > > > >     Received: by psmsu05.aglt2.org (Postfix, from userid 0)
> > > > >             id 7754320DC3; Mon,  9 Feb 2015 10:20:46 -0500 (EST)
> > > > >     Subject: [Fail2Ban] SSH: banned 103.41.124.13 from
> > > > > psmsu05.aglt2.org
> > > > >     Date: Mon, 09 Feb 2015 15:20:46 +0000
> > > > >     From: Fail2Ban 
> > > > > <>
> > > > >     To: 
> > > > >
> > > > >     Message-Id: 
> > > > > <>
> > > > >
> > > > >     Hi,
> > > > >
> > > > >     The IP 103.41.124.13 has just been banned by Fail2Ban after
> > > > >     5 attempts against SSH.
> > > > >
> > > > >
> > > > >     Here is more information about 103.41.124.13:
> > > > >
> > > > >     missing whois program
> > > > >
> > > > >     Regards,
> > > > >
> > > > >     Fail2Ban
> > > > >
> > > > >     --7754320DC3.1423929916/psmsu05.aglt2.org--
> > > > >     *** HEADER EXTRACTED deferred/E/E62D720DC5 ***
> > > > >     *** MESSAGE FILE END deferred/E/E62D720DC5 ***
> > > >
> > > > I know very little about fail2ban... I looked around the config
> > > > files, but could not find where it could be told to send emails.
> > > > I compared a couple likely candidate config files with another
> > > > instance (psum05) and could not find any difference either.
> > > >
> > > >
> > > > I restarted fail2ban (which put 2 more messages in the queue),
> > > > cleared the queue, and will be watching.
> > > >
> > > > Thanks,
> > > >
> > > > Philippe
> > > >
> > > >
> > > > On 17-Feb-2015 4:10 PM, Horst Severini wrote:
> > > > > Hi all,
> > > > >
> > > > > as we have just discussed in the throughput meeting, we are seeing
> > > > > some strange emails in our newly installed 3.4.1 test machine,
> > > > > ps3.ochep.ou.edu .
> > > > >
> > > > > One is that fail2ban is now trying to send email to 
> > > > > ,
> > > > > claiming to be from 
> > > > > ,
> > > > >  so there must be some
> > > > > incorrect
> > > > > initial configuration going on during the installation. There's not
> > > > > a new
> > > > > version of fail2ban out, though, it's from last August:
> > > > >
> > > > > fail2ban-0.8.14-1.el6.noarch
> > > > >
> > > > > But something else must've changed in the mail configuration
> > > > > somehow ...
> > > > >
> > > > > The other thing is that the nightly cron job cron-service_watcher,
> > > > > which runs
> > > > > /opt/perfsonar_ps/toolkit/scripts/watcher_log_archive_cleanup,
> > > > > apparently wasn't quite configured correctly, either, since it still
> > > > > contains
> > > > >
> > > > > STORE_LOCATION=/mnt/store/NPTools/debug
> > > > >
> > > > > which is causing a nightly error mail about that directory not
> > > > > being there.
> > > > >
> > > > > Did I miss a configuration step there somehow?
> > > > >
> > > > > I installed this machine about a week ago, on one of our old KOI boxes
> > > > > which used to be our production PS boxes, before we upgraded to the
> > > > > R310s..
> > > > >
> > > > > Oh, and the other strange thing is that I was trying to install the
> > > > > other
> > > > > KOI box, it wouldn't let me, since during the PS installation,
> > > > > when it comes time to choose where to install the OS on,
> > > > > anaconda can't find the 160 GB hard drive, so there's no way to
> > > > > continue
> > > > > the install.
> > > > >
> > > > > Which makes no sense at all, since I can still boot up the old PS-3.2
> > > > > instance from the hard drive just fine, so there's obviously
> > > > > nothing wrong
> > > > > with the hard drive at all.
> > > > >
> > > > > I also verified all the BIOS settings, they're identical between the
> > > > > two KOI boxes, so why would one of them allow me to install on the
> > > > > hard drive,
> > > > > and the other one won't find it at that time?
> > > > >
> > > > > How can I debug this? We don't immediately need it right now,
> > > > > but I'd like to be able to install a test bandwidth node there at
> > > > > some point,
> > > > > to have a full test set.
> > > > >
> > > > > Thanks,
> > > > >
> > > > >     Horst