perfSONAR User Q&A and Other Discussion

[Horst Severini <>]

Okay,

I decided to blow away all these deferred emails as well, since I can 
always recreate another undeliverable email myself if we wanted to track 
down the reason for these future time stamps.

/var/log/messages hasn't logged another fail2ban event since noon today, 
so we'll have to wait till another one happens to see whether this 
jail.local did the trick, but I'm pretty confident that it did!

Thanks,

        Horst

On 02/18/2015 03:16 PM, Andrew Lake wrote:
> Excellent, thanks!
>
> Andy
>
>
> On Feb 18, 2015, at 3:44 PM, Philippe 
> Laurens<>
>   wrote:
>
> > >Hi Andy,
> > >
> > >Yes, I can confirm that the fail.local solution works
> > >as intended.
> > >
> > >On psmsu01 I installed jail.local, restarted fail2ban,
> > >cleared the mail queue: zero message have been added
> > >to the mail queue since.
> > >
> > >psmsu02 was the control sample and just had fail2ban restarted,
> > >the queue cleared, but the jail files were NOT touched:
> > >more messages have appeared on its mail queue since this morning.
> > >
> > >Cross-check:  the /var/log/messages file on both nodes
> > >show ongoing fail2ban activity during that period.
> > >
> > >Thanks,
> > >
> > >Philippe
> > >
> > >
> > >On 18-Feb-2015 11:55 AM, Philippe Laurens wrote:
> > > >>Hi Andy,
> > > >>
> > > >>Ok, thanks.  I can certainly do that, but being sure it truly helped
> > > >>will take some time as, at least for psmsu05, fail2ban was not
> > > >>generating daily emails.  The mailq is still empty since I cleared
> > > >>it yesterday. This may be because this is a machine lesser known
> > > >>to attackers.
> > > >>
> > > >>Ah! now I see that psmsu01 and 02 are getting similar daily entries
> > > >>in the mailq.  I will install the fix on one of those and we will
> > > >>have a cleaner test.
> > > >>
> > > >>I am still not sure why the psumxx machines have the same jail.conf,
> > > >>no jail.local and seemingly no messages in mailq.
> > > >>What line is no longer commented out in the fail2ban RPM?
> > > >>
> > > >>Thanks,
> > > >>
> > > >>Philippe
> > > >>
> > > >>
> > > >>On 18-Feb-2015 10:53 AM, Andrew Lake wrote:
> > > > >>>Hi,
> > > > >>>
> > > > >>>On the hosts giving you trouble if you copy the attached file to
> > > > >>>/etc/fail2ban/jail.local (note that I said copy it to jail.local NOT
> > > > >>>jail.conf) and run "/sbin/service fail2ban restart" do the mail issues
> > > > >>>subside? It looks like the fail2ban RPM no longer comments out a line
> > > > >>>that says to send mail by default. Copying the attached file to the
> > > > >>>location specified should override this default. If that helps, we
> > > > >>>will squeeze it into the next toolkit release.
> > > > >>>
> > > > >>>Thanks,
> > > > >>>Andy
> > > > >>>
> > > > >>>On Feb 17, 2015, at 5:26 PM, Horst 
> > > > Severini<>
> > > >   wrote:
> > > > >>>
> > > > > >>>>Hi Philippe,
> > > > > >>>>
> > > > > >>>>yes, so we're seeing the same problem with fail2ban.
> > > > > >>>>Not sure why only new installs are affected by that, though,
> > > > > >>>>and not our production instances.
> > > > > >>>>
> > > > > >>>>The other nightly cron email is a different issue, I think I just
> > > > > >>>>haven't completely confugred the mesh yet, since I wasn't sure
> > > > > >>>>whether I should leave that up to the pundit admins to configure.
> > > > > >>>>
> > > > > >>>>So I'm sure that will go away once the mesh is configured.
> > > > > >>>>
> > > > > >>>>Thanks,
> > > > > >>>>
> > > > > >>>>    Horst
> > > > > >>>>
> > > > > >>>>Philippe 
> > > > > Laurens<>
> > > > >   wrote:
> > > > > >>>>
> > > > > > >>>>>Hi Horst et al,
> > > > > > >>>>>
> > > > > > >>>>>I found a similar status on psmsu05.aglt2.org,
> > > > > > >>>>>but only fail2ban messages are in the queue.
> > > > > > >>>>>
> > > > > > >>>>>All 164 entries were from Feb 12, 13, 14
> > > > > > >>>>>while psmsu05 had been installed Feb 6-7 (from SL6 and RPMs)
> > > > > > >>>>>
> > > > > > >>>>>I checked 4-5 of these messages and all those looked like
> > > > > > >>>>>
> > > > > > > >>>>>>[root@psmsu05
> > > > > > >  ~]# postcat -q E62D720DC5
> > > > > > > >>>>>>    *** ENVELOPE RECORDS deferred/E/E62D720DC5 ***
> > > > > > > >>>>>>    message_size:            2420             213
> > > > > > > >>>>>>1               0            2420
> > > > > > > >>>>>>    message_arrival_time: Sat Feb 14 11:05:16 2015
> > > > > > > >>>>>>    create_time: Sat Feb 14 11:05:16 2015
> > > > > > > >>>>>>    named_attribute: log_message_origin=local
> > > > > > > >>>>>>    named_attribute: trace_flags=0
> > > > > > > >>>>>>    sender:
> > > > > > > >>>>>>    
> > > > > > > original_recipient:
> > > > > > > >>>>>>    
> > > > > > > recipient:
> > > > > > > >>>>>>    *** MESSAGE CONTENTS deferred/E/E62D720DC5 ***
> > > > > > > >>>>>>    Received: by psmsu05.aglt2.org (Postfix)
> > > > > > > >>>>>>            id E62D720DC5; Sat, 14 Feb 2015 11:05:16 -0500 (EST)
> > > > > > > >>>>>>    Date: Sat, 14 Feb 2015 11:05:16 -0500 (EST)
> > > > > > > >>>>>>    
> > > > > > > From:
> > > > > > >   (Mail Delivery System)
> > > > > > > >>>>>>    Subject: Undelivered Mail Returned to Sender
> > > > > > > >>>>>>    
> > > > > > > To:
> > > > > > > >>>>>>    Auto-Submitted: auto-replied
> > > > > > > >>>>>>    MIME-Version: 1.0
> > > > > > > >>>>>>    Content-Type: multipart/report; report-type=delivery-status;
> > > > > > > >>>>>>            boundary="7754320DC3.1423929916/psmsu05.aglt2.org"
> > > > > > > >>>>>>    
> > > > > > > Message-Id:<>
> > > > > > > >>>>>>
> > > > > > > >>>>>>    This is a MIME-encapsulated message.
> > > > > > > >>>>>>
> > > > > > > >>>>>>    --7754320DC3.1423929916/psmsu05.aglt2.org
> > > > > > > >>>>>>    Content-Description: Notification
> > > > > > > >>>>>>    Content-Type: text/plain; charset=us-ascii
> > > > > > > >>>>>>
> > > > > > > >>>>>>    This is the mail system at host psmsu05.aglt2.org.
> > > > > > > >>>>>>
> > > > > > > >>>>>>    I'm sorry to have to inform you that your message could not
> > > > > > > >>>>>>    be delivered to one or more recipients. It's attached below.
> > > > > > > >>>>>>
> > > > > > > >>>>>>    For further assistance, please send mail to postmaster.
> > > > > > > >>>>>>
> > > > > > > >>>>>>    If you do so, please include this problem report. You can
> > > > > > > >>>>>>    delete your own text from the attached returned message.
> > > > > > > >>>>>>
> > > > > > > >>>>>>                       The mail system
> > > > > > > >>>>>>
> > > > > > > >>>>>>    
> > > > > > > <>:
> > > > > > >  delivery temporarily suspended: connect to
> > > > > > > >>>>>>        example.com[93.184.216.34]:25: Connection timed out
> > > > > > > >>>>>>
> > > > > > > >>>>>>    --7754320DC3.1423929916/psmsu05.aglt2.org
> > > > > > > >>>>>>    Content-Description: Delivery report
> > > > > > > >>>>>>    Content-Type: message/delivery-status
> > > > > > > >>>>>>
> > > > > > > >>>>>>    Reporting-MTA: dns; psmsu05.aglt2.org
> > > > > > > >>>>>>    X-Postfix-Queue-ID: 7754320DC3
> > > > > > > >>>>>>    X-Postfix-Sender: 
> > > > > > > rfc822;
> > > > > > > >>>>>>    Arrival-Date: Mon,  9 Feb 2015 10:20:46 -0500 (EST)
> > > > > > > >>>>>>
> > > > > > > >>>>>>    Final-Recipient: 
> > > > > > > rfc822;
> > > > > > > >>>>>>    Action: failed
> > > > > > > >>>>>>    Status: 4.4.1
> > > > > > > >>>>>>    Diagnostic-Code: X-Postfix; delivery temporarily suspended:
> > > > > > > >>>>>>connect to
> > > > > > > >>>>>>        example.com[93.184.216.34]:25: Connection timed out
> > > > > > > >>>>>>
> > > > > > > >>>>>>    --7754320DC3.1423929916/psmsu05.aglt2.org
> > > > > > > >>>>>>    Content-Description: Undelivered Message
> > > > > > > >>>>>>    Content-Type: message/rfc822
> > > > > > > >>>>>>
> > > > > > > >>>>>>    
> > > > > > > Return-Path:<>
> > > > > > > >>>>>>    Received: by psmsu05.aglt2.org (Postfix, from userid 0)
> > > > > > > >>>>>>            id 7754320DC3; Mon,  9 Feb 2015 10:20:46 -0500 (EST)
> > > > > > > >>>>>>    Subject: [Fail2Ban] SSH: banned 103.41.124.13 from
> > > > > > > >>>>>>psmsu05.aglt2.org
> > > > > > > >>>>>>    Date: Mon, 09 Feb 2015 15:20:46 +0000
> > > > > > > >>>>>>    From: 
> > > > > > > Fail2Ban<>
> > > > > > > >>>>>>    
> > > > > > > To:
> > > > > > > >>>>>>    
> > > > > > > Message-Id:<>
> > > > > > > >>>>>>
> > > > > > > >>>>>>    Hi,
> > > > > > > >>>>>>
> > > > > > > >>>>>>    The IP 103.41.124.13 has just been banned by Fail2Ban after
> > > > > > > >>>>>>    5 attempts against SSH.
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>>    Here is more information about 103.41.124.13:
> > > > > > > >>>>>>
> > > > > > > >>>>>>    missing whois program
> > > > > > > >>>>>>
> > > > > > > >>>>>>    Regards,
> > > > > > > >>>>>>
> > > > > > > >>>>>>    Fail2Ban
> > > > > > > >>>>>>
> > > > > > > >>>>>>    --7754320DC3.1423929916/psmsu05.aglt2.org--
> > > > > > > >>>>>>    *** HEADER EXTRACTED deferred/E/E62D720DC5 ***
> > > > > > > >>>>>>    *** MESSAGE FILE END deferred/E/E62D720DC5 ***
> > > > > > >>>>>
> > > > > > >>>>>I know very little about fail2ban... I looked around the config
> > > > > > >>>>>files, but could not find where it could be told to send emails.
> > > > > > >>>>>I compared a couple likely candidate config files with another
> > > > > > >>>>>instance (psum05) and could not find any difference either.
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>>I restarted fail2ban (which put 2 more messages in the queue),
> > > > > > >>>>>cleared the queue, and will be watching.
> > > > > > >>>>>
> > > > > > >>>>>Thanks,
> > > > > > >>>>>
> > > > > > >>>>>Philippe
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>>On 17-Feb-2015 4:10 PM, Horst Severini wrote:
> > > > > > > >>>>>>Hi all,
> > > > > > > >>>>>>
> > > > > > > >>>>>>as we have just discussed in the throughput meeting, we are seeing
> > > > > > > >>>>>>some strange emails in our newly installed 3.4.1 test machine,
> > > > > > > >>>>>>ps3.ochep.ou.edu .
> > > > > > > >>>>>>
> > > > > > > >>>>>>One is that fail2ban is now trying to send email 
> > > > > > > ,
> > > > > > > >>>>>>claiming to be 
> > > > > > > ,
> > > > > > >  so there must be some
> > > > > > > >>>>>>incorrect
> > > > > > > >>>>>>initial configuration going on during the installation. There's not
> > > > > > > >>>>>>a new
> > > > > > > >>>>>>version of fail2ban out, though, it's from last August:
> > > > > > > >>>>>>
> > > > > > > >>>>>>fail2ban-0.8.14-1.el6.noarch
> > > > > > > >>>>>>
> > > > > > > >>>>>>But something else must've changed in the mail configuration
> > > > > > > >>>>>>somehow ...
> > > > > > > >>>>>>
> > > > > > > >>>>>>The other thing is that the nightly cron job cron-service_watcher,
> > > > > > > >>>>>>which runs
> > > > > > > >>>>>>/opt/perfsonar_ps/toolkit/scripts/watcher_log_archive_cleanup,
> > > > > > > >>>>>>apparently wasn't quite configured correctly, either, since it still
> > > > > > > >>>>>>contains
> > > > > > > >>>>>>
> > > > > > > >>>>>>STORE_LOCATION=/mnt/store/NPTools/debug
> > > > > > > >>>>>>
> > > > > > > >>>>>>which is causing a nightly error mail about that directory not
> > > > > > > >>>>>>being there.
> > > > > > > >>>>>>
> > > > > > > >>>>>>Did I miss a configuration step there somehow?
> > > > > > > >>>>>>
> > > > > > > >>>>>>I installed this machine about a week ago, on one of our old KOI boxes
> > > > > > > >>>>>>which used to be our production PS boxes, before we upgraded to the
> > > > > > > >>>>>>R310s..
> > > > > > > >>>>>>
> > > > > > > >>>>>>Oh, and the other strange thing is that I was trying to install the
> > > > > > > >>>>>>other
> > > > > > > >>>>>>KOI box, it wouldn't let me, since during the PS installation,
> > > > > > > >>>>>>when it comes time to choose where to install the OS on,
> > > > > > > >>>>>>anaconda can't find the 160 GB hard drive, so there's no way to
> > > > > > > >>>>>>continue
> > > > > > > >>>>>>the install.
> > > > > > > >>>>>>
> > > > > > > >>>>>>Which makes no sense at all, since I can still boot up the old PS-3.2
> > > > > > > >>>>>>instance from the hard drive just fine, so there's obviously
> > > > > > > >>>>>>nothing wrong
> > > > > > > >>>>>>with the hard drive at all.
> > > > > > > >>>>>>
> > > > > > > >>>>>>I also verified all the BIOS settings, they're identical between the
> > > > > > > >>>>>>two KOI boxes, so why would one of them allow me to install on the
> > > > > > > >>>>>>hard drive,
> > > > > > > >>>>>>and the other one won't find it at that time?
> > > > > > > >>>>>>
> > > > > > > >>>>>>How can I debug this? We don't immediately need it right now,
> > > > > > > >>>>>>but I'd like to be able to install a test bandwidth node there at
> > > > > > > >>>>>>some point,
> > > > > > > >>>>>>to have a full test set.
> > > > > > > >>>>>>
> > > > > > > >>>>>>Thanks,
> > > > > > > >>>>>>
> > > > > > > >>>>>>    Horst
> > > > > > > >>>>>>
> > > > >>>