perfSONAR User Q&A and Other Discussion

[Horst Severini <>]

Hi Andy,

okay, I added your jail.local and restarted, let's see what happens.
I added an EOL at the end, I think that can't hurt. :)

So the fact that the modified '[ssh-iptables] action' no longer contains 
a 'sendmail-whois' statement will make it stop sending emails 
altogether? That's not all that intuitive, either. =)

By the way, I looked at the deferred email files (I haven't deleted them 
just yet), and it seems that these files are created with a time stamp 
in the future:

----
/var/spool/postfix/deferred/F:
total 24
drwx------   2 postfix 4096 Feb 18 10:28 .
drwx------. 18 postfix 4096 Feb 15 06:58 ..
-rwx------   1 postfix  757 Feb 18  2015 F03902E0EAB
-rwx------   1 postfix  754 Feb 18  2015 F044E2E1156
-rwx------   1 postfix  751 Feb 18 10:35 F12532E0DF1
-rwx------   1 postfix  757 Feb 18  2015 F184D2E118F
----

How can this happen? What process can write a file and date it in the 
future? ntp is definitely running and synchronized.

Thanks,

        Horst

On 02/18/2015 09:55 AM, Andrew Lake wrote:
> Forgot to attach file….
>
>
>
> On Feb 18, 2015, at 10:53 AM, Andrew 
> Lake<>
>   wrote:
>
> > >Hi,
> > >
> > >On the hosts giving you trouble if you copy the attached file to 
> > /etc/fail2ban/jail.local (note that I said copy it to jail.local NOT jail.conf) and run 
> > "/sbin/service fail2ban restart" do the mail issues subside? It looks like the 
> > fail2ban RPM no longer comments out a line that says to send mail by default. Copying the 
> > attached file to the location specified should override this default. If that helps, we 
> > will squeeze it into the next toolkit release.
> > >
> > >Thanks,
> > >Andy
> > >
> > >On Feb 17, 2015, at 5:26 PM, Horst 
> > Severini<>
> >   wrote:
> > >
> > > >>Hi Philippe,
> > > >>
> > > >>yes, so we're seeing the same problem with fail2ban.
> > > >>Not sure why only new installs are affected by that, though,
> > > >>and not our production instances.
> > > >>
> > > >>The other nightly cron email is a different issue, I think I just
> > > >>haven't completely confugred the mesh yet, since I wasn't sure
> > > >>whether I should leave that up to the pundit admins to configure.
> > > >>
> > > >>So I'm sure that will go away once the mesh is configured.
> > > >>
> > > >>Thanks,
> > > >>
> > > >>        Horst
> > > >>
> > > >>Philippe 
> > > Laurens<>
> > >   wrote:
> > > >>
> > > > >>>Hi Horst et al,
> > > > >>>
> > > > >>>I found a similar status on psmsu05.aglt2.org,
> > > > >>>but only fail2ban messages are in the queue.
> > > > >>>
> > > > >>>All 164 entries were from Feb 12, 13, 14
> > > > >>>while psmsu05 had been installed Feb 6-7 (from SL6 and RPMs)
> > > > >>>
> > > > >>>I checked 4-5 of these messages and all those looked like
> > > > >>>
> > > > > >>>>[root@psmsu05
> > > > >  ~]# postcat -q E62D720DC5
> > > > > >>>>   *** ENVELOPE RECORDS deferred/E/E62D720DC5 ***
> > > > > >>>>   message_size:            2420             213               1          
> > > > >      0            2420
> > > > > >>>>   message_arrival_time: Sat Feb 14 11:05:16 2015
> > > > > >>>>   create_time: Sat Feb 14 11:05:16 2015
> > > > > >>>>   named_attribute: log_message_origin=local
> > > > > >>>>   named_attribute: trace_flags=0
> > > > > >>>>   sender:
> > > > > >>>>   
> > > > > original_recipient:
> > > > > >>>>   
> > > > > recipient:
> > > > > >>>>   *** MESSAGE CONTENTS deferred/E/E62D720DC5 ***
> > > > > >>>>   Received: by psmsu05.aglt2.org (Postfix)
> > > > > >>>>           id E62D720DC5; Sat, 14 Feb 2015 11:05:16 -0500 (EST)
> > > > > >>>>   Date: Sat, 14 Feb 2015 11:05:16 -0500 (EST)
> > > > > >>>>   
> > > > > From:
> > > > >   (Mail Delivery System)
> > > > > >>>>   Subject: Undelivered Mail Returned to Sender
> > > > > >>>>   
> > > > > To:
> > > > > >>>>   Auto-Submitted: auto-replied
> > > > > >>>>   MIME-Version: 1.0
> > > > > >>>>   Content-Type: multipart/report; report-type=delivery-status;
> > > > > >>>>           boundary="7754320DC3.1423929916/psmsu05.aglt2.org"
> > > > > >>>>   
> > > > > Message-Id:<>
> > > > > >>>>
> > > > > >>>>   This is a MIME-encapsulated message.
> > > > > >>>>
> > > > > >>>>   --7754320DC3.1423929916/psmsu05.aglt2.org
> > > > > >>>>   Content-Description: Notification
> > > > > >>>>   Content-Type: text/plain; charset=us-ascii
> > > > > >>>>
> > > > > >>>>   This is the mail system at host psmsu05.aglt2.org.
> > > > > >>>>
> > > > > >>>>   I'm sorry to have to inform you that your message could not
> > > > > >>>>   be delivered to one or more recipients. It's attached below.
> > > > > >>>>
> > > > > >>>>   For further assistance, please send mail to postmaster.
> > > > > >>>>
> > > > > >>>>   If you do so, please include this problem report. You can
> > > > > >>>>   delete your own text from the attached returned message.
> > > > > >>>>
> > > > > >>>>                      The mail system
> > > > > >>>>
> > > > > >>>>   
> > > > > <>:
> > > > >  delivery temporarily suspended: connect to
> > > > > >>>>       example.com[93.184.216.34]:25: Connection timed out
> > > > > >>>>
> > > > > >>>>   --7754320DC3.1423929916/psmsu05.aglt2.org
> > > > > >>>>   Content-Description: Delivery report
> > > > > >>>>   Content-Type: message/delivery-status
> > > > > >>>>
> > > > > >>>>   Reporting-MTA: dns; psmsu05.aglt2.org
> > > > > >>>>   X-Postfix-Queue-ID: 7754320DC3
> > > > > >>>>   X-Postfix-Sender: 
> > > > > rfc822;
> > > > > >>>>   Arrival-Date: Mon,  9 Feb 2015 10:20:46 -0500 (EST)
> > > > > >>>>
> > > > > >>>>   Final-Recipient: 
> > > > > rfc822;
> > > > > >>>>   Action: failed
> > > > > >>>>   Status: 4.4.1
> > > > > >>>>   Diagnostic-Code: X-Postfix; delivery temporarily suspended: connect to
> > > > > >>>>       example.com[93.184.216.34]:25: Connection timed out
> > > > > >>>>
> > > > > >>>>   --7754320DC3.1423929916/psmsu05.aglt2.org
> > > > > >>>>   Content-Description: Undelivered Message
> > > > > >>>>   Content-Type: message/rfc822
> > > > > >>>>
> > > > > >>>>   
> > > > > Return-Path:<>
> > > > > >>>>   Received: by psmsu05.aglt2.org (Postfix, from userid 0)
> > > > > >>>>           id 7754320DC3; Mon,  9 Feb 2015 10:20:46 -0500 (EST)
> > > > > >>>>   Subject: [Fail2Ban] SSH: banned 103.41.124.13 from psmsu05.aglt2.org
> > > > > >>>>   Date: Mon, 09 Feb 2015 15:20:46 +0000
> > > > > >>>>   From: 
> > > > > Fail2Ban<>
> > > > > >>>>   
> > > > > To:
> > > > > >>>>   
> > > > > Message-Id:<>
> > > > > >>>>
> > > > > >>>>   Hi,
> > > > > >>>>
> > > > > >>>>   The IP 103.41.124.13 has just been banned by Fail2Ban after
> > > > > >>>>   5 attempts against SSH.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>   Here is more information about 103.41.124.13:
> > > > > >>>>
> > > > > >>>>   missing whois program
> > > > > >>>>
> > > > > >>>>   Regards,
> > > > > >>>>
> > > > > >>>>   Fail2Ban
> > > > > >>>>
> > > > > >>>>   --7754320DC3.1423929916/psmsu05.aglt2.org--
> > > > > >>>>   *** HEADER EXTRACTED deferred/E/E62D720DC5 ***
> > > > > >>>>   *** MESSAGE FILE END deferred/E/E62D720DC5 ***
> > > > >>>
> > > > >>>I know very little about fail2ban... I looked around the config
> > > > >>>files, but could not find where it could be told to send emails.
> > > > >>>I compared a couple likely candidate config files with another
> > > > >>>instance (psum05) and could not find any difference either.
> > > > >>>
> > > > >>>
> > > > >>>I restarted fail2ban (which put 2 more messages in the queue),
> > > > >>>cleared the queue, and will be watching.
> > > > >>>
> > > > >>>Thanks,
> > > > >>>
> > > > >>>Philippe
> > > > >>>
> > > > >>>
> > > > >>>On 17-Feb-2015 4:10 PM, Horst Severini wrote:
> > > > > >>>>Hi all,
> > > > > >>>>
> > > > > >>>>as we have just discussed in the throughput meeting, we are seeing
> > > > > >>>>some strange emails in our newly installed 3.4.1 test machine,
> > > > > >>>>ps3.ochep.ou.edu .
> > > > > >>>>
> > > > > >>>>One is that fail2ban is now trying to send email 
> > > > > ,
> > > > > >>>>claiming to be 
> > > > > ,
> > > > >  so there must be some incorrect
> > > > > >>>>initial configuration going on during the installation. There's not a new
> > > > > >>>>version of fail2ban out, though, it's from last August:
> > > > > >>>>
> > > > > >>>>fail2ban-0.8.14-1.el6.noarch
> > > > > >>>>
> > > > > >>>>But something else must've changed in the mail configuration somehow ...
> > > > > >>>>
> > > > > >>>>The other thing is that the nightly cron job cron-service_watcher,
> > > > > >>>>which runs /opt/perfsonar_ps/toolkit/scripts/watcher_log_archive_cleanup,
> > > > > >>>>apparently wasn't quite configured correctly, either, since it still
> > > > > >>>>contains
> > > > > >>>>
> > > > > >>>>STORE_LOCATION=/mnt/store/NPTools/debug
> > > > > >>>>
> > > > > >>>>which is causing a nightly error mail about that directory not being 
> > > > > there.
> > > > > >>>>
> > > > > >>>>Did I miss a configuration step there somehow?
> > > > > >>>>
> > > > > >>>>I installed this machine about a week ago, on one of our old KOI boxes
> > > > > >>>>which used to be our production PS boxes, before we upgraded to the 
> > > > > R310s..
> > > > > >>>>
> > > > > >>>>Oh, and the other strange thing is that I was trying to install the other
> > > > > >>>>KOI box, it wouldn't let me, since during the PS installation,
> > > > > >>>>when it comes time to choose where to install the OS on,
> > > > > >>>>anaconda can't find the 160 GB hard drive, so there's no way to continue
> > > > > >>>>the install.
> > > > > >>>>
> > > > > >>>>Which makes no sense at all, since I can still boot up the old PS-3.2
> > > > > >>>>instance from the hard drive just fine, so there's obviously nothing wrong
> > > > > >>>>with the hard drive at all.
> > > > > >>>>
> > > > > >>>>I also verified all the BIOS settings, they're identical between the
> > > > > >>>>two KOI boxes, so why would one of them allow me to install on the hard 
> > > > > drive,
> > > > > >>>>and the other one won't find it at that time?
> > > > > >>>>
> > > > > >>>>How can I debug this? We don't immediately need it right now,
> > > > > >>>>but I'd like to be able to install a test bandwidth node there at some 
> > > > > point,
> > > > > >>>>to have a full test set.
> > > > > >>>>
> > > > > >>>>Thanks,
> > > > > >>>>
> > > > > >>>>        Horst
> > > > > >>>>
> > >