perfSONAR User Q&A and Other Discussion

[Horst Severini <>]

I just looked at this some more:

----
[root@ps3
 ~]# ls -ao --full-time /var/spool/postfix/deferred/*
[...]
/var/spool/postfix/deferred/F:
total 24
drwx------   2 postfix 4096 2015-02-18 10:38:44.800641854 -0600 .
drwx------. 18 postfix 4096 2015-02-15 06:58:53.060907510 -0600 ..
-rwx------   1 postfix  757 2015-02-18 10:55:25.000000000 -0600 F03902E0EAB
-rwx------   1 postfix  754 2015-02-18 11:19:54.000000000 -0600 F044E2E1156
-rwx------   1 postfix  751 2015-02-18 11:45:24.000000000 -0600 F12532E0DF1
-rwx------   1 postfix  757 2015-02-18 11:35:24.000000000 -0600 F184D2E118F

[root@ps3
 ~]# date
Wed Feb 18 10:45:58 CST 2015
----

Why does postfix do that?

        Horst

On 02/18/2015 10:43 AM, Horst Severini wrote:
> Hi Andy,
>
> okay, I added your jail.local and restarted, let's see what happens.
> I added an EOL at the end, I think that can't hurt. :)
>
> So the fact that the modified '[ssh-iptables] action' no longer contains
> a 'sendmail-whois' statement will make it stop sending emails
> altogether? That's not all that intuitive, either. =)
>
> By the way, I looked at the deferred email files (I haven't deleted them
> just yet), and it seems that these files are created with a time stamp
> in the future:
>
> ----
> /var/spool/postfix/deferred/F:
> total 24
> drwx------   2 postfix 4096 Feb 18 10:28 .
> drwx------. 18 postfix 4096 Feb 15 06:58 ..
> -rwx------   1 postfix  757 Feb 18  2015 F03902E0EAB
> -rwx------   1 postfix  754 Feb 18  2015 F044E2E1156
> -rwx------   1 postfix  751 Feb 18 10:35 F12532E0DF1
> -rwx------   1 postfix  757 Feb 18  2015 F184D2E118F
> ----
>
> How can this happen? What process can write a file and date it in the
> future? ntp is definitely running and synchronized.
>
> Thanks,
>
>      Horst
>
> On 02/18/2015 09:55 AM, Andrew Lake wrote:
> > Forgot to attach file….
> >
> >
> >
> > On Feb 18, 2015, at 10:53 AM, Andrew 
> > Lake<>
> >   wrote:
> >
> > > >Hi,
> > > >
> > > >On the hosts giving you trouble if you copy the attached file to
> > > /etc/fail2ban/jail.local (note that I said copy it to jail.local NOT
> > > jail.conf) and run "/sbin/service fail2ban restart" do the mail
> > > issues subside? It looks like the fail2ban RPM no longer comments out
> > > a line that says to send mail by default. Copying the attached file
> > > to the location specified should override this default. If that
> > > helps, we will squeeze it into the next toolkit release.
> > > >
> > > >Thanks,
> > > >Andy
> > > >
> > > >On Feb 17, 2015, at 5:26 PM, Horst 
> > > Severini<>
> > >   wrote:
> > > >
> > > > >>Hi Philippe,
> > > > >>
> > > > >>yes, so we're seeing the same problem with fail2ban.
> > > > >>Not sure why only new installs are affected by that, though,
> > > > >>and not our production instances.
> > > > >>
> > > > >>The other nightly cron email is a different issue, I think I just
> > > > >>haven't completely confugred the mesh yet, since I wasn't sure
> > > > >>whether I should leave that up to the pundit admins to configure.
> > > > >>
> > > > >>So I'm sure that will go away once the mesh is configured.
> > > > >>
> > > > >>Thanks,
> > > > >>
> > > > >>    Horst
> > > > >>
> > > > >>Philippe 
> > > > Laurens<>
> > > >   wrote:
> > > > >>
> > > > > >>>Hi Horst et al,
> > > > > >>>
> > > > > >>>I found a similar status on psmsu05.aglt2.org,
> > > > > >>>but only fail2ban messages are in the queue.
> > > > > >>>
> > > > > >>>All 164 entries were from Feb 12, 13, 14
> > > > > >>>while psmsu05 had been installed Feb 6-7 (from SL6 and RPMs)
> > > > > >>>
> > > > > >>>I checked 4-5 of these messages and all those looked like
> > > > > >>>
> > > > > > >>>>[root@psmsu05
> > > > > >  ~]# postcat -q E62D720DC5
> > > > > > >>>>   *** ENVELOPE RECORDS deferred/E/E62D720DC5 ***
> > > > > > >>>>   message_size:            2420             213
> > > > > > 1               0            2420
> > > > > > >>>>   message_arrival_time: Sat Feb 14 11:05:16 2015
> > > > > > >>>>   create_time: Sat Feb 14 11:05:16 2015
> > > > > > >>>>   named_attribute: log_message_origin=local
> > > > > > >>>>   named_attribute: trace_flags=0
> > > > > > >>>>   sender:
> > > > > > >>>> 
> > > > > > original_recipient:
> > > > > > >>>> 
> > > > > > recipient:
> > > > > > >>>>   *** MESSAGE CONTENTS deferred/E/E62D720DC5 ***
> > > > > > >>>>   Received: by psmsu05.aglt2.org (Postfix)
> > > > > > >>>>           id E62D720DC5; Sat, 14 Feb 2015 11:05:16 -0500 (EST)
> > > > > > >>>>   Date: Sat, 14 Feb 2015 11:05:16 -0500 (EST)
> > > > > > >>>> 
> > > > > > From:
> > > > > >   (Mail Delivery System)
> > > > > > >>>>   Subject: Undelivered Mail Returned to Sender
> > > > > > >>>> 
> > > > > > To:
> > > > > > >>>>   Auto-Submitted: auto-replied
> > > > > > >>>>   MIME-Version: 1.0
> > > > > > >>>>   Content-Type: multipart/report; report-type=delivery-status;
> > > > > > >>>>           boundary="7754320DC3.1423929916/psmsu05.aglt2.org"
> > > > > > >>>>   
> > > > > > Message-Id:<>
> > > > > > >>>>
> > > > > > >>>>   This is a MIME-encapsulated message.
> > > > > > >>>>
> > > > > > >>>>   --7754320DC3.1423929916/psmsu05.aglt2.org
> > > > > > >>>>   Content-Description: Notification
> > > > > > >>>>   Content-Type: text/plain; charset=us-ascii
> > > > > > >>>>
> > > > > > >>>>   This is the mail system at host psmsu05.aglt2.org.
> > > > > > >>>>
> > > > > > >>>>   I'm sorry to have to inform you that your message could not
> > > > > > >>>>   be delivered to one or more recipients. It's attached below.
> > > > > > >>>>
> > > > > > >>>>   For further assistance, please send mail to postmaster.
> > > > > > >>>>
> > > > > > >>>>   If you do so, please include this problem report. You can
> > > > > > >>>>   delete your own text from the attached returned message.
> > > > > > >>>>
> > > > > > >>>>                      The mail system
> > > > > > >>>>
> > > > > > >>>> 
> > > > > > <>:
> > > > > >  delivery temporarily suspended: connect to
> > > > > > >>>>       example.com[93.184.216.34]:25: Connection timed out
> > > > > > >>>>
> > > > > > >>>>   --7754320DC3.1423929916/psmsu05.aglt2.org
> > > > > > >>>>   Content-Description: Delivery report
> > > > > > >>>>   Content-Type: message/delivery-status
> > > > > > >>>>
> > > > > > >>>>   Reporting-MTA: dns; psmsu05.aglt2.org
> > > > > > >>>>   X-Postfix-Queue-ID: 7754320DC3
> > > > > > >>>>   X-Postfix-Sender: 
> > > > > > rfc822;
> > > > > > >>>>   Arrival-Date: Mon,  9 Feb 2015 10:20:46 -0500 (EST)
> > > > > > >>>>
> > > > > > >>>>   Final-Recipient: 
> > > > > > rfc822;
> > > > > > >>>>   Action: failed
> > > > > > >>>>   Status: 4.4.1
> > > > > > >>>>   Diagnostic-Code: X-Postfix; delivery temporarily suspended:
> > > > > > connect to
> > > > > > >>>>       example.com[93.184.216.34]:25: Connection timed out
> > > > > > >>>>
> > > > > > >>>>   --7754320DC3.1423929916/psmsu05.aglt2.org
> > > > > > >>>>   Content-Description: Undelivered Message
> > > > > > >>>>   Content-Type: message/rfc822
> > > > > > >>>>
> > > > > > >>>>   
> > > > > > Return-Path:<>
> > > > > > >>>>   Received: by psmsu05.aglt2.org (Postfix, from userid 0)
> > > > > > >>>>           id 7754320DC3; Mon,  9 Feb 2015 10:20:46 -0500 (EST)
> > > > > > >>>>   Subject: [Fail2Ban] SSH: banned 103.41.124.13 from
> > > > > > psmsu05.aglt2.org
> > > > > > >>>>   Date: Mon, 09 Feb 2015 15:20:46 +0000
> > > > > > >>>>   From: 
> > > > > > Fail2Ban<>
> > > > > > >>>> 
> > > > > > To:
> > > > > > >>>>   
> > > > > > Message-Id:<>
> > > > > > >>>>
> > > > > > >>>>   Hi,
> > > > > > >>>>
> > > > > > >>>>   The IP 103.41.124.13 has just been banned by Fail2Ban after
> > > > > > >>>>   5 attempts against SSH.
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>   Here is more information about 103.41.124.13:
> > > > > > >>>>
> > > > > > >>>>   missing whois program
> > > > > > >>>>
> > > > > > >>>>   Regards,
> > > > > > >>>>
> > > > > > >>>>   Fail2Ban
> > > > > > >>>>
> > > > > > >>>>   --7754320DC3.1423929916/psmsu05.aglt2.org--
> > > > > > >>>>   *** HEADER EXTRACTED deferred/E/E62D720DC5 ***
> > > > > > >>>>   *** MESSAGE FILE END deferred/E/E62D720DC5 ***
> > > > > >>>
> > > > > >>>I know very little about fail2ban... I looked around the config
> > > > > >>>files, but could not find where it could be told to send emails.
> > > > > >>>I compared a couple likely candidate config files with another
> > > > > >>>instance (psum05) and could not find any difference either.
> > > > > >>>
> > > > > >>>
> > > > > >>>I restarted fail2ban (which put 2 more messages in the queue),
> > > > > >>>cleared the queue, and will be watching.
> > > > > >>>
> > > > > >>>Thanks,
> > > > > >>>
> > > > > >>>Philippe
> > > > > >>>
> > > > > >>>
> > > > > >>>On 17-Feb-2015 4:10 PM, Horst Severini wrote:
> > > > > > >>>>Hi all,
> > > > > > >>>>
> > > > > > >>>>as we have just discussed in the throughput meeting, we are
> > > > > > seeing
> > > > > > >>>>some strange emails in our newly installed 3.4.1 test machine,
> > > > > > >>>>ps3.ochep.ou.edu .
> > > > > > >>>>
> > > > > > >>>>One is that fail2ban is now trying to send email
> > > > > > ,
> > > > > > >>>>claiming to be 
> > > > > > ,
> > > > > >  so there must be some
> > > > > > incorrect
> > > > > > >>>>initial configuration going on during the installation.
> > > > > > There's not a new
> > > > > > >>>>version of fail2ban out, though, it's from last August:
> > > > > > >>>>
> > > > > > >>>>fail2ban-0.8.14-1.el6.noarch
> > > > > > >>>>
> > > > > > >>>>But something else must've changed in the mail configuration
> > > > > > somehow ...
> > > > > > >>>>
> > > > > > >>>>The other thing is that the nightly cron job
> > > > > > cron-service_watcher,
> > > > > > >>>>which runs
> > > > > > /opt/perfsonar_ps/toolkit/scripts/watcher_log_archive_cleanup,
> > > > > > >>>>apparently wasn't quite configured correctly, either, since it
> > > > > > still
> > > > > > >>>>contains
> > > > > > >>>>
> > > > > > >>>>STORE_LOCATION=/mnt/store/NPTools/debug
> > > > > > >>>>
> > > > > > >>>>which is causing a nightly error mail about that directory not
> > > > > > being there.
> > > > > > >>>>
> > > > > > >>>>Did I miss a configuration step there somehow?
> > > > > > >>>>
> > > > > > >>>>I installed this machine about a week ago, on one of our old
> > > > > > KOI boxes
> > > > > > >>>>which used to be our production PS boxes, before we upgraded
> > > > > > to the R310s..
> > > > > > >>>>
> > > > > > >>>>Oh, and the other strange thing is that I was trying to
> > > > > > install the other
> > > > > > >>>>KOI box, it wouldn't let me, since during the PS installation,
> > > > > > >>>>when it comes time to choose where to install the OS on,
> > > > > > >>>>anaconda can't find the 160 GB hard drive, so there's no way
> > > > > > to continue
> > > > > > >>>>the install.
> > > > > > >>>>
> > > > > > >>>>Which makes no sense at all, since I can still boot up the old
> > > > > > PS-3.2
> > > > > > >>>>instance from the hard drive just fine, so there's obviously
> > > > > > nothing wrong
> > > > > > >>>>with the hard drive at all.
> > > > > > >>>>
> > > > > > >>>>I also verified all the BIOS settings, they're identical
> > > > > > between the
> > > > > > >>>>two KOI boxes, so why would one of them allow me to install on
> > > > > > the hard drive,
> > > > > > >>>>and the other one won't find it at that time?
> > > > > > >>>>
> > > > > > >>>>How can I debug this? We don't immediately need it right now,
> > > > > > >>>>but I'd like to be able to install a test bandwidth node there
> > > > > > at some point,
> > > > > > >>>>to have a full test set.
> > > > > > >>>>
> > > > > > >>>>Thanks,
> > > > > > >>>>
> > > > > > >>>>    Horst
> > > > > > >>>>
> > > >