perfsonar-user - Re: [perfsonar-user] fail2ban and watcher_log_archive_cleanup cron emails in new installed 3.4.1 instance, and hard drive install issue

Subject: perfSONAR User Q&A and Other Discussion

List archive

Re: [perfsonar-user] fail2ban and watcher_log_archive_cleanup cron emails in new installed 3.4.1 instance, and hard drive install issue

From: Horst Severini <>
To: Andrew Lake <>
Cc: Philippe Laurens <>, , Horst Severini <>
Subject: Re: [perfsonar-user] fail2ban and watcher_log_archive_cleanup cron emails in new installed 3.4.1 instance, and hard drive install issue
Date: Wed, 18 Feb 2015 12:44:52 -0600

Hi Andy,

okay, sounds good!

So should I go ahead and remove all deferred mails from
/var/spool/postfix/defer/ and /var/spool/postfix/deferred/, and make sure no new ones are being created?

I'd still like to figure out why all these queued emails keep getting re-created with time stamps which look to be up to one hour into the future. Maybe that's a CST vs. CDT issue? Shouldn't be, though, since the system clock seems to be perfectly fine ...

Thanks,

Horst

On 02/18/2015 11:57 AM, Andrew Lake wrote:

Hi,

As far as I can tell, perfSONAR never touched that file, it just relied on
what the fail2ban RPM provided. As you saw it seems like at some point
fail2ban changed the default, but it looks like the fail2banRPM is configured
to leave that file alone if it already exists, so updates were unaffected.
Fail2ban recommends that you don't edit jail.conf directly since they may
change it over time and recommend you use a jail.local file like I sent. I'm
planning to have the Toolkit install the jail.local file assuming it works
for you guys so we can avoid this in the future.

Thanks,
Andy

On Feb 18, 2015, at 12:38 PM, Horst
Severini<>
wrote:

>Hi Philippe,
>
>I just had a closer look at our production host, ps1 and ps2, and the latest
fail2ban-0.8.14-1 from last August did create a jail.conf.rpmnew, which does
indeed not comment out this line anymore:
>
>----
>99c77
><
sendmail-whois[name=SSH,,,
sendername="Fail2Ban"]
>---

> ># sendmail-whois[name=SSH,
dest=root,,
sendername="Fail2Ban"]

>----
>
>So this probably means that all hosts which were installed before last
August weren't affected by that, since the .rpmnew file didn't take effect, but
all freshly installed hosts since then are incorrectly configured to send
unconfigured emails?
>
>Then again, it looks like the old production installs also commented this
out during the initial install, since we have all these files here:
>
>----
>-rw-r--r-- 1 root 13769 Feb 4 2014 jail.conf
>-rw-r--r-- 1 root 13768 Dec 3 2013 jail.conf.orig
>-rw-r--r-- 1 root 19313 Jul 29 2014 jail.conf.rpmnew
>-rw-r--r-- 1 root 13769 Feb 4 2014 jail.conf.save
>-rw-r--r-- 1 root 13768 Dec 3 2013 jail.conf~
>----
>
>So some post-install script seems to have made a .orig backup, then
commented this line out with a 'sed -i~' command (or equivalent), and then made
another .save backup. Why is this no longer being done in new installs now? Did
something in the rpmnew file break the 'sed' command maybe? Just guessing here
...
>
>Cheers,
>
> Horst
>
>On 02/18/2015 10:55 AM, Philippe Laurens wrote:

>>Hi Andy,
>>
>>Ok, thanks. I can certainly do that, but being sure it truly helped
>>will take some time as, at least for psmsu05, fail2ban was not
>>generating daily emails. The mailq is still empty since I cleared
>>it yesterday. This may be because this is a machine lesser known
>>to attackers.
>>
>>Ah! now I see that psmsu01 and 02 are getting similar daily entries
>>in the mailq. I will install the fix on one of those and we will
>>have a cleaner test.
>>
>>I am still not sure why the psumxx machines have the same jail.conf,
>>no jail.local and seemingly no messages in mailq.
>>What line is no longer commented out in the fail2ban RPM?
>>
>>Thanks,
>>
>>Philippe
>>
>>
>>On 18-Feb-2015 10:53 AM, Andrew Lake wrote:

>>>Hi,
>>>
>>>On the hosts giving you trouble if you copy the attached file to
>>>/etc/fail2ban/jail.local (note that I said copy it to jail.local NOT
>>>jail.conf) and run "/sbin/service fail2ban restart" do the mail issues
>>>subside? It looks like the fail2ban RPM no longer comments out a line
>>>that says to send mail by default. Copying the attached file to the
>>>location specified should override this default. If that helps, we
>>>will squeeze it into the next toolkit release.
>>>
>>>Thanks,
>>>Andy
>>>
>>>On Feb 17, 2015, at 5:26 PM, Horst
Severini<>
wrote:
>>>

>>>>Hi Philippe,
>>>>
>>>>yes, so we're seeing the same problem with fail2ban.
>>>>Not sure why only new installs are affected by that, though,
>>>>and not our production instances.
>>>>
>>>>The other nightly cron email is a different issue, I think I just
>>>>haven't completely confugred the mesh yet, since I wasn't sure
>>>>whether I should leave that up to the pundit admins to configure.
>>>>
>>>>So I'm sure that will go away once the mesh is configured.
>>>>
>>>>Thanks,
>>>>
>>>> Horst
>>>>
>>>>Philippe
Laurens<>
wrote:
>>>>

>>>>>Hi Horst et al,
>>>>>
>>>>>I found a similar status on psmsu05.aglt2.org,
>>>>>but only fail2ban messages are in the queue.
>>>>>
>>>>>All 164 entries were from Feb 12, 13, 14
>>>>>while psmsu05 had been installed Feb 6-7 (from SL6 and RPMs)
>>>>>
>>>>>I checked 4-5 of these messages and all those looked like
>>>>>

>>>>>>[root@psmsu05
~]# postcat -q E62D720DC5
>>>>>> *** ENVELOPE RECORDS deferred/E/E62D720DC5 ***
>>>>>> message_size: 2420 213
>>>>>>1 0 2420
>>>>>> message_arrival_time: Sat Feb 14 11:05:16 2015
>>>>>> create_time: Sat Feb 14 11:05:16 2015
>>>>>> named_attribute: log_message_origin=local
>>>>>> named_attribute: trace_flags=0
>>>>>> sender:
>>>>>>
original_recipient:
>>>>>>
recipient:
>>>>>> *** MESSAGE CONTENTS deferred/E/E62D720DC5 ***
>>>>>> Received: by psmsu05.aglt2.org (Postfix)
>>>>>> id E62D720DC5; Sat, 14 Feb 2015 11:05:16 -0500 (EST)
>>>>>> Date: Sat, 14 Feb 2015 11:05:16 -0500 (EST)
>>>>>>
From:
(Mail Delivery System)
>>>>>> Subject: Undelivered Mail Returned to Sender
>>>>>>
To:
>>>>>> Auto-Submitted: auto-replied
>>>>>> MIME-Version: 1.0
>>>>>> Content-Type: multipart/report; report-type=delivery-status;
>>>>>> boundary="7754320DC3.1423929916/psmsu05.aglt2.org"
>>>>>>
Message-Id:<>
>>>>>>
>>>>>> This is a MIME-encapsulated message.
>>>>>>
>>>>>> --7754320DC3.1423929916/psmsu05.aglt2.org
>>>>>> Content-Description: Notification
>>>>>> Content-Type: text/plain; charset=us-ascii
>>>>>>
>>>>>> This is the mail system at host psmsu05.aglt2.org.
>>>>>>
>>>>>> I'm sorry to have to inform you that your message could not
>>>>>> be delivered to one or more recipients. It's attached below.
>>>>>>
>>>>>> For further assistance, please send mail to postmaster.
>>>>>>
>>>>>> If you do so, please include this problem report. You can
>>>>>> delete your own text from the attached returned message.
>>>>>>
>>>>>> The mail system
>>>>>>
>>>>>><>:
delivery temporarily suspended: connect to
>>>>>> example.com[93.184.216.34]:25: Connection timed out
>>>>>>
>>>>>> --7754320DC3.1423929916/psmsu05.aglt2.org
>>>>>> Content-Description: Delivery report
>>>>>> Content-Type: message/delivery-status
>>>>>>
>>>>>> Reporting-MTA: dns; psmsu05.aglt2.org
>>>>>> X-Postfix-Queue-ID: 7754320DC3
>>>>>> X-Postfix-Sender:
rfc822;
>>>>>> Arrival-Date: Mon, 9 Feb 2015 10:20:46 -0500 (EST)
>>>>>>
>>>>>> Final-Recipient:
rfc822;
>>>>>> Action: failed
>>>>>> Status: 4.4.1
>>>>>> Diagnostic-Code: X-Postfix; delivery temporarily suspended:
>>>>>>connect to
>>>>>> example.com[93.184.216.34]:25: Connection timed out
>>>>>>
>>>>>> --7754320DC3.1423929916/psmsu05.aglt2.org
>>>>>> Content-Description: Undelivered Message
>>>>>> Content-Type: message/rfc822
>>>>>>
>>>>>>
Return-Path:<>
>>>>>> Received: by psmsu05.aglt2.org (Postfix, from userid 0)
>>>>>> id 7754320DC3; Mon, 9 Feb 2015 10:20:46 -0500 (EST)
>>>>>> Subject: [Fail2Ban] SSH: banned 103.41.124.13 from
>>>>>>psmsu05.aglt2.org
>>>>>> Date: Mon, 09 Feb 2015 15:20:46 +0000
>>>>>> From:
Fail2Ban<>
>>>>>>
To:
>>>>>>
Message-Id:<>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> The IP 103.41.124.13 has just been banned by Fail2Ban after
>>>>>> 5 attempts against SSH.
>>>>>>
>>>>>>
>>>>>> Here is more information about 103.41.124.13:
>>>>>>
>>>>>> missing whois program
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Fail2Ban
>>>>>>
>>>>>> --7754320DC3.1423929916/psmsu05.aglt2.org--
>>>>>> *** HEADER EXTRACTED deferred/E/E62D720DC5 ***
>>>>>> *** MESSAGE FILE END deferred/E/E62D720DC5 ***

>>>>>
>>>>>I know very little about fail2ban... I looked around the config
>>>>>files, but could not find where it could be told to send emails.
>>>>>I compared a couple likely candidate config files with another
>>>>>instance (psum05) and could not find any difference either.
>>>>>
>>>>>
>>>>>I restarted fail2ban (which put 2 more messages in the queue),
>>>>>cleared the queue, and will be watching.
>>>>>
>>>>>Thanks,
>>>>>
>>>>>Philippe
>>>>>
>>>>>
>>>>>On 17-Feb-2015 4:10 PM, Horst Severini wrote:

>>>>>>Hi all,
>>>>>>
>>>>>>as we have just discussed in the throughput meeting, we are seeing
>>>>>>some strange emails in our newly installed 3.4.1 test machine,
>>>>>>ps3.ochep.ou.edu .
>>>>>>
>>>>>>One is that fail2ban is now trying to send email
,
>>>>>>claiming to be
,
so there must be some
>>>>>>incorrect
>>>>>>initial configuration going on during the installation. There's not
>>>>>>a new
>>>>>>version of fail2ban out, though, it's from last August:
>>>>>>
>>>>>>fail2ban-0.8.14-1.el6.noarch
>>>>>>
>>>>>>But something else must've changed in the mail configuration
>>>>>>somehow ...
>>>>>>
>>>>>>The other thing is that the nightly cron job cron-service_watcher,
>>>>>>which runs
>>>>>>/opt/perfsonar_ps/toolkit/scripts/watcher_log_archive_cleanup,
>>>>>>apparently wasn't quite configured correctly, either, since it still
>>>>>>contains
>>>>>>
>>>>>>STORE_LOCATION=/mnt/store/NPTools/debug
>>>>>>
>>>>>>which is causing a nightly error mail about that directory not
>>>>>>being there.
>>>>>>
>>>>>>Did I miss a configuration step there somehow?
>>>>>>
>>>>>>I installed this machine about a week ago, on one of our old KOI boxes
>>>>>>which used to be our production PS boxes, before we upgraded to the
>>>>>>R310s..
>>>>>>
>>>>>>Oh, and the other strange thing is that I was trying to install the
>>>>>>other
>>>>>>KOI box, it wouldn't let me, since during the PS installation,
>>>>>>when it comes time to choose where to install the OS on,
>>>>>>anaconda can't find the 160 GB hard drive, so there's no way to
>>>>>>continue
>>>>>>the install.
>>>>>>
>>>>>>Which makes no sense at all, since I can still boot up the old PS-3.2
>>>>>>instance from the hard drive just fine, so there's obviously
>>>>>>nothing wrong
>>>>>>with the hard drive at all.
>>>>>>
>>>>>>I also verified all the BIOS settings, they're identical between the
>>>>>>two KOI boxes, so why would one of them allow me to install on the
>>>>>>hard drive,
>>>>>>and the other one won't find it at that time?
>>>>>>
>>>>>>How can I debug this? We don't immediately need it right now,
>>>>>>but I'd like to be able to install a test bandwidth node there at
>>>>>>some point,
>>>>>>to have a full test set.
>>>>>>
>>>>>>Thanks,
>>>>>>
>>>>>> Horst
>>>>>>

>>>

>>

Re: [perfsonar-user] fail2ban and watcher_log_archive_cleanup cron emails in new installed 3.4.1 instance, and hard drive install issue, (continued)

List archive

Re: [perfsonar-user] fail2ban and watcher_log_archive_cleanup cron emails in new installed 3.4.1 instance, and hard drive install issue