perfSONAR User Q&A and Other Discussion

[Trey Dockendorf <>]

John-Paul,

Very useful write up, thank you!

My two perfSONAR boxes were also reported to me as running botnets by my campus' network group.  Fortunately we work closely with them on managing the perfSONAR presence at this campus so the fact that I had a comprised system was punishment enough :)

The general fault on my systems was not updating them soon enough.

Brian,

What method of auto-update is being considered?  I ask because one of the RedHat based operating systems surprised me by including "yum-autoupdate" as a default, and enabled, service in minimal installs without the option to go into a "notify only" mode.  Upstream has the "yum-cron" which has such an option.

Jason,

So far of the numerous mailing lists I'm on, the PerfSONAR list has provided the most information regarding this particular vulnerability.  Aside from pushing out updates outside of a site's control, there's not much more that can be done that hasn't already been done.

Unfortunately my perfsonar box became to me an "orphaned" pS box.  I look at it occasionally when users complained about network issues between us and a remote site, but besides that it was 'out-of-mind'.  Would this thread, or a new one, be more appropriate to discuss some of your points (IDS, monitoring of pS, etc)?

- Trey

=============================

Trey Dockendorf 

Systems Analyst I 

Texas A&M University 

Academy for Advanced Telecommunications and Learning Technologies 

Phone: (979)458-2396 

Email:  

Jabber:

On Wed, Oct 1, 2014 at 6:17 PM, Jason Zurawski <> wrote:

Hey John-Paul;

To echo Brian, thanks for your thoughtful note.  One quick point, that some may not be aware of if they have been on this list for 7 years, is that there is a low volume (lower than the user list at least…) announcement list:

        https://lists.internet2.edu/sympa/subscribe/perfsonar-announce

I can also use this opportunity summarize some of the actions we will take based on this, and some other internal discussions occurring of the prior days:

 - As Brian noted, a sensible default method for automatic updates is coming.  This will not be a panacea for security or maintenance of course - and some of our development team has grave concerns about lulling anyone int a false sense of security and making things far far worse then next time a piece of systems software outside of our control barfs.   The bottom line is still that each site is responsible for care, feeding, and sensible sysadmin practices, and we feel that's a statement everyone agrees with.  We do (and will continue to) assist where we can with automated software to observe and upgrade (IDSs, yum, etc.), and we will also augment any technical solution with a 'sysadmin 101' guide for those that need it.  This is in-progress, and will also be something we would encourage community input with over time.

 - As a P.S. to the previous bullet - perfSONAR is not an appliance, and 2014 was the year that really made everyone (community members, stakeholders, developers)  reconsider what it should be.  Recently someone noted that the word 'appliance' strikes images of a box with no seems, that may not have serviceable parts inside.  Linux, unless it has been *drastically* altered, doesn't fit this definition.  As Heartbleed, Shellshock, and other CVEs have shown, no matter how much we work to make something 'easy', it is not invincible, and admitting a failure is part of the road to recovery.  We want to change the perception of the orphaned pS box, as it was the orphaned box that was pegged and owned as an easy target last Friday morning all around the world.

 - We intend to document what a 'normal' list of running toolkit processes look like, to prevent people from having that feeling of panic when they see a strange perl script running (correctly, or maliciously).  This is a very obvious thing that many of us wouldn't have thought about.

 - Some products (e.g. Cacti, JOWAMP) will be removed to reduce (not eliminate) the risk footprint, and those that choose to use them will have to make a conscious choice to download and install the tools.

We do appreciate the communities understanding, feedback, and patience this year.  Please continue to offer feedback as you all see fit, either to the user's list or to the developers directly ().

Thanks;

-jason

On Oct 1, 2014, at 5:14 PM, John-Paul Robinson <> wrote:

> To other shellshocked perfsonar users:
>
> Our perfsonar node did not have automatic yum updates enabled and was impacted by a shellshock-related exploit on Sept 26.   This is was after both bash updates had been announced on the perfsonar-user lists, so we may have survived had automatic updates been enabled by default.
>
>       • Lesson learned: run automatic updates.
>       • Recommendation: It might benefit users to have it default to on in the perfsonar distribution.  Also it would be good if updates were checked for more than once a day.  In our case we would have missed the update mid-day on Sept 26 and may still have gotten exploited before the next run at 4:00 am on Sept 27.  Additionally a perfsonar-announce list might be useful for hearing stuff even when you have -user discussions turned off.
>
> After receiving a local exploit report I went to check on the machine and immediately noticed Apache had restarted.  Alarmingly, a root-owned process called fakewww also started at the same start time and oddly so did one named web100srv.  Both of these processes had open ports and logs open.  Yikes, they got root!  Killed them.  But then they came back after I restarted httpd, even after `rpm -V httpd` showed no corruption. Oh no! They've really gotten a hold of the system.
>       • Lesson learned: not all unfamiliar processes are bad. I later figured out that these are part of the ndt-server rpm and normal parts of perfsonar.
>       • Recommendation: rename fakewww to something meaningful and less scary to the uninitiated.  ndtwwwhelper might be just as good.
>
> Because of the potential for root exploits I ran rpm verifies of core commands (eg rpm -V procps) some were good  some reported prelink inconsistencies.  This caused some concerns at first but as I narrowed down the exploit it became clear the problems were only due to a prelink bug.
>
> https://access.redhat.com/solutions/25215
> https://bugzilla.redhat.com/show_bug.cgi?id=204448
>
>       • Lesson learned: other bugs can make things seem worse than they are.
>       • Recommendation: look up unfamiliar errors before you panic.
>
> Looking further into the state of the system I noticed an '/usr/sbin/sshd -i' process running as apache and an time-wise unrelated httpd process.  lsof showed these were both perl codes running out of /var/tmp/ with established tcp connections off site.  Very suspicious and killed them.
>
>       • Lesson learned: some processes are really bad.  The abrtd logged the event of the first entry into the system via apache and showed the command vector was bash.  This is a very helpful log to determine important time lines.
>       • Recommendation: keep your system up to date.
>
> In the end, I traced the exploit down to the two suspicious perl processes (/var/tmp/x).  They were executing an ircbot as apache.  There was no root access to the system and simply clearing out the installed bots from /var/tmp was a sufficient remedy. There was an attempted install of code to exploit CVE-2013-2094 but thank fully that's a 3.8 kernel bug and perfsonar is still on 2.6.
>
> I hope this experience can be useful to others and that the recommendations can be incorporated into future releases as warranted.
>
> ~jpr