perfSONAR User Q&A and Other Discussion

[John-Paul Robinson <>]

To other shellshocked perfsonar users:

Our perfsonar node did not have automatic yum updates enabled and was impacted by a shellshock-related exploit on Sept 26.   This is was after both bash updates had been announced on the perfsonar-user lists, so we may have survived had automatic updates been enabled by default.

• Lesson learned: run automatic updates.
• Recommendation: It might benefit users to have it default to on in the perfsonar distribution.  Also it would be good if updates were checked for more than once a day.  In our case we would have missed the update mid-day on Sept 26 and may still have gotten exploited before the next run at 4:00 am on Sept 27.  Additionally a perfsonar-announce list might be useful for hearing stuff even when you have -user discussions turned off.

After receiving a local exploit report I went to check on the machine and immediately noticed Apache had restarted.  Alarmingly, a root-owned process called fakewww also started at the same start time and oddly so did one named web100srv.  Both of these processes had open ports and logs open.  Yikes, they got root!  Killed them.  But then they came back after I restarted httpd, even after `rpm -V httpd` showed no corruption. Oh no! They've really gotten a hold of the system.

• Lesson learned: not all unfamiliar processes are bad. I later figured out that these are part of the ndt-server rpm and normal parts of perfsonar.
• Recommendation: rename fakewww to something meaningful and less scary to the uninitiated.  ndtwwwhelper might be just as good.

Because of the potential for root exploits I ran rpm verifies of core commands (eg rpm -V procps) some were good  some reported prelink inconsistencies.  This caused some concerns at first but as I narrowed down the exploit it became clear the problems were only due to a prelink bug.

https://access.redhat.com/solutions/25215
https://bugzilla.redhat.com/show_bug.cgi?id=204448

• Lesson learned: other bugs can make things seem worse than they are.
• Recommendation: look up unfamiliar errors before you panic.

Looking further into the state of the system I noticed an '/usr/sbin/sshd -i' process running as apache and an time-wise unrelated httpd process.  lsof showed these were both perl codes running out of /var/tmp/ with established tcp connections off site.  Very suspicious and killed them.

• Lesson learned: some processes are really bad.  The abrtd logged the event of the first entry into the system via apache and showed the command vector was bash.  This is a very helpful log to determine important time lines.
• Recommendation: keep your system up to date.

In the end, I traced the exploit down to the two suspicious perl processes (/var/tmp/x).  They were executing an ircbot as apache.  There was no root access to the system and simply clearing out the installed bots from /var/tmp was a sufficient remedy. There was an attempted install of code to exploit CVE-2013-2094 but thank fully that's a 3.8 kernel bug and perfsonar is still on 2.6.

I hope this experience can be useful to others and that the recommendations can be incorporated into future releases as warranted.

~jpr