perfSONAR User Q&A and Other Discussion

[Jason Zurawski <>]

Hey John-Paul;

To echo Brian, thanks for your thoughtful note.  One quick point, that some 
may not be aware of if they have been on this list for 7 years, is that there 
is a low volume (lower than the user list at least…) announcement list:

        https://lists.internet2.edu/sympa/subscribe/perfsonar-announce

I can also use this opportunity summarize some of the actions we will take 
based on this, and some other internal discussions occurring of the prior 
days:

 - As Brian noted, a sensible default method for automatic updates is coming. 
 This will not be a panacea for security or maintenance of course - and some 
of our development team has grave concerns about lulling anyone int a false 
sense of security and making things far far worse then next time a piece of 
systems software outside of our control barfs.   The bottom line is still 
that each site is responsible for care, feeding, and sensible sysadmin 
practices, and we feel that's a statement everyone agrees with.  We do (and 
will continue to) assist where we can with automated software to observe and 
upgrade (IDSs, yum, etc.), and we will also augment any technical solution 
with a 'sysadmin 101' guide for those that need it.  This is in-progress, and 
will also be something we would encourage community input with over time.   

 - As a P.S. to the previous bullet - perfSONAR is not an appliance, and 2014 
was the year that really made everyone (community members, stakeholders, 
developers)  reconsider what it should be.  Recently someone noted that the 
word 'appliance' strikes images of a box with no seems, that may not have 
serviceable parts inside.  Linux, unless it has been *drastically* altered, 
doesn't fit this definition.  As Heartbleed, Shellshock, and other CVEs have 
shown, no matter how much we work to make something 'easy', it is not 
invincible, and admitting a failure is part of the road to recovery.  We want 
to change the perception of the orphaned pS box, as it was the orphaned box 
that was pegged and owned as an easy target last Friday morning all around 
the world.  

 - We intend to document what a 'normal' list of running toolkit processes 
look like, to prevent people from having that feeling of panic when they see 
a strange perl script running (correctly, or maliciously).  This is a very 
obvious thing that many of us wouldn't have thought about.  

 - Some products (e.g. Cacti, JOWAMP) will be removed to reduce (not 
eliminate) the risk footprint, and those that choose to use them will have to 
make a conscious choice to download and install the tools.  

We do appreciate the communities understanding, feedback, and patience this 
year.  Please continue to offer feedback as you all see fit, either to the 
user's list or to the developers directly 
().
  

Thanks;

-jason

On Oct 1, 2014, at 5:14 PM, John-Paul Robinson 
<>
 wrote:

> To other shellshocked perfsonar users:
> 
> Our perfsonar node did not have automatic yum updates enabled and was 
> impacted by a shellshock-related exploit on Sept 26.   This is was after 
> both bash updates had been announced on the perfsonar-user lists, so we may 
> have survived had automatic updates been enabled by default.
> 
>       • Lesson learned: run automatic updates.
>       • Recommendation: It might benefit users to have it default to on in 
> the perfsonar distribution.  Also it would be good if updates were checked 
> for more than once a day.  In our case we would have missed the update 
> mid-day on Sept 26 and may still have gotten exploited before the next run 
> at 4:00 am on Sept 27.  Additionally a perfsonar-announce list might be 
> useful for hearing stuff even when you have -user discussions turned off.
> 
> After receiving a local exploit report I went to check on the machine and 
> immediately noticed Apache had restarted.  Alarmingly, a root-owned process 
> called fakewww also started at the same start time and oddly so did one 
> named web100srv.  Both of these processes had open ports and logs open.  
> Yikes, they got root!  Killed them.  But then they came back after I 
> restarted httpd, even after `rpm -V httpd` showed no corruption. Oh no! 
> They've really gotten a hold of the system.
>       • Lesson learned: not all unfamiliar processes are bad. I later 
> figured out that these are part of the ndt-server rpm and normal parts of 
> perfsonar.
>       • Recommendation: rename fakewww to something meaningful and less 
> scary to the uninitiated.  ndtwwwhelper might be just as good.
> 
> Because of the potential for root exploits I ran rpm verifies of core 
> commands (eg rpm -V procps) some were good  some reported prelink 
> inconsistencies.  This caused some concerns at first but as I narrowed down 
> the exploit it became clear the problems were only due to a prelink bug. 
> 
> https://access.redhat.com/solutions/25215
> https://bugzilla.redhat.com/show_bug.cgi?id=204448
> 
>       • Lesson learned: other bugs can make things seem worse than they are.
>       • Recommendation: look up unfamiliar errors before you panic.
> 
> Looking further into the state of the system I noticed an '/usr/sbin/sshd 
> -i' process running as apache and an time-wise unrelated httpd process.  
> lsof showed these were both perl codes running out of /var/tmp/ with 
> established tcp connections off site.  Very suspicious and killed them.
> 
>       • Lesson learned: some processes are really bad.  The abrtd logged 
> the event of the first entry into the system via apache and showed the 
> command vector was bash.  This is a very helpful log to determine important 
> time lines.
>       • Recommendation: keep your system up to date.
> 
> In the end, I traced the exploit down to the two suspicious perl processes 
> (/var/tmp/x).  They were executing an ircbot as apache.  There was no root 
> access to the system and simply clearing out the installed bots from 
> /var/tmp was a sufficient remedy. There was an attempted install of code to 
> exploit CVE-2013-2094 but thank fully that's a 3.8 kernel bug and perfsonar 
> is still on 2.6.
> 
> I hope this experience can be useful to others and that the recommendations 
> can be incorporated into future releases as warranted.
> 
> ~jpr