perfSONAR User Q&A and Other Discussion

[Jason Zurawski <>]

Hey Trey;

It looks like the yum-cron thing was asked and answered, so no need to double 
answer that.  

Start some new threads on your other points - no need to muddy the waters 
since those issues are greater than the bash shell. 

Thanks;

-jason

On Oct 1, 2014, at 8:59 PM, Trey Dockendorf 
<>
 wrote:

> John-Paul,
> 
> Very useful write up, thank you!
> 
> My two perfSONAR boxes were also reported to me as running botnets by my 
> campus' network group.  Fortunately we work closely with them on managing 
> the perfSONAR presence at this campus so the fact that I had a comprised 
> system was punishment enough :)
> 
> The general fault on my systems was not updating them soon enough.
> 
> Brian,
> 
> What method of auto-update is being considered?  I ask because one of the 
> RedHat based operating systems surprised me by including "yum-autoupdate" 
> as a default, and enabled, service in minimal installs without the option 
> to go into a "notify only" mode.  Upstream has the "yum-cron" which has 
> such an option.
> 
> Jason,
> 
> So far of the numerous mailing lists I'm on, the PerfSONAR list has 
> provided the most information regarding this particular vulnerability.  
> Aside from pushing out updates outside of a site's control, there's not 
> much more that can be done that hasn't already been done.
> 
> Unfortunately my perfsonar box became to me an "orphaned" pS box.  I look 
> at it occasionally when users complained about network issues between us 
> and a remote site, but besides that it was 'out-of-mind'.  Would this 
> thread, or a new one, be more appropriate to discuss some of your points 
> (IDS, monitoring of pS, etc)?
> 
> - Trey
> 
> 
> =============================
> 
> Trey Dockendorf 
> Systems Analyst I 
> Texas A&M University 
> Academy for Advanced Telecommunications and Learning Technologies 
> Phone: (979)458-2396 
> Email: 
> 
>  
> Jabber: 
> 
> 
> On Wed, Oct 1, 2014 at 6:17 PM, Jason Zurawski 
> <>
>  wrote:
> Hey John-Paul;
> 
> To echo Brian, thanks for your thoughtful note.  One quick point, that some 
> may not be aware of if they have been on this list for 7 years, is that 
> there is a low volume (lower than the user list at least…) announcement 
> list:
> 
>         https://lists.internet2.edu/sympa/subscribe/perfsonar-announce
> 
> I can also use this opportunity summarize some of the actions we will take 
> based on this, and some other internal discussions occurring of the prior 
> days:
> 
>  - As Brian noted, a sensible default method for automatic updates is 
> coming.  This will not be a panacea for security or maintenance of course - 
> and some of our development team has grave concerns about lulling anyone 
> int a false sense of security and making things far far worse then next 
> time a piece of systems software outside of our control barfs.   The bottom 
> line is still that each site is responsible for care, feeding, and sensible 
> sysadmin practices, and we feel that's a statement everyone agrees with.  
> We do (and will continue to) assist where we can with automated software to 
> observe and upgrade (IDSs, yum, etc.), and we will also augment any 
> technical solution with a 'sysadmin 101' guide for those that need it.  
> This is in-progress, and will also be something we would encourage 
> community input with over time.
> 
>  - As a P.S. to the previous bullet - perfSONAR is not an appliance, and 
> 2014 was the year that really made everyone (community members, 
> stakeholders, developers)  reconsider what it should be.  Recently someone 
> noted that the word 'appliance' strikes images of a box with no seems, that 
> may not have serviceable parts inside.  Linux, unless it has been 
> *drastically* altered, doesn't fit this definition.  As Heartbleed, 
> Shellshock, and other CVEs have shown, no matter how much we work to make 
> something 'easy', it is not invincible, and admitting a failure is part of 
> the road to recovery.  We want to change the perception of the orphaned pS 
> box, as it was the orphaned box that was pegged and owned as an easy target 
> last Friday morning all around the world.
> 
>  - We intend to document what a 'normal' list of running toolkit processes 
> look like, to prevent people from having that feeling of panic when they 
> see a strange perl script running (correctly, or maliciously).  This is a 
> very obvious thing that many of us wouldn't have thought about.
> 
>  - Some products (e.g. Cacti, JOWAMP) will be removed to reduce (not 
> eliminate) the risk footprint, and those that choose to use them will have 
> to make a conscious choice to download and install the tools.
> 
> We do appreciate the communities understanding, feedback, and patience this 
> year.  Please continue to offer feedback as you all see fit, either to the 
> user's list or to the developers directly 
> ().
> 
> Thanks;
> 
> -jason
> 
> On Oct 1, 2014, at 5:14 PM, John-Paul Robinson 
> <>
>  wrote:
> 
> > To other shellshocked perfsonar users:
> >
> > Our perfsonar node did not have automatic yum updates enabled and was 
> > impacted by a shellshock-related exploit on Sept 26.   This is was after 
> > both bash updates had been announced on the perfsonar-user lists, so we 
> > may have survived had automatic updates been enabled by default.
> >
> >       • Lesson learned: run automatic updates.
> >       • Recommendation: It might benefit users to have it default to on 
> > in the perfsonar distribution.  Also it would be good if updates were 
> > checked for more than once a day.  In our case we would have missed the 
> > update mid-day on Sept 26 and may still have gotten exploited before the 
> > next run at 4:00 am on Sept 27.  Additionally a perfsonar-announce list 
> > might be useful for hearing stuff even when you have -user discussions 
> > turned off.
> >
> > After receiving a local exploit report I went to check on the machine and 
> > immediately noticed Apache had restarted.  Alarmingly, a root-owned 
> > process called fakewww also started at the same start time and oddly so 
> > did one named web100srv.  Both of these processes had open ports and logs 
> > open.  Yikes, they got root!  Killed them.  But then they came back after 
> > I restarted httpd, even after `rpm -V httpd` showed no corruption. Oh no! 
> > They've really gotten a hold of the system.
> >       • Lesson learned: not all unfamiliar processes are bad. I later 
> > figured out that these are part of the ndt-server rpm and normal parts of 
> > perfsonar.
> >       • Recommendation: rename fakewww to something meaningful and less 
> > scary to the uninitiated.  ndtwwwhelper might be just as good.
> >
> > Because of the potential for root exploits I ran rpm verifies of core 
> > commands (eg rpm -V procps) some were good  some reported prelink 
> > inconsistencies.  This caused some concerns at first but as I narrowed 
> > down the exploit it became clear the problems were only due to a prelink 
> > bug.
> >
> > https://access.redhat.com/solutions/25215
> > https://bugzilla.redhat.com/show_bug.cgi?id=204448
> >
> >       • Lesson learned: other bugs can make things seem worse than they 
> > are.
> >       • Recommendation: look up unfamiliar errors before you panic.
> >
> > Looking further into the state of the system I noticed an '/usr/sbin/sshd 
> > -i' process running as apache and an time-wise unrelated httpd process.  
> > lsof showed these were both perl codes running out of /var/tmp/ with 
> > established tcp connections off site.  Very suspicious and killed them.
> >
> >       • Lesson learned: some processes are really bad.  The abrtd logged 
> > the event of the first entry into the system via apache and showed the 
> > command vector was bash.  This is a very helpful log to determine 
> > important time lines.
> >       • Recommendation: keep your system up to date.
> >
> > In the end, I traced the exploit down to the two suspicious perl 
> > processes (/var/tmp/x).  They were executing an ircbot as apache.  There 
> > was no root access to the system and simply clearing out the installed 
> > bots from /var/tmp was a sufficient remedy. There was an attempted 
> > install of code to exploit CVE-2013-2094 but thank fully that's a 3.8 
> > kernel bug and perfsonar is still on 2.6.
> >
> > I hope this experience can be useful to others and that the 
> > recommendations can be incorporated into future releases as warranted.
> >
> > ~jpr