perfSONAR User Q&A and Other Discussion

[Jason Zurawski <>]

Hey John-Paul;

Your suggestion did generate some internal discussion today (and the 
statement 'I want cattle not pets.' made my day).  

Its been on our mind recently to adapt to a similar use case, once some of 
the pieces that the upcoming 3.4 release addresses are in place.  Some of 
components of this exist (e.g. collections of scripts for backup and 
restoration, the ability to offload data to external storage), but its all 
ad-hoc.  

We will certainly take the suggestion under advisement for our next round of 
milestone planning.  

Thanks;

-jason

On Oct 2, 2014, at 11:16 AM, John-Paul Robinson 
<>
 wrote:

> Jason and others,
> 
> Thanks for the very helpful replies and links.  I had missed the
> -announce list but am on it now.  I'm also happy about the forthcoming
> yum cron default on.  The suggestion about multiple updates per day was
> more about when the cron job runs rather than some manual checking.  We
> use puppet for much of the admin here and may incorporate that for
> perfsonar in the future.
> 
> Let me clarify that perfsonar is an invaluable resource to our effort to
> assess networking and build a science dmz.  I couldn't do without it.
> 
> Regarding instances in the orphanage, I too keep only a light overview
> on my system.  It works. I'm happy.  I trust the platform and the
> community building it. I don't expect perfection or that nothing will
> fail.  I accept that from time to time things will need fixing or may be
> even re-installing.  I'm ok with that.
> 
> I'd rather take a more cloud-like or scale-up perspective on the
> platform itself. I want cattle not pets. In other words, I'd rather
> flush it down the toilet and get another one when it fails than worry
> about maintaining the one precious instance I have now.  I don't have
> much time to dedicate to care and feeding.  One of the functions of
> personar is a sensor.  I'd like it if i could just throw a failed sensor
> out and install a new one.
> 
> What I don't want is to lose my telemetry data. I have been running a
> regular batch of tests for several months to gather a performance
> profile from various points of interest.  I look at those throughput
> reports regularly.  They are building a narrative for what we have and
> what we need.
> 
> I would like it If there were a way to secure my history of the data
> collected so that i can either move it forward when i reinstall or view
> it else where.   I don't want a failed sensor to threaten the life of my
> data.   The rebuild recommendations I've seen so far don't appear to
> protect my data.  That's why it was worth the hours I spent verifying my
> existing platform's integrity.
> 
> It would be very helpful to have a way to preserve test history data off
> the platform so that I can look at it with the same interface elsewhere
> (another box or a future box).
> 
> Again, I'm very happy with personar. I've found it very reliable.  Then
> again, I may just be benefiting from others pain, so thanks to those who
> have suffered.
> 
> Keep up the good work and thanks for making network performance data
> gathering so much easier than it was in the past.
> 
> ~jpr
> 
> On 10/01/2014 06:17 PM, Jason Zurawski wrote:
>> Hey John-Paul;
>> 
>> To echo Brian, thanks for your thoughtful note.  One quick point, that 
>> some may not be aware of if they have been on this list for 7 years, is 
>> that there is a low volume (lower than the user list at least…) 
>> announcement list:
>> 
>>      https://lists.internet2.edu/sympa/subscribe/perfsonar-announce
>> 
>> I can also use this opportunity summarize some of the actions we will take 
>> based on this, and some other internal discussions occurring of the prior 
>> days:
>> 
>> - As Brian noted, a sensible default method for automatic updates is 
>> coming.  This will not be a panacea for security or maintenance of course 
>> - and some of our development team has grave concerns about lulling anyone 
>> int a false sense of security and making things far far worse then next 
>> time a piece of systems software outside of our control barfs.   The 
>> bottom line is still that each site is responsible for care, feeding, and 
>> sensible sysadmin practices, and we feel that's a statement everyone 
>> agrees with.  We do (and will continue to) assist where we can with 
>> automated software to observe and upgrade (IDSs, yum, etc.), and we will 
>> also augment any technical solution with a 'sysadmin 101' guide for those 
>> that need it.  This is in-progress, and will also be something we would 
>> encourage community input with over time.   
>> 
>> - As a P.S. to the previous bullet - perfSONAR is not an appliance, and 
>> 2014 was the year that really made everyone (community members, 
>> stakeholders, developers)  reconsider what it should be.  Recently someone 
>> noted that the word 'appliance' strikes images of a box with no seems, 
>> that may not have serviceable parts inside.  Linux, unless it has been 
>> *drastically* altered, doesn't fit this definition.  As Heartbleed, 
>> Shellshock, and other CVEs have shown, no matter how much we work to make 
>> something 'easy', it is not invincible, and admitting a failure is part of 
>> the road to recovery.  We want to change the perception of the orphaned pS 
>> box, as it was the orphaned box that was pegged and owned as an easy 
>> target last Friday morning all around the world.  
>> 
>> - We intend to document what a 'normal' list of running toolkit processes 
>> look like, to prevent people from having that feeling of panic when they 
>> see a strange perl script running (correctly, or maliciously).  This is a 
>> very obvious thing that many of us wouldn't have thought about.  
>> 
>> - Some products (e.g. Cacti, JOWAMP) will be removed to reduce (not 
>> eliminate) the risk footprint, and those that choose to use them will have 
>> to make a conscious choice to download and install the tools.  
>> 
>> We do appreciate the communities understanding, feedback, and patience 
>> this year.  Please continue to offer feedback as you all see fit, either 
>> to the user's list or to the developers directly 
>> ().
>>   
>> 
>> Thanks;
>> 
>> -jason
>> 
>> On Oct 1, 2014, at 5:14 PM, John-Paul Robinson 
>> <>
>>  wrote:
>> 
>>> To other shellshocked perfsonar users:
>>> 
>>> Our perfsonar node did not have automatic yum updates enabled and was 
>>> impacted by a shellshock-related exploit on Sept 26.   This is was after 
>>> both bash updates had been announced on the perfsonar-user lists, so we 
>>> may have survived had automatic updates been enabled by default.
>>> 
>>>     • Lesson learned: run automatic updates.
>>>     • Recommendation: It might benefit users to have it default to on in 
>>> the perfsonar distribution.  Also it would be good if updates were 
>>> checked for more than once a day.  In our case we would have missed the 
>>> update mid-day on Sept 26 and may still have gotten exploited before the 
>>> next run at 4:00 am on Sept 27.  Additionally a perfsonar-announce list 
>>> might be useful for hearing stuff even when you have -user discussions 
>>> turned off.
>>> 
>>> After receiving a local exploit report I went to check on the machine and 
>>> immediately noticed Apache had restarted.  Alarmingly, a root-owned 
>>> process called fakewww also started at the same start time and oddly so 
>>> did one named web100srv.  Both of these processes had open ports and logs 
>>> open.  Yikes, they got root!  Killed them.  But then they came back after 
>>> I restarted httpd, even after `rpm -V httpd` showed no corruption. Oh no! 
>>> They've really gotten a hold of the system.
>>>     • Lesson learned: not all unfamiliar processes are bad. I later 
>>> figured out that these are part of the ndt-server rpm and normal parts of 
>>> perfsonar.
>>>     • Recommendation: rename fakewww to something meaningful and less 
>>> scary to the uninitiated.  ndtwwwhelper might be just as good.
>>> 
>>> Because of the potential for root exploits I ran rpm verifies of core 
>>> commands (eg rpm -V procps) some were good  some reported prelink 
>>> inconsistencies.  This caused some concerns at first but as I narrowed 
>>> down the exploit it became clear the problems were only due to a prelink 
>>> bug. 
>>> 
>>> https://access.redhat.com/solutions/25215
>>> https://bugzilla.redhat.com/show_bug.cgi?id=204448
>>> 
>>>     • Lesson learned: other bugs can make things seem worse than they are.
>>>     • Recommendation: look up unfamiliar errors before you panic.
>>> 
>>> Looking further into the state of the system I noticed an '/usr/sbin/sshd 
>>> -i' process running as apache and an time-wise unrelated httpd process.  
>>> lsof showed these were both perl codes running out of /var/tmp/ with 
>>> established tcp connections off site.  Very suspicious and killed them.
>>> 
>>>     • Lesson learned: some processes are really bad.  The abrtd logged 
>>> the event of the first entry into the system via apache and showed the 
>>> command vector was bash.  This is a very helpful log to determine 
>>> important time lines.
>>>     • Recommendation: keep your system up to date.
>>> 
>>> In the end, I traced the exploit down to the two suspicious perl 
>>> processes (/var/tmp/x).  They were executing an ircbot as apache.  There 
>>> was no root access to the system and simply clearing out the installed 
>>> bots from /var/tmp was a sufficient remedy. There was an attempted 
>>> install of code to exploit CVE-2013-2094 but thank fully that's a 3.8 
>>> kernel bug and perfsonar is still on 2.6.
>>> 
>>> I hope this experience can be useful to others and that the 
>>> recommendations can be incorporated into future releases as warranted.
>>> 
>>> ~jpr