perfSONAR User Q&A and Other Discussion

[Jason Zurawski <>]

Hi Stephan;

Aside from the fact that you have baited your mail with plurium 
interrogationum, I can answer as such:

 - Any and all devices that feature bash in their list of ingredients are in 
the same boat with regards to the base risk - servers, appliances, embedded 
controllers.  The solution for everyone is to patch with what is available 
from the powers that be, and wait for further instructions if that is 
insufficient to mitigate what was discovered.  As you know, we don't have 
control over that part.  

 - The part we do have control over is the additional packages we add to make 
CentOS 6 into a perfSONAR toolkit.  As with all forms of vulnerability, there 
are attack vectors build around the original risk.  As communicated in a mail 
yesterday - Redhat released several good FAQs in one of their writeups (read 
the 'common configuration examples' section): 
https://access.redhat.com/articles/1200223.  Attack vectors have actors; 
people on the outside looking in, or the inside looking out.  For some of the 
items described in the writeup above, a trusted user would need to be the one 
to use the exploit in many circumstances.  E.g. someone with an account that 
exploits via SSH, or sudo.  Others are public facing, namely the HTTPD vector 
through CGI scripts that invoked bash functionality.  The fine people at F5 
have a great diagram, you can find it here:

https://f5.com/solutions/mitigation/mitigating-the-bash-shellshock-cve-2014-6271-and-cve-2014-7169-vulnerabilities

The bottom line is that invoking any bash functionality from something that 
faces outward leaves an unpatched system at risk to the disclosed CVEs.  It 
could be use of a variable, it could be a call to a function, it could be a 
single line of code that featured a metacharacter interpreted by the parser.  
 

 - Other applications have faced similar risks against this attack vector.  I 
didn't do an exhaustive search, but any web-based application that needed to 
use bash for some reason was hit with the same frying pan that we were.  Even 
some of our software that didn't specifically call bash may have been hurt - 
in our case we never explicitly used bash, but a call to /bin/sh can turn 
into a call to bash on certain systems.  Something we relayed over the prior 
days were changes that were made to the portions of the toolkit that were 
found to specifically or inadvertently expose bash in this manner.  We will 
take the demerit for being sloppy, the cardinal rule of increasing exposure 
to a systems internals to outside fiddling was broken in these regards.  We 
weren't the first, and we won't be the last. 

 - We have done our due diligence, as have others.  No one wants to spend the 
day apologizing, but experience has shown that salt cannot mine itself.  2 
weeks ago, many people would not have thought twice about this (the smart 
security people would have, in their dashing capes).  Many may have also not 
wanted to believe botnets scanning every corner of the internet running a 
simple HTTP get that featured some malicious intent.  As per the HTTP logs 
from some of you, we have seen this occurred Friday - only shortly after the 
patch to prevent it became 'live'.  

I will close this out with the statement that all software development comes 
with risk when it is integrated from an outsiders perspective.  Many people 
trusted that openssl was impenetrable, and heartbleed changed that.  
Shellshock pointed out a similar fault and we are all cleaning up the pieces. 
 Next week could bring something else catastrophic in some other major 
software package.  In all cases we can play wednesday morning quarterback and 
find faults in applications like perfSONAR, call them insecure, threaten to 
walk away, bemoan extra work needed to keep them operational.  I don't have 
answers for those that are in strong disagreement, and those in strong 
disagreement will stand on the side after walking away still shouting 
incendiary statements no matter what.  We would expect nothing less.  

The point of this, the prior, and other [TL;DR] previous mails is to 
acknowledge that we don't have all the answers.  We want to make a useful 
product, and will try our hardest to succeed in doing so, minimizing the risk 
where we can - these are our foremost two concerns.  We attempt to minimize 
the maintenance variable, but this will be a 'pick 2' solution, it is nearly 
impossible to guarantee all three.  

Have the discussion with your local people if you feel the cost benefit on 
perfSONAR has disappeared, and as always let us know how we can be helpful.  

Thanks;

-jason

On Oct 2, 2014, at 9:49 AM, Stefan Piperov 
<>
 wrote:

> Jason, indeed Brown is running central perfSONAR servers. We installed an 
> additional, dedicated one, on our network segment, because we wanted the 
> ability to test the throughput down to the very last point before it 
> reaches our Tier3 machines.
> 
> I guess the question that I am asking - and probably many others are - is 
> why is perfSONAR so much more vulnerable in its default installation than 
> our other servers, given that they are based basicaly on the same operating 
> system?  Shouldn't the defaults - like the ones for httpd you have 
> currently described on the homepage of the project - be tightened up to the 
> max, and then having local admins releave them as needed?
> Because most of us here - the large majority of these 700 users that you 
> quote - approach perfSONAR as users, and not as developers of any kind.
> We know how to administer systems, and we can tell a problematic one when 
> we see one. I'll put it this way: In its current state and configuration, 
> my perfSONAR server requires a disproportionate amount of effort to 
> maintain secure.
> 
> Perhaps the project should aim at releasing a (probably scaled down?) 
> production version, which can be run in a typical low-maintenance mode as 
> we are used to see from other SL/OSG-based packages, and separate the new 
> and fancy features in a development version?
> 
> Regards,
> Stefan.
> 
> 
> On Thu, 2 Oct 2014, Jason Zurawski wrote:
> 
>> Hi Stefan;
>> 
>> The argument cannot be boiled down to 'will it cause people to flee if it 
>> is hard', it comes down to putting out the best quality product that we 
>> can.  'Product' here is a combination of the software and the BCP that 
>> goes with it, and when we are wrong we need to own up to it - the 
>> appliance mentality is wrong.  We cannot guarantee the 'appliance' use 
>> case - it does a disservice to our product and our customers to pretend we 
>> can.  We are not a 24/7/365 effort.  We have about 20 core people, 
>> devoting anywhere from 1/64th to all to all of their time, and we are 
>> split between many timezones.  Appliances require almost centralized 
>> control to deliver on their promises, its just not something a pokey open 
>> source project can do without significant people and funding.
>> 
>> We can continue to be responsive to user suggestions, complaints, 
>> questions, and praise - we are a long way from the 1st release of this 
>> tool and there has been significant upward progress.  As you note, It is a 
>> tool like a hammer, you use it to solve a task.  There are many forms of 
>> hammers - lets go back to imagining the manual version, not the 
>> hydraulically operated kind.
>> 
>> One my of my previous notes made reference to 'sysadmin 101'.  What 
>> follows is not meant to be an elitist statement: not everyone can be a 
>> sysadmin.  The world needs scientists, sysadmins, and people to set up 
>> pins in a bowling alley.  If maintenance of the machine is too hard for 
>> one person to manage - finding someone else who can assist is the best 
>> course of action.  Someone who knows how to be a sysadmin can set up 
>> automated monitoring and maintenance routines - this is their job after 
>> all.  They don't log on to the machine 2 times a day to apply patches, 
>> that would be ludicrous.  They do watch security lists, react when needed, 
>> and trust that automated methods are doing what they need to be doing.  
>> There needs to be a better range of care between 'never patch since its an 
>> appliance' and 'patch hourly', and it is possible to strike a balance.
>> 
>> The proper answer, that I am going to read into from your mail below, is 
>> that you need to sit down and talk with your network security and systems 
>> administration people about reaching a better way to handle the care and 
>> feeding of this machine.
>> 
>> - Perhaps the answer is 'you shouldn't use it', and maybe that is best.  
>> If we look at other servers on a campus (DNS, Mail, OSG Compute nodes) 
>> they all have an administrator that need to grease the gears from time to 
>> time.  If the perfSONAR node is treated in a different manner because the 
>> word 'appliance' was associated with it, that needs to change toward a 
>> model where its a server like the others.  The number of users of 
>> perfSONAR is not as important to us as the value people see in using it 
>> along with the ability they possess to maintain it.
>> 
>> - Perhaps the answer will be 'this machine increases the value of CMS 
>> science - so let us help you maintain it'.  Cyberinfrastructure matters to 
>> everyone, not just one overworked physicist.  If there is not 
>> campus/laboratory buy-in, things are going to fail.  It may be the case 
>> your local support staff (who really are helpful people - science and 
>> networking do not need to have an adversarial relationship) can assist.  
>> Maybe it means integrating the machine into a CFEngine/Puppet/etc. system. 
>>  Maybe it means moving it out of your facility to the border of campus.  
>> Maybe it means reducing the number of distributed machines from many to 1 
>> - there are lots of answers.  I don't have these, your local support staff 
>> will.
>> 
>> The networking staff at Brown U are very reasonable, and I have worked 
>> with them (and you) on several issues over the years.  If you would like 
>> someone from the perfSONAR project to help you make the case for this 
>> tool, we would be happy to do so (and that goes for anyone else that is 
>> reading this - all 700 of you).  We make a crappy 'appliance', which is 
>> true, we do however want to try to make a useful tool that doesn't ruin 
>> someone's Friday.
>> 
>> I hope this helps you and others in a similar situation;
>> 
>> Thanks;
>> 
>> -jason
>> 
>> On Oct 2, 2014, at 8:19 AM, Stefan Piperov 
>> <>
>>  wrote:
>> 
>>> On Wed, 1 Oct 2014, Jason Zurawski wrote:
>>> 
>>>> - As a P.S. to the previous bullet - perfSONAR is not an appliance...
>>> 
>>> Jason, are you not afraid that a statement like this will push away many 
>>> users? In our collaboration (CMS) perfSONAR was recommended as a _tool_ 
>>> for diagnosing network problems. A (good) tool normally does not require 
>>> maintenance or babysitting.
>>> 
>>> If I am to spend any significant amount of time (like patching the system 
>>> twice per day, as suggested in one of the responses), I'd rather not use 
>>> perfSONAR at all.
>>> 
>>> My server has been hacked already twice, which puts me in a really bad 
>>> situation in relation to the network security people.
>>> Plus the machine was found 'frozen' a couple of more times during this 
>>> last year that I have uesd perfSONAR.
>>> Quite disappointing.
>>> 
>>> Regards,
>>> Stefan.