perfSONAR User Q&A and Other Discussion

["Roderick Mooi" <>]

Hi,

Jason and team - thanks for your efforts - going above and beyond really 
means a lot to us.

Some comments on the thread without repeating what others have said:

A colleague and I maintain a deployment of 12 perfSONAR servers. We don't 
have automated updating (we prefer being in control). I watch the mailing 
list closely (no need to read every email every day - just scan the subjects) 
and respond when needed (like installing the patches for these 
vulnerabilities). It was one of the first places I heard about bashapocalpse 
and shellshock. It took about 15 mins total to run and verify the two yum 
updates required for each patch. That's not too much to ask (and the fast 
notices on the mailing list meant I could do it in time).

Note: the mailing list is invaluable. Someone with access to your server 
should be watching it regularly (filtering for important subjects if needed).

-- 
Senior Engineer: South African National Research Network (SANReN)
Meraka Institute, CSIR


 | +27 12 841 4111 | www.sanren.ac.za


>>> On 2014-10-02 at 15:49, Stefan Piperov 
>>> <>
>>>  wrote:
> I'll put it this way: In its current state and configuration, 
> my perfSONAR server requires a disproportionate amount of effort to 
> maintain secure.

Between bash and openssl vulnerabilities - the only serious security issues 
this year - it took about 2 hours of our time to keep our servers secure. We 
do consider them high-risk machines due to the bandwidth they can consume and 
also being outside of the security parameter.

I believe the perfsonar team has done an admirable job of balancing security 
with performance. We added a few extras of our own but recent updates to 
iptables and including fail2ban made this largely unnecessary.

>>> On 2014-10-02 at 17:16, John-Paul Robinson 
>>> <>
>>>  wrote:

> I want cattle not pets.

Honestly, I don't mind my "pets" - they're not very demanding (and are very 
rewarding ;) ).
But I like this idea (we are almost done with figuring out automated 
installations - kickstart scripts, etc.) so that will help.

> What I don't want is to lose my telemetry data. I have been running a
> regular batch of tests for several months to gather a performance
> profile from various points of interest.  I look at those throughput
> reports regularly.  They are building a narrative for what we have and
> what we need.
> 
> I would like it If there were a way to secure my history of the data
> collected so that i can either move it forward when i reinstall or view
> it else where.  

What about using a central MA? We do and it works well.

>>> On 2014-10-02 at 17:49, Jason Zurawski 
>>> <>
>>>  wrote:

> We want to make a useful 
> product, and will try our hardest to succeed in doing so, minimizing the 
> risk 
> where we can - these are our foremost two concerns.  We attempt to minimize 
> the maintenance variable, but this will be a 'pick 2' solution, it is 
> nearly 
> impossible to guarantee all three.  

I vote for 1, then 2...

> Have the discussion with your local people if you feel the cost benefit on 
> perfSONAR has disappeared, and as always let us know how we can be helpful. 
>  

For a free toolkit we really can't complain :D

Thanks again,

Roderick


>>> On 2014-10-04 at 20:44, Trey Dockendorf 
>>> <>
>>>  wrote:
> I think the reason perfsonar systems got hit harder or may have been
> targeted first is the nature of where these systems live.
> 
> Perfsonar boxes are typically on good quality, high speed networks.  Prime
> target for botnets.
> 
> They also typically reside outside an organization's firewall perimeter.
> At least this is the case at my university.
> 
> The subnets used for such traffic like perfsonar are well known to it seems
> for my campus as non perfsonar boxes on same network are constantly probed
> by people trying to exploit my systems.  Being at a university makes my
> systems a big target due to quality of networking, etc.
> 
> I like idea of cattle vs pets and I plan to make that a reality with
> perfsonar when I can get cycles to do so.  Being able to automate the
> deployment of the systems, as well as backups of metrics collected ,
> configurations, etc.  I plan to start threads on the subject and hope to
> work with community and devs to touch on these subjects.  I'd like to be
> able to treat perfsonar like any system I manage in that it's fully
> redeployable without data loss and with integration to local monitoring and
> security applications.
> 
> Just my 0.02.
> 
> - Trey
> On Oct 2, 2014 10:51 AM, "Alan Whinery" 
> <>
>  wrote:
> 
>> I find the question "why is perfSONAR so much more vulnerable (...)?"
>> surprising. I don't think that it is. Having had zero break-ins in 10 or
>> more boxes, over 4 years, and loss-of-contact only when there was
>> underlying hardware/configuration issues, I have to say tjat your incident
>> numbers are unusually high, and that there probably are unidentified causal
>> factors, other than perfSONAR flaws.
>>
>> Also, the aftermath of shellshock is not a typical scenario. The earth
>> just shifted under our feet. Several kinds of security strategy just sprang
>> to life, like avoiding over-reliance on a single code-base. (I never
>> thought I'd want to install busybox on full sized servers).
>>
>> -Alan
>>
>> On October 2, 2014 3:49:06 AM HST, Stefan Piperov 
>> <>
>> wrote:
>>>
>>>
>>> Jason, indeed Brown is running central perfSONAR servers. We installed an
>>> additional, dedicated one, on our network segment, because we wanted the
>>> ability to test the throughput down to the very last point before it
>>> reaches our Tier3 machines.
>>>
>>> I guess the question that I am asking - and probably many others are - is
>>> why is perfSONAR so much more vulnerable in its default installation than
>>> our other servers, given that they are based basicaly on the same
>>> operating system?  Shouldn't the defaults - like the ones for httpd you
>>> have currently described on the homepage of the project - be tightened up
>>> to the max, and then having local admins releave them as needed?
>>> Because most of us here - the large majority of these 700 users that you
>>> quote - approach perfSONAR as users, and not as developers of any kind.
>>> We know how to administer systems, and we can tell a problematic one when 
>>> <
>>>  br />we
>>> see one. I'll put it this way: In its current state and configuration,
>>> my perfSONAR server requires a disproportionate amount of effort to
>>> maintain secure.
>>>
>>> Perhaps the project should aim at releasing a (probably scaled down?)
>>> production version, which can be run in a typical low-maintenance mode as
>>> we are used to see from other SL/OSG-based packages, and separate the new
>>> and fancy features in a development version?
>>>
>>> Regards,
>>> Stefan.
>>>
>>>
>>> On Thu, 2 Oct 2014, Jason Zurawski wrote:
>>>
>>>  Hi Stefan;
>>>>
>>>>  The argument cannot be boiled down to 'will it cause people to flee if 
>>>> it 
> is hard', it comes down to putting out the best quality product that we 
> can.  
> 'Product' here is a combination of the software and the BCP that goes with 
> it, and when we are wrong we need to own up to it - the ap
>>>>  pliance
>>>> mentality is wrong.  We cannot guarantee the 'appliance' use case - it 
>>>> does 
> a disservice to our product and our customers to pretend we can.  We are 
> not 
> a 24/7/365 effort.  We have about 20 core people, devoting anywhere from 
> 1/64th to all to all of their time, and we are split between many 
> timezones.  
> Appliances require almost centralized control to deliver on their promises, 
> its just not something a pokey open source project can do without 
> significant 
> people and funding.
>>>>
>>>>  We can continue to be responsive to user suggestions, complaints, 
> questions, and praise - we are a long way from the 1st release of this tool 
> and there has been significant upward progress.  As you note, It is a tool 
> like a hammer, you use it to solve a task.  There are many forms of hammers 
> - 
> lets go back to imagining the manual version, not the hydraulically 
> operated 
> kind.
>>>>
>>>>  One my of my previous notes made reference to 'sysadmin 101'.  What 
>>>> follows 
> is not meant to be an elitist stat
>>>>  ement:
>>>> not everyone can be a sysadmin.  The world needs scientists, sysadmins, 
>>>> and 
> people to set up pins in a bowling alley.  If maintenance of the machine is 
> too hard for one person to manage - finding someone else who can assist is 
> the best course of action.  Someone who knows how to be a sysadmin can set 
> up 
> automated monitoring and maintenance routines - this is their job after 
> all.  
> They don't log on to the machine 2 times a day to apply patches, that would 
> be ludicrous.  They do watch security lists, react when needed, and trust 
> that automated methods are doing what they need to be doing.  There needs 
> to 
> be a better range of care between 'never patch since its an appliance' and 
> 'patch hourly', and it is possible to strike a balance.
>>>>
>>>>  The proper answer, that I am going to read into from your mail below, 
>>>> is 
> that you need to sit down and talk with your network security and systems 
> administration people about reaching a better way to handle the care and 
> feeding of this
>>>> machine.
>>>>
>>>>  - Perhaps the answer is 'you shouldn't use it', and maybe that is best. 
>>>>  If 
> we look at other servers on a campus (DNS, Mail, OSG Compute nodes) they 
> all 
> have an administrator that need to grease the gears from time to time.  If 
> the perfSONAR node is treated in a different manner because the word 
> 'appliance' was associated with it, that needs to change toward a model 
> where 
> its a server like the others.  The number of users of perfSONAR is not as 
> important to us as the value people see in using it along with the ability 
> they possess to maintain it.
>>>>
>>>>  - Perhaps the answer will be 'this machine increases the value of CMS 
> science - so let us help you maintain it'.  Cyberinfrastructure matters to 
> everyone, not just one overworked physicist.  If there is not 
> campus/laboratory buy-in, things are going to fail.  It may be the case 
> your 
> local support staff (who really are helpful people - science and networking 
> do not need to have an adversarial relations
>>>>  hip)
>>>> can assist.  Maybe it means integrating the machine into a 
> CFEngine/Puppet/etc. system.  Maybe it means moving it out of your facility 
> to the border of campus.  Maybe it means reducing the number of distributed 
> machines from many to 1 - there are lots of answers.  I don't have these, 
> your local support staff will.
>>>>
>>>>  The networking staff at Brown U are very reasonable, and I have worked 
>>>> with 
> them (and you) on several issues over the years.  If you would like someone 
> from the perfSONAR project to help you make the case for this tool, we 
> would 
> be happy to do so (and that goes for anyone else that is reading this - all 
> 700 of you).  We make a crappy 'appliance', which is true, we do however 
> want 
> to try to make a useful tool that doesn't ruin someone's Friday.
>>>>
>>>>  I hope this helps you and others in a similar situation;
>>>>
>>>>  Thanks;
>>>>
>>>>  -jason
>>>>
>>>>  On Oct 2, 2014, at 8:19 AM, Stefan Piperov 
>>>> <>
>>>>  wrote:
>>>>
>>>>  On Wed, 1 Oct 2014, Jason Zurawski wrote:
>>>>
>>>>  - As a P.S. to the previous bullet - perfSONAR is not an appliance...
>>>>>
>>>>
>>>>  Jason, are you not afraid that a statement like this will push away 
>>>> many 
> users? In our collaboration (CMS) perfSONAR was recommended as a _tool_ for 
> diagnosing network problems. A (good) tool normally does not require 
> maintenance or babysitting.
>>>>
>>>>  If I am to spend any significant amount of time (like patching the 
>>>> system 
> twice per day, as suggested in one of the responses), I'd rather not use 
> perfSONAR at all.
>>>>
>>>>  My server has been hacked already twice, which puts me in a really bad 
> situation in relation to the network security people.
>>>>  Plus the machine was found 'frozen' a couple of mo
>>>>  re
>>>> times during this last year that I have uesd perfSONAR.
>>>>  Quite disappointing.
>>>>
>>>>  Regards,
>>>>  Stefan.
>>>>
>>>
>>>
>> --
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>
> 
> -- 
> This message is subject to the CSIR's copyright terms and conditions, 
> e-mail 
> legal notice, and implemented Open Document Format (ODF) standard. 
> The full disclaimer details can be found at 
> http://www.csir.co.za/disclaimer.html.
> 
> This message has been scanned for viruses and dangerous content by 
> MailScanner, 
> and is believed to be clean.
> 
> Please consider the environment before printing this email.

-- 
This message is subject to the CSIR's copyright terms and conditions, e-mail 
legal notice, and implemented Open Document Format (ODF) standard. 
The full disclaimer details can be found at 
http://www.csir.co.za/disclaimer.html.

This message has been scanned for viruses and dangerous content by 
MailScanner, 
and is believed to be clean.

Please consider the environment before printing this email.