perfSONAR User Q&A and Other Discussion

["Uhl, George D. (GSFC-423.0)[SGT INC]" <>]

Hi Roderick,

I¹ve implemented an email alerting system in Nagios using same OWAMP
checks that are used in Maddash.  It uses the same check time intervals as
my dashboard but because it runs independently the alerts are not
necessarily in sync with dashboard status changes. I thought about
enabling alerts on throughput tests, but I run them on 4 hour intervals so
alerts over that period of time wouldn¹t be very helpful.  It was
important to limit Nagios alerts so that they wouldn¹t spam my inbox, so I
made sure that the alerts would be emailed only after three consecutive
checks indicate a status change similar to what is done in Maddash.

George



On 6/30/15, 9:20 AM, 
"
 on behalf of
Roderick Mooi" 
<
 on behalf of
>
 wrote:

>Hi Andy
>
>Thanks. We want to avoid the need for anyone to keep the MaDDash page
>open all the time (or put it on the NOC wall) so email/sms/etc alerts to
>say something is wrong prompting them to open MaDDash for more info would
>work - this is what we¹re aiming for. You raise some important
>considerations (flooding, etc.) which we¹ll have to work outŠ
>
>Regards,
>
>Roderick
>
>
>
>> On 30 Jun 2015, at 12:36 PM, Andrew Lake 
>> <>
>>  wrote:
>> 
>> Hi Roderick,
>> 
>> I wouldn¹t necessarily call it recommended, but you can do it. It might
>>take some work for you to get the settings right so you aren¹t flooded
>>with emails every time a host goes down or similar, but it should
>>technically work. We generally only use nagios for things like standard
>>health checks at the moment (e.g. is something listening on port X, etc)
>>and use MaDDash to run the performance checks. We tried nagios with the
>>performance checks for awhile, but we never found the right
>>configuration that captured the inherent two-dimensial nature of the
>>checks (since they have two endpoints) and felt like we were having
>>trouble pinpointing problems. This was one of the initial motivators for
>>MaDDash. This is the approach that groups like ESnet and the WLCG use
>>after experimenting with various options, but it¹s certainly not the
>>only way to do it. There are probably things we haven¹t tried and are
>>always open to new ideas.
>> 
>> Thanks,
>> Andy
>> 
>> 
>> 
>> 
>> On Tue, Jun 30, 2015 at 2:55 AM, Roderick Mooi 
>> <>
>>  wrote:
>> 
>> Sorry, perl scripts (not python)Š
>> 
>> > On 30 Jun 2015, at 8:51 AM, Roderick Mooi 
>> > <>
>> >  wrote:
>> > 
>> > Hi Andy, 
>> > 
>> > Thanks for the clarification. In terms of a temporary solution - is
>>the Nagios integration using the python scripts the way to go?
>> > 
>> > Thanks, 
>> > 
>> > Roderick 
>> > 
>> >> On 29 Jun 2015, at 4:43 PM, Andrew Lake 
>> >> <>
>> >>  wrote:
>> >> 
>> >> Hi, 
>> >> 
>> >> Those will not be in 3.5 unfortunately. They will be in the next
>>version of MaDDash which is likely Q4 of this year at the soonest. A
>>MaDDash 2.0 planing doc is on the 3.5 page, but in terms of he final
>>timeline it won¹t be out until later this year since it¹s release cycle
>>isn¹t necessarily the same as the rest of the toolkit. Also, for future
>>reference, we no longer use google code, and our wiki is now at
>>https://github.com/perfsonar/project/wiki.It is one of the most
>>requested features, we just want to make sure we do it right when we do
>>it. In particular there is already some effort underway to identify
>>common patterns in the dashboards and alert on those as opposed to
>>flooding administrators with alerts. Hope that answers your questions.
>> >> 
>> >> Thanks, 
>> >> Andy 
>> >> 
>> >> 
>> >> 
>> >> 
>> >> On Mon, Jun 29, 2015 at 10:24 AM, Roderick Mooi 
>> >> <>
>>wrote: 
>> >> 
>> >> Hi Jason, Andy
>> >> 
>> >> We would very much like the ability to have emails sent to us for
>>the various MaDDash thresholds. Our current plan is doing some kind of
>>Nagios integration. I¹ve found some information on this here:
>> >> https://code.google.com/p/perfsonar-ps/wiki/NagiosDevelopment
>> >> and 
>> >> https://github.com/perfsonar/nagios
>> >> 
>> >> While looking around though, I came across the 3.5 roadmap and also
>> >> 
>>https://code.google.com/p/perfsonar-ps/wiki/LargeScaleDeploymentFeatures#
>>Notifications_/_Alarms
>> >> 
>> >> which is exactly what we¹re looking for :)
>> >> 
>> >> So my questions:
>> >> While we are eagerly waiting for this (in 3.5 hopefully), should we
>>follow the first two links and effectively get our Nagios system to
>>generate the email alerts based on what it reads from our central MA? Is
>>this the best approach and are there any snags we should know about?
>> >> 
>> >> Thanks very much,
>> >> 
>> >> Roderick 
>> >> -- 
>> >> This message is subject to the CSIR's copyright terms and
>>conditions, e-mail legal notice, and implemented Open Document Format
>>(ODF) standard. 
>> >> The full disclaimer details can be found at
>>http://www.csir.co.za/disclaimer.html.
>> >> 
>> >> This message has been scanned for viruses and dangerous content by
>>MailScanner, 
>> >> and is believed to be clean.
>> >> 
>> >> Please consider the environment before printing this email.
>> >> 
>> >> 
>> >> 
>> >> 
>> >> -- 
>> >> This message is subject to the CSIR's copyright terms and
>>conditions, e-mail legal notice, and implemented Open Document Format
>>(ODF) standard. 
>> >> The full disclaimer details can be found at
>>http://www.csir.co.za/disclaimer.html.
>> >> 
>> >> This message has been scanned for viruses and dangerous content by
>>MailScanner, 
>> >> and is believed to be clean.
>> >> 
>> >> 
>> >> Please consider the environment before printing this email.
>> >> 
>> > 
>> 
>> 
>> -- 
>> This message is subject to the CSIR's copyright terms and conditions,
>>e-mail legal notice, and implemented Open Document Format (ODF)
>>standard. 
>> The full disclaimer details can be found at
>>http://www.csir.co.za/disclaimer.html.
>> 
>> This message has been scanned for viruses and dangerous content by
>>MailScanner, 
>> and is believed to be clean.
>> 
>> Please consider the environment before printing this email.
>> 
>> 
>> 
>> 
>> -- 
>> This message is subject to the CSIR's copyright terms and conditions,
>>e-mail legal notice, and implemented Open Document Format (ODF)
>>standard. 
>> The full disclaimer details can be found at
>>http://www.csir.co.za/disclaimer.html.
>> 
>> This message has been scanned for viruses and dangerous content by
>>MailScanner, 
>> and is believed to be clean.
>> 
>> 
>> Please consider the environment before printing this email.
>> 
>
>
>-- 
>This message is subject to the CSIR's copyright terms and conditions,
>e-mail legal notice, and implemented Open Document Format (ODF) standard.
>The full disclaimer details can be found at
>http://www.csir.co.za/disclaimer.html.
>
>This message has been scanned for viruses and dangerous content by
>MailScanner, 
>and is believed to be clean.
>
>Please consider the environment before printing this email.
>