perfSONAR User Q&A and Other Discussion

[Casey Russell <>]

Andrew,

     yeah, so um...  I knew that.   Seriously though, wow.  Good catch, I had no idea.  It did it so seamlessly before the upgrade that I never even noticed anything amiss. 

     I'll start doing some more thorough manual testing in addition to seeing if anyone has suggestions on my last email.  Thank you.

Casey Russell

Network Engineer

Kansas Research and Education Network

2029 Becker Drive, Suite 282

Lawrence, KS  66047

(785)856-9820  ext 9809

On Fri, Oct 2, 2015 at 9:20 AM, Andrew Lake <> wrote:

Hi,

Actually I think maybe your dashboard was lying to you before the upgrade. Unless you had configured something in MA by hand, the MeshConfig did not previously split out UDP and TCP tests when it built the MaDDash config. As of 3.5 it does. It was probably just grabbing the TCP results so things were green. Looking at the MaDDash history (clicking on a box and selecting the “History” tab) I am almost postive this is what was happening. The throughput results that were green have throughput values that match exactly between corresponding TCP and UDP boxes. 

 Looking at your MA I don’t see any UDP throughput results either (not just recenlt, ever). This would return some json other than [] if it did have results: http://ps-dashboard.perfsonar.kanren.net/esmond/perfsonar/archive/?format=json&event-type=throughput&ip-transport-protocol=udp

Probably the thing to do is debug why UDP tests are failing on your host. Did your by hand tests match your configured tests in bandwidth and duration? It’s commonly limits issues with UDP tests. Some good logs to look in would be /var/log/perfsonar/owamp_bwctl.log and /var/log/perfsonar/regular_testing.log. Feel free to send those to me as well.

Thanks,

Andy

On Thu, Oct 1, 2015 at 8:36 PM, Humberto Galiza <> wrote:

Hi Casey,

I had the same issue after upgrading. Then, I commented out these lines below on file /opt/perfsonar_ps/mesh_config/etc/gui_agent_configuration.conf, generated my mesh-config again (./opt/perfsonar_ps/mesh_config/bin/generate_gui_configuration), and got my Maddash back.

   <ma_filter>
            ma_filter_name  bw-ignore-first-seconds
            mesh_parameter_name  omit_interval
   </ma_filter>

To be honest, I didn't understand why these lines have caused this issue. But solved. I hope it could help you.

Thanks,

Humberto Galiza ..::.. AmLight - Americas Lightpaths
E-mail:
P:+1 (786) 288-3367
M:+55 (19) 971-445-570
Skype:humbertogaliza

---

De: "Casey Russell" <>
Para:
Enviadas: Quinta-feira, 1 de outubro de 2015 16:02:06
Assunto: [perfsonar-user] Broken Maddash dashboard after updates yesterday.

Group, 

     I updated my systems early on Monday to 3.5 and worked through a couple of early issues to get everything stable by Tuesday.  However yesterday morning I came in to find that my Maddash dashboard had gone yellow again overnight.  I suppose it may (or may not) have been related to updates released the evening of the 29th (I do have automatic updates enabled).

     In short, the latency grid, and the TCP bandwidth testing grids still work fine for most hosts.  On my internal hosts, I've lost (in the Maddash dashboard) all UDP bandwidth testing.  The dashboard shows yellow with the warning text:  " Unable to find any tests with data in the given time range where source is ps-bryant-bw.perfsonar.kanren.net and destination is ps-wsu-bw.perfsonar.kanren.net"

     I've verified that I can manually run these tests between the hosts.  I've also gone back and verified that the bugs I've seen reported on the list over the last couple of days don't seem to be the cause of my issue.  However, I'm stumped as to just what IS going on.  I don't know how to query esmond well enough to see if the UDP test data is there, but this feels to me like a situation where the data is still there, but Maddash has lost it's ability to find it.  In situations where a test was running, but stops, you can typically still click on that Maddash square and see the old data, even if there's a recent blank spot.  In this case, there is not even any historical data. 

     Anyone have any thoughts on where to point me next?  http://ps-dashboard.kanren.net/maddash-webui

Maddash log data:
level=INFO ts=2015-10-01T13:43:47.066342Z event=maddash.RunCheckJob.execute.runCheck.end guid=4159a7e8-c611-42b7-80d3-977a971b2abe resultMsg=" Unable to find any tests with data in the given time range where source is ps-bryant-bw.perfsonar.kanren.net and destination is ps-wsu-bw.perfsonar.kanren.net" col=ps-wsu-bw.perfsonar.kanren.net status=0 resultCode=3 grid="KanREN Mesh - KanREN iPerf Bandwidth UDP testing" row=ps-bryant-bw.perfsonar.kanren.net
level=INFO ts=2015-10-01T13:43:47.071295Z event=maddash.RunCheckJob.execute.runCheck.end guid=c78228e2-32c0-4a24-a463-51c15c4dc067 resultMsg=" Unable to find any tests with data in the given time range where source is ps-bryant-bw.perfsonar.kanren.net and destination is ps-ku-bw.perfsonar.kanren.net" col=ps-ku-bw.perfsonar.kanren.net status=0 resultCode=3 grid="KanREN Mesh - KanREN iPerf Bandwidth UDP testing" row=ps-bryant-bw.perfsonar.kanren.net

Lots more log data where that came from, but I won't pollute the list with it.  Ask if there's something you need to see to help and I'll provide.

Thank you in advance

Casey Russell

Network Engineer

Kansas Research and Education Network

2029 Becker Drive, Suite 282

Lawrence, KS  66047

(785)856-9820  ext 9809