perfSONAR User Q&A and Other Discussion

[Pedro Queirós <>]

Hey all,

I'm getting several errors in my logs, and would like to know the
probable causes for this. Some errors include:

---
Jul 23 01:01:13 perfsonar-tmp1 powstream[8964]: _OWPWriteFetchSession
called in wrong state.
Jul 23 01:01:13 perfsonar-tmp1 powstream[8964]:
OWPScheduleContextGenerateNextDelta: Schedule complete
Jul 23 01:01:14 perfsonar-tmp1 last message repeated 599 times
---

I suspect the lack of memory in this particular virtual machine is
causing this (512Mb RAM). Today, I've also got:

---
Jul 23 05:22:25 perfsonar-tmp1 kernel: owampd invoked oom-killer:
gfp_mask=0x200d2, order=0, oomkilladj=0
(...)
Jul 23 05:22:27 perfsonar-tmp1 kernel: Out of memory: Killed process
23097, UID 100, (daemon.pl).
Jul 23 05:34:43 perfsonar-tmp1 kernel: sendmail invoked oom-killer:
gfp_mask=0xd0, order=0, oomkilladj=0
(...)
Jul 23 05:34:47 perfsonar-tmp1 kernel: Out of memory: Killed process
23161, UID 100, (daemon.pl).
Jul 23 05:36:18 perfsonar-tmp1 kernel: owampd invoked oom-killer:
gfp_mask=0x200d2, order=0, oomkilladj=0
---


In another virtual machine, with 2Gb RAM, I've also got some errors:

---
Jul 23 01:06:48 perfsonar-tmp2 powmaster.pl[28051]: send_data died,
restarting! at /opt/perfsonar_ps/perfsonarbuoy_ma/bin/powmaster.pl
line 690.
Jul 23 01:06:48 perfsonar-tmp2 powmaster.pl[28051]: Handling SIGHUP...
Stop processing...
Jul 23 01:06:48 perfsonar-tmp2 powstream[9152]: _OWPReadFetchAck:
Unable to read from socket.Jul 23 01:06:48 perfsonar-tmp2
powstream[9152]: SIGTERM/SIGINT Caught: Exiting.
Jul 23 01:06:48 perfsonar-tmp2 powstream[9150]: SIGTERM/SIGINT Caught: 
Exiting.
Jul 23 01:06:48 perfsonar-tmp2 powstream[9149]: SIGTERM/SIGINT Caught:
Exiting.Jul 23 01:06:48 perfsonar-tmp2 powmaster.pl[28051]:
Restarting...: /opt/perfsonar_ps/perfsonarbuoy_ma/bin/powmaster.pl -c
/opt/perfsonar_ps/perfsonarbuoy_ma/etc -Z at /opt/p
erfsonar_ps/perfsonarbuoy_ma/bin/powmaster.pl line 713.
---

In a physical machine, with 512Mb RAM, I've also got some errors, e.g.:

---
Jul 22 06:00:12 sondauminho powstream[2073]: _OWPReadFetchAck: Unable
to read from socket.
Jul 22 06:00:12 sondauminho powstream[2073]: _OWPWriteFetchSession
called in wrong state.
Jul 22 06:00:12 sondauminho last message repeated 39 times
Jul 22 06:00:12 sondauminho powstream[2070]: Read failed: (Connection
reset by peer)
Jul 22 06:00:12 sondauminho powstream[2070]:
OWPControlOpen([193.136.7.81]:861): Couldn't open 'control' connection
to server: Connection reset by peer
Jul 22 06:00:12 sondauminho powstream[2073]: Read failed: (Connection
reset by peer)
Jul 22 06:00:12 sondauminho powstream[2073]:
OWPControlOpen([193.136.7.81]:861): Couldn't open 'fetch' connection
to server: Connection reset by peer
Jul 22 06:00:12 sondauminho powstream[2073]:
OWPControlOpen([193.136.7.81]:861): Waiting 9.796890000 seconds before
retrying
Jul 22 06:00:12 sondauminho powstream[2070]:
OWPControlOpen([193.136.7.81]:861): Couldn't open 'control' connection
to server: Connection refused
Jul 22 06:00:12 sondauminho powstream[2070]:
OWPControlOpen([193.136.7.81]:861): Waiting 8.707051999 seconds before
retrying
Jul 22 06:00:13 sondauminho powstream[2074]: Read failed: (Connection
reset by peer)
(...)
Jul 23 06:05:34 sondauminho powstream[2073]: _OWPWriteFetchSession
called in wrong state.
Jul 23 06:05:34 sondauminho powstream[2073]:
OWPScheduleContextGenerateNextDelta: Schedule complete
Jul 23 06:05:34 sondauminho last message repeated 599 times
---

These _OWPReadFetchAck and _OWPWriteFetchSession /
OWPScheduleContextGenerateNextDelta are the most common.

The only enabled services are perfSONAR-BUOY Latency Testing,
perfSONAR-BUOY Measurement Archive, OWAMP and SSH. I'm only running
OWAMP tests from the physical machine to the 2 VMs and between the
VMs, with the default values.

Feedback would be appreciated, thanks for your time!


Kind Regards,
Pedro