perfSONAR User Q&A and Other Discussion

["Jeff W. Boote" <>]

I would recommend attempting to run bwctl tests on the command line.  
And if that doesn't work, try iperf directly with each one of the  
ports that bwctl is defined to use.

Every single one of the errors I see in this email looks like the  
iperf data is null - either because the iperf connection didn' t  
happen at all, or because no data was received.

My off-the-cuff, not enough data to really determine, guess is that  
there is a firewall interfering with the iperf communication.

jeff

On Feb 27, 2009, at 8:13 AM, Richard Carlson wrote:

> All;
>
> John sent me a tarball with the contents of the /var/log directory.   
> I looked through the syslog files and see the bwcollector.pl script  
> writing log entries.  I can put this tarball somewhere, any  
> suggestions (my home dir on packrat?).
>
> I did notice that the BNL node is contacting the UC node, but it  
> usually fails.  Most of the time it says
> Feb 23 08:49:28 NPToolkit bwcollector.pl[3983]: Use of uninitialized  
> value in numeric lt (<) at /usr/local/AMI/script/bwcollector.pl line  
> 960, <SESS> line 16.
> Feb 23 08:49:28 NPToolkit bwcollector.pl[3983]: Use of uninitialized  
> value in subtraction (-) at /usr/local/AMI/script/bwcollector.pl  
> line 960, <SESS> line 16.
> Feb 23 08:49:28 NPToolkit bwcollector.pl[3983]: IGNORED: termination  
> problem KNOPPIX-BWTCP4_UCT2-NET1_UCHICAGO_EDU_KNOPPIX file = /var/ 
> lib/pSB_MP/bwctl/upload/p8gFNrjVZI at /usr/local/AMI/script/ 
> bwcollector.pl line 961, <SESS> line 16.
>
> A few times it recorded a contact with the UC node, but no test was  
> run
> Feb 23 09:27:07 NPToolkit bwcollector.pl[3983]: BEGIN IGNORED: empty  
> KNOPPIX-BWTCP4_KNOPPIX_UCT2-NET1_UCHICAGO_EDU file = /var/lib/pSB_MP/ 
> bwctl/upload/lt0YQJGqAN at /usr/local/AMI/script/bwcollector.pl line  
> 941, <SESS> line 10.
> Feb 23 09:27:07 NPToolkit bwcollector.pl[3983]: /var/lib/pSB_MP/ 
> bwctl/upload/lt0YQJGqAN: bwctl: exec_line: /usr/local/bin/iperf -B  
> 192.12.15.23 -s -f b -m -p 5008 -t 10 -i 2
> Feb 23 09:27:07 NPToolkit bwcollector.pl[3983]: /var/lib/pSB_MP/ 
> bwctl/upload/lt0YQJGqAN: bwctl: start_tool: 3444388022.863175
> Feb 23 09:27:07 NPToolkit bwcollector.pl[3983]: /var/lib/pSB_MP/ 
> bwctl/upload/lt0YQJGqAN: bind failed: Address already in use
> Feb 23 09:27:07 NPToolkit bwcollector.pl[3983]: /var/lib/pSB_MP/ 
> bwctl/upload/lt0YQJGqAN:  
> ------------------------------------------------------------
> Feb 23 09:27:07 NPToolkit bwcollector.pl[3983]: /var/lib/pSB_MP/ 
> bwctl/upload/lt0YQJGqAN: Server listening on TCP port 5008
> Feb 23 09:27:07 NPToolkit bwcollector.pl[3983]: /var/lib/pSB_MP/ 
> bwctl/upload/lt0YQJGqAN: Binding to local address 192.12.15.23
> Feb 23 09:27:07 NPToolkit bwcollector.pl[3983]: /var/lib/pSB_MP/ 
> bwctl/upload/lt0YQJGqAN: TCP window size: 87380 Byte (default)
> Feb 23 09:27:07 NPToolkit bwcollector.pl[3983]: /var/lib/pSB_MP/ 
> bwctl/upload/lt0YQJGqAN:  
> ------------------------------------------------------------
> Feb 23 09:27:07 NPToolkit bwcollector.pl[3983]: /var/lib/pSB_MP/ 
> bwctl/upload/lt0YQJGqAN: bwctl: remote peer cancelled test
> Feb 23 09:27:07 NPToolkit bwcollector.pl[3983]: /var/lib/pSB_MP/ 
> bwctl/upload/lt0YQJGqAN: bwctl: stop_exec: 3444388027.477541
> Feb 23 09:27:07 NPToolkit bwcollector.pl[3983]: END IGNORED: empty  
> KNOPPIX-BWTCP4_KNOPPIX_UCT2-NET1_UCHICAGO_EDU file = /var/lib/pSB_MP/ 
> bwctl/upload/lt0YQJGqAN at /usr/local/AMI/script/bwcollector.pl line  
> 946, <SESS> line 20.
>
> I did find at least 1 example where the test successfully ran.
>
> What's the next step?
>
> Rich
>
> On Feb 25, 2009, at 1:27 PM, Jeff W. Boote wrote:
>
> > Then initially, I would suggest that the BNL host is not even  
> > contacting the chicago host. (At least not the one that corresponds  
> > to this log file.) Otherwise, there would be more connections logged.
> >
> > jeff
> >
> > On Feb 25, 2009, at 12:20 PM, Richard Carlson wrote:
> >
> > > Jeff;
> > >
> > > These are the log files form the receiving side.  The pS-SB is  
> > > running on the BNL node.
> > >
> > > John, would you please create a tarball of the /var/log dir on  
> > > lhcmon and email/post it on a URL?
> > > Thanks.
> > >
> > > Rich
> > >
> > > On Feb 25, 2009, at 12:32 PM, Jeff W. Boote wrote:
> > >
> > > > Rich - in looking at that log dir, I don't see any of the  
> > > > regularly scheduled tests... Only some on-demand ones. These logs  
> > > > don't seem to match the data you describe below... where are you  
> > > > seeing this data?
> > > >
> > > > jeff
> > > >
> > > > On Feb 25, 2009, at 10:21 AM, Richard Carlson wrote:
> > > >
> > > > > Hi Aaron;
> > > > >
> > > > > Charles created a tarball with a complete dump from the /var/log  
> > > > > directory on the UC machine.  Unfortunately I can't tell what is  
> > > > > causing the tests to fail.  I can see requests coming in from  
> > > > > the BNL node, but nothing jumps out at me as to why the test  
> > > > > failed.  What should we be looking for?  (I can also point you  
> > > > > at the tarball if that would help).
> > > > >
> > > > > Rich
> > > > >
> > > > > On Feb 25, 2009, at 9:02 AM, Aaron Brown wrote:
> > > > >
> > > > > > On Feb 25, 2009, at 9:57 AM, Richard Carlson wrote:
> > > > > >
> > > > > > > Hi John;
> > > > > > >
> > > > > > > Thanks for the logs.  I see from the log extract that your  
> > > > > > > server is running 15 sec tests.  However I see other 10 & 60  
> > > > > > > sec requests, so I'm assuming that the 10 sec tests are from  
> > > > > > > people using the command line version (10 sec is the default  
> > > > > > > test period) and some other servers are scheduling tests via  
> > > > > > > the perfSONAR-Buoy interface on their machine.
> > > > > > >
> > > > > > > Here's what I see right now.  Testing to/from MSU is working,  
> > > > > > > but we are loosing data.  The current graphs show 7 successful  
> > > > > > > tests from MSU --> BNL starting at 1:47 am and ending at  
> > > > > > > 8:27.  The 4:45 am test is missing.  The graphs also show 4  
> > > > > > > tests in the opposite direction missing 1, 3, & 6 am.
> > > > > > >
> > > > > > > I also see tests to/from OU with 4 tests from OU -> BNL at 1,  
> > > > > > > 2, 7, & 9 am with 3 in the opposite direction at midnight, 1,  
> > > > > > > & 7 am.  The rest were unsuccessful.
> > > > > > >
> > > > > > > I did notice that you are testing to the psum01 node, which I  
> > > > > > > think Shawn setup as the delay server so that's why none of  
> > > > > > > those work.  I think you want to test to the psum02 node.
> > > > > > >
> > > > > > > I'm still wondering if your server is just trying to run too  
> > > > > > > many tests (19 peers in your config) and I don't know how many  
> > > > > > > other servers are requesting tests.  As I see it, we can try  
> > > > > > > reducing the number of peers, or we can get the pS folks to  
> > > > > > > help guide us on determining why some tests are failing and  
> > > > > > > why some peers aren't responding at all.
> > > > > > >
> > > > > > > Jeff/Aaron, what log files should we be looking at to  
> > > > > > > determine what is going on?
> > > > > >
> > > > > > I'd take a look in /var/log/messages, the output should be  
> > > > > > going there.
> > > > > >
> > > > > > Cheers,
> > > > > > Aaron
> > > > >
> > > > > Richard Carlson
> > > > >
> > > > > 1000 Oakbrook Dr
> > > > > Ann Arbor, MI  48104
> > > > >
> > > > > P: 734-352-7043
> > > > > C: 630-251-4572
> > >
> > > Richard Carlson
> > >
> > > 1000 Oakbrook Dr
> > > Ann Arbor, MI  48104
> > >
> > > P: 734-352-7043
> > > C: 630-251-4572
>
> Richard Carlson
>
> 1000 Oakbrook Dr
> Ann Arbor, MI  48104
>
> P: 734-352-7043
> C: 630-251-4572