perfSONAR User Q&A and Other Discussion

[Ben Nelson <>]

Hi All,

We've seen an issue pop up a few times where consecutive bwctl tests on
the same host manage to use the same port or iperf testing.  If the
tests are close enough, it seems like the port doesn't get freed up
quickly enough for the second test to run, and ends up failing.  Here
are some log snippets of what we see:

May 24 17:01:53 pas.bldc.grnoc.iu.edu bwcollector.pl[45056]: BEGIN
IGNORED: empty AL3S-BW4_CHIC_LOSA file =
/var/lib/perfsonar/perfsonarbuoy_ma/bwctl/upload/K5pQCMJl0S at
/opt/perfsonar_ps/perfsonarbuoy_m
a/bin/bwcollector.pl line 891, <SESS> line 17.#012
May 24 17:01:53 pas.bldc.grnoc.iu.edu bwcollector.pl[45056]:
/var/lib/perfsonar/perfsonarbuoy_ma/bwctl/upload/K5pQCMJl0S: bwctl:
start_endpoint: 3609939653.158082#012#012
May 24 17:01:53 pas.bldc.grnoc.iu.edu bwcollector.pl[45056]:
/var/lib/perfsonar/perfsonarbuoy_ma/bwctl/upload/K5pQCMJl0S: bwctl:
run_endpoint: receiver: 198.71.46.56#012#012
May 24 17:01:53 pas.bldc.grnoc.iu.edu bwcollector.pl[45056]:
/var/lib/perfsonar/perfsonarbuoy_ma/bwctl/upload/K5pQCMJl0S: bwctl:
run_endpoint: sender: 198.71.46.50#012#012
May 24 17:01:53 pas.bldc.grnoc.iu.edu bwcollector.pl[45056]:
/var/lib/perfsonar/perfsonarbuoy_ma/bwctl/upload/K5pQCMJl0S: bwctl:
exec_line: iperf -B em-2.400.pas-tst.chic.net.internet2.edu -s -f b -l 4487
2 -m -p 5001 -u -w 100663296 -t 20 -i 1.000000#012#012
May 24 17:01:53 pas.bldc.grnoc.iu.edu bwcollector.pl[45056]:
/var/lib/perfsonar/perfsonarbuoy_ma/bwctl/upload/K5pQCMJl0S: bwctl:
run_tool: tester: iperf#012#012
May 24 17:01:53 pas.bldc.grnoc.iu.edu bwcollector.pl[45056]:
/var/lib/perfsonar/perfsonarbuoy_ma/bwctl/upload/K5pQCMJl0S: bwctl:
run_tool: receiver: 198.71.46.56#012#012
May 24 17:01:53 pas.bldc.grnoc.iu.edu bwcollector.pl[45056]:
/var/lib/perfsonar/perfsonarbuoy_ma/bwctl/upload/K5pQCMJl0S: bwctl:
run_tool: sender: 198.71.46.50#012#012
May 24 17:01:53 pas.bldc.grnoc.iu.edu bwcollector.pl[45056]:
/var/lib/perfsonar/perfsonarbuoy_ma/bwctl/upload/K5pQCMJl0S: bwctl:
start_tool: 3609939684.650671#012#012
May 24 17:01:53 pas.bldc.grnoc.iu.edu bwcollector.pl[45056]:
/var/lib/perfsonar/perfsonarbuoy_ma/bwctl/upload/K5pQCMJl0S: bind
failed: Address already in use#012#012
May 24 17:01:53 pas.bldc.grnoc.iu.edu bwcollector.pl[45056]:
/var/lib/perfsonar/perfsonarbuoy_ma/bwctl/upload/K5pQCMJl0S:
------------------------------------------------------------#012#012
May 24 17:01:53 pas.bldc.grnoc.iu.edu bwcollector.pl[45056]:
/var/lib/perfsonar/perfsonarbuoy_ma/bwctl/upload/K5pQCMJl0S: Server
listening on UDP port 5001#012#012
May 24 17:01:53 pas.bldc.grnoc.iu.edu bwcollector.pl[45056]:
/var/lib/perfsonar/perfsonarbuoy_ma/bwctl/upload/K5pQCMJl0S: Binding to
local address em-2.400.pas-tst.chic.net.internet2.edu#012#012
May 24 17:01:53 pas.bldc.grnoc.iu.edu bwcollector.pl[45056]:
/var/lib/perfsonar/perfsonarbuoy_ma/bwctl/upload/K5pQCMJl0S: Receiving
44872 byte datagrams#012#012
May 24 17:01:53 pas.bldc.grnoc.iu.edu bwcollector.pl[45056]:
/var/lib/perfsonar/perfsonarbuoy_ma/bwctl/upload/K5pQCMJl0S: UDP buffer
size: 201326592 Byte (WARNING: requested 100663296 Byte)#012#012
May 24 17:01:53 pas.bldc.grnoc.iu.edu bwcollector.pl[45056]:
/var/lib/perfsonar/perfsonarbuoy_ma/bwctl/upload/K5pQCMJl0S:
------------------------------------------------------------#012#012
May 24 17:01:53 pas.bldc.grnoc.iu.edu bwcollector.pl[45056]:
/var/lib/perfsonar/perfsonarbuoy_ma/bwctl/upload/K5pQCMJl0S: bwctl:
stop_tool: 3609939713.725873#012#012
May 24 17:01:53 pas.bldc.grnoc.iu.edu bwcollector.pl[45056]:
/var/lib/perfsonar/perfsonarbuoy_ma/bwctl/upload/K5pQCMJl0S: bwctl:
stop_endpoint: 3609939713.726364#012#012

Can anything be done to unbind the port quicker, so the second test is
able to finish?  Or possible a change in bwctl testing to choose a port
that hasn't been used recently?

Thanks,
Ben


-- 
Ben Nelson, Systems Engineer
Indiana University GlobalNOC
<>