perfSONAR User Q&A and Other Discussion

[Hyojoon Kim <>]

I was able to fix the problem manually. Hope someone finds this useful when encountered with the same problem: 

Running the following command manually fixes the problem (after running "usr/lib/perfsonar/scripts/mod_interface_route --command add --device em3 --ipv4_gateway 1.2.7.1”):

[root@perfsonar]# ip route add 1.2.7.0/22 dev em3 table em3_source_route

This makes “em3_source_route” to have an additional line as following: 

[root@perfsonar]# ip route list table em3_source_route

default via 1.2.7.1 dev em3 

1.2.7.0/22 dev em3  scope link 

[root@perfsonar]# 

Instead of: 

[root@perfsonar]# ip route list table em3_source_route

default via 1.2.7.1 dev em3 

[root@perfsonar]# 

Thanks,

Joon 

On May 11, 2017, at 11:30 AM, Hyojoon Kim <> wrote:

update: 

Removing both routes improve the performance, back to normal. 

# /usr/lib/perfsonar/scripts/mod_interface_route --command delete --device em2

# /usr/lib/perfsonar/scripts/mod_interface_route --command delete --device em3

-Joon 

On May 11, 2017, at 10:59 AM, Hyojoon Kim <> wrote:

Just to add, 

Even removing routes do not improve the performance. Throughput stays around 9~10 Mbps. 

# /usr/lib/perfsonar/scripts/mod_interface_route --command delete --device em2

Only rebooting the node with a single interface configured makes the throughput measurement go up to normal (950 Mbps). 

Thanks, 

Joon 

On May 11, 2017, at 10:27 AM, Hyojoon Kim <> wrote:

Hello, 

We are getting degraded throughput numbers (< 10 Mbps) when using perfSONAR 4.0 that has multiple interfaces. We have installed perfSONAR 4.0 using the perfSONAR CentOS 7 ISO, and did an ‘yum update’ yesterday.

When we go back to using a single interface, the performance go back to normal (~ 950 Mbps). I think the routing gets funky within the perfSONAR node when we try to use two or more interfaces. Other perfSONAR nodes that use multiple interfaces with same settings work fine, but those are CentOS 6s that got perfSONAR 4.0 via auto-update. 

If anyone has an idea, please let me know. 

We follow this guideline. Each interface is on a different subnet, as recommended. 

http://docs.perfsonar.net/manage_dual_xface.html

= Configuration =

* IP address for each interface 

  - em2: 1.2.2.37/24.   Gateway: 1.2.2.1
  - em3: 1.2.7.37/22. Gateway: 1.2.7.1

We run below to add routes: 

[root@perfsonar-l-m]# /usr/lib/perfsonar/scripts/mod_interface_route --command add --device em2 --ipv4_gateway 1.2.2.1

[root@perfsonar-l-m]# /usr/lib/perfsonar/scripts/mod_interface_route --command add --device em3 --ipv4_gateway 1.2.7.1

* ip route show

[root@perfsonar-l-m]# ip route show

default via 1.2.7.1 dev em3 

1.2.7.0/22 dev em3  proto kernel  scope link  src 1.2.7.37 

1.2.2.0/24 dev em2  proto kernel  scope link  src 1.2.2.37 

[root@perfsonar-l-m]# 

* ip rule list

[root@perfsonar-l-m]# ip rule list

0:      from all lookup local 

200:    from 1.2.2.37 lookup em2_source_route 

200:    from 1.2.7.37 lookup em3_source_route 

32766:  from all lookup main 

32767:  from all lookup default 

[root@perfsonar-l-m]# 

* ip route list for each table

[root@perfsonar-l-m]# ip route list table em2_source_route

default via 1.2.2.1 dev em2 

[root@perfsonar-l-m]# ip route list table em3_source_route

default via 1.2.7.1 dev em3 

[root@perfsonar-l-m]# 

= Performance =

* Bad performance within subnet 

  - 1.2.2.0/24 subnet to 1.2.2.0/24 subnet (# pscheduler task throughput --source 1.2.2.37 --dest 1.2.2.38): 7 Mbps. 
  - 1.2.7.0/22 subnet to 1.2.7.0/22 subnet (# pscheduler task throughput --source 1.2.7.37 --dest 1.2.7.38): 14 Mbps

* Interestingly, cross-subnet measurements are good. 
  - 1.2.2.0/24 subnet to 1.2.7.0/22 subnet (# pscheduler task throughput --source 1.2.2.37 --dest 1.2.7.38): 940 Mbps
  - 1.2.7.0/22 subnet to 1.2.2.0/24 subnet (# pscheduler task throughput --source 1.2.7.37 --dest 1.2.2.38): 945 Mbps

Thanks,

Joon