Skip to Content.
Sympa Menu

perfsonar-user - Re: [perfsonar-user] Degraded throughput result when using perfsonar 4.0 with multiple interfaces

Subject: perfSONAR User Q&A and Other Discussion

List archive

Re: [perfsonar-user] Degraded throughput result when using perfsonar 4.0 with multiple interfaces


Chronological Thread 
  • From: Hyojoon Kim <>
  • To: perfsonar-user <>
  • Subject: Re: [perfsonar-user] Degraded throughput result when using perfsonar 4.0 with multiple interfaces
  • Date: Thu, 11 May 2017 14:59:46 +0000
  • Accept-language: en-US
  • Ironport-phdr: 9a23:e1AnRR3pJIcZb+WGsmDT+DRfVm0co7zxezQtwd8ZseISKvad9pjvdHbS+e9qxAeQG96KtLQc1qGJ4ujJYi8p2d65qncMcZhBBVcuqP49uEgeOvODElDxN/XwbiY3T4xoXV5h+GynYwAOQJ6tL1LdrWev4jEMBx7xKRR6JvjvGo7Vks+7y/2+94fdbghMhjexe69+IRW5oQjTssQdnJdvJLs2xhbVuHVDZv5YxXlvJVKdnhb84tm/8Zt++ClOuPwv6tBNX7zic6s3UbJXAjImM3so5MLwrhnMURGP5noHXWoIlBdDHhXI4wv7Xpf1tSv6q/Z91SyHNsD4Ubw4RTKv5LptRRT1iikIKiQ5/XnXhMJukaxbvByvqR9xw4HWYYGaKPVxc7jHct8GQGpMRNpdWzBDD466coABD/ABPeFdr4TlpVUPrRy+BQurBOPozD9Dm3r43aw/0+88FgzG3gsgH9UIsHTXt9j1LL0SUeGzzKbU0zrDc/RW2THn5IfWbx8hvOiBULRtesTfzkkvEhnKjlSWqYH9JzOayP4Ns2aA7+pnU+KgkXQnpxt3ojex2schjJTCiIENyl3c6Cl13oU4KcelREJmf9KoCoZcuz+ZOodsXM8uXm5ltD4nxrEYupO3ZjUGxIk5yxLFdvCKcYaF7gjtWeqMOTt0mXNodbalixqs8UWtxffwWtSp3FpQrydIkd/BvW0X2RPJ8MiIUP5981+h2TmR0wDT7flJL1oularaLZ4u3qA/loYUsUTFBCP4l0v4g6GOekUl/Oin9fjnb637qpOCNIJ4kBzyPrkwlsClHOg0LAsDU3KF9eih27Dv5Uj5T69Ljv0ynKnZqpfaJcEDq667BA9V1YAj6xWkAje+ytQYnH8HI05bdxKbiIjpPUvCL+7lAveim1isiitkx+jaPr39BZXANmDDn6n7fbZz8E5czwwzwspF555NF7EBO+z8WlXqudzcDx85KBC0w/35BNln14MeX36PDbGDMKPUr1CI+vwjL/OSa4AIpTauY8Qisubjl3EinlkUZ+y0xpYNICSgE+5oOEKfaGCpn8wMC08LuBYzVurnlAfEXDJOMSWcRaU5sxEqD4ChCoGLZYGoyOiD2yC9HbVOfGFAAV2QFnGueomZDaRfIBmOK9Nsx2RXHYOqTJUsgEmj

Just to add, 

Even removing routes do not improve the performance. Throughput stays around 9~10 Mbps. 
# /usr/lib/perfsonar/scripts/mod_interface_route --command delete --device em2

Only rebooting the node with a single interface configured makes the throughput measurement go up to normal (950 Mbps). 

Thanks, 
Joon 

On May 11, 2017, at 10:27 AM, Hyojoon Kim <> wrote:

Hello, 

We are getting degraded throughput numbers (< 10 Mbps) when using perfSONAR 4.0 that has multiple interfaces. We have installed perfSONAR 4.0 using the perfSONAR CentOS 7 ISO, and did an ‘yum update’ yesterday.

When we go back to using a single interface, the performance go back to normal (~ 950 Mbps). I think the routing gets funky within the perfSONAR node when we try to use two or more interfaces. Other perfSONAR nodes that use multiple interfaces with same settings work fine, but those are CentOS 6s that got perfSONAR 4.0 via auto-update. 

If anyone has an idea, please let me know. 

We follow this guideline. Each interface is on a different subnet, as recommended. 

= Configuration =
* IP address for each interface 
  - em2: 1.2.2.37/24.   Gateway: 1.2.2.1
  - em3: 1.2.7.37/22. Gateway: 1.2.7.1

We run below to add routes: 
[root@perfsonar-l-m]# /usr/lib/perfsonar/scripts/mod_interface_route --command add --device em2 --ipv4_gateway 1.2.2.1
[root@perfsonar-l-m]# /usr/lib/perfsonar/scripts/mod_interface_route --command add --device em3 --ipv4_gateway 1.2.7.1

* ip route show
[root@perfsonar-l-m]# ip route show
default via 1.2.7.1 dev em3 
1.2.7.0/22 dev em3  proto kernel  scope link  src 1.2.7.37 
1.2.2.0/24 dev em2  proto kernel  scope link  src 1.2.2.37 
[root@perfsonar-l-m]# 

* ip rule list
[root@perfsonar-l-m]# ip rule list
0:      from all lookup local 
200:    from 1.2.2.37 lookup em2_source_route 
200:    from 1.2.7.37 lookup em3_source_route 
32766:  from all lookup main 
32767:  from all lookup default 
[root@perfsonar-l-m]# 

* ip route list for each table
[root@perfsonar-l-m]# ip route list table em2_source_route
default via 1.2.2.1 dev em2 
[root@perfsonar-l-m]# ip route list table em3_source_route
default via 1.2.7.1 dev em3 
[root@perfsonar-l-m]# 


= Performance =
* Bad performance within subnet 
  - 1.2.2.0/24 subnet to 1.2.2.0/24 subnet (# pscheduler task throughput --source 1.2.2.37 --dest 1.2.2.38): 7 Mbps. 
  - 1.2.7.0/22 subnet to 1.2.7.0/22 subnet (# pscheduler task throughput --source 1.2.7.37 --dest 1.2.7.38): 14 Mbps

* Interestingly, cross-subnet measurements are good. 
  - 1.2.2.0/24 subnet to 1.2.7.0/22 subnet (# pscheduler task throughput --source 1.2.2.37 --dest 1.2.7.38): 940 Mbps
  - 1.2.7.0/22 subnet to 1.2.2.0/24 subnet (# pscheduler task throughput --source 1.2.7.37 --dest 1.2.2.38): 945 Mbps


Thanks,
Joon 




Archive powered by MHonArc 2.6.19.

Top of Page