perfSONAR User Q&A and Other Discussion

[Antoine Delvaux <>]

Hello Doug,

Thank you for sharing your patch with us.  Can you also let us know which 
linux driver you're using for this Broadcom card?

Thanks,

Antoine.

> Le 18 janv. 2018 à 13:56, Doug Wussler 
> <>
>  a écrit :
> 
> I posted about this a couple weeks ago.  My system throws the same error 
> because the
> card does not support a value of 0 for the rx-usecs parameter.  Since a 
> value of 0 means
> to use the value of rx-frames, I looked up that value and then set rx-usecs 
> to that value.
> so on my system, the solution is to modify the script in two places as 
> shown between the
> asterisks.
>  
>     if [ $DISABLE_INTERRUPT_COALESCING ]; then
>         # test if this interface supports IC adjustments
>         /sbin/ethtool -c $interface > /dev/null 2>&1
>         RETVAL=$?
>         if [ $RETVAL -eq 0 ]; then
>             # apply multi-ethtool settings
>             echo -n $"disabling interrupt coalescing for interface 
> $interface: "
>             ######################################################
>             # 2017.09.29 - Doug Wussler - See comment below in IC_OFF_em 
> subroutine
>             if echo $interface | grep -q '^em[12]$'; then
>                 IC_OFF_em $interface
>             else
>                 IC_OFF $interface
>             fi
>             ######################################################
>             if [ $? -eq 0 ]; then
>                 success; echo
>             else
>                 failure; echo; ret=1
>             fi
>         fi
>     fi
>  
> ###########################################################################
> # 2017.09.29 - Doug Wussler - The Broadcom Limited NetXtreme BCM5720 
> Gigabit Ethernet PCIe
> # card does not support a value of 0 for the rx-usecs parameter,
> # which says to use the value of rx-frames, so we must set it to 1.
> IC_OFF_em() {
> IC_RET_em=0
> for i in "$1"; do
>     if ! /usr/sbin/ethtool -c $i | grep -q '^rx-frames: 1$'; then
>          /usr/sbin/ethtool -C $i rx-frames 1 || IC_RET_em=$?
>     fi
>  
>     if ! /usr/sbin/ethtool -c $i | grep -q '^rx-usecs: 1$'; then
>          /usr/sbin/ethtool -C $i rx-usecs 1  || IC_RET_em=$?
>     fi
> done
> return $IC_RET_em
> }
> ###########################################################################
>  
>  
> Doug Wussler
> Florida State University
>  
> From: 
> <>
>  on behalf of dan dev 
> <>
> Date: Saturday, January 13, 2018 at 3:12 AM
> To: Valentin Vidic 
> <>
> Cc: 
> ""
>  
> <>
> Subject: Re: Re: [perfsonar-user] Installation errors in perfSONAR-toolkit
>  
> Hi Valentin, 
>  
> Here is the output for the command
>  
> # SYSTEMCTL_SKIP_REDIRECT=1 bash -x 
> /etc/init.d/perfsonar-configure_nic_parameters start
>  
>  
> + . /lib/lsb/init-functions
> +++ run-parts --lsbsysinit --list /lib/lsb/init-functions.d
> ++ for hook in '$(run-parts --lsbsysinit --list /lib/lsb/init-functions.d 
> 2>/dev/null)'
> ++ '[' -r /lib/lsb/init-functions.d/01-upstart-lsb ']'
> ++ . /lib/lsb/init-functions.d/01-upstart-lsb
> +++ unset UPSTART_SESSION
> +++ _RC_SCRIPT=/etc/init.d/perfsonar-configure_nic_parameters
> +++ '[' -r /etc/init//etc/init.d/perfsonar-configure_nic_parameters.conf ']'
> +++ _UPSTART_JOB=perfsonar-configure_nic_parameters
> +++ '[' -r /etc/init/perfsonar-configure_nic_parameters.conf ']'
> ++ for hook in '$(run-parts --lsbsysinit --list /lib/lsb/init-functions.d 
> 2>/dev/null)'
> ++ '[' -r /lib/lsb/init-functions.d/20-left-info-blocks ']'
> ++ . /lib/lsb/init-functions.d/20-left-info-blocks
> ++ for hook in '$(run-parts --lsbsysinit --list /lib/lsb/init-functions.d 
> 2>/dev/null)'
> ++ '[' -r /lib/lsb/init-functions.d/40-systemd ']'
> ++ . /lib/lsb/init-functions.d/40-systemd
> +++ _use_systemctl=0
> +++ '[' -d /run/systemd/system ']'
> +++ prog=perfsonar-configure_nic_parameters
> +++ service=perfsonar-configure_nic_parameters.service
> ++++ systemctl -p LoadState show perfsonar-configure_nic_parameters.service
> +++ state=LoadState=loaded
> +++ '[' LoadState=loaded = LoadState=masked ']'
> +++ '[' 18266 -ne 1 ']'
> +++ '[' -z '' ']'
> +++ '[' -z 1 ']'
> +++ export _SYSTEMCTL_SKIP_REDIRECT=true
> +++ _SYSTEMCTL_SKIP_REDIRECT=true
> +++ '[' 0 = 1 ']'
> ++ for hook in '$(run-parts --lsbsysinit --list /lib/lsb/init-functions.d 
> 2>/dev/null)'
> ++ '[' -r /lib/lsb/init-functions.d/50-ubuntu-logging ']'
> ++ . /lib/lsb/init-functions.d/50-ubuntu-logging
> +++ LOG_DAEMON_MSG=
> ++ for hook in '$(run-parts --lsbsysinit --list /lib/lsb/init-functions.d 
> 2>/dev/null)'
> ++ '[' -r /lib/lsb/init-functions.d/99-plymouth ']'
> ++ . /lib/lsb/init-functions.d/99-plymouth
> +++ plymouth --ping
> +++ return
> ++ FANCYTTY=
> ++ '[' -e /etc/lsb-base-logging.sh ']'
> ++ true
> + DISABLE_TCP_OFFLOAD=1
> + DISABLE_INTERRUPT_COALESCING=1
> + TXQUEUELEN=10000
> ++ /sbin/ifquery --all --list
> ++ grep -v -w lo
> + interfaces='eth0
> eth1
> xenbr0
> xenbr1
> br-ex'
> ++ /sbin/ifquery --allow hotplug --list
> + interfaces='eth0
> eth1
> xenbr0
> xenbr1
> br-ex '
> + case "$1" in
> + start
> + ret=0
> + for interface in '$interfaces'
> + /sbin/ip link show eth0
> + '[' 0 -ne 0 ']'
> + '[' 1 ']'
> + /sbin/ethtool -c eth0
> + RETVAL=0
> + '[' 0 -eq 0 ']'
> + log_action_begin_msg 'Disabling interrupt coalescing for interface eth0'
> + log_daemon_msg 'Disabling interrupt coalescing for interface eth0...'
> + '[' -z 'Disabling interrupt coalescing for interface eth0...' ']'
> + log_use_fancy_output
> + TPUT=/usr/bin/tput
> + EXPR=/usr/bin/expr
> + '[' -t 1 ']'
> + '[' xxterm-256color '!=' x ']'
> + '[' xxterm-256color '!=' xdumb ']'
> + '[' -x /usr/bin/tput ']'
> + '[' -x /usr/bin/expr ']'
> + /usr/bin/tput hpa 60
> + /usr/bin/tput setaf 1
> + '[' -z ']'
> + FANCYTTY=1
> + case "$FANCYTTY" in
> + true
> + /usr/bin/tput xenl
> ++ /usr/bin/tput cols
> + COLS=80
> + '[' 80 ']'
> + '[' 80 -gt 6 ']'
> ++ /usr/bin/expr 80 - 7
> + COL=73
> + log_use_plymouth
> + '[' n = y ']'
> + plymouth --ping
> + printf ' * Disabling interrupt coalescing for interface eth0...       '
>  * Disabling interrupt coalescing for interface eth0...       ++ 
> /usr/bin/expr 80 - 1
> + /usr/bin/tput hpa 79
>                                                                             
>    + printf ' '
>  + IC_OFF eth0
> + IC_RET=0
> + for i in '"$1"'
> + /sbin/ethtool -c eth0
> + grep -q '^rx-frames: 1$'
> + /sbin/ethtool -C eth0 rx-frames 1
> Cannot set device coalesce parameters: Unknown error 524
> + IC_RET=81
> + /sbin/ethtool -c eth0
> + grep -q '^rx-usecs: 0$'
> + return 81
> + '[' 81 -eq 0 ']'
> + log_action_end_msg 1
> + log_end_msg 1
> + '[' -z 1 ']'
> + '[' 73 ']'
> + '[' -x /usr/bin/tput ']'
> + log_use_plymouth
> + '[' n = y ']'
> + plymouth --ping
> + printf '\r'
> + /usr/bin/tput hpa 73
>                                                                          + 
> '[' 1 -eq 0 ']'
> + printf '['
> [+ /usr/bin/tput setaf 1
> + printf fail
> fail+ /usr/bin/tput op
> + echo ']'
> ]
> + return 1
> + true
> + ret=1
> + '[' 1 ']'
> + log_action_begin_msg 'Disabling TCP offload for interface eth0'
> + log_daemon_msg 'Disabling TCP offload for interface eth0...'
> + '[' -z 'Disabling TCP offload for interface eth0...' ']'
> + log_use_fancy_output
> + TPUT=/usr/bin/tput
> + EXPR=/usr/bin/expr
> + '[' -t 1 ']'
> + '[' xxterm-256color '!=' x ']'
> + '[' xxterm-256color '!=' xdumb ']'
> + '[' -x /usr/bin/tput ']'
> + '[' -x /usr/bin/expr ']'
> + /usr/bin/tput hpa 60
> + /usr/bin/tput setaf 1
> + '[' -z 1 ']'
> + true
> + case "$FANCYTTY" in
> + true
> + /usr/bin/tput xenl
> ++ /usr/bin/tput cols
> + COLS=80
> + '[' 80 ']'
> + '[' 80 -gt 6 ']'
> ++ /usr/bin/expr 80 - 7
> + COL=73
> + log_use_plymouth
> + '[' n = y ']'
> + plymouth --ping
> + printf ' * Disabling TCP offload for interface eth0...       '
>  * Disabling TCP offload for interface eth0...       ++ /usr/bin/expr 80 - 1
> + /usr/bin/tput hpa 79
>                                                                             
>    + printf ' '
>  + TSO_OFF eth0
> + /sbin/ethtool -K eth0 tso off
> + '[' 0 -eq 0 ']'
> + log_action_end_msg 0
> + log_end_msg 0
> + '[' -z 0 ']'
> + '[' 73 ']'
> + '[' -x /usr/bin/tput ']'
> + log_use_plymouth
> + '[' n = y ']'
> + plymouth --ping
> + printf '\r'
> + /usr/bin/tput hpa 73
>                                                                          + 
> '[' 0 -eq 0 ']'
> + echo '[ OK ]'
> [ OK ]
> + return 0
> + '[' 10000 -ne 0 ']'
> + log_action_begin_msg 'Setting txqueuelen for eth0 to 10000'
> + log_daemon_msg 'Setting txqueuelen for eth0 to 10000...'
> + '[' -z 'Setting txqueuelen for eth0 to 10000...' ']'
> + log_use_fancy_output
> + TPUT=/usr/bin/tput
> + EXPR=/usr/bin/expr
> + '[' -t 1 ']'
> + '[' xxterm-256color '!=' x ']'
> + '[' xxterm-256color '!=' xdumb ']'
> + '[' -x /usr/bin/tput ']'
> + '[' -x /usr/bin/expr ']'
> + /usr/bin/tput hpa 60
> + /usr/bin/tput setaf 1
> + '[' -z 1 ']'
> + true
> + case "$FANCYTTY" in
> + true
> + /usr/bin/tput xenl
> ++ /usr/bin/tput cols
> + COLS=80
> + '[' 80 ']'
> + '[' 80 -gt 6 ']'
> ++ /usr/bin/expr 80 - 7
> + COL=73
> + log_use_plymouth
> + '[' n = y ']'
> + plymouth --ping
> + printf ' * Setting txqueuelen for eth0 to 10000...       '
>  * Setting txqueuelen for eth0 to 10000...       ++ /usr/bin/expr 80 - 1
> + /usr/bin/tput hpa 79
>                                                                             
>    + printf ' '
>  + TXQUEUELEN_SET eth0 10000
> + /sbin/ip link set eth0 txqueuelen 10000
> + '[' 0 -eq 0 ']'
> + log_action_end_msg 0
> + log_end_msg 0
> + '[' -z 0 ']'
> + '[' 73 ']'
> + '[' -x /usr/bin/tput ']'
> + log_use_plymouth
> + '[' n = y ']'
> + plymouth --ping
> + printf '\r'
> + /usr/bin/tput hpa 73
>                                                                          + 
> '[' 0 -eq 0 ']'
> + echo '[ OK ]'
> [ OK ]
> + return 0
> + for interface in '$interfaces'
> + /sbin/ip link show eth1
> + '[' 0 -ne 0 ']'
> + '[' 1 ']'
> + /sbin/ethtool -c eth1
> + RETVAL=0
> + '[' 0 -eq 0 ']'
> + log_action_begin_msg 'Disabling interrupt coalescing for interface eth1'
> + log_daemon_msg 'Disabling interrupt coalescing for interface eth1...'
> + '[' -z 'Disabling interrupt coalescing for interface eth1...' ']'
> + log_use_fancy_output
> + TPUT=/usr/bin/tput
> + EXPR=/usr/bin/expr
> + '[' -t 1 ']'
> + '[' xxterm-256color '!=' x ']'
> + '[' xxterm-256color '!=' xdumb ']'
> + '[' -x /usr/bin/tput ']'
> + '[' -x /usr/bin/expr ']'
> + /usr/bin/tput hpa 60
> + /usr/bin/tput setaf 1
> + '[' -z 1 ']'
> + true
> + case "$FANCYTTY" in
> + true
> + /usr/bin/tput xenl
> ++ /usr/bin/tput cols
> + COLS=80
> + '[' 80 ']'
> + '[' 80 -gt 6 ']'
> ++ /usr/bin/expr 80 - 7
> + COL=73
> + log_use_plymouth
> + '[' n = y ']'
> + plymouth --ping
> + printf ' * Disabling interrupt coalescing for interface eth1...       '
>  * Disabling interrupt coalescing for interface eth1...       ++ 
> /usr/bin/expr 80 - 1
> + /usr/bin/tput hpa 79
>                                                                             
>    + printf ' '
>  + IC_OFF eth1
> + IC_RET=0
> + for i in '"$1"'
> + /sbin/ethtool -c eth1
> + grep -q '^rx-frames: 1$'
> + /sbin/ethtool -C eth1 rx-frames 1
> Cannot set device coalesce parameters: Unknown error 524
> + IC_RET=81
> + /sbin/ethtool -c eth1
> + grep -q '^rx-usecs: 0$'
> + return 81
> + '[' 81 -eq 0 ']'
> + log_action_end_msg 1
> + log_end_msg 1
> + '[' -z 1 ']'
> + '[' 73 ']'
> + '[' -x /usr/bin/tput ']'
> + log_use_plymouth
> + '[' n = y ']'
> + plymouth --ping
> + printf '\r'
> + /usr/bin/tput hpa 73
>                                                                          + 
> '[' 1 -eq 0 ']'
> + printf '['
> [+ /usr/bin/tput setaf 1
> + printf fail
> fail+ /usr/bin/tput op
> + echo ']'
> ]
> + return 1
> + true
> + ret=1
> + '[' 1 ']'
> + log_action_begin_msg 'Disabling TCP offload for interface eth1'
> + log_daemon_msg 'Disabling TCP offload for interface eth1...'
> + '[' -z 'Disabling TCP offload for interface eth1...' ']'
> + log_use_fancy_output
> + TPUT=/usr/bin/tput
> + EXPR=/usr/bin/expr
> + '[' -t 1 ']'
> + '[' xxterm-256color '!=' x ']'
> + '[' xxterm-256color '!=' xdumb ']'
> + '[' -x /usr/bin/tput ']'
> + '[' -x /usr/bin/expr ']'
> + /usr/bin/tput hpa 60
> + /usr/bin/tput setaf 1
> + '[' -z 1 ']'
> + true
> + case "$FANCYTTY" in
> + true
> + /usr/bin/tput xenl
> ++ /usr/bin/tput cols
> + COLS=80
> + '[' 80 ']'
> + '[' 80 -gt 6 ']'
> ++ /usr/bin/expr 80 - 7
> + COL=73
> + log_use_plymouth
> + '[' n = y ']'
> + plymouth --ping
> + printf ' * Disabling TCP offload for interface eth1...       '
>  * Disabling TCP offload for interface eth1...       ++ /usr/bin/expr 80 - 1
> + /usr/bin/tput hpa 79
>                                                                             
>    + printf ' '
>  + TSO_OFF eth1
> + /sbin/ethtool -K eth1 tso off
> + '[' 0 -eq 0 ']'
> + log_action_end_msg 0
> + log_end_msg 0
> + '[' -z 0 ']'
> + '[' 73 ']'
> + '[' -x /usr/bin/tput ']'
> + log_use_plymouth
> + '[' n = y ']'
> + plymouth --ping
> + printf '\r'
> + /usr/bin/tput hpa 73
>                                                                          + 
> '[' 0 -eq 0 ']'
> + echo '[ OK ]'
> [ OK ]
> + return 0
> + '[' 10000 -ne 0 ']'
> + log_action_begin_msg 'Setting txqueuelen for eth1 to 10000'
> + log_daemon_msg 'Setting txqueuelen for eth1 to 10000...'
> + '[' -z 'Setting txqueuelen for eth1 to 10000...' ']'
> + log_use_fancy_output
> + TPUT=/usr/bin/tput
> + EXPR=/usr/bin/expr
> + '[' -t 1 ']'
> + '[' xxterm-256color '!=' x ']'
> + '[' xxterm-256color '!=' xdumb ']'
> + '[' -x /usr/bin/tput ']'
> + '[' -x /usr/bin/expr ']'
> + /usr/bin/tput hpa 60
> + /usr/bin/tput setaf 1
> + '[' -z 1 ']'
> + true
> + case "$FANCYTTY" in
> + true
> + /usr/bin/tput xenl
> ++ /usr/bin/tput cols
> + COLS=80
> + '[' 80 ']'
> + '[' 80 -gt 6 ']'
> ++ /usr/bin/expr 80 - 7
> + COL=73
> + log_use_plymouth
> + '[' n = y ']'
> + plymouth --ping
> + printf ' * Setting txqueuelen for eth1 to 10000...       '
>  * Setting txqueuelen for eth1 to 10000...       ++ /usr/bin/expr 80 - 1
> + /usr/bin/tput hpa 79
>                                                                             
>    + printf ' '
>  + TXQUEUELEN_SET eth1 10000
> + /sbin/ip link set eth1 txqueuelen 10000
> + '[' 0 -eq 0 ']'
> + log_action_end_msg 0
> + log_end_msg 0
> + '[' -z 0 ']'
> + '[' 73 ']'
> + '[' -x /usr/bin/tput ']'
> + log_use_plymouth
> + '[' n = y ']'
> + plymouth --ping
> + printf '\r'
> + /usr/bin/tput hpa 73
>                                                                          + 
> '[' 0 -eq 0 ']'
> + echo '[ OK ]'
> [ OK ]
> + return 0
> + for interface in '$interfaces'
> + /sbin/ip link show xenbr0
> + '[' 1 -ne 0 ']'
> + continue
> + for interface in '$interfaces'
> + /sbin/ip link show xenbr1
> + '[' 1 -ne 0 ']'
> + continue
> + for interface in '$interfaces'
> + /sbin/ip link show br-ex
> + '[' 0 -ne 0 ']'
> + '[' 1 ']'
> + /sbin/ethtool -c br-ex
> + RETVAL=82
> + '[' 82 -eq 0 ']'
> + '[' 1 ']'
> + log_action_begin_msg 'Disabling TCP offload for interface br-ex'
> + log_daemon_msg 'Disabling TCP offload for interface br-ex...'
> + '[' -z 'Disabling TCP offload for interface br-ex...' ']'
> + log_use_fancy_output
> + TPUT=/usr/bin/tput
> + EXPR=/usr/bin/expr
> + '[' -t 1 ']'
> + '[' xxterm-256color '!=' x ']'
> + '[' xxterm-256color '!=' xdumb ']'
> + '[' -x /usr/bin/tput ']'
> + '[' -x /usr/bin/expr ']'
> + /usr/bin/tput hpa 60
> + /usr/bin/tput setaf 1
> + '[' -z 1 ']'
> + true
> + case "$FANCYTTY" in
> + true
> + /usr/bin/tput xenl
> ++ /usr/bin/tput cols
> + COLS=80
> + '[' 80 ']'
> + '[' 80 -gt 6 ']'
> ++ /usr/bin/expr 80 - 7
> + COL=73
> + log_use_plymouth
> + '[' n = y ']'
> + plymouth --ping
> + printf ' * Disabling TCP offload for interface br-ex...       '
>  * Disabling TCP offload for interface br-ex...       ++ /usr/bin/expr 80 - 
> 1
> + /usr/bin/tput hpa 79
>                                                                             
>    + printf ' '
>  + TSO_OFF br-ex
> + /sbin/ethtool -K br-ex tso off
> + '[' 0 -eq 0 ']'
> + log_action_end_msg 0
> + log_end_msg 0
> + '[' -z 0 ']'
> + '[' 73 ']'
> + '[' -x /usr/bin/tput ']'
> + log_use_plymouth
> + '[' n = y ']'
> + plymouth --ping
> + printf '\r'
> + /usr/bin/tput hpa 73
>                                                                          + 
> '[' 0 -eq 0 ']'
> + echo '[ OK ]'
> [ OK ]
> + return 0
> + '[' 10000 -ne 0 ']'
> + log_action_begin_msg 'Setting txqueuelen for br-ex to 10000'
> + log_daemon_msg 'Setting txqueuelen for br-ex to 10000...'
> + '[' -z 'Setting txqueuelen for br-ex to 10000...' ']'
> + log_use_fancy_output
> + TPUT=/usr/bin/tput
> + EXPR=/usr/bin/expr
> + '[' -t 1 ']'
> + '[' xxterm-256color '!=' x ']'
> + '[' xxterm-256color '!=' xdumb ']'
> + '[' -x /usr/bin/tput ']'
> + '[' -x /usr/bin/expr ']'
> + /usr/bin/tput hpa 60
> + /usr/bin/tput setaf 1
> + '[' -z 1 ']'
> + true
> + case "$FANCYTTY" in
> + true
> + /usr/bin/tput xenl
> ++ /usr/bin/tput cols
> + COLS=80
> + '[' 80 ']'
> + '[' 80 -gt 6 ']'
> ++ /usr/bin/expr 80 - 7
> + COL=73
> + log_use_plymouth
> + '[' n = y ']'
> + plymouth --ping
> + printf ' * Setting txqueuelen for br-ex to 10000...       '
>  * Setting txqueuelen for br-ex to 10000...       ++ /usr/bin/expr 80 - 1
> + /usr/bin/tput hpa 79
>                                                                             
>    + printf ' '
>  + TXQUEUELEN_SET br-ex 10000
> + /sbin/ip link set br-ex txqueuelen 10000
> + '[' 0 -eq 0 ']'
> + log_action_end_msg 0
> + log_end_msg 0
> + '[' -z 0 ']'
> + '[' 73 ']'
> + '[' -x /usr/bin/tput ']'
> + log_use_plymouth
> + '[' n = y ']'
> + plymouth --ping
> + printf '\r'
> + /usr/bin/tput hpa 73
>                                                                          + 
> '[' 0 -eq 0 ']'
> + echo '[ OK ]'
> [ OK ]
> + return 0
> + return 1
> + RETVAL=1
> + exit 1
>  
>  
> On Fri, Jan 12, 2018 at 10:46 PM, Valentin Vidic 
> <>
>  wrote:
> On Fri, Jan 12, 2018 at 10:03:30PM +0900, dan dev wrote:
> >  # SYSTEMCTL_SKIP_REDIRECT=1 bash -x
> > /etc/init.d/perfsonar-configure_nic_parameters
> > start
> 
> Sorry, for Ubuntu this should be:
> 
>   # _SYSTEMCTL_SKIP_REDIRECT=1 bash -x 
> /etc/init.d/perfsonar-configure_nic_parameters start
> 
> --
> Valentin Vidic
> Computer Systems Engineer - Expert
> Department of Computer Infrastructure and Services
> Croatian Academic and Research Network - CARNET
> Josipa Marohnica 5, HR-10000 Zagreb, Croatia
> tel: +385 1 6661 714, fax. +385 1 6661 635
> gsm: +385 91 2480 919
> www.CARNet.hr
>