perfSONAR User Q&A and Other Discussion

[Olalekan ALLADE <>]

Hello Szymon,

Limit configuration is something quite discreet that we couldn't initially imagine to be the cause of the issue. The page clearly states, "pScheduler will stop running on update if your limits are not valid!"

We have executed the proposed script on both nodes, and they produced the same output. Please refer to the file: Limit_check_output.txt. We have also executed the limit example file attached to the check script. You can find the output in the file: example_Limit_file.txt.

Now, in order to customize our current Limits file (Actual_limit_file.txt), we are proposing some modifications. You can find them in the "Limit" section of the file "Proposed_limit_file.txt". We kindly request your valuable input on this matter.

Here is more information about the attached files:

- Output of the check script: Limit_check_output.txt

- Output of running the limit example: example_Limit_file.txt

- Currently running /etc/pscheduler/limits.conf: Actual_limit_file.txt

- Proposed new /etc/pscheduler/limits.conf: Proposed_limit_file.txt

Thank you, Szymon TROCHA, for your greatly appreciated help!

With regards

On 9/1/23 12:09 PM, Szymon Trocha wrote:

W dniu 31.08.2023 o 18:35, Olalekan ALLADE (via perfsonar-user Mailing List) pisze:

Hello Szymon,

We are definitely going to get the tests results up with your help.

We have run "pscheduler troubleshoot" on each server. These are the outputs:

• server A

• • 
    

Hi,

I think we have found it.

There was a breaking change in the limit configuration format in 5.0. See https://www.perfsonar.net/releasenotes-2023-04-17-5-0-0.html

If your limits file is the one shipped with 4.4, it should have been replaced with a valid one during the upgrade but something could go wrong.

If your limits file has been customized, we have a script that you can download and run to recommend changes https://github.com/perfsonar/pscheduler-limit-checker-for-5.0

Regards,

--
Szymon Trocha
Poznań Supercomputing & Networking Center
General NOC phone +48 61-858-2015 | noc.pcss.pl
Personal desk phone +48 61-858-2022 Wysłaliśmy do Ciebie ten e-mail w odpowiedzi na Twoje zapytanie lub w związku z oferowaną usługą. Przesłanie korespondencji do Centrum Zarządzania PCSS lub zgłoszenie telefoniczne jest równoznaczne z wyrażeniem zgody na przetwarzanie danych osobowych przez Instytut Chemii Bioorganicznej Polskiej Akademii Nauk w Poznaniu adres: ul. Z. Noskowskiego 12/14, 61-704 Poznań. Szczegółowe informacje znajdują się w naszej Polityce prywatności. | This message has been sent as a part of communication with PSNC NOC or your service request sent to us. For more information read our Privacy Policy.

-- 
Network Operation Center Engineer @ WACREN |
 | +22994096312 | wacren.net |
|To dare is to do|

[admin@server]$ ./check-limits 
Examining limit configuration at /etc/pscheduler/limits.conf

NOTE: This limit configration could not be fully validated on a perfSONAR 
5.0.4  system.  This program will still attempt to find problems related to 
the removal of the 'test' limit.

ACTION REQUIRED.

This limit system configuration contains limits that are deprecated in
the 4.x family and will not be supported in release 5.0.

Limit 'idleex-default':

This limit has been removed in 5.0.  The idleex test is now disallowed for 
untrusted hosts by the 'allowed-tests' limit found in the new default limit 
configuration shipped with the toolkit.  That
configuration can be found at 
https://github.com/perfsonar/toolkit/blob/master/toolkit/perfsonar-toolkit/etc/default_service_configs/pscheduler_limits.conf.
  If your limit configuration was only
lightly-customized, re-customizing the new version would be worth the effort. 
 Also, make sure to remove any additional references to idleex-default 
elsewhere in the configuration file.


Limit 'throughput-default-time':

This limit has been rewritten to be compatible with perfSONAR 5.0.  If yours 
has not been customized, it can be replaced in its entirety with the 
following:
    
    {
        "name": "throughput-default-time",
        "description": "Throughput tests of reasonable duration",
        "type": "jq",
        "data": {
            "script": [
                "import \"pscheduler/iso8601\" as iso;",
                "if .test.type != \"throughput\"",
                "then true  # Don't care.",
                "else",
            "  if .test.spec.duration != null",
            "  and iso::duration_as_seconds(.test.spec.duration) > 60",
                "  then \"Duration for throughput must be 60 seconds or 
less.\"",
                "  else true end",
                "end"
            ]
        }
    }
    


Note that any of the modifications described above will be fully-compatible 
with 4.4.x  Getting a revised version tested and into production prior to 
upgrading to 5.0 is strongly-recommended.
[admin@server]$ ./check-limits example.conf 
Examining limit configuration at example.conf

NOTE: This limit configration could not be fully validated on a perfSONAR 
5.0.4  system.  This program will still attempt to find problems related to 
the removal of the 'test' limit.

ACTION REQUIRED.

This limit system configuration contains limits that are deprecated in
the 4.x family and will not be supported in release 5.0.

Limit 'idleex-default':

This limit has been removed in 5.0.  The idleex test is now disallowed for 
untrusted hosts by the 'allowed-tests' limit found in the new default limit 
configuration shipped with the toolkit.  That
configuration can be found at 
https://github.com/perfsonar/toolkit/blob/master/toolkit/perfsonar-toolkit/etc/default_service_configs/pscheduler_limits.conf.
  If your limit configuration was only
lightly-customized, re-customizing the new version would be worth the effort. 
 Also, make sure to remove any additional references to idleex-default 
elsewhere in the configuration file.


Limit 'some-limit-i-wrote':

This limit appears to be a customization and needs to be rewritten using the 
'jq' limit.  Documentation on how to do that may be found at
https://docs.perfsonar.net/config_pscheduler_limits.html#jq-use-a-jq-script-to-make-a-pass-fail-decision.


Limit 'throughput-default-time':

This limit has been rewritten to be compatible with perfSONAR 5.0.  If yours 
has not been customized, it can be replaced in its entirety with the 
following:
    
    {
        "name": "throughput-default-time",
        "description": "Throughput tests of reasonable duration",
        "type": "jq",
        "data": {
            "script": [
                "import \"pscheduler/iso8601\" as iso;",
                "if .test.type != \"throughput\"",
                "then true  # Don't care.",
                "else",
            "  if .test.spec.duration != null",
            "  and iso::duration_as_seconds(.test.spec.duration) > 60",
                "  then \"Duration for throughput must be 60 seconds or 
less.\"",
                "  else true end",
                "end"
            ]
        }
    }
    


Limit 'throughput-default-udp':

This limit has been rewritten to be compatible with perfSONAR 5.0.  If yours 
has not been customized, it can be replaced in its entirety with the 
following:
    
    {
        "name": "throughput-default-udp",
        "description": "UDP throughput tests of reasonable bandwidth",
        "type": "jq",
        "data": {
            "script": [
                "import \"pscheduler/iso8601\" as iso;",
                "if .test.type != \"throughput\"",
                "then true  # Don't care.",
                "else",
                "   if .test.spec.udp == true",
                "     and (.test.spec.bandwidth == null",
                "          or.test.spec.bandwidth > 50000000)",
                "   then",
                "     \"UDP throughput bandwidth must be less than 50 Mb/s\"",
                "   else true end",
                "end"
            ]
        }
    }
    


Note that any of the modifications described above will be fully-compatible 
with 4.4.x  Getting a revised version tested and into production prior to 
upgrading to 5.0 is strongly-recommended.

{
    "#": 
"-------------------------------------------------------------------",
    "#": "                 PSCHEDULER LIMIT CONFIGURATION                    
",
    "#": "                                                                   
",
    "#": "This file is a sample which contains fairly sane defaults.  It     
",
    "#": "should be customized according to the needs of your site.          
",
    "#": 
"-------------------------------------------------------------------",

    "schema": 2,

    "#": 
"-------------------------------------------------------------------",
    "#": "IDENTIFIERS:  WHO'S ASKING?                                        
",
    "#": "                                                                   
",
    "#": "These identify who's asking to run the test.  One requester can    
",
    "#": "map to zero or more identifiers.                                   
",
    "#": 
"-------------------------------------------------------------------",

    "identifiers": [
        {
            "name": "everybody",
            "description": "An identifier that always identifies",
            "type": "always",
            "data": { }
        },
        {
            "name": "local-interfaces",
            "description": "Requests coming from this system",
            "type": "localif",
            "data": { }
        },
        {
            "#": "NOTE: This only works if the host can resolve DNS",
            "#": "on the public Internet.",

            "name": "bogons",
            "description": "Bogon/Martian IPv4 addresses without private 
networks",
            "type": "ip-cymru-bogon",
            "data": {
                "exclude": [
                    "10.0.0.0/8",
                    "127.0.0.0/8",
                    "172.16.0.0/12",
                    "192.168.0.0/16",
                    "169.254.0.0/16",
                    "::/8",
                    "fe80::/10"
                    ],
                "timeout": "PT1S",
                "fail-result": false
            }
        },
        {
            "name": "hackers",
            "description": "Blocks that have tried to hack us (actually 
TEST-NET-2)",
            "type": "ip-cidr-list",
            "data": {
                "cidrs": [ "198.51.100.0/24" ]
            }
        }
    ],


    "#": 
"-------------------------------------------------------------------",
    "#": "CLASSIFIERS:  HOW DO WE CLASSIFY THE IDENTIFIERS?                  
",
    "#": "                                                                   
",
    "#": "These collect identifiers into groups.                             
",
    "#": 
"-------------------------------------------------------------------",

    "classifiers": [
        {
            "name": "default",
            "description": "Everybody",
            "identifiers": [ "everybody" ]
        },
        {
            "name": "friendlies",
            "description": "Identifiers we find friendly",
            "identifiers": [ "local-interfaces" ]
        },
        {
            "name": "hostiles",
            "description": "Identifiers we find unfriendly",
            "identifiers": [ "hackers", "bogons" ]
        }
    ],


    "#": 
"-------------------------------------------------------------------",
    "#": "REWRITE:  WHAT CHANGES ARE MADE TO INCOMING TASKS?                 
",
    "#": "                                                                   
",
    "#": "This is a jq transform that makes changes to incoming tasks prior  
",
    "#": "to limit enforcement.                                              
",
    "#": 
"-------------------------------------------------------------------",

    "rewrite": {
        "script": [
            "import \"pscheduler/iso8601\" as iso;",

            "# This does nothing but is recommended so the statements below",
            "# all begin with |.  (This makes editing easier.)",
            ".",

            "# Hold this for use later.",
            "| .test.type as $testtype",

            "# Throttle non-friendlies wanting UDP throughput to 50 Mb/s",
            "| if $testtype == \"throughput\"",
            "    and .test.spec.udp",
            "    and (",
            "      (.test.spec.bandwidth == null)",
            "      or (.test.spec.bandwidth > 50000000)",
            "    )",
            "    and (classifiers | contains([\"friendlies\"]) | not)",
            "  then",
            "    .test.spec.bandwidth = 50000000",
            "    | change(\"Throttled UDP bandwidth to 50 Mb/s\")",
            "  else",
            "    .",
            "  end",

            "# Make some tests run a minimum of 5 seconds",
            "| if ( [\"idle\", \"idlebgm\", \"idleex\", \"latency\", 
\"latencybg\", \"throughput\" ]",
            "       | contains([$testtype]) )",
            "    and .test.spec.duration != null",
            "    and iso::duration_as_seconds(.test.spec.duration) < 5",
            "  then",
            "    .test.spec.duration = \"PT5S\"",
            "    | change(\"Bumped duration to 5-second minimum\")",
            "  else",
            "    .",
            "  end",

            "# The end.  (This takes care of the no-comma-at-end problem)"
        ]
    },



    "#": 
"-------------------------------------------------------------------",
    "#": "LIMITS:  WHAT ARE THE RESTRICTIONS?                                
",
    "#": "                                                                   
",
    "#": "These are comparisons made against the type of test being 
proposed,",
    "#": "the paramaters for the run and when it is proposed to be run.      
",
    "#": 
"-------------------------------------------------------------------",

    "limits": [
        {
            "name": "always",
            "description": "Always passes",
            "type": "pass-fail",
            "data": {
                "pass": true
            }
        },
        {
            "name": "never",
            "description": "Always fails",
            "type": "pass-fail",
            "data": {
                "pass": false
            }
        },
        {
            "#": "This prevents denial of service by scheduling long tasks.",
            "name": "idleex-default",
            "description": "Default limits for idleex",
            "type": "test",
            "data": {
                "test": "idleex",
                "limit": {
                    "duration": {
                        "range": {
                            "lower": "PT1S",
                            "upper": "PT2S"
                        }
                     }
                }
            }
        },

        {
            "name": "innocuous-tests",
            "description": "Tests considered harmless",
            "type": "test-type",
            "data": {
                "#": "Resource hogs, which will be inverted below",
                "types": [ "throughput", "idleex" ]
            },
            "invert": true
        },
        {
            "name": "throughput-default-time",
            "description": "Throughput time limits",
            "type": "test",
            "data": {
                "test": "throughput",
                "limit": {
                    "duration": {
                        "range": {
                            "lower": "PT5S",
                            "upper": "PT60S"
                        }
                     }
                }
            }
        }
    ],


    "#": 
"-------------------------------------------------------------------",
    "#": "APPLICATIONS:  TO WHOM DO WE APPLY THE LIMITS?                     
",
    "#": "                                                                   
",
    "#": "These are processed in order until one passes all of the           
",
    "#": "requirements.  The run will be rejected if one fails with          
",
    "#": "stop-on-failure set to true or none of them passes.                
",
    "#": 
"-------------------------------------------------------------------",

    "applications": [
        {
            "description": "Hosts we don't want running any tests",
            "classifier": "hostiles",
            "apply": [
                { "require": "all", "limits": [ "never" ] }
            ],
            "stop-on-failure": true
        },
        {
            "description": "Hosts we trust to do everything",
            "classifier": "friendlies",
            "apply": [
                { "require": "all", "limits": [ "always" ] }
            ]
        },
        {
            "description": "Defaults applied to non-friendly hosts",
            "classifier": "default",
            "apply": [
                {
                    "require": "any",
                    "limits": [
                        "innocuous-tests",
                        "throughput-default-time",
                        "idleex-default"
                    ]
                }
            ]
        }
    ]
}

{
    "#": 
"-------------------------------------------------------------------",
    "#": "                 PSCHEDULER LIMIT CONFIGURATION                    
",
    "#": "                                                                   
",
    "#": "This file is a sample which contains fairly sane defaults.  It     
",
    "#": "should be customized according to the needs of your site.          
",
    "#": 
"-------------------------------------------------------------------",

    "schema": 2,

    "#": 
"-------------------------------------------------------------------",
    "#": "IDENTIFIERS:  WHO'S ASKING?                                        
",
    "#": "                                                                   
",
    "#": "These identify who's asking to run the test.  One requester can    
",
    "#": "map to zero or more identifiers.                                   
",
    "#": 
"-------------------------------------------------------------------",

    "identifiers": [
        {
            "name": "everybody",
            "description": "An identifier that always identifies",
            "type": "always",
            "data": { }
        },
        {
            "name": "local-interfaces",
            "description": "Requests coming from this system",
            "type": "localif",
            "data": { }
        },
        {
            "#": "NOTE: This only works if the host can resolve DNS",
            "#": "on the public Internet.",

            "name": "bogons",
            "description": "Bogon/Martian IPv4 addresses without private 
networks",
            "type": "ip-cymru-bogon",
            "data": {
                "exclude": [
                    "10.0.0.0/8",
                    "127.0.0.0/8",
                    "172.16.0.0/12",
                    "192.168.0.0/16",
                    "169.254.0.0/16",
                    "::/8",
                    "fe80::/10"
                    ],
                "timeout": "PT1S",
                "fail-result": false
            }
        },
        {
            "name": "hackers",
            "description": "Blocks that have tried to hack us (actually 
TEST-NET-2)",
            "type": "ip-cidr-list",
            "data": {
                "cidrs": [ "198.51.100.0/24" ]
            }
        }
    ],


    "#": 
"-------------------------------------------------------------------",
    "#": "CLASSIFIERS:  HOW DO WE CLASSIFY THE IDENTIFIERS?                  
",
    "#": "                                                                   
",
    "#": "These collect identifiers into groups.                             
",
    "#": 
"-------------------------------------------------------------------",

    "classifiers": [
        {
            "name": "default",
            "description": "Everybody",
            "identifiers": [ "everybody" ]
        },
        {
            "name": "friendlies",
            "description": "Identifiers we find friendly",
            "identifiers": [ "local-interfaces" ]
        },
        {
            "name": "hostiles",
            "description": "Identifiers we find unfriendly",
            "identifiers": [ "hackers", "bogons" ]
        }
    ],


    "#": 
"-------------------------------------------------------------------",
    "#": "REWRITE:  WHAT CHANGES ARE MADE TO INCOMING TASKS?                 
",
    "#": "                                                                   
",
    "#": "This is a jq transform that makes changes to incoming tasks prior  
",
    "#": "to limit enforcement.                                              
",
    "#": 
"-------------------------------------------------------------------",

    "rewrite": {
        "script": [
            "import \"pscheduler/iso8601\" as iso;",

            "# This does nothing but is recommended so the statements below",
            "# all begin with |.  (This makes editing easier.)",
            ".",

            "# Hold this for use later.",
            "| .test.type as $testtype",

            "# Throttle non-friendlies wanting UDP throughput to 50 Mb/s",
            "| if $testtype == \"throughput\"",
            "    and .test.spec.udp",
            "    and (",
            "      (.test.spec.bandwidth == null)",
            "      or (.test.spec.bandwidth > 50000000)",
            "    )",
            "    and (classifiers | contains([\"friendlies\"]) | not)",
            "  then",
            "    .test.spec.bandwidth = 50000000",
            "    | change(\"Throttled UDP bandwidth to 50 Mb/s\")",
            "  else",
            "    .",
            "  end",

            "# Make some tests run a minimum of 5 seconds",
            "| if ( [\"idle\", \"idlebgm\", \"idleex\", \"latency\", 
\"latencybg\", \"throughput\" ]",
            "       | contains([$testtype]) )",
            "    and .test.spec.duration != null",
            "    and iso::duration_as_seconds(.test.spec.duration) < 5",
            "  then",
            "    .test.spec.duration = \"PT5S\"",
            "    | change(\"Bumped duration to 5-second minimum\")",
            "  else",
            "    .",
            "  end",

            "# The end.  (This takes care of the no-comma-at-end problem)"
        ]
    },



    "#": 
"-------------------------------------------------------------------",
    "#": "LIMITS:  WHAT ARE THE RESTRICTIONS?                                
",
    "#": "                                                                   
",
    "#": "These are comparisons made against the type of test being 
proposed,",
    "#": "the paramaters for the run and when it is proposed to be run.      
",
    "#": 
"-------------------------------------------------------------------",

    "limits": [
        {
        "name": "throughput-default-time",
        "description": "Throughput tests of reasonable duration",
        "type": "jq",
        "data": {
            "script": [
                "import \"pscheduler/iso8601\" as iso;",
                "if .test.type != \"throughput\"",
                "then true  # Don't care.",
                "else",
            "  if .test.spec.duration != null",
            "  and iso::duration_as_seconds(.test.spec.duration) > 60",
                "  then \"Duration for throughput must be 60 seconds or 
less.\"",
                "  else true end",
                "end"
            ]
        }
    }
    {
        "name": "throughput-default-udp",
        "description": "UDP throughput tests of reasonable bandwidth",
        "type": "jq",
        "data": {
            "script": [
                "import \"pscheduler/iso8601\" as iso;",
                "if .test.type != \"throughput\"",
                "then true  # Don't care.",
                "else",
                "   if .test.spec.udp == true",
                "     and (.test.spec.bandwidth == null",
                "          or.test.spec.bandwidth > 50000000)",
                "   then",
                "     \"UDP throughput bandwidth must be less than 50 Mb/s\"",
                "   else true end",
                "end"
            ]
        }
    }
    ],


    "#": 
"-------------------------------------------------------------------",
    "#": "APPLICATIONS:  TO WHOM DO WE APPLY THE LIMITS?                     
",
    "#": "                                                                   
",
    "#": "These are processed in order until one passes all of the           
",
    "#": "requirements.  The run will be rejected if one fails with          
",
    "#": "stop-on-failure set to true or none of them passes.                
",
    "#": 
"-------------------------------------------------------------------",

    "applications": [
        {
            "description": "Hosts we don't want running any tests",
            "classifier": "hostiles",
            "apply": [
                { "require": "all", "limits": [ "never" ] }
            ],
            "stop-on-failure": true
        },
        {
            "description": "Hosts we trust to do everything",
            "classifier": "friendlies",
            "apply": [
                { "require": "all", "limits": [ "always" ] }
            ]
        },
        {
            "description": "Defaults applied to non-friendly hosts",
            "classifier": "default",
            "apply": [
                {
                    "require": "any",
                    "limits": [
                        "innocuous-tests",
                        "throughput-default-time",
                        "idleex-default"
                    ]
                }
            ]
        }
    ]
}