Mar 15 11:06:57 enpl-max-ps runner INFO     24189829: Run succeeded.
Mar 16 11:06:47 enpl-max-ps scheduler DEBUG    26483: Trying to schedule 84b5408c-93c9-4656-b790-ad0771d73f06 for 2018-03-16 11:06:57.043489-04:00
Mar 16 11:06:47 enpl-max-ps scheduler DEBUG    26483: Posting https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/84b5408c-93c9-4656-b790-ad0771d73f06 at 2018-03-16 11:06:57.043489-04:00
Mar 16 11:06:47 enpl-max-ps scheduler DEBUG    26483: Possible run range {'start': '2018-03-16T11:06:57-04:00', 'end': '2018-03-16T11:13:08-04:00'}
Mar 16 11:06:47 enpl-max-ps pscheduler-api DEBUG    Response 200+JSON: [{"upper": "2018-03-16T11:09:03-04:00", "lower": "2018-03-16T11:06:57-04:00"}, {"upper": "2018-03-16T11:13:08-04:00", "lower": "2018-03-16T11:10:34-04:00"}]
Mar 16 11:06:47 enpl-max-ps scheduler DEBUG    26483: Ranges: [R(2018-03-16 11:06:57-04:00..2018-03-16 11:09:03-04:00), R(2018-03-16 11:10:34-04:00..2018-03-16 11:13:08-04:00)]
Mar 16 11:06:47 enpl-max-ps scheduler DEBUG    26483: Ranges in common: [R(2018-03-16 11:06:57-04:00..2018-03-16 11:09:03-04:00), R(2018-03-16 11:10:34-04:00..2018-03-16 11:13:08-04:00)]
Mar 16 11:06:47 enpl-max-ps scheduler DEBUG    26483: Selected range R(2018-03-16 11:06:57-04:00..2018-03-16 11:09:03-04:00)
Mar 16 11:06:47 enpl-max-ps scheduler DEBUG    26483: Data {'start-time': '2018-03-16T11:06:57-04:00'}
Mar 16 11:06:47 enpl-max-ps pscheduler-api DEBUG    Run POST: https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/84b5408c-93c9-4656-b790-ad0771d73f06/runs --> {"start-time": "2018-03-16T11:06:57-04:00"}
Mar 16 11:06:47 enpl-max-ps pscheduler-api DEBUG    Checking limits against {'type': u'throughput', 'spec': {u'dest': u'perfnsidc.colorado.edu', u'source': u'enpl-pt2-10g.eos.nasa.gov', u'schema': 1}, 'schedule': {'duration': 'PT1M11S', 'start': '2018-03-16T11:06:57-04:00'}}
Mar 16 11:06:48 enpl-max-ps pscheduler-api DEBUG    Posting run for task 84b5408c-93c9-4656-b790-ad0771d73f06 starting 2018-03-16 11:06:57-04:00
Mar 16 11:06:49 enpl-max-ps pscheduler-api DEBUG    Response 200+JSON: {"result-href": "https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/84b5408c-93c9-4656-b790-ad0771d73f06/runs/18a23044-8273-4b36-9779-9d42d988ef11/result", "end-time": "2018-03-16T11:08:08-04:00", "participant": 0, "participant-data": {}, "result-merged": null, "participant-data-full": null, "participants": ["enpl-pt2-10g.eos.nasa.gov"], "state-display": "Pending", "result-full": null, "state": "pending", "errors": null, "href": "https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/84b5408c-93c9-4656-b790-ad0771d73f06/runs/18a23044-8273-4b36-9779-9d42d988ef11", "task-href": "https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/84b5408c-93c9-4656-b790-ad0771d73f06", "duration": "PT1M11S", "start-time": "2018-03-16T11:06:57-04:00", "added": "2018-03-16T11:06:48-04:00", "result": null}
Mar 16 11:06:49 enpl-max-ps pscheduler-api DEBUG    Response 200+JSON: {"result-href": "https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/84b5408c-93c9-4656-b790-ad0771d73f06/runs/18a23044-8273-4b36-9779-9d42d988ef11/result", "end-time": "2018-03-16T11:08:08-04:00", "participant": 0, "participant-data": {}, "result-merged": null, "participant-data-full": null, "participants": ["enpl-pt2-10g.eos.nasa.gov"], "state-display": "Pending", "result-full": null, "state": "pending", "errors": null, "href": "https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/84b5408c-93c9-4656-b790-ad0771d73f06/runs/18a23044-8273-4b36-9779-9d42d988ef11", "task-href": "https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/84b5408c-93c9-4656-b790-ad0771d73f06", "duration": "PT1M11S", "start-time": "2018-03-16T11:06:57-04:00", "added": "2018-03-16T11:06:48-04:00", "result": null}
Mar 16 11:06:49 enpl-max-ps scheduler DEBUG    26483: Scheduled for 2018-03-16 11:06:57-04:00 - 2018-03-16 11:08:08-04:00 at https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/84b5408c-93c9-4656-b790-ad0771d73f06/runs/18a23044-8273-4b36-9779-9d42d988ef11
Mar 16 11:06:49 enpl-max-ps runner DEBUG    Run 24296018, starts at 2018-03-16 11:06:57-04:00
Mar 16 11:06:49 enpl-max-ps runner DEBUG    Start at 2018-03-16 11:06:57-04:00
Mar 16 11:06:50 enpl-max-ps pscheduler-api DEBUG    Response 200+JSON: {"result-href": "https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/84b5408c-93c9-4656-b790-ad0771d73f06/runs/18a23044-8273-4b36-9779-9d42d988ef11/result", "end-time": "2018-03-16T11:08:08-04:00", "participant": 0, "participant-data": {}, "result-merged": null, "participant-data-full": [{}], "participants": ["enpl-pt2-10g.eos.nasa.gov"], "state-display": "On Deck", "result-full": null, "state": "on-deck", "errors": null, "href": "https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/84b5408c-93c9-4656-b790-ad0771d73f06/runs/18a23044-8273-4b36-9779-9d42d988ef11", "task-href": "https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/84b5408c-93c9-4656-b790-ad0771d73f06", "duration": "PT1M11S", "start-time": "2018-03-16T11:06:57-04:00", "added": "2018-03-16T11:06:48-04:00", "result": null}
Mar 16 11:06:50 enpl-max-ps pscheduler-api DEBUG    Response 200+JSON: {"result-href": "https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/84b5408c-93c9-4656-b790-ad0771d73f06/runs/18a23044-8273-4b36-9779-9d42d988ef11/result", "end-time": "2018-03-16T11:08:08-04:00", "participant": 0, "participant-data": {}, "result-merged": null, "participant-data-full": [{}], "participants": ["enpl-pt2-10g.eos.nasa.gov"], "state-display": "On Deck", "result-full": null, "state": "on-deck", "errors": null, "href": "https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/84b5408c-93c9-4656-b790-ad0771d73f06/runs/18a23044-8273-4b36-9779-9d42d988ef11", "task-href": "https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/84b5408c-93c9-4656-b790-ad0771d73f06", "duration": "PT1M11S", "start-time": "2018-03-16T11:06:57-04:00", "added": "2018-03-16T11:06:48-04:00", "result": null}
Mar 16 11:06:57 enpl-max-ps runner INFO     24296018: Running https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/84b5408c-93c9-4656-b790-ad0771d73f06/runs/18a23044-8273-4b36-9779-9d42d988ef11
Mar 16 11:06:57 enpl-max-ps runner INFO     24296018: With bwctliperf3: throughput --source enpl-pt2-10g.eos.nasa.gov --dest perfnsidc.colorado.edu
Mar 16 11:06:57 enpl-max-ps runner DEBUG    24296018: Tool input: {"task-uuid": "84b5408c-93c9-4656-b790-ad0771d73f06", "participant": 0, "schedule": {"duration": "PT1M11S", "start": "2018-03-16T11:06:57-04:00"}, "participant-data": [{}], "test": {"type": "throughput", "spec": {"dest": "perfnsidc.colorado.edu", "source": "enpl-pt2-10g.eos.nasa.gov", "schema": 1}}, "limits-passed": [], "schema": 1}
Mar 16 11:06:57 enpl-max-ps ticker DEBUG    QM: Queue change.
Mar 16 11:06:57 enpl-max-ps ticker DEBUG    QM: Processing
Mar 16 11:06:57 enpl-max-ps runner DEBUG    Schedule change.
Mar 16 11:06:57 enpl-max-ps runner DEBUG    24296018: Start time difference is 0:00:00.012030
Mar 16 11:06:57 enpl-max-ps runner DEBUG    Got 27 upcoming rows
Mar 16 11:06:57 enpl-max-ps runner DEBUG    Skipped 27 already-running runs
Mar 16 11:06:57 enpl-max-ps runner DEBUG    Next run or check in 0:01:00
Mar 16 11:06:57 enpl-max-ps runner DEBUG    24267532: Got result: {"max-clock-error": 0.23000000000000001, "packets-duplicated": 0, "succeeded": true, "histogram-latency": {"41.64": 12, "41.65": 7, "41.62": 29, "41.63": 29, "41.60": 29, "41.61": 40, "41.66": 3, "41.67": 1, "41.59": 33, "41.58": 32, "41.57": 38, "41.56": 31, "41.55": 12, "41.54": 2, "41.53": 2}, "histogram-ttl": {"249": 300}, "packets-sent": 300, "packets-reordered": 0, "packets-lost": 0, "packets-received": 300, "schema": 1}
Mar 16 11:06:57 enpl-max-ps runner DEBUG    24267048: Got result: {"max-clock-error": 0.40999999999999998, "packets-duplicated": 0, "succeeded": true, "histogram-latency": {"31.90": 1, "31.58": 1, "31.78": 4, "31.79": 5, "31.72": 14, "31.73": 15, "31.70": 19, "31.71": 17, "31.76": 17, "31.77": 12, "31.74": 15, "31.75": 10, "31.88": 1, "31.83": 4, "31.82": 1, "31.81": 1, "31.80": 7, "31.85": 3, "31.84": 3, "31.69": 30, "31.68": 18, "31.61": 1, "31.60": 2, "31.63": 6, "31.62": 9, "31.65": 18, "31.64": 21, "31.67": 19, "31.66": 26}, "histogram-ttl": {"245": 300}, "packets-sent": 300, "packets-reordered": 0, "packets-lost": 0, "packets-received": 300, "schema": 1}
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Notifications: archiving_change
Mar 16 11:06:57 enpl-max-ps runner DEBUG    Schedule change.
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Refreshing default archivers from /etc/pscheduler/default-archives
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Nothing has changed; not updating.
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Got 1 rows
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566045: Starting worker
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566045: Thread running
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Pool esmond: Skimming
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566045: Task is https://localhost/pscheduler/tasks/80b613cc-7758-4beb-9d70-85a1d791dc32
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Waiting 15.0 for change or notification
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566045: Run is https://localhost/pscheduler/tasks/80b613cc-7758-4beb-9d70-85a1d791dc32/runs/e5b51ed6-9230-47b8-bd19-5efea4804f8f
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Notifications: archiving_change
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Pool esmond: Got a process
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Refreshing default archivers from /etc/pscheduler/default-archives
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Nothing has changed; not updating.
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Got 2 rows
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566045: Already running a worker
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    Found metadata key 25f6fd88d56244b390d24a8258a763e5 in cache for https://localhost/pscheduler/tasks/80b613cc-7758-4beb-9d70-85a1d791dc32@https://archive.eos.nasa.gov/esmond/perfsonar/archive/, so running in fast mode
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    Found metadata key 25f6fd88d56244b390d24a8258a763e5 in cache for https://localhost/pscheduler/tasks/80b613cc-7758-4beb-9d70-85a1d791dc32@https://archive.eos.nasa.gov/esmond/perfsonar/archive/, so running in fast mode
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    fast_mode is True
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    fast_mode is True
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    Putting data to https://archive.eos.nasa.gov/esmond/perfsonar/archive/25f6fd88d56244b390d24a8258a763e5/: {'data': [{'ts': 1521212817, 'val': [{'val': 0.23000000000000001, 'event-type': 'time-error-estimates'}, {'val': 0, 'event-type': 'packet-duplicates'}, {'val': {u'41.64': 12, u'41.65': 7, u'41.62': 29, u'41.63': 29, u'41.60': 29, u'41.61': 40, u'41.66': 3, u'41.67': 1, u'41.59': 33, u'41.58': 32, u'41.57': 38, u'41.56': 31, u'41.55': 12, u'41.54': 2, u'41.53': 2}, 'event-type': 'histogram-owdelay'}, {'val': {u'249': 300}, 'event-type': 'histogram-ttl'}, {'val': 300, 'event-type': 'packet-count-sent'}, {'val': 0, 'event-type': 'packet-reorders'}, {'val': 0, 'event-type': 'packet-count-lost'}, {'val': {'denominator': 300, 'numerator': 0}, 'event-type': 'packet-loss-rate'}, {'val': {'href': 'https://localhost/pscheduler/tasks/80b613cc-7758-4beb-9d70-85a1d791dc32/runs/e5b51ed6-9230-47b8-bd19-5efea4804f8f'}, 'event-type': 'pscheduler-run-href'}]}]}
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    Putting data to https://archive.eos.nasa.gov/esmond/perfsonar/archive/25f6fd88d56244b390d24a8258a763e5/: {'data': [{'ts': 1521212817, 'val': [{'val': 0.23000000000000001, 'event-type': 'time-error-estimates'}, {'val': 0, 'event-type': 'packet-duplicates'}, {'val': {u'41.64': 12, u'41.65': 7, u'41.62': 29, u'41.63': 29, u'41.60': 29, u'41.61': 40, u'41.66': 3, u'41.67': 1, u'41.59': 33, u'41.58': 32, u'41.57': 38, u'41.56': 31, u'41.55': 12, u'41.54': 2, u'41.53': 2}, 'event-type': 'histogram-owdelay'}, {'val': {u'249': 300}, 'event-type': 'histogram-ttl'}, {'val': 300, 'event-type': 'packet-count-sent'}, {'val': 0, 'event-type': 'packet-reorders'}, {'val': 0, 'event-type': 'packet-count-lost'}, {'val': {'denominator': 300, 'numerator': 0}, 'event-type': 'packet-loss-rate'}, {'val': {'href': 'https://localhost/pscheduler/tasks/80b613cc-7758-4beb-9d70-85a1d791dc32/runs/e5b51ed6-9230-47b8-bd19-5efea4804f8f'}, 'event-type': 'pscheduler-run-href'}]}]}
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566046: Starting worker
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Waiting 15.0 for change or notification
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566046: Thread running
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566046: Task is https://localhost/pscheduler/tasks/389d562a-794f-4858-a122-fd5511dc9a04
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566046: Run is https://localhost/pscheduler/tasks/389d562a-794f-4858-a122-fd5511dc9a04/runs/99363258-2f59-467d-840a-81c4740a9390
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Pool esmond: Got a process
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    Found metadata key 49ec40961896445c99a8bb00bfdf13df in cache for https://localhost/pscheduler/tasks/389d562a-794f-4858-a122-fd5511dc9a04@https://archive.eos.nasa.gov/esmond/perfsonar/archive/, so running in fast mode
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    Found metadata key 49ec40961896445c99a8bb00bfdf13df in cache for https://localhost/pscheduler/tasks/389d562a-794f-4858-a122-fd5511dc9a04@https://archive.eos.nasa.gov/esmond/perfsonar/archive/, so running in fast mode
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    fast_mode is True
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    fast_mode is True
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    Putting data to https://archive.eos.nasa.gov/esmond/perfsonar/archive/49ec40961896445c99a8bb00bfdf13df/: {'data': [{'ts': 1521212817, 'val': [{'val': 0.40999999999999998, 'event-type': 'time-error-estimates'}, {'val': 0, 'event-type': 'packet-duplicates'}, {'val': {u'31.90': 1, u'31.58': 1, u'31.78': 4, u'31.79': 5, u'31.72': 14, u'31.73': 15, u'31.70': 19, u'31.71': 17, u'31.76': 17, u'31.77': 12, u'31.74': 15, u'31.75': 10, u'31.88': 1, u'31.83': 4, u'31.82': 1, u'31.81': 1, u'31.80': 7, u'31.85': 3, u'31.84': 3, u'31.69': 30, u'31.68': 18, u'31.61': 1, u'31.60': 2, u'31.63': 6, u'31.62': 9, u'31.65': 18, u'31.64': 21, u'31.67': 19, u'31.66': 26}, 'event-type': 'histogram-owdelay'}, {'val': {u'245': 300}, 'event-type': 'histogram-ttl'}, {'val': 300, 'event-type': 'packet-count-sent'}, {'val': 0, 'event-type': 'packet-reorders'}, {'val': 0, 'event-type': 'packet-count-lost'}, {'val': {'denominator': 300, 'numerator': 0}, 'event-type': 'packet-loss-rate'}, {'val': {'href': 'https://localhost/pscheduler/tasks/389d562a-794f-4858-a122-fd5511dc9a04/runs/99363258-2f59-467d-840a-81c4740a9390'}, 'event-type': 'pscheduler-run-href'}]}]}
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    Putting data to https://archive.eos.nasa.gov/esmond/perfsonar/archive/49ec40961896445c99a8bb00bfdf13df/: {'data': [{'ts': 1521212817, 'val': [{'val': 0.40999999999999998, 'event-type': 'time-error-estimates'}, {'val': 0, 'event-type': 'packet-duplicates'}, {'val': {u'31.90': 1, u'31.58': 1, u'31.78': 4, u'31.79': 5, u'31.72': 14, u'31.73': 15, u'31.70': 19, u'31.71': 17, u'31.76': 17, u'31.77': 12, u'31.74': 15, u'31.75': 10, u'31.88': 1, u'31.83': 4, u'31.82': 1, u'31.81': 1, u'31.80': 7, u'31.85': 3, u'31.84': 3, u'31.69': 30, u'31.68': 18, u'31.61': 1, u'31.60': 2, u'31.63': 6, u'31.62': 9, u'31.65': 18, u'31.64': 21, u'31.67': 19, u'31.66': 26}, 'event-type': 'histogram-owdelay'}, {'val': {u'245': 300}, 'event-type': 'histogram-ttl'}, {'val': 300, 'event-type': 'packet-count-sent'}, {'val': 0, 'event-type': 'packet-reorders'}, {'val': 0, 'event-type': 'packet-count-lost'}, {'val': {'denominator': 300, 'numerator': 0}, 'event-type': 'packet-loss-rate'}, {'val': {'href': 'https://localhost/pscheduler/tasks/389d562a-794f-4858-a122-fd5511dc9a04/runs/99363258-2f59-467d-840a-81c4740a9390'}, 'event-type': 'pscheduler-run-href'}]}]}
Mar 16 11:06:57 enpl-max-ps runner DEBUG    Got 27 upcoming rows
Mar 16 11:06:57 enpl-max-ps runner DEBUG    Skipped 27 already-running runs
Mar 16 11:06:57 enpl-max-ps runner DEBUG    Next run or check in 0:01:00
Mar 16 11:06:57 enpl-max-ps runner DEBUG    Schedule change.
Mar 16 11:06:57 enpl-max-ps runner DEBUG    24267655: Got result: {"max-clock-error": 0.22, "packets-duplicated": 0, "succeeded": true, "histogram-latency": {"49.84": 1, "49.85": 1, "49.86": 5, "49.87": 5, "49.83": 1, "49.88": 18, "49.89": 29, "50.04": 2, "50.02": 1, "50.03": 3, "50.00": 5, "50.01": 3, "49.99": 10, "49.98": 7, "49.93": 37, "49.92": 39, "49.91": 37, "49.90": 24, "49.97": 6, "49.96": 13, "49.95": 22, "49.94": 31}, "histogram-ttl": {"246": 300}, "packets-sent": 300, "packets-reordered": 0, "packets-lost": 0, "packets-received": 300, "schema": 1}
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Notifications: archiving_change
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Refreshing default archivers from /etc/pscheduler/default-archives
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Nothing has changed; not updating.
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Got 3 rows
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566045: Already running a worker
Mar 16 11:06:57 enpl-max-ps runner DEBUG    24267038: Got result: {"max-clock-error": 1.0800000000000001, "packets-duplicated": 0, "succeeded": true, "histogram-latency": {"27.46": 1, "26.09": 1, "26.08": 1, "26.03": 3, "26.02": 4, "26.01": 2, "26.00": 6, "26.21": 1, "26.05": 2, "26.04": 1, "26.89": 1, "26.72": 1, "25.91": 45, "25.90": 10, "25.93": 50, "25.92": 77, "25.95": 17, "25.94": 45, "25.97": 9, "25.96": 15, "25.99": 4, "25.98": 4}, "histogram-ttl": {"243": 300}, "packets-sent": 300, "packets-reordered": 0, "packets-lost": 0, "packets-received": 300, "schema": 1}
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566046: Already running a worker
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566047: Starting worker
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566047: Thread running
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Waiting 15.0 for change or notification
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566047: Task is https://localhost/pscheduler/tasks/83161a5f-c3ec-4e7f-9782-19aaca576258
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566047: Run is https://localhost/pscheduler/tasks/83161a5f-c3ec-4e7f-9782-19aaca576258/runs/f281f895-cef1-4c9b-8eaa-e0a0f2fc1aaa
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Pool esmond: Got a process
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Notifications: archiving_change
Mar 16 11:06:57 enpl-max-ps tool-bwctliperf3/run DEBUG    Debug started
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Refreshing default archivers from /etc/pscheduler/default-archives
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Nothing has changed; not updating.
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Got 4 rows
Mar 16 11:06:57 enpl-max-ps tool-bwctl/run DEBUG    Debug started
Mar 16 11:06:57 enpl-max-ps tool-bwctl/run DEBUG    starting bwctl tool
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566045: Already running a worker
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    Found metadata key f0a52756d81d44d3a7d91f0a825ac44a in cache for https://localhost/pscheduler/tasks/83161a5f-c3ec-4e7f-9782-19aaca576258@https://archive.eos.nasa.gov/esmond/perfsonar/archive/, so running in fast mode
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    Found metadata key f0a52756d81d44d3a7d91f0a825ac44a in cache for https://localhost/pscheduler/tasks/83161a5f-c3ec-4e7f-9782-19aaca576258@https://archive.eos.nasa.gov/esmond/perfsonar/archive/, so running in fast mode
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    fast_mode is True
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    fast_mode is True
Mar 16 11:06:57 enpl-max-ps tool-bwctl/run DEBUG    Input is {u'task-uuid': u'84b5408c-93c9-4656-b790-ad0771d73f06', u'participant': 0, u'schedule': {u'duration': u'PT1M11S', u'start': u'2018-03-16T11:06:57-04:00'}, u'participant-data': [{}], u'test': {u'type': u'throughput', u'spec': {u'dest': u'perfnsidc.colorado.edu', u'source': u'enpl-pt2-10g.eos.nasa.gov', u'schema': 1}}, u'limits-passed': [], u'schema': 1}
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    Putting data to https://archive.eos.nasa.gov/esmond/perfsonar/archive/f0a52756d81d44d3a7d91f0a825ac44a/: {'data': [{'ts': 1521212817, 'val': [{'val': 0.22, 'event-type': 'time-error-estimates'}, {'val': 0, 'event-type': 'packet-duplicates'}, {'val': {u'49.84': 1, u'49.85': 1, u'49.86': 5, u'49.87': 5, u'49.83': 1, u'49.88': 18, u'49.89': 29, u'50.04': 2, u'50.02': 1, u'50.03': 3, u'50.00': 5, u'50.01': 3, u'49.99': 10, u'49.98': 7, u'49.93': 37, u'49.92': 39, u'49.91': 37, u'49.90': 24, u'49.97': 6, u'49.96': 13, u'49.95': 22, u'49.94': 31}, 'event-type': 'histogram-owdelay'}, {'val': {u'246': 300}, 'event-type': 'histogram-ttl'}, {'val': 300, 'event-type': 'packet-count-sent'}, {'val': 0, 'event-type': 'packet-reorders'}, {'val': 0, 'event-type': 'packet-count-lost'}, {'val': {'denominator': 300, 'numerator': 0}, 'event-type': 'packet-loss-rate'}, {'val': {'href': 'https://localhost/pscheduler/tasks/83161a5f-c3ec-4e7f-9782-19aaca576258/runs/f281f895-cef1-4c9b-8eaa-e0a0f2fc1aaa'}, 'event-type': 'pscheduler-run-href'}]}]}
Mar 16 11:06:57 enpl-max-ps tool-bwctl/run DEBUG    Waiting 1 sec for server on other side to start
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    Putting data to https://archive.eos.nasa.gov/esmond/perfsonar/archive/f0a52756d81d44d3a7d91f0a825ac44a/: {'data': [{'ts': 1521212817, 'val': [{'val': 0.22, 'event-type': 'time-error-estimates'}, {'val': 0, 'event-type': 'packet-duplicates'}, {'val': {u'49.84': 1, u'49.85': 1, u'49.86': 5, u'49.87': 5, u'49.83': 1, u'49.88': 18, u'49.89': 29, u'50.04': 2, u'50.02': 1, u'50.03': 3, u'50.00': 5, u'50.01': 3, u'49.99': 10, u'49.98': 7, u'49.93': 37, u'49.92': 39, u'49.91': 37, u'49.90': 24, u'49.97': 6, u'49.96': 13, u'49.95': 22, u'49.94': 31}, 'event-type': 'histogram-owdelay'}, {'val': {u'246': 300}, 'event-type': 'histogram-ttl'}, {'val': 300, 'event-type': 'packet-count-sent'}, {'val': 0, 'event-type': 'packet-reorders'}, {'val': 0, 'event-type': 'packet-count-lost'}, {'val': {'denominator': 300, 'numerator': 0}, 'event-type': 'packet-loss-rate'}, {'val': {'href': 'https://localhost/pscheduler/tasks/83161a5f-c3ec-4e7f-9782-19aaca576258/runs/f281f895-cef1-4c9b-8eaa-e0a0f2fc1aaa'}, 'event-type': 'pscheduler-run-href'}]}]}
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566048: Starting worker
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566048: Thread running
Mar 16 11:06:57 enpl-max-ps runner DEBUG    Got 27 upcoming rows
Mar 16 11:06:57 enpl-max-ps runner DEBUG    Skipped 27 already-running runs
Mar 16 11:06:57 enpl-max-ps runner DEBUG    Next run or check in 0:01:00
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566046: Already running a worker
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566048: Task is https://localhost/pscheduler/tasks/bb99eb89-2da4-4a96-b278-2b66bf7eb078
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566047: Already running a worker
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566048: Run is https://localhost/pscheduler/tasks/bb99eb89-2da4-4a96-b278-2b66bf7eb078/runs/1c4a5aaa-a3d6-4b70-8853-8b051f720fe3
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Waiting 15.0 for change or notification
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Pool esmond: Got a process
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    Found metadata key 6b52a70ad209467a942b0ddd3212fe4c in cache for https://localhost/pscheduler/tasks/bb99eb89-2da4-4a96-b278-2b66bf7eb078@https://archive.eos.nasa.gov/esmond/perfsonar/archive/, so running in fast mode
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    Found metadata key 6b52a70ad209467a942b0ddd3212fe4c in cache for https://localhost/pscheduler/tasks/bb99eb89-2da4-4a96-b278-2b66bf7eb078@https://archive.eos.nasa.gov/esmond/perfsonar/archive/, so running in fast mode
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    fast_mode is True
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    fast_mode is True
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    Putting data to https://archive.eos.nasa.gov/esmond/perfsonar/archive/6b52a70ad209467a942b0ddd3212fe4c/: {'data': [{'ts': 1521212817, 'val': [{'val': 1.0800000000000001, 'event-type': 'time-error-estimates'}, {'val': 0, 'event-type': 'packet-duplicates'}, {'val': {u'27.46': 1, u'26.09': 1, u'26.08': 1, u'26.03': 3, u'26.02': 4, u'26.01': 2, u'26.00': 6, u'26.21': 1, u'26.05': 2, u'26.04': 1, u'26.89': 1, u'26.72': 1, u'25.91': 45, u'25.90': 10, u'25.93': 50, u'25.92': 77, u'25.95': 17, u'25.94': 45, u'25.97': 9, u'25.96': 15, u'25.99': 4, u'25.98': 4}, 'event-type': 'histogram-owdelay'}, {'val': {u'243': 300}, 'event-type': 'histogram-ttl'}, {'val': 300, 'event-type': 'packet-count-sent'}, {'val': 0, 'event-type': 'packet-reorders'}, {'val': 0, 'event-type': 'packet-count-lost'}, {'val': {'denominator': 300, 'numerator': 0}, 'event-type': 'packet-loss-rate'}, {'val': {'href': 'https://localhost/pscheduler/tasks/bb99eb89-2da4-4a96-b278-2b66bf7eb078/runs/1c4a5aaa-a3d6-4b70-8853-8b051f720fe3'}, 'event-type': 'pscheduler-run-href'}]}]}
Mar 16 11:06:57 enpl-max-ps archiver-esmond/archive DEBUG    Putting data to https://archive.eos.nasa.gov/esmond/perfsonar/archive/6b52a70ad209467a942b0ddd3212fe4c/: {'data': [{'ts': 1521212817, 'val': [{'val': 1.0800000000000001, 'event-type': 'time-error-estimates'}, {'val': 0, 'event-type': 'packet-duplicates'}, {'val': {u'27.46': 1, u'26.09': 1, u'26.08': 1, u'26.03': 3, u'26.02': 4, u'26.01': 2, u'26.00': 6, u'26.21': 1, u'26.05': 2, u'26.04': 1, u'26.89': 1, u'26.72': 1, u'25.91': 45, u'25.90': 10, u'25.93': 50, u'25.92': 77, u'25.95': 17, u'25.94': 45, u'25.97': 9, u'25.96': 15, u'25.99': 4, u'25.98': 4}, 'event-type': 'histogram-owdelay'}, {'val': {u'243': 300}, 'event-type': 'histogram-ttl'}, {'val': 300, 'event-type': 'packet-count-sent'}, {'val': 0, 'event-type': 'packet-reorders'}, {'val': 0, 'event-type': 'packet-count-lost'}, {'val': {'denominator': 300, 'numerator': 0}, 'event-type': 'packet-loss-rate'}, {'val': {'href': 'https://localhost/pscheduler/tasks/bb99eb89-2da4-4a96-b278-2b66bf7eb078/runs/1c4a5aaa-a3d6-4b70-8853-8b051f720fe3'}, 'event-type': 'pscheduler-run-href'}]}]}
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566046: Returned JSON from archiver: {u'succeeded': True}
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566046: Succeeded: 99363258-2f59-467d-840a-81c4740a9390 to esmond
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566045: Returned JSON from archiver: {u'succeeded': True}
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566045: Succeeded: e5b51ed6-9230-47b8-bd19-5efea4804f8f to esmond
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566046: Thread finished
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566045: Thread finished
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Notifications: archiving_change
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Refreshing default archivers from /etc/pscheduler/default-archives
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Nothing has changed; not updating.
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Got 2 rows
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566047: Already running a worker
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566048: Already running a worker
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Waiting 15.0 for change or notification
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566047: Returned JSON from archiver: {u'succeeded': True}
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566047: Succeeded: f281f895-cef1-4c9b-8eaa-e0a0f2fc1aaa to esmond
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566047: Thread finished
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Notifications: archiving_change
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Refreshing default archivers from /etc/pscheduler/default-archives
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Nothing has changed; not updating.
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Got 1 rows
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566048: Already running a worker
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Waiting 15.0 for change or notification
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566048: Returned JSON from archiver: {u'succeeded': True}
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566048: Succeeded: 1c4a5aaa-a3d6-4b70-8853-8b051f720fe3 to esmond
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    23566048: Thread finished
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Notifications: archiving_change
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Nothing to archive; finding time until next archiving.
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Next archiving in 139.459908
Mar 16 11:06:57 enpl-max-ps archiver DEBUG    Waiting 15.0 for change or notification
Mar 16 11:08:08 enpl-max-ps pscheduler-api DEBUG    Response 200+JSON: {"result-href": "https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/84b5408c-93c9-4656-b790-ad0771d73f06/runs/18a23044-8273-4b36-9779-9d42d988ef11/result", "end-time": "2018-03-16T11:06:58-04:00", "participant": 0, "participant-data": {}, "result-merged": {"diags": "", "succeeded": false, "error": "Unable to parse iperf3 output as JSON: No JSON object could be decoded"}, "participant-data-full": [{}], "participants": ["enpl-pt2-10g.eos.nasa.gov"], "state-display": "Finished", "result-full": [{"diags": "", "succeeded": false, "error": "Unable to parse iperf3 output as JSON: No JSON object could be decoded"}], "state": "finished", "errors": null, "href": "https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/84b5408c-93c9-4656-b790-ad0771d73f06/runs/18a23044-8273-4b36-9779-9d42d988ef11", "task-href": "https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/84b5408c-93c9-4656-b790-ad0771d73f06", "duration": "PT1S", "start-time": "2018-03-16T11:06:57-04:00", "added": "2018-03-16T11:06:48-04:00", "result": {"diags": "", "succeeded": false, "error": "Unable to parse iperf3 output as JSON: No JSON object could be decoded"}}