perfSONAR User Q&A and Other Discussion

[Mark Feit <>]

I wrote:

 

Paul MEBALE-ASSAKO writes:

  

Limit processor is not initialized: ‘ascii’ codec can't decode byte 0xc3 in position 11: ordinal not in range(128)

 

This is a bug in pScheduler.  There’s some old code in the JSON parser that tries to decode byte streams as ASCII when it shouldn’t be so picky.

 

After further investigation as I went to implement the fix, it turns out that there is a bug in the production pScheduler code, but not quite what I thought it was.

 

The current RFC describing JSON (https://www.rfc-editor.org/rfc/rfc8259, released in 2017) requires that “JSON text exchanged between systems that are not part of a closed ecosystem must be encoded using UTF-8.”  ECMA 404 does not call this out explicitly but does say that it and RFC 8259 are intended to be identical.  Since perfSONAR is not a closed ecosystem and there’s an established standard, pScheduler in 5.2 will abide by it and decode as UTF-8.

 

However, the byte in the file that caused the problem, 0xc3, will still cause it to be rejected.  The existing code (incorrectly) attempts to decode the byte stream as  ASCII, so that byte isn’t valid.  (It is valid ISO 8859-1, but that’s not in the picture here.)  UTF-8 maps the first 128 codepoints to 7-bit ASCII, but anything in the 0x80-0xff range is called a continuation byte, which indicates that there is at least one additional byte to follow.  The byte 0xc3 has bits set indicating that there is one more byte to follow that must have bits 10xxxxxx to be valid UTF-8.  The moral here is that the upper half of ISO 8859-1 character set shouldn’t be used in JSON.

 

Paul, if you’d be so kind as to send me the limit configuration you’re using off-list, I’ll use it as a test case.

 

--Mark