Encoding is in reality UTF-8 for PF 6000

[ekdahl]

Even though the specification is only mentioning ASCII as string encoding, when naming things in a PF 6000 controller and using characters not present in plain ASCII, it is actually transferred using UTF-8.

Examples using Swedish characters ÅÄÖ:

MID 0013 Parameter set data upload reply
Parameter set name = Pset1ÅÄÖ
3031303430303133303031203030303020202020303130303130325073657431c385c384c39620202020202020202020202020203033313034303030353030303830303036303031323030303730303130303030383030303030303930303939393130303033363000

MID 0002 Application Communication start acknowledge
Controller Name = TestLabbPF6000ÅÄÖ
3032323130303032303036203030303020202020303130303030303230303033546573744c616262504636303030c385c384c39620202020203034414354303576322e3020202020202020202020202020202030365046365f525f332e362e31322e31303834325f3037312e302e323320202020202020202020202020303850726f6365737320506c757320436f6e74726f6c202020203039413438323034353320203130303033313130303131323031333031343030303030303032363431355669727475616c2053746174696f6e2031202020202020202031363000

[Rickedb]

Hello, thank you for submitting.

I've worked recently with PF6K and the messages were sent all in ASCII. Could you confirm if there is an option or configuration in which the controller can be configured to use ASCII or UTF-8?

I've never seen that behaviour.
Also, if possible, please provide firmware version and if there is a new documentation version of Open Protocol.

[ekdahl]

As ASCII is a subset of UTF-8 this is not noticed until a character that is not part of ASCII is used.
Like when entering the Swedish characters ÅÄÖ in the web site of a PF6000, like below.

Software version of this controller is 3.6.12.10842

A few years ago I asked R&D at Atlas for a confirmation regarding encoding used and why it is not mentioned in the specification. The following is what I sent:

I noticed something that I would like to clarify regarding text encoding in Open Protocol.
This example comes from the PF6000, but I don’t know how it is for the PF4000 or other controllers.
If you name a controller with characters that are not part of ASCII, you will notice that it is "encoded" as UTF-8 in the communication.
Here, the controller is named "Svänghjulsmontering":


According to the specification, this field should be 25 ASCII characters. However, it seems that the PF6000 does not follow this requirement, as it actually behaves more like:
"The controller name is 25 bytes long and specified by UTF-8 encoded characters."
This means it is not necessarily 25 characters but rather 25 bytes.

This raises a few follow-up questions:

Is UTF-8 generally used as the encoding? Or only for certain fields?
Does this apply to all controllers?
Since the specification does not mention encoding (except for ASCII), how can one determine in advance how the data should be interpreted?

The answer I got was not so clear and more like an explanation of what UTF-8 is, which was not what I asked for:

The controller name like you mentioned the length is 25 ASCII characters as specified in the documentation, the ASCII characters are single-byte and fall within the decimal range of 0 to 127. Any character with a value greater than 127 suggests it is UTF-8 character, it can also be multibyte.

For example, in the case of Japanese keyboards, certain characters are represented using two bytes in UTF-8. In UTF-8 encoding, multibyte characters start with a byte pattern where the leftmost bits are set to 110xxxxx for the first byte and 10xxxxxx for subsequent bytes.

To handle these UTF-8 characters, particularly those requiring two bytes, the software needs to recognize the byte pattern and interpret the characters accordingly.

Perhaps an optional encoding setting would be a good idea to have. I guess there could be other brands that have used other encodings like Latin-1 for example, since the specification only mentions ASCII.

[Rickedb]

Until PF6K I believe that there were no other way of configuring it without a proper software or directly at the panel, now they are exposing a web page for it. Maybe there were some kind of limitations that enforces the ASCII back then.

Here in Brazil we use Latin, but most of the time we don't using accents and other characters like cedil (Ç) at configurations to avoid unexpected behaviours. That's why I've never seen this case. I might try something with the PF4K we have at the lab when I get there someday and give the feedback about it. (which probably is in a very old firmware version)

---

But about the library itself: the documentation says that it should be ASCII, however I see no problem in enable parsing it from UTF8 since, like you said, ASCII is a subset of UTF8.
Moreover, it would be nice to enable a feature/improvement to user to set a "global encoding" or an encoding that he wants to use at a single message.

Since I'm planning to improve the library to use Span<char> for performance purposes, I'll as well try to provide this feature.

For now if you use the bytes overload and decode it as UTF-8 on your own might help you out with your problem (at least with controller name).

Does this encoding problem affects you only when parsing the controller name or does it affect other mids?