Clarify encoding status of strings

[ralokt]

String encodings are currently technically unspecified

Currently, the EVF spec doesn't actually specify whether or not the c strings in the header need to be valid utf-8 (at least not in the english translation).

It does state that utf-8 is recommended in the design goals, but the actual specification only mentions them as strings. Even if we take the design goals to be part of the spec, a recommendation doesn't mean that clone authors can't simply decide to not follow it.

Also, there is no place in the file format that specifies the encoding used.

So, in its current state, EVF strings aren't actually well specified. Sure, we can just parse them as utf-8, but it would be possible to create a compliant clone that writes some other encoding.

Proposal: specify that strings are utf-8 (or add encoding info to the format)

I would propose one of the following two changes to the standard:

• Specify that all c strings are utf-8. Then, if a clone writes strings in a different encoding, it is non-compliant, and replays it produces should be rejected by other clones/replay viewers.
• Add a way to specify the encoding in the format.

I prefer the first option, as I don't think it's actually very limiting:

• Requiring a new clone, a program with an international audience, to use a unicode format in the year 2024 is not too much to ask.
• The strings are c strings, eliminating multi-byte encodings where characters can contain '\0'. As much as these might be appealing to windows developers, they're already not an option.
• So, as industry standards go, we're pretty much left with utf-8 anyway.
• Even if utf-8 is too much of an ask for a given application, ASCII definitely isn't, and valid ASCII is valid utf-8.

What about translating old replays?

The one drawback to making utf-8 part of the spec instead of a recommendation is that it can complicate translating replays from other clones that use code pages. However, this seems like a kinda niche application to me - also, since vsweep 4.0.0, all strings in RMVs are utf-8, and in Arbiter, some inference via the player's country could probably be done to produce pretty good results.

That leaves old RMV replays, but tools that take an extra input for the encoding are still a feasible fallback to process these - especially as no new data should be generated in that format in any place where it gets submitted to an automated service that needs to make a decision without human intervention.

Also, in general, only supporting utf-8 doesn't make EVF less universal - it's not like code pages can't be mapped to utf-8, it's just that requiring utf-8 would make that necessary at the time a replay is converted.

So, altogether, I'd argue that the burden of dealing with multiple possibilities for encodings should not be dragged on to the next generation of replay processors.

[eee555]

In the original design, the encoding should be determined according to the "source of the recording". For example, if a video claims to be from "Meta 3.1", then the encoding is "utf-8", which is beyond doubt.

The rationality of this design is that the number of clones will be very limited, and the encoding can be hard-coded.

But now we find that if someone wants to parse evf format (as is currently the case), this design will be in trouble. As he needs to have all knowledge about all clones encoding.

Therefore, to make EVFs easier to parse, this knowledge of encoding should be provided in some way.

[eee555]

Requiring a new clone, a program with an international audience, to use a unicode format in the year 2024 is not too much to ask.

The task of pushing clones to adopt unicode format should be left to ranking websites (For example, I would recommend open minesweeper to you), but not the format. If someone just wants to use a niche code, and his purpose is not to serve international audience, we should not stop it.

[eee555]

In my opinion, a recommendation has the same effect as a requirement. A recommendation essentially reminds developers that only UTF-8 can reduce the likelihood of garbled text on other clients. If a developer doesn't mind this outcome, they probably won't care if their replays can't be played due to not using UTF-8.

If the developer wants to disrupt, then he can do whatever he wants, and there is no strong limitation in this video format.

[ralokt]

In my opinion, a recommendation has the same effect as a requirement.

Hard disagree on this. As a library author, it makes a huge difference to me.

A recommendation means that my library needs to be able to handle these inputs as well. If it doesn't, my library is broken.

A requirement means that my library can assume that it is followed, and refuse to parse inputs that don't conform to the requirement. If a clone produces these inputs, the clone is broken.

Now, of course, clone authors can disregard the requirement anyway, and library authors can treat it as a requirement anyway. But what is still changed is who is responsible if things break.

Assume that I as a library author treat it as a requirement, and a clone author disregards the recommendation, and that I get a complaint from a library user that things don't work:

• If it's a requirement in the spec, I can tell the user to complain to the clone author, as the clone author disregarded the spec. I can rightfully say that it's not my responsibility to support this behavior. The clone author needs to fix it, or live with tooling not working with their replays.
• If it's only a recommendation, the library user is right. My library is broken, and I need to fix it.

Now, what incentive structure for clone authors does that create?

• If it's a requirement, clone authors are incentivized to follow the spec. Their replays will not work with common tooling otherwise. If their broken output becomes common enough to warrant libraries implementing it in spite of the standard, fellow devs will be rightfully annoyed at them for making the minesweeper software ecosystem worse.
• If it's a recommendation, clone authors are incentivized to be lazy, throw whatever the framework they are working with uses as strings in there, and call it a day. This makes the ecosystem worse by default.

By far not all disruption is good. Disruption can be good if new technologies emerge that work better than what came before - but code page encodings are most definitely not an example of that.

If this is about supporting GB 18030 for legal reasons, there's still the option for the format to require explicitly specifying the encoding.

[eee555]

Option 1: Add a field for specifying the encoding.
Option 2: Enforce the use of "UTF-8".
Option 3: Recommend using "UTF-8".

For EVF files translated from AVF, only Option 3 can handle the situation because, in reality, I don't know the encoding of AVF. I can only write the original array in the EVF file. Even if I knew the encoding, to avoid requiring ms-toollib to include a decoding toolset, only Options 1 and 3 would be feasible.

The positioning of ms-toollib is as a low-level tool. As long as the strings do not affect other algorithms, low-level tools should avoid excessive processing.

Of course, as an improvement, the EVF spec can provide standard hints about the encoding used in formats like AVF, making it easier for the UI layer to handle and display them.

[ralokt]

I do kinda see where you're coming from. It is a difficult problem, because in a sense, there's no way to win - we'd need to always know the encoding for that...

But I do have some ideas.

More options

Option 4: Enforce the use of "UTF-8" for replays generated by clones only, allow translated replays to contain any encoding, but enforce that they have metadata that will mark them as translated replays, and at least provide clues to parsers.

Option 5: Option 2, but add new fields for untranslatable metadata, have translators write to those fields and leave the main fields empty where they don't know the source encoding. No data is better than garbage data (and having the data somewhere else where it's clearly marked as incomplete is a lot better than no data).

Option 6: Option 2, but add an encoding_guessed field that specifies which encoding was assumed during translation. If empty, the encoding was known, so this is only needed for translated replays. That way, if a translated replay contains garbage, the original data can be reconstructed by re-encoding as the wrongly guessed encoding. For example, let's say that the translation guessed that cp-1252 was used, but cp-1250 was actually correct. To fix it, one could parse the utf-8, re-encode it as cp-1252, then take that binary data (which is the same as in the original replay), interpret it as cp-1250 instead, and then encode that as utf-8 again. There are no losses here - if the utf-8 comes from cp-1252, then all characters are representable in cp-1252.

A case for option 2

As crazy as that might sound, I still think that option 2 is viable and actually the most correct one. All it requires is that replay translation takes another input - the encoding for the source replay.

Now, of course, that source encoding isn't always known, so you might say that this is impossible. But the same is true once you have an EVF that has been translated that way. And once it's a translated EVF replay, it only gets harder. At best, the EVF will contain just as much metadata as the original, and translation tools will have the same amount of information to consider when guessing the encoding. At worst, something gets lost in translation.

End of the day, by allowing EVF to contain unspecified text data, you'd just be pushing the problem down the road, and diluting the usefulness of EVF as a universal replay format (more on that later).

And I think that there's another reason that the best chance at guessing encodings is at the source: If you have an EVF, but not the original, it's pretty unlikely that you're the player that produced the original. But if you're making an EVF, there's a decent chance that you ARE the original player translating the replay for others, and can supply the encoding. This can be partially automated by looking at the system language, asking users to confirm that their player name is being rendered correctly, then, as a fallback if it isn't correct, asking them to look at a list of player names decoded as different encodings, and pick the one that is correct.

An issue with all options BUT option 2 and 6

As soon as a field can contain text data in any encoding, you shouldn't be using C strings. UTF-16 contains null bytes that will be parsed as the terminator. It's also a reasonable choice for windows GUI applications, as it's what win32 wants you to use for unicode. Even if we don't know of any clones that do this rn, it isn't future proof at all.

Universal formats vs low-level tools

EVF is meant to be a universal format, right? Because if so, its specification technically shouldn't follow the needs of a specific tool (ms-toollib), the spec should be written to be applicable universally, and then tools should be written in a way that takes that into account.

Now, I'm aware that in practice, this gets blurry, and also I'd be lying if I claimed that RMV were free from vsweep-specific details. Actually having an implementation is worth some compromises, and so is being somewhat backwards-compatible with existing parsers.

But I do think that there shouldn't be any fundamental issues with the spec that are tied to a specific tool. Player names are a pretty important part of the format. And yeah, ms-toollib may have a focus on analyzing the game itself, but player names tell us who actually played a game, they're super important for the social context the game exists in, and a universal format needs to get them right. In general, having a field that is allowed by the format to contain garbage isn't a great look for a universal format.

Now, you might be arguing that this doesn't just apply to ms-toollib, but low-level tools in general. But I see three cases here:

• a low-level tool that just wants to read. Here, there is an advantage to having just utf-8. It's compatible with c strings, you can just pass data around as binary because people can assume that it's utf-8, and if you want to process it as text, it's highly likely that you can just do it out of the box (rust supports it, for example). This is the most common case, and it gets a lot easier.
• a low-level tool that wants to write, and can control the source encoding. Pretty much the same as before applies here. Using UTF-8 is a great idea anyway; For the C++ on windows (or probably anywhere that uses UTF-16) people, nowide is amazing, I used it to convert vsweep (which is a MFC application) to use UTF-8 internally and only convert to/from UTF-16 at application boundaries, and it worked like a charm. This case is fairly common and gets a bit harder.
• a low-level tool that wants to write, and can't control the source encoding - ie, translation. Here, I'd argue that dealing with encodings is just unavoidable if you want to do it right. This case is fairly niche, and the only one that gets significantly harder.

As for ms-toollib, I see two cases:

• it's meant to be a universal library for EVF, including translation. Then it should deal with text encoding properly - perhaps as an optional, feature-gated thing (I'm sure the Python/JS libraries wouldn't mind having them).
• it's meant to be low-level and focus on game analysis. Then perhaps it should be a tool that only reads (or only writes utf-8), and there should be a separate tool that translates replays and uses ms-toollib to do the low-level heavy lifting.

I actually really like the idea of separating the two in general:

• It adheres to the "do one thing, and do it well" aspect of the UNIX design philosophy
• It cleanly separates the concerns of generating replay files that have correct metadata and doing analysis on their contents
• It gives library users a choice of whether or not they want to include the overhead for the translation stuff

ms-toollib would not need to be encoding-aware. The only change needed would be providing an API that allows libraries to pass a re-encoding function along with whatever else is needed for translation. This re-encoding function would be binary in, binary out, and guarantee that its output is UTF-8. Then a higher-level library could use that API together with a crate that can deal with character encodings, and put it all together to provide a high-level API that is suitable for replay translation, not just parsing and generation.

I'm aware that both of these mean work, but I'd be willing to help with that, or maybe even do it by myself. If you're open to the idea of going with option 2 or 6, I can make a proof of concept PR.

[eee555]

I agree with a principle that EVF and ms_toollib should be separate; making the format is one thing, and implementation is another.

Therefore, it is necessary to improve the usability of EVF data. However, there are two obstacles: encoding methods and start, end timestamps. While ms_toollib does not include encoding and decoding modules.

When involving FFI, passing closures may be difficult and needs to be evaluated. That is, changing from pub fn generate_evf_v3_raw_data(&mut self) to pub fn generate_evf_v3_raw_data(&mut self, decoder, utf8_encoder). In Rust, we know how to do this, but how should it be done when using Rust in Python?

If these difficulties can be overcome, I believe Option 4 is appropriate. And these issues may require your assistance.

[ralokt]

I agree with a principle that EVF and ms_toollib should be separate; making the format is one thing, and implementation is another.

Yay, happy to hear that! :)

Therefore, it is necessary to improve the usability of EVF data. However, there are two obstacles: encoding methods and start, end timestamps. While ms_toollib does not include encoding and decoding modules.

Start and end timestamps seem beyond the scope here, let's make another issue for that?

Decoding isn't an issue if the encoding is always utf-8 (rust supports that without extra libraries) or if we just return binary data (suboptimal from an interface perspective, but usable if we can also distinguish between translated replays where we need to guess, and non-translated replays where we know that we can decode it as utf-8 (which is probably what we'd need to do with option 4 - that's part of why I'd prefer 2 or 6, but I can live with 4)).

Re: encoding:

When involving FFI, passing closures may be difficult and needs to be evaluated. That is, changing from pub fn generate_evf_v3_raw_data(&mut self) to pub fn generate_evf_v3_raw_data(&mut self, decoder, utf8_encoder). In Rust, we know how to do this, but how should it be done when using Rust in Python?

PyO3/pyo3#120 this seems very doable! And I'd guess that wasm-bindgen is similar.

There is also the option to not expose the whole write API in ms-toollib, and instead do that in ms-toollib-translate (or whatever we call the library that is meant to be used for writing). Then library users don't have to mess around with callbacks and can just call a library function where they need to pass an encoding, and be done with it.

So:

• ms-toollib has generate_evf_v3_raw_data_with_utf8(&mut self)
• ms-toollib-translate has generate_evf_v3_raw_data_with_encoding(replay: &mut BaseReplay, encoding: &str)

If these difficulties can be overcome, I believe Option 4 is appropriate. And these issues may require your assistance.

You'll have it! A heads-up, it might be that the next few weeks end up being super busy for me (christmas means a lot of social obligations), and I think this is something where I have to take some time to properly focus on it. Plus, I gotta finish RMV2 and vsweep (and some related things), and do some things for the world championship. So it might not happen immediately - but absolute worst case, I'll make room in January!

By the way, if we're going with option 4 - how do you want to deal with multibyte encodings that could contain null bytes?

[eee555]

Ok, specifying the length before the string allows compatibility with encodings such as "UTF-16" and "UTF-32." These changes will all be included in version 4, and earlier versions will no longer be modified. A new format specification will be provided shortly.

[ralokt]

Ok, specifying the length before the string allows compatibility with encodings such as "UTF-16" and "UTF-32." These changes will all be included in version 4, and earlier versions will no longer be modified. A new format specification will be provided shortly.

Awesome! Yeah, was thinking myself that a new format version would be necessary (for all the options, actually).

[eee555]

The v0.4 version proposal of EVF has been added. If there are no issues, we can start implementing it in the code.
evf-v0.4

[ralokt]

Amazing! Looks great :)

For now, I'm working on a MR that cleans up the warnings, btw. Your change straight up dropped a commit out of my branch as it did the same thing anyway - rename cal3BV to cal_bbbv:

ralokt@archlinux ~/p/m/wasm (fix-warnings)> git rebase main
dropping bb5bbf269cd1b2e119b3ffa2ca255487b8382fbf fix c version still using cal3BV instead of cal_bbbv. NOTE: we seem to have accidentally modified something public there. Take a closer look at what else may be affected. -- patch contents already upstream
Successfully rebased and updated refs/heads/fix-warnings.
ralokt@archlinux ~/p/m/wasm (fix-warnings)>

(ignore the note, that was me thinking I'd done the rename and broken sth).

Great minds think alike ^^

[ralokt]

Oh wait, there's one thing I missed: Does the v0.4 spec specify anywhere that non-translated replays must use utf-8? I can only find the part where it says that the encoding field is only available for translated replays, but the spec should also include that if a replay isn't translated, the encoding will always be utf-8.