Source files decode to Text before lexing (decodeAutoEncoding, BOM-sniffed, lenient), so a byte sequence that is not valid UTF-8 reaches the lexer as replacement characters. Upstream lexes raw bytes, and a string literal containing such bytes carries them into the eval string verbatim - where nova-nix's eval strings are byte-indexed (VStr ByteString) precisely to match that model. The divergence is observable only for sources that are not valid UTF-8 (or UTF-16 with the BOM), so real nixpkgs is unaffected, but a byte-preserving lexing path would close the last text-shaped gap between source and eval string semantics.
Weigh against the UTF-16 auto-detection support, which a bytes-first lexer must keep: transcoding UTF-16 sources to UTF-8 bytes at the decode gate preserves both properties.
From the #85 audit (parser subsystem).
Source files decode to
Textbefore lexing (decodeAutoEncoding, BOM-sniffed, lenient), so a byte sequence that is not valid UTF-8 reaches the lexer as replacement characters. Upstream lexes raw bytes, and a string literal containing such bytes carries them into the eval string verbatim - where nova-nix's eval strings are byte-indexed (VStr ByteString) precisely to match that model. The divergence is observable only for sources that are not valid UTF-8 (or UTF-16 with the BOM), so real nixpkgs is unaffected, but a byte-preserving lexing path would close the last text-shaped gap between source and eval string semantics.Weigh against the UTF-16 auto-detection support, which a bytes-first lexer must keep: transcoding UTF-16 sources to UTF-8 bytes at the decode gate preserves both properties.
From the #85 audit (parser subsystem).