Add offset tracking to Encoding

[anidoesdev]

Hi @nico-martin and @xenova,

Adds offsets: [number, number][] to the Encoding object returned by
tokenizer.encode(). Each entry is a half-open character span [start, end]
in the original (pre-normalization) input string. Special tokens injected by
post-processors ([CLS], [SEP], </s>, etc.) receive the sentinel [0, 0].

Adapted from huggingface/transformers.js#1702.

Changes:

• All pre-tokenizers now carry (token, [start, end]) tuples through the
  pipeline instead of bare strings
• New build_alignment_map(original, normalized) in src/utils/core.ts maps
  positions in the normalizer's output string back to the original input,
  handling NFD expansion, NFC compression, and accent/diacritic stripping
• offsets: Array<[number, number]> added to the Encoding interface in
  src/static/types.ts and re-exported from types/index.d.ts

Tokenizer families with dedicated offset tests:

• BERT (BertNormalizer → BertPreTokenizer → WordPiece → BertProcessing)
• GPT-2 (ByteLevel pre-tokenizer → BPE)
• RoBERTa (ByteLevel → BPE → RobertaProcessing)
• T5 (Precompiled normalizer → Metaspace → Unigram → TemplateProcessing)
• Edge cases: empty input, whitespace-only, NFC vs NFD normalization, text pairs

Known limitations:

• ByteLevel.trim_offsets is not yet implemented (marked @todo). When
  trim_offsets: true is set in a config (RoBERTa, GPT-2 variants), the
  leading space before a word is currently absorbed into that token's span
  start rather than being trimmed to the word boundary.
• The Precompiled normalizer (used by T5/SentencePiece models) is
  approximated. build_alignment_map correctly handles the common Unicode
  transformations (NFC recomposition, whitespace normalisation) but may
  diverge from the Rust reference implementation for rare character mappings
  not expressible as standard Unicode normalization steps.

[xenova]

Thanks for the PR! 🤗 Regarding implementation, it does seem to touch a lot of files (not necessarily a bad thing), but there are some points where it does certainly change API, e.g., pre_tokenize_text returning Array<[string, [number, number]] now. Are you following the rust implementation for this?

[anidoesdev]

Good point changing pre_tokenize_text's return type is a breaking change for anyone extending PreTokenizer, which I should have flagged upfront.

Regarding the Rust implementation: not exactly. I looked at the Rust implementation NormalizedString tracks alignment internally and PreTokenizedString lets pre-tokenizers mutate in place, so offsets flow through without changing return types.

Would you prefer I take that approach introducing something closer to NormalizedString to carry alignment internally or
is there a lighter fix you have in mind that avoids the breaking API change?

[anidoesdev]

Good point changing pre_tokenize_text's return type is a breaking change for anyone extending PreTokenizer, which I should have flagged upfront.

Regarding the Rust implementation: not exactly. I looked at the Rust implementation NormalizedString tracks alignment internally and PreTokenizedString lets pre-tokenizers mutate in place, so offsets flow through without changing return types.

Would you prefer I take that approach introducing something closer to NormalizedString to carry alignment internally or is there a lighter fix you have in mind that avoids the breaking API change?

Hi @xenova
Can I get follow up on this? So that I can start working on this..