Migrate high-level serialization to System.Text.Json

[YiyuanMiao]

Migrate high-level serialization to System.Text.Json

Related to #388.

Migrates the OpenSearch.Client high-level serializer (and the low-level OpenSearch.Net serializer) from the bundled, unmaintained Utf8Json fork to System.Text.Json (STJ), and makes STJ the default engine. The legacy Utf8Json engine is kept as a fully supported opt-out fallback.

Note: there are other in-flight PRs targeting the same migration (#982, #996,
#995). This is an independent, complete implementation submitted for
comparison / consideration; happy to consolidate with whichever approach the
maintainers prefer.

Why

Utf8Json is deprecated and unmaintained. System.Text.Json is the official, supported, actively-maintained .NET JSON stack. This removes the reliance on the bundled fork for the default serialization path while preserving backward compatibility through an opt-out.

What changed

• New STJ engines: a low-level SystemTextJsonSerializer and a high-level SystemTextJsonHighLevelSerializer driven by a settings-aware IJsonTypeInfoResolver (HighLevelContractResolver / InterfaceDataContractResolver) that reproduces the legacy engine's runtime-config-driven behaviour (field-name inference, per-member property mappings, [DataMember]/ [InterfaceDataContract], ShouldSerialize conventions, explicit-interface and
  non-public members).
• ~150 formatters migrated from IJsonFormatter<T> to JsonConverter<T>, across simple values, dates/times (ISO 8601 + epoch/ticks), collections and dictionaries, inference types, unions, geo, analysis, queries (field-name, fuzzy, terms, range, geo, span), aggregations, properties/mappings, proxy requests, and responses. Open generics are wired via JsonConverterFactory.
• Behaviour parity work (the bulk of the branch): integral double/float/ decimal keep the legacy trailing .0; [StringEnum] values serialize with the verbatim field name; member- and type-level ShouldSerialize are honoured
  (empty bool queries and empty routing are omitted); [PropertyName] / OSC mapping-attribute / property-mapping-provider naming precedence; object-keyed and read-only dictionaries; ValueTuple; single-or-array coercions; document bodies (index/create/bulk/terms) delegate to the configured SourceSerializer; Error/ServerError deserialization.
• Default switch: ConnectionSettings now defaults to STJ. Set OSC_USE_UTF8JSON=true (or OSC_USE_STJ=false) to fall back to the legacy Utf8Json engine.
• Docs: CHANGELOG breaking-change + added entries; an UPGRADING section on the default switch and the fallback.
• CI: the unit test job now runs the full suite against both engines (STJ default + OSC_USE_UTF8JSON=true) via a serializer matrix, so neither path can silently regress.

The bundled Utf8Json library is intentionally retained — it is the implementation behind the OSC_USE_UTF8JSON fallback, which is a permanently supported escape hatch.

Compatibility

• A custom source serializer supplied via ConnectionSettings(..., sourceSerializer: ...) (e.g. the OpenSearch.Client.JsonNetSerializer package) is unaffected under either engine.
• Recommended path: use the STJ default; opt out only on a concrete incompatibility.

Testing

• Unit tests: the full suite passes on both engines. Both produce the same small set of pre-existing failures that are unrelated to serialization (naming-convention code standards, a role-detection sniffing test, and the JSON.NET low-level exception-parity test) — i.e. STJ is at parity with Utf8Json.

• Real-cluster integration (manual): index / get / search (match + term on .keyword) / bulk / update / average aggregation / delete-by-query round-trips against a 4-node OpenSearch cluster passed 16/16 on STJ and 16/16 on Utf8Json. (OpenSearch publishes no macOS distribution, so the repo's ephemeral integration harness cannot run natively on macOS; this was validated against a Docker cluster instead.)

• Benchmarks (BenchmarkDotNet, .NET 10, arm64): STJ is ~1.2–1.4× slower than Utf8Json with slightly higher allocations — expected, since Utf8Json was a hand-tuned library; absolute costs are microsecond-scale and dominated by network/cluster time in practice. Performance-sensitive callers can opt into the Utf8Json fallback.

  Benchmark	STJ	Utf8Json	ratio
  Serialize single doc	1080 ns	805 ns	1.34×
  Deserialize single doc	1284 ns	1079 ns	1.19×
  Serialize 100 docs	112 µs	78 µs	1.44×
  Deserialize 100 docs	143 µs	114 µs	1.25×

Check list

• New functionality includes testing
• New functionality has been documented (CHANGELOG, UPGRADING)
• Commits are signed per the DCO using --signoff