Consider supporting Traditional Chinese segmentation with rime/rime-essay

[frankslin]

cppjieba currently ships with Simplified Chinese dictionaries. It would be useful to also provide an official Traditional Chinese dictionary.

A good candidate is the data from https://github.com/rime/rime-essay. From a technical perspective, there doesn't appear to be any major obstacle -- the dictionary format can be converted to the format expected by cppjieba.

The main consideration is licensing. rime-essay is licensed under LGPL-3.0, so vendoring the dictionary into this repository may not be ideal. Instead, it might be preferable to include it as a Git submodule (or otherwise keep it as an external dependency), so that the code and dictionary remain clearly separated.

Would the maintainer of cppjieba be open to supporting Traditional Chinese segmentation in this way?

[frankslin]

Reference implementation: b685fa1

[yanyiwu]

Thanks for the proposal and reference implementation. I am open to supporting Traditional Chinese segmentation, but I would prefer to keep it optional and avoid vendoring the dictionary into cppjieba by default. I also do not want to add another submodule before the maintenance and distribution boundaries are clear.

A useful next step would be a small prototype PR that includes:

1. a reproducible conversion script from a pinned rime/rime-essay revision to cppjieba dictionary format;
2. clear upstream license, attribution, and redistribution documentation (without assuming that using a submodule alone resolves all LGPL obligations);
3. a Traditional Chinese test set with documented provenance and expected segmentation output;
4. instructions for users to obtain/build the optional dictionary without changing the existing default Simplified Chinese setup; and
5. an estimate of generated dictionary size and the expected update/maintenance process.

With those pieces we can compare a documented external dictionary workflow, an optional download/build step, and a submodule-based integration on concrete evidence. The default build and existing dictionary behavior should remain unchanged.