Update documentation for tokenizer: add detailed usage examples and s…#560

yash2002vardhan wants to merge 1 commit intogoogle-deepmind:maingoogle-deepmind/gemma:mainfrom

yash2002vardhan:doc/add-tokenizer-documentationyash2002vardhan/gemma:doc/add-tokenizer-documentationCopy head branch name to clipboard