text-prep-lite

Lightweight text preprocessing utilities for Natural Language Processing (NLP) written in TypeScript.

text-prep-lite provides two core helpers:

• normalizeText – clean & normalise raw text into a predictable representation.
• tokenize – break text into lowercase word tokens.

The library is intentionally dependency-free and suitable for browsers, Node.js, and serverless environments.

---

Why?

Natural-language data is messy. Before tokenisation or feeding text into an NLP model you often need to:

• normalise case & whitespace
• expand contractions ("can't" → "cannot")
• strip punctuation / emojis

text-prep-lite does those common steps with zero runtime dependencies.

---

Installation

npm install text-prep-lite
# or
yarn add text-prep-lite

---

Usage

import { normalizeText, tokenize } from "text-prep-lite";

const raw = "  I can't believe it's not butter! 🧈  ";

const cleaned = normalizeText(raw, {
  expandContractions: true,
  removePunctuation: true,
  removeEmojis: true,
});
// → "i cannot believe it is not butter"

const tokens = tokenize(raw);
// → ["i", "can", "t", "believe", "it", "s", "not", "butter"]

---

API

normalizeText(input: string, options?: NormalizeOptions): string

Returns a cleaned version of input.

NormalizeOptions:

Option	Default	Description
expandContractions	false	Expand contractions for the selected locale.
removePunctuation	false	Strip punctuation characters.
removeEmojis	false	Remove Unicode emoji characters.
locale	'en'	BCP-47 language tag for locale-specific rules (currently: en, sq, fr, de, he).

Supported locales

• en – English (default)
• sq – Albanian
• fr – French
• de – German
• he – Hebrew
• es – Spanish
• zh – Chinese (Mandarin)
• yue – Chinese (Cantonese)

// French example
normalizeText("C'est incroyable!", { expandContractions: true, locale: "fr" });
// → "ce est incroyable!"  (punctuation kept in this call)

---

tokenize(input: string): string[]

1. Converts text to lowercase.
2. Removes punctuation & emojis.
3. Splits by whitespace / word boundaries.

Returns an array of tokens.

tokenize has no options – it always lowercases, strips punctuation & emojis, and splits on whitespace.

---

🔗 Related

1. 👉 Need word embeddings for semantic analysis?
   Check out wink-embeddings-small-en-50d

2. 👉 Need a simple and robust PDF text extraction utility with a quality interface? Check out [pdf-worker-package]https://www.npmjs.com/package/pdf-worker-package

---

Development

# run tests
npm test

# build library
npm run build

---

License

MIT © Cavani21/thegreatbey

---

Contributing

1. Fork & clone the repo
2. npm i
3. npm test – run lint & unit tests
4. Submit pull-request 🚀

Please add tests for any new feature or bug-fix.