Sign Language Interpreter

A desktop-based sign language interpreter built with Python, MediaPipe, TensorFlow, OpenCV, and Tkinter.

The app captures hand signs through a webcam, predicts the sign using a trained neural network, and builds a sentence in real time.

Features

• Real-time webcam-based sign detection
• Desktop UI built with Tkinter
• Hand landmark extraction using MediaPipe
• Trained classifier for letters and common words
• Sentence builder with add space, delete, clear, and copy actions
• Improved prediction stability using confidence thresholds and short-term smoothing

Supported Signs

The current model is trained on:

• Letters: A-Z
• Words: hello, thanks, yes, no, please, sorry, help, iloveyou

Tech Stack

• Python 3.11
• MediaPipe
• TensorFlow
• OpenCV
• NumPy
• scikit-learn
• Pillow
• Tkinter

Project Structure

sign-language-interpreter/
├── app/
│   ├── collect_data.py
│   ├── train_model.py
│   └── interpreter.py
├── data/
├── model/
│   ├── classes.npy
│   └── sign_model.keras
├── requirements.txt
└── README.md

Setup

1. Clone the repository

git clone https://github.com/Shreyaaaaaak/Sign-Language-interpreter.git
cd Sign-Language-interpreter

2. Create and activate a virtual environment

On Windows:

python -m venv .venv
.venv\Scripts\activate

3. Install dependencies

pip install -r requirements.txt

Run the App

From the project root:

cd app
python interpreter.py

Then:

1. Click Start Camera
2. Show a supported hand sign to the webcam
3. Hold the sign steady until it is detected
4. Build a sentence using the recognized signs

Collect Your Own Data

If you want to collect your own training dataset:

cd app
python collect_data.py

How it works:

• Opens the webcam
• Collects hand landmark samples for each sign
• Stores .npy files inside the data/ folder

Train the Model

To train the classifier again using your own collected data:

cd app
python train_model.py

This saves:

• model/classes.npy
• model/sign_model.keras

Notes

• The current project is designed for Windows and was tested with Python 3.11.
• Good lighting and a clear hand pose improve detection accuracy.
• If the webcam does not open, close other apps that may be using the camera.
• If you retrain the model, run the interpreter again after training completes.

Future Improvements

• Add text-to-speech output
• Show top 3 predictions in the UI
• Improve landmark normalization for better accuracy
• Add more words and dynamic signs
• Support exporting recognized sentences

Author

Shreya Kaushik