<<<<<<< Updated upstream
=======
Stashed changes
AirController is a hands-free PC interaction utility that enables users to control mouse navigation, simulate keystrokes, trigger keyboard shortcuts, and dictate text without physical contact or network dependencies. It integrates real-time hand-gesture tracking using Google MediaPipe Hand Landmarker and OpenCV with offline voice recognition via the Vosk speech API.
- Real-Time Smoothing: Employs a OneEuroFilter (OEF) algorithm with customizable cutoff/beta parameters to filter tracking jitter and deliver responsive cursor coordinates.
- Smart Magnet Precision: Leverages a target-attraction logic (MAGNET_STRENGTH, MAGNET_RADIUS) that pulls the cursor toward active elements during slow movements, assisting in precise object selection.
- Offline Dictation & Shortcut Execution: Utilizes a dedicated background thread running the Vosk offline model for localized transcription and execution of mapped system macros.
- HUD Visualizer: A customizable debug display showing landmarks, tracking states (PAUSED, NAVIGATION, INPUT), dragging boundaries, and voice feedback messages.
The gesture engine maps the following verified triggers in Navigation Mode:
- Pointer: Cursor movement. Moves the mouse cursor relative to the normalized coordinate delta of the hand landmarks, scaling speed between MOUSE_SPEED_NORMAL and MOUSE_SPEED_FAST depending on speed threshold limits.
- TwoFingers: Left Mouse Click. Triggered once the two-finger (V-sign) configuration is confirmed.
- Pinch: Split action based on duration. If the pinch (index + thumb) duration exceeds PINCH_THRESHOLD, it starts a drag-and-drop mode. If released under the threshold, it triggers a Right Mouse Click.
- OpenHand: Absolute cursor tracking with Drag-and-Hold active. Relieves dragging when the hand state changes.
- Fist: Scroll navigation. Hand movements in the Y-axis control vertical scrolling, while movements in the X-axis trigger horizontal scrolling.
- Swipe (Left/Right): Alt-Tab. Triggers task switching when swipe velocity exceeds SWIPE_THRESHOLD.
- Fists Zoom: Dual fists movement. Spreading hands zooms in; bringing hands together zooms out.
- Thumbs Up: Triggers mode switch to INPUT mode, activating the microphone.
- Fist (in Input Mode): Triggers mode switch back to NAVIGATION, muting the microphone.
Active only in INPUT Mode:
- "enter", "submit", "go", "next line": Simulates the Enter keypress.
- "tab", "next field", "move on": Simulates the Tab keypress.
- "delete", "back", "backspace": Triggers Backspace (5 times).
- "clear", "clear all", "erase": Clears current input text (Ctrl + A -> Delete).
- "copy" / "paste": Executes standard Ctrl + C and Ctrl + V shortcuts.
- "zoom in", "enlarge", "bigger": Browser zoom in command.
- "zoom out", "smaller", "shrink": Browser zoom out command.
- "scroll up", "go up", "page up": Scrolls viewport up.
- "scroll down", "go down", "page down": Scrolls viewport down.
- "escape", "cancel", "close": Simulates the Escape keypress.
- "system off", "exit", "quit": Shuts down the background process loop.
- Any other phrase: Transcribes the spoken words directly as text input.
Adjust parameters inside config.py or settings.json:
- MOUSE_SPEED_NORMAL / MOUSE_SPEED_FAST: Mouse cursor speed scaling.
- PINCH_THRESHOLD: Proximity click-drag boundary limits.
- OEF_MIN_CUTOFF / OEF_BETA / OEF_D_CUTOFF: OneEuroFilter parameters.
- GESTURE_CONFIRM_FRAMES: Minimum consecutive frames required to execute a gesture.
- VOSK_MODEL_PATH: Folder path containing the local Vosk acoustic model.
- HAND_MODEL_PATH: Folder path pointing to hand_landmarker.task.
- Install Python 3.8+
- Install dependencies:
pip install -r requirements.txt
- Place
hand_landmarker.taskand the extractedvosk-model-small-en-us-0.15model in the root directory.
Run the script to start the interface:
python main.pyTo enable the camera capture and HUD display window, verify that debug_mode is enabled in settings.json.