Speech to Sign Language

The short version

Most assistive speech tools stop at transcription. A deaf user who receives a Telugu sentence as text still has to translate it into a visual language themselves.

This project goes one step further:

1. Listens to Telugu speech from a microphone or an audio file.
2. Recognises the words with Telugu ASR (te-IN).
3. Translates the Telugu text into Indian Sign Language glosses using a curated dictionary, an English pivot, and fuzzy stem matching.
4. Animates each gloss with ordered frames from an ISL corpus.
5. Falls back gracefully to a text card or alphabet spelling when a sign is missing.

The result is a desktop application that shows the recognised Telugu, the ISL gloss sequence, and the sign animation side by side.

The honest architectural summary: the runtime is production-shaped, the model branch is research-shaped. The default app does not pretend that a synthetic or unlabeled model is a trained Telugu recogniser.

Why this is harder than transcription

Three constraints shape the whole design.

Telugu ASR is only the first step. Speech recognition gives text in Telugu script. ISL is a separate visual language with its own grammar — both Telugu and ISL prefer subject-object-verb order, which helps, but ISL also uses non-manual features, classifiers and spatial grammar that a word-level system cannot represent.

A gloss is not a translation of a word in isolation. The mapping has to survive inflection, compound words, and vocabulary the dictionary has never seen. Dropping an unknown word is worse than showing it untranslated, because it silently changes the meaning of the sentence.

A missing sign must never hide the meaning. If the frame corpus has no entry for a gloss, the application shows the untranslated word as a readable text card. That fallback is a feature, not a crash path.

The pipeline

Translation layers

The shipped mapping contains 214 Telugu entries. Everything outside it depends on the English pivot and the corpus vocabulary. The fallback path is visible in the UI, so coverage gaps are observable rather than hidden.

The model: Res-BiLSTM-AM

The acoustic branch follows the paper's architecture.

Each frame is described by 143 features:

Sequences are padded or truncated to 200 frames. Training uses Adam (1e-3), sparse categorical cross-entropy, early stopping, best-checkpoint saving, and learning-rate reduction on plateau. The paper's evaluation split is reproduced as 75% train / 25% test.

The architecture is deliberately small enough to train on a single machine. The residual connection around the attention block and the residual BiLSTM block both address the same failure: a deep sequence model that stops learning once the signal has to pass through too many transformations.

The split that makes it usable

The repository ships two paths, and keeping them separate is the main product decision.

The GUI does not need TensorFlow. That keeps installation light and means the app still runs on a laptop that cannot train a model. The acoustic branch is implemented and trainable in parallel for experiments and for a future fully learned recognition path.

This is also why the repository separates requirements.txt, requirements-train.txt and requirements-dev.txt. The runtime dependency set is not hostage to the training stack.

Honest handling of the model

• No pretrained checkpoint is shipped. --mode classify is only meaningful after training on labelled audio.
• Synthetic data is a smoke test, not a result. If fewer than ten audio files are present, training falls back to synthetic data so the loop can be exercised. Those numbers are never reported as model quality.
• The paper's metrics are not reproduced. The paper reports ESTOI 97.2%, PCC 96.1% and similar figures; this repository does not claim them. The README says so explicitly.

That last point matters more than it might seem. An academic capstone that claims the paper's numbers without reproducing the evaluation is not a stronger project; it is a less trustworthy one.

Engineering for failure

The application is designed to be useful before the ideal data exists.

• No corpus? It still starts. The right panel shows a text card for every sign instead of a blank animation.
• Unknown word? It is shown, not dropped. The TEXT_ marker keeps the original Telugu visible.
• Offline ASR? Direct dictionary translation and animation playback still work; only recognition and the English pivot need the network.
• Corpus licensing. The frame corpus is not redistributed. The app expects a bring-your-own corpus under data/ISL_Corpus/Frames_Word_Level/, and the README tells the user to check the licence of anything they add.
• Inspectable runtime. python app.py --mode info prints the loaded mappings, indexed signs, corpus frame count and ASR backend, so a support conversation starts with facts.
• Tests are offline. The pytest suite uses temporary frame directories, so it does not need the real corpus or network access. The model test is skipped automatically when TensorFlow is absent.
• Cross-platform entry points. scripts/run_app.bat and scripts/run_app.sh remove the "which Python?" step for a demo.

Verification

pytest -q

A representative run:

$ python app.py --translate "నమస్కారం ధన్యవాదాలు"

  నమస్కారం   → hello_hi   [✓ direct]
  ధన్యవాదాలు → thank      [✓ direct]

Gloss: hello_hi → thank
Animation: 0 ready, 2 text fallback(s)

The last line is the design in miniature: the translation succeeded, the corpus is incomplete, and the application says both things plainly.

Limitations

These are stated in the README because a case study that hides them is marketing, not engineering.

• Word-level, not continuous. The output is a sequence of glosses with frame playback, not a continuous signing avatar.
• Dictionary coverage. 214 entries cover common phrases; everything else leans on the English pivot.
• Simplified grammar. Telugu and ISL both prefer SOV order, so token order is preserved. The runtime does not yet model ISL non-manual features, classifiers or spatial grammar.
• Online dependencies. Google ASR and the English pivot require internet access.
• Corpus licensing. The ISL frame corpus is not bundled.
• No pretrained model. The Res-BiLSTM-AM branch needs labelled data and a training run before classification is meaningful.

What I would do next

• Add a larger, licensed ISL frame corpus and expand the Telugu mapping beyond 214 entries.
• Wire the trained Res-BiLSTM-AM classifier into the GUI as a selectable recognition backend.
• Add ISL grammar transformations: time-first ordering, copula dropping, question markers.
• Add non-manual feature overlays — facial expression and head movement — for grammatical negation and questions.
• Build an evaluation harness for WER, gloss accuracy and playback coverage, so future model work has a baseline.
• Export sequences as video/GIF and add a web playback surface.
• Add a continuous-signing path rather than word-level concatenation.

What this project demonstrates

Three things beyond the model itself:

1. Applied sequence modelling. A paper architecture translated into a trainable Keras implementation, with the feature pipeline, padding strategy, training loop and checkpointing that make it reproducible.
2. Product-shaped fallbacks. The system is usable with a partial corpus, an unknown word, no network, or no GPU. Each of those states has a designed response rather than an exception.
3. Honest evaluation. The runtime and the research branch are clearly separated, synthetic results are labelled as smoke tests, and the paper's metrics are not claimed. That restraint is part of the engineering.