HANDY LOCAL DICTATION — REPLICATION SPECIFICATION
Give this document to your coding assistant.

GOAL
Build this on my computer: local English and Greek dictation in any focused application, with toggle recording, hold-to-talk, paste-only and paste-plus-Enter. English uses NVIDIA GPU acceleration when available and automatically falls back to the CPU. Greek uses a separate resident local CPU model. Normal dictation needs no server, cloud speech API, LLM or subscription.

Inspect my OS, RAM, GPU/driver, microphone and shortcut conflicts first. The implementation described below uses Linux/X11. Adapt keyboard capture, paste, startup services and IPC for Windows/macOS; do not pretend Linux-specific code works unchanged there. CUDA acceleration requires NVIDIA hardware. Keep CPU operation available on other machines.

ARCHITECTURE
Handy: https://github.com/cjpais/Handy
The reference custom implementation is based on Handy v0.9.8 with backend extensions. These features are not all stock Handy settings. Use a suitable upstream version and port the behavior explicitly.

Flow:
shortcut -> Handy microphone capture/VAD/resampling -> mono16kHz float32 audio -> selected worker -> transcript -> Handy history/postprocessing -> focused application paste -> optional Enter.

Handy retains ownership of capture, settings, history and paste. Python workers only recognize audio. Use separate workers for English GPU and Greek CPU. Retain the complete captured audio for error recovery.

1. EXACT MODELS AND DOWNLOADS

A. ENGLISH GPU: Parakeet TDT0.6B v3, FP16 ONNX
Base model: https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3
Export: https://huggingface.co/visuall/parakeet-tdt-0.6b-v3-onnx-fp16
Pinned revision: 125d44237abd9a53d291a3104a563fc0ba104ecb
Download these four files into one model directory:
encoder-model.fp16.onnx
https://huggingface.co/visuall/parakeet-tdt-0.6b-v3-onnx-fp16/resolve/125d44237abd9a53d291a3104a563fc0ba104ecb/encoder-model.fp16.onnx

decoder_joint-model.fp16.onnx
https://huggingface.co/visuall/parakeet-tdt-0.6b-v3-onnx-fp16/resolve/125d44237abd9a53d291a3104a563fc0ba104ecb/decoder_joint-model.fp16.onnx

nemo128.onnx
https://huggingface.co/visuall/parakeet-tdt-0.6b-v3-onnx-fp16/resolve/125d44237abd9a53d291a3104a563fc0ba104ecb/nemo128.onnx

vocab.txt
https://huggingface.co/visuall/parakeet-tdt-0.6b-v3-onnx-fp16/resolve/125d44237abd9a53d291a3104a563fc0ba104ecb/vocab.txt

Create config.json locally; this export does NOT supply it:
{"model_type":"nemo-conformer-tdt","features_size":128,"subsampling_factor":8}

Use this FP16 model for GPU inference. It is not the English CPU fallback model.

B. ENGLISH CPU FALLBACK: Handy Parakeet v3 INT8
Exact archive used by Handy:
https://blob.handy.computer/parakeet-v3-int8.tar.gz
Archive SHA256:
43d37191602727524a7d8c6da0eef11c4ba24320f5b4730f1a2497befc2efa77

Prefer Handy's normal model manager to download and install it. Model ID: parakeet-tdt-0.6b-v3. Model directory: parakeet-tdt-0.6b-v3-int8.
Required contents:
- encoder-model.int8.onnx
- decoder_joint-model.int8.onnx
- nemo128.onnx
- vocab.txt
- config.json

The CPU fallback uses Handy's existing native Parakeet/transcribe-rs implementation. Keep its CPU backend configured and usable even while external GPU acceleration is enabled. Verify the archive checksum before extraction.

C. GREEK CPU: specialist Parakeet TDT0.6B v3 Greek, FP32
https://huggingface.co/myned-ai/parakeet-tdt-0.6b-v3-greek
Pinned revision: c314e68a3935a5bb8e398681790ff13efb99432f
Download these files into a separate directory:
config.json
https://huggingface.co/myned-ai/parakeet-tdt-0.6b-v3-greek/resolve/c314e68a3935a5bb8e398681790ff13efb99432f/config.json

vocab.txt
https://huggingface.co/myned-ai/parakeet-tdt-0.6b-v3-greek/resolve/c314e68a3935a5bb8e398681790ff13efb99432f/vocab.txt

encoder-model.onnx
https://huggingface.co/myned-ai/parakeet-tdt-0.6b-v3-greek/resolve/c314e68a3935a5bb8e398681790ff13efb99432f/encoder-model.onnx

encoder-model.onnx.data
https://huggingface.co/myned-ai/parakeet-tdt-0.6b-v3-greek/resolve/c314e68a3935a5bb8e398681790ff13efb99432f/encoder-model.onnx.data

decoder_joint-model.onnx
https://huggingface.co/myned-ai/parakeet-tdt-0.6b-v3-greek/resolve/c314e68a3935a5bb8e398681790ff13efb99432f/decoder_joint-model.onnx

nemo128.onnx
https://huggingface.co/myned-ai/parakeet-tdt-0.6b-v3-greek/resolve/c314e68a3935a5bb8e398681790ff13efb99432f/nemo128.onnx

LICENSE-nemo128.txt
https://huggingface.co/myned-ai/parakeet-tdt-0.6b-v3-greek/resolve/c314e68a3935a5bb8e398681790ff13efb99432f/LICENSE-nemo128.txt

Keep encoder-model.onnx.data alongside encoder-model.onnx: the graph depends on those external weights. Do not substitute the similarly named INT8 files for this FP32 configuration.

The specialist adds Greek vocabulary support, including final sigma (ς). It is used for Greek, not English. Preserve its CC BY4.0 attribution and the preprocessor's license notice. The .nemo training-framework checkpoint is not needed for this ONNX deployment.

D. HANDY VOICE ACTIVITY DETECTOR
https://blob.handy.computer/silero_vad_v4.onnx
Use the version expected by the chosen Handy build. The reference development resource path is src-tauri/resources/models/silero_vad_v4.onnx.

2. PYTHON AND NVIDIA RUNTIME

Reference Python runtime:3.10, in a private virtual environment.
Relevant installed package versions:
- numpy==2.2.6
- onnx-asr==0.11.0
- onnxruntime-gpu==1.20.2
- nvidia-cublas-cu12==12.2.5.6
- nvidia-cuda-runtime-cu12==12.2.140
- nvidia-cuda-nvrtc-cu12==12.2.140
- nvidia-cudnn-cu12==9.1.0.70
- nvidia-cufft-cu12==11.0.8.103

Example installation for a compatible NVIDIA/Linux machine:
python3 -m venv ~/.local/share/handy-gpu/venv
~/.local/share/handy-gpu/venv/bin/python -m pip install numpy==2.2.6 onnx-asr==0.11.0 onnxruntime-gpu==1.20.2 nvidia-cublas-cu12==12.2.5.6 nvidia-cuda-runtime-cu12==12.2.140 nvidia-cuda-nvrtc-cu12==12.2.140 nvidia-cudnn-cu12==9.1.0.70 nvidia-cufft-cu12==11.0.8.103

Adjust installation to platform/driver compatibility. On a CPU-only installation, use onnxruntime rather than onnxruntime-gpu. Do not install both packages into the same environment. CPUExecutionProvider is also available inside the GPU runtime, so the reference Greek worker shares that environment without using CUDA.

For Linux, expose the environment's nvidia/{cublas,cuda_runtime,cuda_nvrtc,cudnn,cufft}/lib directories through the worker's LD_LIBRARY_PATH. Calculate the actual site-packages path; do not hard-code somebody else's username or Python version. The machine still needs a compatible NVIDIA driver.

3. ENGLISH GPU SESSION SETTINGS

Use onnx_asr.models.nemo.NemoConformerTdt with a custom encoder method implementing the long-recording algorithm below. Load:
config -> config.json
vocab -> vocab.txt
encoder -> encoder-model.fp16.onnx
decoder_joint -> decoder_joint-model.fp16.onnx

Run nemo128.onnx preprocessing on CPU, with one intra-op and one inter-op thread. It takes waveforms(float32,[batch,samples]) and waveforms_lens(int64,[batch]) and returns features and feature lengths. Use batch size1 for this worker.

Encoder/decoder ONNX Runtime SessionOptions:
intra_op_num_threads=4
inter_op_num_threads=1

CUDAExecutionProvider options:
gpu_mem_limit=str(1792*1024**2)
arena_extend_strategy="kSameAsRequested"
cudnn_conv_algo_search="HEURISTIC"
cudnn_conv_use_max_workspace="0"
cudnn_conv1d_pad_to_nc1d="1"

The last setting is important for this export/runtime combination. The memory limit governs the CUDA allocator, not all process/GPU memory. Leave headroom for the desktop and other applications.

Provider list: CUDAExecutionProvider with those options, followed by CPUExecutionProvider for operations the CUDA backend does not execute. Require CUDAExecutionProvider to remain present on BOTH encoder and decoder sessions and call session.disable_fallback(). This disables runtime provider fallback; it does not mean every individual graph operation executes on CUDA.

For each encoder run, use RunOptions with:
memory.enable_memory_arena_shrinkage="gpu:0"

Keep a recognizer resident between requests. The reference worker releases its model after300 idle seconds, checking idle state every30 seconds. A preload option warms it with one second of zero audio. Model loading on the next request adds latency after unloading.

4. EXACT LONG-RECORDING GPU ALGORITHM

Work on acoustic features, not on separately recognized text. Do not require a pause to split: Handy's VAD can already have removed silence.

Preprocess the FULL audio once. Features have128 mel bins and a10ms frame step. The encoder subsamples by8, giving one output frame per80ms.

Let F be the valid feature length, as an integer.
Let N=(F+7)//8 be the expected valid encoder-frame count.

Short input: if F<=3000, make one intact encoder call.
Long input: plan windows in ENCODER-FRAME units:
- owned core:200 frames =16 seconds
- left context:125 frames =10 seconds
- right context:25 frames =2 seconds
- maximum encoder window:350 frames =28 seconds

For each core_start in range(0,N,200):
  core_end=min(N,core_start+200)
  window_start=max(0,core_start-125)
  window_end=min(N,core_end+25)
  feature_start=window_start*8
  feature_end=min(F,window_end*8)
  window_features=features[:,:,feature_start:feature_end]
  window_length=feature_end-feature_start
  encoded=encode(window_features,int64([window_length]))
  owned_start=core_start-window_start
  owned_end=core_end-window_start
  retain encoded[:,owned_start:owned_end,:]

Here encoded is arranged as [batch,time,hidden]. The raw encoder returns [batch,hidden,time], so transpose it first.

Concatenate ONLY the retained owned frames, in order. Require the concatenated length to equal N. Then invoke ONE continuous TDT decoding pass over that assembled encoder sequence. Keep its recurrent prediction state across all frames; do not reset it at window boundaries.

Why overlap works: the encoder gets neighboring speech as context, while deterministic frame ownership prevents duplicate output. Keeping one decoder state preserves continuity. There is no fuzzy text matching, no removal of repeated words, and no concatenation of independently decoded transcripts.

FP16 LENGTH CORRECTION — REQUIRED
The export can round its encoded_lengths arithmetic through FP16. Do not trust that value alone. Recompute the valid length for EVERY encoder call in integer arithmetic:
  expected=(feature_length-1)//8+1

Read encoder outputs with names "outputs" and "encoded_lengths". Validate:
- batch size is1;
- expected>0;
- expected does not exceed the raw output time dimension;
- encoder values are finite;
- each owned slice is within the validated valid length;
- assembled output contains exactly N frames.

Use the integer expected value as the decoder's valid length. If these checks fail, reject the GPU request and retry the full original recording on the CPU. Never clamp away genuine speech to make dimensions fit and never invent frames.

Implement this by subclassing NemoConformerTdt and overriding _encode; let recognize_batch perform the normal single decoding pass afterward. This relies on internal onnx-asr interfaces, which is why the package version is pinned.

This windowing design is for the English/general Parakeet export. The Greek specialist below uses whole-recording CPU inference; do not automatically reuse this encoder-window method for that fine-tune.

5. AUTOMATIC ENGLISH CPU FALLBACK

In Handy's Parakeet transcription branch, when active_model is parakeet-tdt-0.6b-v3:
- Try the optional external GPU worker.
- If it returns a valid success response, use that transcript.
- If the socket is unconfigured/unavailable, initialization fails, the request times out, CUDA fails, memory allocation fails or the worker returns an error, invoke Handy's existing CPU Parakeet recognizer on the SAME complete captured audio.

No manual model selection should be needed when there is no compatible GPU. Do not attempt to run the CUDA FP16 export as the CPU fallback; use the separate native INT8 model above.

The worker returns either the complete transcript or an error. It must not return partial text on failure. Keep English CPU fallback inside Handy so it continues working if the Python GPU service stops.

6. GREEK RESIDENT CPU WORKER

Load with:
onnx_asr.load_model("nemo-conformer-tdt",greek_model_directory,
  providers=["CPUExecutionProvider"],sess_options=options)

SessionOptions:
intra_op_num_threads=2
inter_op_num_threads=1
enable_cpu_mem_arena=False
graph_optimization_level=ORT_DISABLE_ALL

Set CUDA_VISIBLE_DEVICES to an empty value for this service. Require the encoder and decoder to report CPUExecutionProvider only.

For each request:
text=model.recognize(audio_float32,sample_rate=16000)
text=text.replace("?",";").strip()

Keep the model loaded; do not load it afresh for each recording. Budget approximately3GB RAM. The reference service requires4GiB available RAM before initial loading and has a6GiB service memory ceiling. Adapt those guards to the target OS.

Greek recognition errors must be reported, not silently routed to English or a remote server. Keep the full recording available for retry. This fine-tune's accuracy/completeness is not guaranteed for arbitrary speech; selecting it is a model choice, not a guarantee of complete output.

7. EXACT LOCAL WORKER PROTOCOL

Unix stream sockets on Linux, in private0700 same-user directories. Socket files should be user-only. Verify ownership, reject symlinks, check connecting peer credentials and prevent duplicate workers.

Request:
-4 bytes: unsigned big-endian JSON header length.
-UTF8 JSON header.
-Audio bytes: samples*4 bytes, little-endian float32, mono16kHz.

Example English header:
{"version":1,"sample_rate":16000,"samples":160000,"model":"parakeet-tdt-0.6b-v3"}
Greek wire model identifier: "parakeet-greek".

Response:
-4 bytes: unsigned big-endian JSON response length.
-UTF8 JSON: {"ok":true,"text":"..."}
-Or: {"ok":false,"error":"..."}
Greek may additionally return compute_seconds.

Use exact-read loops: one socket read is not guaranteed to return a whole message. Reject nonfinite audio and invalid sizes.
Reference bounds: header<=4096 bytes, response<=2MiB, English audio<=20minutes, Greek audio<=3minutes. These are implementation limits, not suggested silent truncation points. Reject oversized requests explicitly.
Native socket timeouts:90s read,10s write. Adapt timeouts coherently for slower CPUs; avoid launching overlapping retries on a worker still processing an earlier request.

Launcher variables:
HANDY_GPU_SOCKET=$XDG_RUNTIME_DIR/handy-gpu/worker.sock
HANDY_GREEK_SOCKET=$XDG_RUNTIME_DIR/handy-greek-local/worker.sock
HANDY_CTRL_SUPER_PTT=1
HANDY_GREEK_PTT=1
HANDY_SPOKEN_SEND=1

These are CUSTOM extension variables, not stock Handy features. Implement the matching native code. The reference native socket bridge is Linux-specific; use an equivalent private IPC transport elsewhere.

8. KEYBOARD AND OUTPUT BEHAVIOR

Super means Windows key on Linux. Adapt to conflicts/platform conventions.
Ctrl+Space: toggle English start/stop; paste.
Ctrl+Super: hold English; release to paste+Enter.
Alt+Super: hold English; release to paste only.
Ctrl+Super+G: hold Greek CPU; release to paste+Enter.
Alt+Super+G: hold Greek CPU; release to paste only.

Track a recording's owner separately from its language route. G may arrive after Ctrl+Super has started recording: update the selected route without stopping/restarting capture. Latch language and submit behavior for that session. Releasing the chord stops exactly once; modifier release order must not start a second recording. Intercept the registered G chord so it does not type into the focused app.

Optional spoken-send suffix: recognize trailing "submit now" or "submitnow", remove the suffix and request Enter. Apply only to fresh dictation; an empty command-only transcript must not submit existing application text.

An optional Ctrl+Super+S route can select a separate server Greek CPU worker. It is not required for local operation. If implemented, use a separate socket variable HANDY_GREEK_SERVER_SOCKET and native route parakeet-greek-server, while serializing wire model parakeet-greek. Keep G strictly local, S explicitly remote, and do not silently fall back between them.

9. RESPONSIVE GUI AND CORRECT PASTE

Keep capture work, inference, clipboard reads and sleeps off the GUI thread. The popup must not steal focus. Serialize clipboard transactions. Guard delayed completion using a recording-session counter so an old paste cannot hide a newer recording's popup.

Reuse the upstream paste-worker approach:
https://github.com/cjpais/Handy/pull/2302
Upstream merge:5a9451e70aedc90648ce2befd9d7e0304746f9c1
It uses a dedicated paste thread, a paste mutex, main-thread marshaling for Enigo calls where required, and session-aware cleanup. Preserve these platform requirements when adding custom paste+Enter wrappers.

Linux/X11 uses xdotool; the reference paste chord is Ctrl+Shift+V. Make paste configurable because many nonterminal applications use Ctrl+V and macOS uses Command+V. Handle held modifiers explicitly so Enter is not sent with Ctrl/Super accidentally held.

Choose clipboard handling explicitly. Keeping the transcript on the clipboard avoids restoring stale content before the target reads it. If restoring the previous clipboard is required, a short fixed sleep is not proof that the target has received the new text. A successful input-injection command confirms only that the keystroke was sent, not that the application consumed the clipboard.

Reference timing values are150ms before paste,60ms after the paste command and50ms before auto-Enter. These are timing settings, not a transfer acknowledgement or a correctness guarantee. Do not rely on the60ms value to make old-clipboard restoration reliable.

Do not send Enter when recognition fails or the transcript is empty. Keep automatic Enter tied to the selected send action, not all application output.

10. INSTALLATION, STARTUP AND BUILD

Keep custom binaries and workers separate from the system Handy package. Suggested layout:
~/.local/share/handy-gpu/handy
~/.local/share/handy-gpu/venv/
~/.local/share/handy-gpu/model/
~/.local/share/handy-gpu/worker.py
~/.local/share/handy-gpu/segments.py
~/.local/share/handy-greek/local_worker.py
~/.local/share/handy-greek/model/

On Linux, run supervised systemd user services for the English GPU worker and Greek CPU worker. Use Restart=on-failure, a restart delay, UMask=0077 and duplicate-instance protection. Start them with the user's graphical session. Make GPU preload failure nonfatal so native CPU fallback remains available.

Preserve Handy's single-instance behavior. Remote-control invocations must signal the existing GUI instance rather than launching multiple microphones or workers.

Build the frontend with Bun and the native backend with Rust/Tauri. A normal production command is bun run tauri build. For a custom cargo build with an already-built frontend, include --features tauri/custom-protocol; otherwise the GUI can still expect a development server.

Use application-specific scaling and overlay positioning only if the target desktop needs it. Do not copy somebody else's DPI scaling or screen coordinates as universal settings.

DELIVERABLE
Install a usable local app with the above routes and shortcuts, automatic English CPU fallback, startup services, version-controlled source, rollback instructions and a short user guide. Preserve model/license notices and keep recordings local. Explain clearly which behavior required custom Handy patches. Use this specification to implement the system; it is not a prebuilt installer.