Local speech recognition from Unity audio.

Overview

VoxScribe is for offline or local speech workflows that start with microphone input or an existing Unity AudioClip. The source converts samples into a mono representation and invokes native whisper-style functions through P/Invoke.

The recognizer has OnPartialText and OnFinalText events for Unity integration, while the included sample registers listeners and starts microphone listening. The shown recognizer implementation does not currently invoke those events itself.

Requirements

• Target Unity baseline: 2021.3 LTS and newer.
• A native whisper library is required for the native path. The source declares whisper imports and comments that the target bundle provides the library.
• WhisperNative.IsNativeAvailable returns true only for Windows, macOS and Linux editor/player runtime platforms in this source.
• Microphone capture needs platform microphone permission. Planned site platform messaging is broader than this native availability check.
• Mono/IL2CPP builds need native plugin import settings and native binding validation; this prototype does not include a link.xml or stripping configuration.

Setup

1. Copy the plugin folder into Assets/Denisitree/VoxScribe/.
2. Place the required native library under the project's Unity native plugin location appropriate to the target.
3. Place the model file at the path assigned to VoxScribeRecognizer.modelPath; the source default is Assets/Models/ggml-base.en.bin.
4. Add VoxScribeRecognizer to a GameObject, assign the model and language fields, and register listeners for the declared UnityEvents.

Quick start

This is the complete listener and lifecycle shape from VoxScribeDemo.

recognizer = GetComponent<VoxScribeRecognizer>();
recognizer.OnPartialText.AddListener(text => Debug.Log("Partial: " + text));
recognizer.OnFinalText.AddListener(text => Debug.Log("Final: " + text));
recognizer.StartListening();

1. Add the recognizer component.
2. Register listeners.
3. Call StartListening.
4. Call StopListening during teardown.

Core concepts

Native boundary

WhisperNative declares native functions for model initialization, inference, segment text and cleanup. CreateContext validates a path and wraps the returned pointer in WhisperContext.

Audio normalization

TranscribeAsync reads interleaved clip samples and passes them through AudioResampler.ToMono16KHz. The implementation averages channel values but does not use sampleRate to resample time; treat the method as the source's current channel-folding helper.

Inference thread

Full inference is wrapped in Task.Run, and the result is returned as a string. MainThreadDispatcher provides a queued callback mechanism, but the recognizer source does not call it.

Microphone windows

MicrophoneStreamer owns a queue, clears it on Start and Stop, and returns fixed-size windows only when enough samples are present. The current source does not add microphone samples to that queue.

API reference

VoxScribeRecognizer

public string modelPath = "Assets/Models/ggml-base.en.bin";
public string language = "auto";
public bool translateToEnglish;
public UnityEvent<string> OnPartialText;
public UnityEvent<string> OnFinalText;
public void StartListening()
public void StopListening()
public async Task<string> TranscribeAsync(AudioClip clip)

WhisperNative and WhisperContext

public struct WhisperContext { public IntPtr handle; }
static bool IsNativeAvailable()
WhisperContext CreateContext(string modelPath)
string RunFullInference(WhisperContext context, float[] samples, string language, bool translateToEnglish)

AudioResampler

static float[] ToMono16KHz(float[] samples, int channels, int sampleRate)

MicrophoneStreamer

void Start()
void Stop()
float[] CaptureWindow()

MainThreadDispatcher

static void Post(Action action)
static void Update()

VoxScribeDemo

This public sample class has no public fields, properties, methods or events. Its private Start method registers listeners and starts listening; its private OnDestroy method stops listening.

Recipes

Live subtitles shape

Use the sample's event listener pattern for UI text. The current recognizer declares the events, but the source does not invoke them, so actual live partial subtitles are planned rather than available.

Transcribe an AudioClip

Call TranscribeAsync(clip) from an async method and use the returned string after native availability and model initialization succeed.

Control microphone lifecycle

Call StartListening once the component is configured, then call StopListening on teardown, matching the sample's OnDestroy path.

Post a Unity callback

Queue an Action with MainThreadDispatcher.Post and call Update from a Unity update path. The recognizer does not wire this helper automatically.

Performance and platform notes

Full inference runs inside Task.Run, while native model creation and pointer handling remain explicit. No benchmark or memory claim is made by the source.

Native support is currently guarded to Windows, macOS and Linux editor/player values. Android and iOS microphone workflows require additional native validation and are not implied by the current binding check.

Troubleshooting / FAQ

Planned

• Working partial and final event emission from the streaming path.
• Population of the microphone sample queue and complete streaming window feed.
• Native packaging and validation for additional platforms.
• Build stripping guidance such as [Preserve] or link.xml, if needed after validation.