Ecosystem
VoxScribe -> PocketMind -> Vocalis.
This guide explains the intended offline voice assistant shape using the actual VoiceAssistantDemo source and the adjacent current APIs. SonicLoad sits beside the voice path for file and buffer import/export.
Pipeline at a glance
1. VoxScribe
Capture and prepare audio, then expose the recognizer path and text events.
2. PocketMind
Send a prompt to local generation and receive the implemented completed response.
3. Vocalis
Turn response text into an AudioClip and play it through an AudioSource.
Side path: SonicLoad
Load files or bytes, inspect clips and export WAV data around the voice workflow.
The real VoiceAssistantDemo path
The source sample in Assets/Denisitree/PocketMind/Samples/VoiceAssistantDemo.cs contains two active operations:
chat = GetComponent<PocketMindChat>();
chat.SendAsync("What is the next objective in this scene?");That sample does not call VoxScribe or Vocalis directly. The complete pipeline below is therefore an integration shape built from the real public APIs, not a claim that one existing sample already wires all three plugins together.
Step by step
1. Prepare audio input with VoxScribe
Add VoxScribeRecognizer, configure modelPath and language, then call StartListening. For an existing clip, call TranscribeAsync and await the returned string.
2. Send recognized text to PocketMind
Pass the recognized text to PocketMindChat.SendAsync. The class appends a user message, formats Llama 3 markers and calls native generation. Listen to OnCompleted for the implemented response path.
3. Speak the completed response with Vocalis
Give the completed response to VocalisSpeaker.SpeakAsync, with target set to an AudioSource. The speaker checks native availability, synthesizes and plays the returned clip.
4. Use SonicLoad around the edges
Use AudioLoader for file, byte or URL clip input and WavEncoder for the implemented WAV export path. This is useful for saved prompts, recorded buffers or debug artifacts, but it is not called by the provided VoiceAssistantDemo.
Integration shape
string transcript = await recognizer.TranscribeAsync(clip);
await chat.SendAsync(transcript);
// Handle chat.OnCompleted, then call speaker.SpeakAsync(response).Ownership and threading
- VoxScribe wraps full native inference in
Task.Runand has a separate queuedMainThreadDispatcherhelper. - PocketMind wraps native generation in
Task.Runand invokesOnCompletedafter it returns. - Vocalis wraps native synthesis in
Task.Runand creates an AudioClip after the result returns. - SonicLoad uses asynchronous file/network APIs and a background task for WAV decoding.
- Unity object access, event listeners and AudioSource playback still need a Unity-safe lifecycle in the integrating project.
What is implemented vs planned
| Area | Implemented in source | Not implemented or not wired |
|---|---|---|
| Speech input | AudioClip conversion path and recognizer lifecycle methods. | Recognizer event emission and queue population are not shown. |
| Local reasoning | Completed generation and OnCompleted event. | OnToken is declared but not invoked. |
| Speech output | Native synthesis to AudioClip and AudioSource playback. | Validated cross-platform packaging is not present. |
| Audio files | WAV decode and 16-bit PCM WAV encode. | MP3, FLAC and OGG export are planned in the SonicLoad sample. |
Troubleshooting / FAQ
No. VoxScribe, Vocalis and PocketMind each declare native bindings for their core paths. SonicLoad does not declare a native library in the inspected source.
The provided VoiceAssistantDemo is a PocketMind-only sample. The end-to-end sequence here is an integration guide based on the separate public APIs.
The VoxScribe events are declared and listeners can be registered, but the current recognizer code does not invoke them.
PocketMind appends native token text internally and only invokes OnCompleted. OnToken streaming is planned.
Vocalis requires native availability and a non-null AudioSource target. Check the generated clip and target assignment.
Planned
- A single sample scene that wires microphone input, recognized text, local response completion and speech playback.
- VoxScribe partial/final event emission and microphone queue feeding.
- PocketMind token event emission.
- Validated native library packaging across target platforms.
Something unclear?