Local speech synthesis into Unity AudioClip output.

Overview

Vocalis is for local NPC dialogue, narration and accessibility paths where text should become an AudioClip without a cloud request. VocalisSpeaker owns the inspector-facing voice and playback fields, while NativeTtsEngine calls the native synthesis functions.

The source also includes VoiceCatalog for JSON voice definitions and PcmUtility for clip conversion and a simple speed operation. Native synthesis is the implemented path; voice catalog loading and utility methods are separate APIs.

Requirements

• Target Unity baseline: 2021.3 LTS and newer.
• A native TTS library is required. The source imports vocalis_tts and comments that platform libraries are placed in the Unity native plugin folder.
• NativeTtsEngine.IsNativeAvailable returns true for Windows, macOS and Linux editor/player platforms only.
• Voice model data is read from Assets/StreamingAssets/Vocalis/model.onnx by the native engine.
• Mono/IL2CPP native import settings and stripping behavior require target validation. The source does not include a link.xml or preservation attribute.

Setup

1. Copy the plugin folder into Assets/Denisitree/Vocalis/.
2. Provide the native library and the model path expected by NativeTtsEngine.Synthesize.
3. Add VocalisSpeaker to a GameObject and assign target, voiceId, speed and pitch.
4. For voice definitions, place JSON files under a StreamingAssets subfolder and call VoiceCatalog.Load with its root path.

Quick start

The sample constructs a speaker with Inspector-assigned playback and explicit voice settings.

var speaker = new VocalisSpeaker { target = target, voiceId = "en-US-voice-01", speed = 1f, pitch = 0f };
await speaker.SpeakAsync(prompt);

1. Assign an AudioSource to target.
2. Choose a voice ID.
3. Set speed and pitch.
4. Await SpeakAsync.

Core concepts

Native synthesis

NativeTtsEngine initializes the ONNX model path, calls native synthesis with text, voice ID, speed and pitch, copies the output buffer and frees the native context.

AudioClip conversion

VocalisSpeaker.SynthesizeAsync runs native synthesis in Task.Run, then calls PcmUtility.ToAudioClip. SpeakAsync assigns and plays the result only when both clip and target are non-null.

Voice catalog

VoiceCatalog.Load combines Application.streamingAssetsPath with a root path, reads JSON files and deserializes VoiceDefinition values containing id, name and language.

Speed helper

PcmUtility.ApplySpeed allocates a new array based on speed and linearly interpolates neighboring samples. It is not called by VocalisSpeaker in the current source.

API reference

VocalisSpeaker

public string voiceId = "en-US-voice-01";
public float speed = 1f;
public float pitch = 0f;
public AudioSource target;
Task<AudioClip> SynthesizeAsync(string text)
Task SpeakAsync(string text)

ITtsEngine and NativeTtsEngine

void Initialize(string modelPath)
(float[] samples, int sampleRate) Synthesize(string text, string voiceId, float speed, float pitch)
IReadOnlyList<string> GetVoices()
static bool IsNativeAvailable()
(float[] samples, int sampleRate) Synthesize(string text, string voiceId, float speed, float pitch)

VoiceCatalog and VoiceDefinition

public string id;
public string name;
public string language;
IReadOnlyList<VoiceDefinition> Load(string rootPath)

PcmUtility

static AudioClip ToAudioClip(float[] samples, int sampleRate, string clipName)
static float[] ApplySpeed(float[] samples, float speed)

VocalisDemo

The public sample class exposes two public fields:

Recipes

Speak with Inspector values

Use VocalisSpeaker with an assigned AudioSource, then call SpeakAsync. The sample's default voice, speed and pitch are explicit in its object initializer.

Change voice speed at runtime

Set the public speed field before calling SynthesizeAsync or SpeakAsync. The value is passed to the native engine.

Load voice definitions

Instantiate VoiceCatalog and call Load(rootPath). The returned list contains only successfully deserialized JSON definitions.

Apply the PCM speed helper

Call PcmUtility.ApplySpeed directly when you own a float sample buffer. This helper is available but not wired into the speaker path.

Performance and platform notes

Native synthesis is placed inside Task.Run; clip creation and AudioSource playback happen after the task returns. The source provides no benchmark or memory guarantee.

Native availability is limited to the desktop platforms listed in the source. StreamingAssets paths and native library import settings need target-specific validation, especially for IL2CPP.

Troubleshooting / FAQ

Planned

• Validated native packaging and target-platform coverage.
• Wiring ApplySpeed into the speaker path, if that becomes part of the API contract.
• Expanded voice catalog validation and runtime voice selection behavior.