On-device prompt generation and chat history.

Overview

PocketMind is for local dialogue, game companion prompts and context-aware NPC experiments. PocketMindChat owns the Inspector-facing model, prompt and generation fields, while ChatHistory and ChatTemplate prepare the native request.

The source declares an OnToken event, but the current generation method invokes only OnCompleted after the full output is returned. Token-by-token delivery is therefore planned, not an available behavior.

Requirements

• Target Unity baseline: 2021.3 LTS and newer.
• A native llama library is required. The source declares imports for llama and comments that the native files live under the Unity native plugin folder.
• LlamaNative.IsNativeAvailable returns true for Windows, macOS and Linux editor/player platforms only.
• The source loads Assets/StreamingAssets/PocketMind/phi-3-mini.gguf inside Generate, regardless of the public modelPath field value.
• Mono/IL2CPP native import and stripping settings need target validation. No link.xml or preservation attribute appears in the source.

Setup

1. Copy the plugin folder into Assets/Denisitree/PocketMind/.
2. Provide the native llama library and place the GGUF model at the StreamingAssets path used by LlamaNative.Generate.
3. Add PocketMindChat to a GameObject and configure modelPath, systemPrompt, temperature, maxTokens and contextSize.
4. Register OnCompleted for completed responses. Keep OnToken marked as a declared but currently unwired event.

Quick start

The included sample gets the component and sends one prompt.

chat = GetComponent<PocketMindChat>();
chat.SendAsync("What is the next objective in this scene?");

1. Add PocketMindChat to the same GameObject.
2. Assign the system prompt and generation fields.
3. Call SendAsync.
4. Observe the completed output through OnCompleted.

Core concepts

Prompt history

SendAsync appends a user message, trims history with TrimToBudget, formats the messages with ChatTemplate.FormatLlama3 and sends the formatted string to native generation.

Context budget

ChatHistory.TrimToBudget returns all messages when the requested budget is non-positive; otherwise it keeps the last twelve messages according to the current source. The contextBudget value is not used to calculate token length.

Native generation

LlamaNative.Generate loads the fixed GGUF path, creates a context with four threads, loops to maxTokens, appends native token text and frees the context.

Completion events

The source invokes OnCompleted with the final output. OnToken is declared but not invoked by the current loop.

Editor model manager

ModelManagerWindow.Open opens an editor window that lists GGUF files in StreamingAssets and contains an Add by URL button path. The source example uses a placeholder URL; this documentation does not treat it as a real distribution link.

API reference

PocketMindChat

public string modelPath = "Assets/StreamingAssets/PocketMind/phi-3-mini.gguf";
public string systemPrompt = "You are a helpful game companion.";
public float temperature = 0.8f;
public int maxTokens = 256;
public int contextSize = 2048;
public UnityEvent<string> OnToken;
public UnityEvent<string> OnCompleted;
Task SendAsync(string prompt)
void Cancel()

LlamaNative

static bool IsNativeAvailable()
string Generate(string prompt, float temperature, int maxTokens)
void Cancel()

ChatTemplate and ChatMessage

static string FormatChatML(IReadOnlyList<ChatMessage> messages, string systemPrompt)
static string FormatLlama3(IReadOnlyList<ChatMessage> messages, string systemPrompt)
public string role;
public string text;

ChatHistory

IReadOnlyList<ChatMessage> Messages { get; }
void AppendUserMessage(string text)
void AppendAssistantMessage(string text)
IReadOnlyList<ChatMessage> TrimToBudget(int contextBudget)

ModelManagerWindow

static void Open()

VoiceAssistantDemo

This public sample class has no public fields, properties, methods or events. Its private Start method gets PocketMindChat and sends one prompt.

Recipes

Request a completed NPC reply

Get PocketMindChat, call SendAsync with a prompt, and listen to OnCompleted. This is the current implemented event path.

Format a ChatML prompt

Call ChatTemplate.FormatChatML directly with ChatMessage values when the ChatML output shape is useful outside the default Llama 3 path.

Cancel native work

Call Cancel on PocketMindChat or LlamaNative to free the current context. The source does not expose a cooperative cancellation token.

Stream a reply token by token

This is not implemented in the current source. OnToken exists, but the native loop appends token text internally and emits only OnCompleted; token events belong under Planned.

Performance and platform notes

Generation is placed inside Task.Run, but native model loading and context creation still consume memory and native resources. The code publishes no throughput or memory benchmark.

The availability check covers Windows, macOS and Linux editor/player values. Test GGUF loading, native imports and IL2CPP stripping on every target build.

Troubleshooting / FAQ

Planned

• Invoke OnToken during generation for actual token streaming.
• Honor the public modelPath field instead of the fixed native path.
• Validate broader platform packaging and native model lifecycle behavior.