Voice & appearance¶

Custom TTS¶

Replace the default text-to-speech engine with a custom implementation. The TTS engine should stream audio in real-time.

Extend uni.TtsEngine and register via @uni.tts:

Custom TTS engine

@uni.tts
class MyCustomTts(uni.TtsEngine):

    def __init__(self):
        self._sample_rate = 24000
        self._bytes_per_sample = 2  # 16-bit PCM

    @property
    def sample_rate(self) -> int:
        """Sample rate in Hz."""
        return self._sample_rate

    @property
    def bytes_per_sample(self) -> int:
        """Bytes per sample (2 for 16-bit PCM)."""
        return self._bytes_per_sample

    def load(self) -> None:
        """Load resources needed by the TTS engine."""
        logger.info("Custom TTS loaded")

    def synthesize(self, text: str) -> Iterator[bytes]:
        """Generate and stream audio for the given text."""
        for chunk in generate_audio_chunks(text):
            yield chunk

    def shutdown(self) -> None:
        """Release resources held by the TTS engine."""
        pass

If your model produces float32 samples, convert them with uni.float32_to_pcm16.

Custom avatars¶

Avatars control UNI's visual representation. They can react to TTS audio levels (pulse/lip sync) and support expressions ("happy", "listening", "surprised").

Since avatars render client-side, they're built in JavaScript. The JS module exposes a mount(context) function. On the Python side, use @uni.avatar to register it.

PNG-based avatar example¶

plugins/my_png_avatar/__init__.py

from pathlib import Path

import uni_plugin_sdk as uni

images_dir = Path(__file__).parent / "static" / "images"

@uni.avatar(module="~/avatar.js")
def create_avatar() -> dict[str, Any]:
    return {
        "expressions": [f.stem for f in images_dir.glob("*.png")]
    }

The object returned from create_avatar is passed to your JavaScript module.

Automatic expressions

If expressions is present in the returned object, UNI triggers them automatically during interactions based on sentiment analysis. Use descriptive names.

plugins/my_png_avatar/static/avatar.js

/**
 * @typedef {Object} AvatarContext
 * @property {HTMLElement} container - The container element.
 * @property {Object} config - Config object from the server.
 * @property {{base: string, static: string}} paths - Plugin URL roots: base for routes and uploaded data, static for bundled assets.
 * @property {function(src:string):Promise<void>} loadScript - Load a static script.
 * @property {function(eventName: string, callback: Function): void} on - Register event listener.
 */

/**
 * @param {AvatarContext} context
 * @returns {Promise<Function|undefined>}
 */
export const mount = async (context) => {
  const getImagePath = (fileName) => {
    return `${context.paths.static}/expressions/${fileName}.png`;
  };

  const img = document.createElement("img");
  img.src = getImagePath("default");
  img.style.transition = "transform 0.2s ease-out";
  context.container.appendChild(img);

  context.on("expression", (name) => (img.src = getImagePath(name)));
  context.on("audio", (lv) => (img.style.transform = `scale(${1 + lv * 0.1})`));

  await Promise.all(
    context.config.expressions.map((name) => {
      return new Promise((resolve) => {
        const img = new Image();
        img.onload = img.onerror = resolve;
        img.src = getImagePath(name);
      });
    })
  );

  return () => img.remove();
};

Advanced example

Check out the included uni_avatar_live2d plugin for an example with user-provided assets and external libraries.

Expressions¶

Avatars can optionally support expressions (emotes). They're triggered automatically:

Expression	Trigger
`"default"`	Active by default (otherwise first one is used)
`"sleeping"`	Active during the sleep cycle
`"listening"`	Active while the user is speaking
Other names	Auto-fire while UNI speaks (sentiment-based)

Naming matters

Use descriptive expression names for sentiment analysis to work correctly.

Zoom and pan¶

Users can zoom (wheel or pinch), drag to pan, and double-click to reset the avatar view. The host tracks this per device and saves it locally. Subscribe to transform to apply it in your avatar:

context.on("transform", ({ scale, x, y, grabbing }) => {
  // scale multiplies your own fit. x and y are viewport fractions.
  // grabbing is true mid-gesture, so pause any pointer-driven motion.
});

Wake word detection¶

Provide custom wake word detection by extending uni.WakeWordEngine and registering via @uni.wake_word:

Custom wake word engine

@uni.wake_word
class MyWakeWordEngine(uni.WakeWordEngine):

    def load(self) -> None:
        config = uni.get_config()
        words = config.plugin.get("words", str, "").split(",")
        sensitivity = config.plugin.get("sensitivity", float, 0.5)
        self._detector = load_model(words, sensitivity)

    def detect(self, audio_chunk: bytes, sample_rate: int) -> bool:
        return self._detector.detected(audio_chunk, sample_rate)

    def shutdown(self) -> None:
        self._detector.close()