robonix.service.speech

service package v0.1.0

Voice interaction service (ASR + TTS + Dialog) under robonix/service/speech.

View source Open README on GitHub JSON
README robonix@dev

speech_service

Robonix speech service for robonix/service/speech/*. It exposes ASR, streaming ASR, TTS, streaming TTS, dialog state, and speaker helper capabilities.

Backends:

  • local (default): Whisper one-shot ASR, sherpa-onnx int8 Paraformer streaming ASR, Edge TTS.
  • tencent: Tencent Cloud real-time ASR WebSocket and TextToVoice TTS API.
  • custom: load ASR/TTS backend classes from module:Class environment variables without changing Liaison or speech capability contracts.

Local Edge TTS requires edge-tts>=7.3.1 for the current upstream handshake. Rebuild Speech to update an existing package environment. If synthesis returns HTTP 403, check its installed version before retrying; older versions may use the rejected handshake. See the upstream fix.

Tencent Cloud mode:

  1. Open the Tencent Cloud ASR console and activate Automatic Speech Recognition.
  2. Open CAM / Access Management -> Access Key -> API Key Management and create an API key. Save SecretKey immediately; Tencent Cloud only shows it when the key is created. The same page/account information provides the AppID.
  3. Select the backend in the deployment manifest so rbnx build also knows that no local model or GPU dependencies are needed:
env:
  SPEECH_BACKEND: tencent

service:
  - name: speech
    path: ${ROBONIX_SOURCE_PATH}/services/speech
    config:
      speech_backend: tencent
      tencent_asr_appid: "1234567890"
      tencent_asr_engine: 16k_zh
      tencent_tts_voice_type: 502003
      tencent_tts_region: ap-guangzhou
  1. Export only the credentials on the operator machine before rbnx boot:
export TENCENTCLOUD_SECRET_ID=...
export TENCENTCLOUD_SECRET_KEY=...

Do not commit Tencent credentials. Put them in the operator shell, a local ignored env file, or a machine-local boot wrapper. The AppID and non-secret backend settings belong in the deployment manifest. Useful optional knobs: TENCENT_ASR_ENGINE (default 16k_zh), TENCENT_TTS_VOICE_TYPE (default 1001), TENCENT_TTS_REGION (default ap-guangzhou), TENCENT_TTS_MAX_CHARS (default 140), and TENCENT_TTS_MAX_TOTAL_CHARS (default 5000). The per-request setting accepts 1–150 characters, using Tencent's Chinese limit as a conservative ceiling for mixed text. Long input is split at sentence punctuation, commas or whitespace, and finally a hard boundary, with its PCM segments returned in order. The service rejects input above the configured total limit, input requiring more than 40 provider requests, and separator runs that cannot be sent without a punctuation-only request. The Speech contract requires TENCENT_TTS_CODEC=pcm; encoded formats such as WAV or MP3 are rejected because their independently wrapped segments cannot be exposed as one pcm_s16le response. One-shot synthesis and speech/speak also stop below 4 MiB of decoded PCM so the result fits the core client's default gRPC receive limit; use the streaming TTS capability for larger output.

With SPEECH_BACKEND=tencent, the build installs cloud client, audio adaptation and wake-word dependencies. It does not install or warm Whisper, Torch, CUDA, or Edge TTS. The local backend keeps those dependencies in the local optional dependency set and preloads its models during build.

The TTS contract returns 16 kHz mono pcm_s16le bytes in both local and Tencent modes so existing audio speaker primitives can play the response directly.

Local streaming ASR

The build downloads the bilingual int8 Paraformer model into rbnx-build/models/sherpa-onnx-streaming-paraformer-bilingual-zh-en. Streaming inference defaults to CPU without loading torch. SHERPA_ASR_DIR selects a different model directory; SHERPA_ASR_ENCODER, SHERPA_ASR_DECODER and SHERPA_ASR_TOKENS override individual files. SHERPA_ASR_THREADS defaults to 2.

SHERPA_ASR_PROVIDER selects the sherpa-onnx execution provider (default: cpu). To request CUDA, install a GPU-enabled sherpa-onnx 1.13.4 build compatible with the board, driver and model into this package's venv after rbnx build, then export SHERPA_ASR_PROVIDER=cuda before starting the service. Reapply the GPU runtime installation after rebuilding: the build script installs the default CPU wheel set. Installing onnxruntime-gpu alone does not replace the native runtime linked by sherpa-onnx.

Provider strings are passed directly to sherpa-onnx, without board-specific selection. Its build and model determine which providers work. Logs show the requested provider; unsupported providers or operators may fall back to CPU. The bundled int8 Paraformer can execute partly on CPU even when CUDA is selected, so profile the target board before choosing it. This setting does not change Whisper, wake-word detection or TTS.

Wake word

Wake-word recognition is part of the Speech package, not Liaison. The default phrase is the bundled Mandarin wake phrase (YAML \u7f57\u4f2f\u7279); it is selected because the bundled KWS model recognizes it from a 16 kHz Tencent TTS test stream. A deployment can replace it without changing code:

service:
  - name: speech
    config:
      wake_words: ["\u7f57\u4f2f\u7279"]
      wake_word_boost: 2.0
      wake_word_threshold: 0.45

SPEECH_WAKE_WORDS is the environment equivalent (comma-separated). The Speech build downloads the KWS model; runtime does not download model weights.

Adding a new ASR/TTS algorithm

Liaison depends only on the speech service capabilities:

  • robonix/service/speech/asr
  • robonix/service/speech/asr_stream
  • robonix/service/speech/tts
  • robonix/service/speech/tts_stream

Algorithm code stays inside this package. To add a new backend, create a new file such as services/speech/speech_service/my_asr.py and implement the backend method shape you need:

from collections.abc import Iterable, Iterator


class MyASRBackend:
    def recognize(self, audio_bytes: bytes, encoding: str, sample_rate: int, language: str) -> dict:
        return {"text": "...", "confidence": 0.9}

    def recognize_stream(self, pcm_chunks: Iterable[bytes]) -> Iterator[dict]:
        for chunk in pcm_chunks:
            # Forward chunk to your local model, cloud API, or gRPC stub.
            yield {
                "event_type": 0,
                "text": "...",
                "confidence": 0.9,
                "is_final": False,
            }


class MyTTSBackend:
    async def synthesize(self, text: str, voice: str = "", speed: float = 1.0) -> bytes:
        return b""  # 16 kHz mono pcm_s16le

    async def synthesize_stream(self, text: str, voice: str = "", speed: float = 1.0):
        yield await self.synthesize(text, voice, speed)

Then select it at boot:

export SPEECH_BACKEND=custom
export SPEECH_ASR_BACKEND_CLASS=speech_service.my_asr:MyASRBackend
export SPEECH_ASR_STREAM_BACKEND_CLASS=speech_service.my_asr:MyASRBackend
export SPEECH_TTS_BACKEND_CLASS=speech_service.my_tts:MyTTSBackend

The backend receives already-adapted ASR audio as 16 kHz mono pcm_s16le. Streaming ASR yields event dictionaries; one-shot ASR returns a dictionary with text and confidence. TTS must return or yield 16 kHz mono pcm_s16le bytes.

The backend may call any implementation internally: a local model, a cloud API, or an existing gRPC stub. Do not change Liaison for algorithm swaps.

Streaming configuration and events

Manifest sherpa_asr_dir, sherpa_asr_threads (default 2), and sherpa_asr_provider (default cpu) override SHERPA_ASR_DIR, SHERPA_ASR_THREADS, and SHERPA_ASR_PROVIDER. The build installs the same sherpa-onnx 1.13.4 wheel family as Voiceprint. GPU providers require a compatible runtime; the requested provider alone does not prove GPU execution.

Streaming partial events carry text deltas. The final event carries the full transcript and replaces all earlier partials, including recognition revisions. Custom streaming backends must follow this rule. Upgrade Liaison and Speech together; older Liaison versions append final text and can duplicate words.

Model downloads honor RBNX_GH_MIRROR (empty selects GitHub directly). KWS and ASR release archives have pinned SHA256 checksums; only required runtime files are extracted into a staging directory. A failed transfer, checksum or extraction leaves the previous model directory intact. Verified runtime files are checked again before reuse. With ROBONIX_MODEL_CACHE_DIR set, verified archives are cached under its speech/ directory by SHA256, checked again on each cache hit, and published atomically under a per-digest lock. A digest-keyed mirror request and KWS's official GitHub release asset API provide alternate downloads. KWS also tries gh-proxy.org after the official endpoints fail; every response must match the pinned archive checksum. Checksum failures report both expected and actual SHA256 values. ASR retains the int8 encoder and decoder; unused FP32 graphs are not extracted.

At end of input, Paraformer receives lookahead padding before final decoding, following the upstream streaming example. This flushes words near the end of a recording; it does not wait for additional microphone audio.