We ported 8.2-billion-parameter MisoTTS hierarchical RVQ transformer model to Apple's native MLX framework for local, real-time speech generation on macOS. This implementation achieves up to 6.7x speedups on Apple Silicon GPUs, bypasses memory bandwidth limits through 4-bit quantization, and addresses numerical precision differences between Metal and PyTorch CPU that can cause premature generation termination.

The PyTorch CPU Bottleneck on macOS

MisoTTS 8B is a hierarchical RVQ transformer that generates speech conditioned on both text and preceding audio context, allowing for context-aware voice synthesis. However, executing the PyTorch reference implementation on Apple Silicon forces the model onto the CPU:

[!WARNING]
The float64 GPU Limitation: The original PyTorch implementation uses double-precision float64 operations. Because the PyTorch Metal Performance Shaders (MPS) backend does not support float64 arithmetic, execution falls back entirely to the CPU.

Running an 8.2-billion-parameter model on the macOS CPU is computationally expensive. It causes thermal throttling, high fan noise, and high latency, producing only ~0.5 to 1.0 second of audio per second of processing time. This makes real-time voice synthesis and interactive dialogue impractical.

To resolve this bottleneck, we ported the model's 7.7B parameter Llama backbone, its 300M parameter autoregressive decoder, and its key-value (KV) caches to Apple's native MLX framework. Running the entire pipeline on the Unified GPU memory yields a 6.7x speedup, generating more than 20 frames of audio per second (1.6x real-time generation speed).

This post describes the implementation, the JIT compilation and quantization strategies used to reduce latency, and the debugging of a numerical precision divergence that caused premature sequence termination.

Architecture of MisoTTS

MisoTTS synthesizes speech in three stages:

Flowchart detailing a conversational audio generation architecture. Input text and a short audio prompt are tokenized into Llama 3.2 BPE and Mimi Codec tokens, then interleaved into a 7.7B parameter Temporal Backbone Transformer. The resulting frame representation is processed sequentially by a 300M parameter Autoregressive Audio Decoder, a 32x RVQ Codebook Predictor, and a Mimi Neural Decoder to reconstruct a 24kHz waveform, finally outputting a studio-quality WAV file.

  1. The Mimi Audio Codec: Compresses 24 kHz audio into 32 Residual Vector Quantization (RVQ) codebook channels at a frame rate of 12.5 Hz. Each frame represents 80ms of audio.
  2. The 7.7B Llama Backbone: Processes interleaved sequences of text and the first RVQ codebook level over time, modeling dialogue state and style context.
  3. The 300M Autoregressive Audio Decoder: Receives the hidden states from the backbone and executes 31 autoregressive micro-steps per frame to generate the remaining 31 higher-order RVQ codebooks.

Traced Inner-Decoder JIT Compilation

Compiling autoregressive sequence models often introduces JIT compilation overhead. As conversation history grows, input shapes to the 7.7B parameter backbone change at each outer generation step. Under compilers like MLX (@mx.compile) or PyTorch (torch.compile), these changing shapes trigger frequent recompilations.

However, the 31-step autoregressive loop within the audio decoder uses purely static shapes:

  • The decoder's input sizes remain fixed for every step of the outer generator loop.
  • The key-value (KV) attention caches for the decoder layers are cleared at the start of each frame and grow through the same 31 shapes.

Applying @mx.compile to the inner decoder loop (referencing the model via closure) compiles the 31-step sequence into a fused Metal GPU shader. This eliminates Python interpreter overhead for the 31 micro-steps per frame, executing them as a single operation on the GPU.

Performance Impact of Inner-Decoder Compilation

Benchmarks for a 10-second audio generation run (125 total frames) on an Apple Silicon GPU:

Metric Uncompiled MLX Baseline Compiled Inner Decoder Impact
Total Generation Time 318.98 seconds 169.97 seconds 1.87x speedup
Step Latency (After JIT) 2,531.5 ms / frame 1,246.7 ms / frame 2.03x speedup
First-Step JIT Compile 5.06 seconds 15.38 seconds One-time compilation cost
Real-Time Factor (RTF) 33.14 18.10 45.4% latency reduction
Peak Memory Footprint 12.41 GB 12.47 GB Negligible change (+0.06 GB)

The compilation overhead on the first step is offset after 10 steps, saving 149 seconds on a standard 10-second audio generation run.

Bypassing the Unified Memory Bandwidth Constraint via 4-Bit Quantization

Even with JIT compilation, unquantized model execution is limited by memory bandwidth. The unquantized MisoTTS 8B weights (model.safetensors) are approximately 16.38 GB. During generation, the GPU must stream these 16.38 billion parameters from Unified Memory into the GPU registers on each step.

  • On an Apple Silicon SoC with 100 GB/s memory bandwidth, streaming 16.38 GB requires at least 164 milliseconds per step, limiting synthesis throughput to 6 steps per second.
  • On an SoC with 150 GB/s memory bandwidth, weight streaming requires a minimum of 109 milliseconds per step.

To resolve this memory transfer limit, we applied 4-bit weight quantization using mlx.core.quantize to the linear projection and transformer layers. This reduces the weight size from 16.38 GB to 5.52 GB (a 2.97x reduction), lowering memory bandwidth pressure and yielding:

  1. Compilation Warmup Reduction: JIT compilation time on the first step drops from 6.28 seconds to 0.53 seconds (an 11.8x reduction).
  2. Throughput Increase: Step inference speed increases by 3.82x. This reduces the Real-Time Factor (RTF) below 1.0 on standard Apple Silicon configurations, enabling real-time local dialogue synthesis.

Numerical Divergence and Autoregressive Drift

During parity testing against the PyTorch reference implementation under strict deterministic argmax (greedy) sampling, the MLX port generated the first word correctly, but then emitted silence for the remainder of the audio. The script printed [EOS] reached and exited without crashing.

An activations trace comparing intermediate values and token logits step-by-step against the PyTorch CPU reference showed that MLX layer outputs matched the reference with a relative error under 1% ($< 0.01$).

The Step-2 Logit Divergence

An analysis of the raw output logits at Step 2 of generation revealed a narrow margin between the top content token (ID 1484) and the End-of-Sequence token (ID 0):

Token ID Token Role PyTorch CPU Logit MLX GPU (Metal) Logit
1484 Content Token 10.3750 (Selected) 10.3992
1021 Content Token 10.3125 10.3759
0 EOS (End-of-Sequence) 10.2500 10.4015 (Selected)

Because the logits differed by less than 0.02, floating-point precision differences between PyTorch on CPU and MLX on GPU (Metal) reordered the predicted outputs:

  • PyTorch CPU selected token 1484, continuing active speech synthesis.
  • MLX GPU selected token 0 (EOS), immediately halting generation.

Autoregressive Error Propagation

In autoregressive models, a single early EOS or silent token is fed back into the backbone as context for subsequent steps. The model then adapts its attention weights to this silence, causing it to continue predicting silence and terminating early.

Mitigation: Parameter Scheduling and Guidance

Deterministic argmax sampling is highly sensitive to minor numerical variances. Real-world speech synthesis is more robust when using probabilistic sampling with adjusted generation parameters:

  1. Dynamic Temperature Decay: Sampling starts at a higher temperature (temp-start 0.7) to allow vocal variation, and decays to temp-min 0.4 over the first 30 frames. This avoids early silent attractor states and limits high-temperature acoustic artifacts (such as sibilant hiss).
  2. Classifier-Free Guidance (CFG): Applying a CFG scale of 2.0 guides the model toward the text prompt, pushing content token logits ahead of the EOS token at critical boundaries.
  3. SilentCipher Bypass: A toggle bypasses the silent acoustic watermarking pass, eliminating low-frequency background hums in the generated audio stream.

Automated Quality Auditing

To replace subjective listening tests with reproducible metrics, we added two evaluation scripts:

  1. compare_audio.py: A mathematical utility that compares the MLX output to the PyTorch reference, computing peak amplitude, RMS energy, Log-Mel Spectrogram Mean Absolute Error (MAE), spectral cosine similarity, and temporal envelope correlation.
  2. audio_evaluator.py: An evaluation harness that streams generated audio bytes to Gemini 3.1 Flash Lite on Google Cloud AI to transcribe the speech, detect phonetic drift (word substitutions), grade acoustic clarity, and output a structured performance scorecard:
{
  "transcription": "Hello from local GPU. This is widely variable speech.",
  "expected_text": "Hello from local GPU! this is highly variable speech.",
  "acoustic_clarity_rating": "Clean, crisp, no robotic static or clipping.",
  "completeness": "100% complete, no trailing silent loops.",
  "phonetic_drift": "Substituted 'highly' with 'widely' (1/9 words deviated).",
  "alignment_score": 89,
  "overall_quality_score": 70
}

Command-Line Interface

The MLX implementation and tools are available in the repository. The command-line interface supports hardware diagnostics, weight conversion, loop compilation, and 4-bit quantization:

# Synthesize speech on GPU with 4-bit quantization and temperature scheduling
uv run python miso_mlx/miso_mlx_cli.py speak \
  --text "Hello! Ported to Apple Silicon Metal GPU with 4-bit quantization." \
  --mlx \
  --quant \
  --temp-start 0.7 \
  --temp-min 0.4 \
  --temp-decay-steps 30 \
  --cfg-scale 2.0 \
  --output outputs/hello_mlx.wav

Compiling inner-decoder loops, quantizing weights to 4-bit, and scheduling generation parameters allows this 8.2B parameter voice model to run locally on Apple Silicon GPUs without memory bandwidth exhaustion.

https://github.com/ghchinoy/misotts-mlx

User interface of MisoTTS Studio showing a completed text-to-speech generation. The left panel displays the input script, 'Speaker 24' selection, and model parameters set to 4-bit quantized. The right panel features a terminal output detailing MLX profiling breakdowns and memory usage, followed by a successful audio playback widget accompanied by Metal GPU inference telemetry showing metrics like Real-Time Factor, total latency, and peak RAM.