smdesai/Nemotron-3.5-ASR-Streaming-Multilingual-0.6b

Nemotron 3.5 ASR Streaming Multilingual 0.6B

This repo is an Apple CoreML and Core AI port of NVIDIA's nemotron-3.5-asr-streaming-0.6b. The base model is a 600M-parameter cache-aware FastConformer encoder with an RNN-T decoder and a language-ID prompt. It is built for streaming, on-device speech recognition on iPhone, iPad and Mac. The encoder runs on the Neural Engine.

• Four streaming latency tiers (560 / 1120 / 2240 / 4480 ms). Each tier is a self-contained bundle with both a CoreML build (iOS 18+ / macOS 15+) and a Core AI build (iOS 27+ / macOS 27+).
• INT8 encoder (weight-only, per-channel symmetric). About 570 MB, down from about 1.2 GB in fp16. It is split into a small pre-encode stage plus four 6-layer shards, so it compiles for and runs on the Neural Engine on iOS as well as macOS.
• Full 13,087-token vocabulary with no pruning. One bundle covers every language the base model supports. You select the language per stream with prompt_id.
• Numerically faithful to the base model. On FLEURS, WER/CER falls within the noise of NVIDIA's fp32 NeMo reference (see Accuracy).

Weights were converted and quantized after training. There was no retraining, fine-tuning or calibration data. The conversion used the current upstream .nemo checkpoint (SHA-256 210214ed…5436a74).

Repository layout

multilingual/<tier>ms/                       # tier ∈ {560, 1120, 2240, 4480}; ~1.4 GB each
├── metadata.json                            # shapes, prompt_dictionary, lang-tag ids, blank id
├── tokenizer.json                           # id → SentencePiece piece
├── coreml/                                  # CoreML — iOS 18+ / macOS 15+   (~706 MB)
│   ├── preprocessor.mlmodelc                # audio → 128-bin log-mel (per chunk)
│   ├── encoder_pre_encode.mlmodelc          # INT8 3× strided-conv subsampling (4.4 MB)
│   ├── encoder_shard_{0,1,2}.mlmodelc       # INT8, 6 conformer layers each (140 MB each)
│   ├── encoder_shard_3.mlmodelc             # INT8, last 6 layers + prompt kernel (144 MB)
│   ├── decoder.mlmodelc                     # RNN-T prediction net, 2× LSTM 640 (29 MB)
│   ├── decoder_joint_argmax.mlmodelc        # fused decoder + joint + argmax, one call per step (47 MB)
│   ├── joint_noencproj_batched.mlmodelc     # smart-spec: 8-frame-window joint, no encoder projection (17 MB)
│   └── native_weights/                      # fp16 encoder-projection weights for the host
│       ├── weights.bin                      #   (47 MB)
│       └── weights_index.json
└── coreai/                                  # Core AI — iOS 27+ / macOS 27+   (~682 MB)
    ├── encoder_shard_{0..3}_int8.aimodel    # INT8 weights / fp16 activations, 6 layers each
    ├── decoder_joint.aimodel                # fused decoder+joint over all frames of a chunk (93 MB)
    └── mel_filterbank.f32, mel_window.f32   # host-side log-mel front-end constants

The Apple Watch CoreML bundle (2240 ms only) is staged separately under models/watch/2240ms/ (--watch / --watch-only in convert_models.sh). It has the split encoder, preprocessor, decoder and decoder_joint_argmax, plus metadata.json and tokenizer.json. It has no joint_noencproj_batched or native_weights, because those expand to about 98 MB of fp32 in memory and caused ANE timeouts on Apple Watch Ultra 3.

Use either coreml/ or coreai/. metadata.json and tokenizer.json serve both.

Which tier?

Tier	Mel frames / chunk	Encoder frames / chunk	New audio per step	Notes
560 ms	56	7	0.56 s	lowest latency; half of a trained 14-frame attention chunk
1120 ms	112	14	1.12 s	one trained attention chunk per call
2240 ms ⭐	224	28	2.24 s	default; best ANE efficiency
4480 ms	448	56	4.48 s	long-form / offline

All tiers use att_context_size = [42, 13]: 42 frames of left attention context, 13 frames of right context (80 ms per encoder frame). Every tier prepends a 9-frame pre-encode mel cache from the previous chunk.

Accuracy vs. the base model

The CoreML bundle was run end-to-end (CoreML preprocessor → encoder → greedy RNN-T), with the encoder on .cpuAndNeuralEngine. Results were compared against NVIDIA NeMo fp32 cache-aware streaming inference with the same attention context [42, 13], the same checkpoint, the same audio and the same scoring.

Test set: FLEURS test, the first 100 distinct sentences per language (sorted by FLEURS id, one recording per sentence). That is 10 languages, 1,000 utterances and 3.6 h of audio. The language was forced through prompt_id; es_419 used the es-US prompt. Scoring: NFKC, lowercase, Unicode punctuation and symbols removed, language tags removed. WER is used for all languages except CER for ja/zh.

2240 ms tier, CoreML (INT8 encoder)

Language	Metric	NeMo fp32 [42,13]	This model (INT8, ANE)	Δ	Identical transcripts¹
English (en_us)	WER	10.53	10.75	+0.21	61 %
German (de_de)	WER	11.08	11.25	+0.18	52 %
Spanish (es_419)	WER	7.82	8.52	+0.69	67 %
French (fr_fr)	WER	12.89	12.82	−0.07	49 %
Italian (it_it)	WER	6.91	6.91	0.00	71 %
Portuguese (pt_br)	WER	10.64	10.72	+0.08	73 %
Russian (ru_ru)	WER	14.48	14.86	+0.39	42 %
Hindi (hi_in)	WER	15.50	15.74	+0.23	56 %
Japanese (ja_jp)	CER	14.53	14.27	−0.25	37 %
Chinese (cmn_hans_cn)	CER	19.62	19.47	−0.15	37 %
Pooled WER (8 langs)		11.12	11.33	+0.21 (95 % CI −0.04…+0.45)
Pooled CER (ja, zh)		16.74	16.54	−0.21 (95 % CI −0.81…+0.31)

¹ This is the share of utterances where the normalized CoreML transcript is character-for-character identical to NeMo fp32. The CIs come from a paired bootstrap with 2,000 resamples, stratified by language. Neither pooled difference is statistically distinguishable from zero.

Where the small residual difference comes from. Three controlled runs on the same 1,000 utterances show it:

Variant	Pooled WER	Pooled CER	Divergence from NeMo fp32²
NeMo fp32 [42,13] (reference)	11.12	16.74	0
CoreML fp16 encoder (unquantized)	11.33	16.80	1.6–5.3 %
CoreML INT8 encoder, single-file	11.33	16.54	1.9–5.2 %
INT8 encoder fed NeMo's full-utterance mel instead of the per-chunk CoreML preprocessor	11.35	16.69	1.8–5.3 %
CoreML INT8 sharded encoder (pre-encode + 4 shards; this repo)	11.33	16.54	0.00 % vs. single-file (1000/1000 identical)

² Per-language WER/CER of the CoreML transcript scored against the NeMo fp32 transcript.

• INT8 quantization costs nothing measurable. The fp16 and INT8 encoders drift from fp32 by the same amount.
• The per-chunk CoreML mel front-end is not a source of error. Swapping in NeMo's own full-utterance features does not change the result.
• The drift that remains comes from fp16 activations on the Neural Engine versus fp32 PyTorch. Greedy RNN-T decoding occasionally flips a near-tie token, and the flip moves WER in both directions.
• Sharding changes nothing. The shipped sharded encoder produces exactly the same transcripts as a single-file encoder built from the same weights.

Absolute WERs here are higher than those on NVIDIA's model card (e.g. en 7.91 % at 1.12 s) for three reasons: this simple normalizer does not canonicalize numbers or spelling variants, the test set is a 100-sentence subset, and the left context is [42, 13] rather than the upstream [56, 13]. Read the tables as base vs. port under identical conditions, not as leaderboard numbers.

About the [42, 13] attention context. The checkpoint was trained with a left context of 56 frames ([56, {0, 3, 6, 13}]). This port uses 42, which shrinks the attention cache by 25%. To measure the cost of that choice, NeMo fp32 was also run at the upstream [56, 13] on the same 1,000 utterances:

NeMo fp32	Pooled WER	Pooled CER
[56, 13] (upstream, 1.12 s)	11.00	16.81
[42, 13] (this port's context)	11.12	16.74
Δ (95 % CI)	+0.13 (−0.07…+0.37)	−0.07 (−0.32…+0.19)

The reduced left context has no statistically significant accuracy cost.

Usage

Runtime contract (CoreML)

Model	Inputs	Outputs
preprocessor	audio fp32 [1, N], audio_length int32 [1]	mel fp32 [1, 128, frames], mel_length
encoder_pre_encode	mel fp32 [1, 128, total_mel_frames], length int32 [1]	hidden [1, T, 1024], length_out
encoder_shard_k (k = 0…3)	hidden [1, T, 1024], length, cache_channel fp32 [1, 6, 42, 1024], cache_time fp32 [1, 6, 1024, 8], cache_len int32 [1], prompt_id int32 [1]	hidden_out [1, T, 1024] (shards 0–2) or encoded [1, 1024, T] (shard 3), length_out, cache_channel_out, cache_time_out, cache_len_out
decoder_joint_argmax	token int32 [1, 1], token_length int32 [1], h_in / c_in fp32 [2, 1, 640], encoder fp32 [1, 1024, 1] (one encoder frame)	token_id int32 [1, 1, 1] (argmax over 13,088 logits, blank = 13087), h_out, c_out [2, 1, 640]
decoder	token int32 [1, 1], token_length int32 [1], h_in / c_in fp32 [2, 1, 640]	decoder_out [1, 640, 1], h_out, c_out
joint_noencproj_batched	encoder_proj fp32 [1, 8, 640] (the joint's encoder projection, applied host-side), decoder fp32 [1, 640, 1]	logits [1, 8, 1, 13088]

• total_mel_frames = chunk_mel_frames + 9, the 9-frame pre-encode cache. T = chunk_mel_frames / 8. All values are in metadata.json.
• Shard k owns layers 6k … 6k+5. Its cache inputs are that slice of the full [1, 24, …] caches in metadata.json (cache_channel_shape, cache_time_shape).
• Run pre-encode once per chunk, then shards 0 → 1 → 2 → 3. Each shard's hidden_out feeds the next shard's hidden.
• Greedy decoding runs decoder_joint_argmax once per step with the last emitted token (blank at the start) and the current encoder frame. On a non-blank token_id, adopt h_out / c_out and step again on the same frame. On blank, drop them and move to the next frame.
• decoder, joint_noencproj_batched and native_weights/ serve the windowed "smart-spec" decode: one decoder call plus one 8-frame joint call per window, with the encoder projection computed on the host from native_weights/. The example below does not use them.

Load the encoder shards with .cpuAndNeuralEngine, never .all. With .all, CoreML can route the INT8 encoder to a much slower GPU path.

Language selection

Pass the language's prompt_id from metadata.json["prompt_dictionary"] (for example en-US→0, es-US→3, zh-CN→4, ja-JP→10, de-DE→9) on every encoder call. auto (101) lets the model choose. The model emits a <xx-XX> language-tag token somewhere in the output. Strip every id in metadata.json["lang_tag_token_ids"] from the transcript. With forced prompts, the language is already known. Auto-mode language identification is unreliable on short utterances and weak for zh.

Python (coremltools) streaming reference

import json, numpy as np, coremltools as ct
from huggingface_hub import snapshot_download

d = snapshot_download("smdesai/Nemotron-3.5-ASR-Streaming-Multilingual-0.6b",
                      allow_patterns=["multilingual/2240ms/*.json", "multilingual/2240ms/coreml/*"])
d += "/multilingual/2240ms"
md, vocab = json.load(open(f"{d}/metadata.json")), json.load(open(f"{d}/tokenizer.json"))
def load(name, cu=ct.ComputeUnit.CPU_ONLY):
    return ct.models.CompiledMLModel(f"{d}/coreml/{name}.mlmodelc", compute_units=cu)
ane = ct.ComputeUnit.CPU_AND_NE
pre, dec_joint = load("preprocessor"), load("decoder_joint_argmax")
pre_enc = load("encoder_pre_encode", ane)
shards = [load(f"encoder_shard_{k}", ane) for k in range(4)]

def encode(mel, T, cc, ct_, cl, pid):                               # pre-encode + 4 shards
    o = pre_enc.predict({"mel": mel, "length": np.array([T], np.int32)})
    x, ln, cco, cto = o["hidden"], o["length_out"], [], []
    for k, s in enumerate(shards):
        r = s.predict({"hidden": x, "length": ln, "cache_channel": cc[:, 6*k:6*k+6],
                       "cache_time": ct_[:, 6*k:6*k+6], "cache_len": cl, "prompt_id": pid})
        cco.append(r["cache_channel_out"]); cto.append(r["cache_time_out"])
        x = r["encoded"] if k == 3 else r["hidden_out"]
    return x, np.concatenate(cco, 1), np.concatenate(cto, 1), r["cache_len_out"]

def transcribe(audio: np.ndarray, lang: str = "en-US") -> str:          # 16 kHz mono float32
    blank, T, pc = md["blank_idx"], md["total_mel_frames"], md["pre_encode_cache"]
    n = md["chunk_mel_frames"] * 160
    pid = np.array([md["prompt_dictionary"][lang]], np.int32)
    cc = np.zeros(md["cache_channel_shape"], np.float32)
    ct_ = np.zeros(md["cache_time_shape"], np.float32)
    cl = np.zeros(1, np.int32)
    h = c = np.zeros((2, 1, 640), np.float32)
    last, toks, mel_cache = blank, [], None
    tags = set(md["lang_tag_token_ids"])
    for off in range(0, len(audio), n):
        x = np.pad(audio[off:off + n], (0, max(0, off + n - len(audio))))[None].astype(np.float32)
        mel = pre.predict({"audio": x, "audio_length": np.array([n], np.int32)})["mel"]
        m = (np.concatenate([mel_cache, mel], 2) if mel_cache is not None
             else np.pad(mel, ((0, 0), (0, 0), (pc, 0))))
        m = np.pad(m, ((0, 0), (0, 0), (0, max(0, T - m.shape[2]))))[:, :, :T]
        mel_cache = mel[:, :, -pc:]
        enc, cc, ct_, cl = encode(m, T, cc, ct_, cl, pid)
        for t in range(enc.shape[2]):                                    # greedy RNN-T
            e = enc[:, :, t:t + 1].astype(np.float32)
            for _ in range(10):                                          # max symbols per frame
                r = dec_joint.predict({"token": np.array([[last]], np.int32),
                                       "token_length": np.array([1], np.int32),
                                       "h_in": h, "c_in": c, "encoder": e})
                k = int(r["token_id"].reshape(-1)[0])
                if k == blank:                                           # blank: keep decoder state
                    break
                toks.append(k); last = k; h, c = r["h_out"], r["c_out"]
    return "".join(vocab[str(t)] for t in toks if t not in tags).replace("▁", " ").strip()

Core AI (iOS 27+ / macOS 27+)

The coreai/ bundle has the same pipeline with three differences:

• Mel is computed on the host. The STFT can't be lowered to Core AI. Use mel_window.f32 (Hann-400 placed at offset 56 in a 512-sample frame) and mel_filterbank.f32 ([128, 257]): STFT with n_fft 512 and hop 160 → power re² + im² → filterbank → log(mel + 5.96e-08).
• The encoder has four shards and no separate pre-encode. encoder_shard_0_int8 takes the mel [1, 128, total_mel_frames] directly. Inputs and caches are fp16; caches are per-shard slices, as in CoreML.
• Decoding uses one fused model. decoder_joint.aimodel takes token, token_length, h_in, c_in and the whole chunk's encoder output [1, 1024, T]. It returns logits for every frame of the chunk under the current decoder state, plus h_out and c_out. Call it again after each emitted token.

Model details

Base model	nvidia/nemotron-3.5-asr-streaming-0.6b (EncDecRNNTBPEModelWithPrompt)
Parameters	638 M (encoder 609 M, prediction net 15 M, joint 9.5 M, prompt kernel 4.5 M)
Encoder	24-layer cache-aware FastConformer, d_model 1024, 8× subsampling (80 ms/frame)
Decoder	RNN-T, 2× LSTM (640), greedy; blank id 13087
Vocabulary	13,087 SentencePiece tokens + blank (includes <xx-XX> language tags)
Front-end	16 kHz, 128-bin log-mel, 25 ms window / 10 ms hop, pre-emphasis 0.97, no normalization
Language conditioning	one-hot prompt_id (128 slots) concatenated with the encoder output → 2-layer MLP
Streaming	att_context_size [42, 13], 9-frame pre-encode cache, per-layer attention (42) + conv (8) caches
Quantization	encoder weights INT8 linear symmetric, per-channel; activations fp16; decoder/joint/preprocessor fp16 weights + compute (fp32 I/O)
Conversion	CoreML: NeMo (NVIDIA-NeMo/Speech) → torch.jit.trace → coremltools 9 mlprogram → coremltools.optimize INT8 → coremlcompiler. Core AI: NeMo → coreai-torch 0.4.2 → coreai-opt INT8 (fp16 activations) → .aimodel (coreai-core 1.0.0b2)

Limitations

• [42, 13] is not one of the checkpoint's trained attention contexts. The trained contexts are [56, {0, 3, 6, 13}]. On this test set the reduced left context showed no significant cost (see above), but it has not been validated on long-form or out-of-domain audio.
• The evaluation covers the 2240 ms CoreML build only. The other tiers share the conversion recipe and weights but were not re-benchmarked here. The 560 ms tier splits a 14-frame trained attention chunk, so expect a small quality cost there.
• The coreai/ builds were not part of this FLEURS evaluation. Core AI requires iOS 27 / macOS 27.
• Greedy decoding only. Beam search, word boosting and timestamps are left to the host.
• Auto language detection (prompt_id 101) is much less accurate than forcing the language.
• zh-TW has a prompt but no vocabulary tag. The model emits <zh-CN> while conditioning on Traditional script.
• The evaluation is a 1,000-utterance FLEURS subset (read speech). Performance on conversational, noisy or far-field audio was not measured.

License & attribution

This is a derivative of NVIDIA's nemotron-3.5-asr-streaming-0.6b, released under the OpenMDW-1.1 license. This CoreML/Core AI port is distributed under the same license. Keep this attribution to NVIDIA and the OpenMDW-1.1 notice when you redistribute it.