Ternary-Bonsai-8B — LiteRT-LM (INT2 / ternary, GPU + CPU)

Ternary (INT2) .litertlm bundles of prism-ml/Ternary-Bonsai-8B-unpacked, packaged for the LiteRT-LM runtime and Google AI Edge Gallery.

These run on the GPU backend (recommended — fastest) or the CPU backend. Not the NPU.

An 8B that fits in ~2.2–2.6 GB — ternary weights are what make a model this size practical on a phone GPU at all.

Recommended: bonsai-8b-int2pc-4k-gpu.litertlm. The plain, unmodified build with no optimizations applied. Everything else in this repo is experimental — longer context or lighter activations, but whether a given one loads depends on your app's LiteRT / LiteRT-LM version and dependencies. Start with the recommended build; reach for an experimental one only if you specifically need what it offers.

Builds

file	ctx	weights	activations	size	notes
bonsai-8b-int2pc-4k-gpu.litertlm	4096	INT2 per-channel	fp32	2.56 GB	RECOMMENDED — plain, no optimizations
bonsai-8b-sdpa-fp16-int2all-32k-fixed2.litertlm	32768	INT2 (all)	fp16	2.16 GB	experimental — fp16 activations, fused SDPA, smallest
bonsai-8b-int2pc-32k-crashfix.litertlm	32768	INT2 per-channel	fp32	2.56 GB	experimental — long context, no SDPA fusion

Ternary-Bonsai-8B's max_position_embeddings is 65536, so the 32k builds here do not reach the model's full context. A 64k build is possible and simply hasn't been made yet.

All bundles carry Bonsai's own chat template (Qwen3 ChatML with the reasoning block intact).

Why several variants

The GPU accelerator runs a float graph — INT2 is a storage format, and compute happens in fp16/fp32. Two consequences shape this list:

• Per-channel ternary dequantizes coherently. Block-quantized weights mix scales inside a single GEMM, which is why the per-channel builds are the conservative choice.
• fp16 activations are lighter and smaller but less widely supported. The sdpa-fp16 bundle is the most compact here; it is also the most likely to meet a runtime that won't take it.

Support ranges by app, so the full set is published rather than a single "best" build.

Backends

backend	status
GPU	✅ recommended, fastest
CPU	✅ works — verified on a desktop x86 CPU via the litert-lm Python package (~10 tok/s); phone CPU not yet tested
NPU	❌ not supported by these bundles

Why CPU works even though XNNPACK has no INT2 kernel: XNNPACK takes the float ops, and the INT2 FULLY_CONNECTED ops fall back to LiteRT's built-in kernels, which handle INT2 directly. Expect CPU to be noticeably slower than GPU. A YNNPACK-enabled runtime could speed up CPU INT2, but it isn't required.

Sampling defaults

Every bundle ships these in its LlmMetadata, so a LiteRT-LM host picks them up without any configuration:

parameter	value
type	TOP_P
top-k	20
top-p	0.85
temperature	0.5

These are the values the bundles were built with. Override them in your host if you want different behaviour.

Usage

Any LiteRT-LM host — the AI Edge Gallery app, or litert_lm_main — with the GPU backend (fastest) or the CPU backend selected.

Provenance and licensing

• Base model: prism-ml/Ternary-Bonsai-8B-unpacked, Apache-2.0
• Built from Qwen3-8B, Copyright 2024 Alibaba Cloud, Apache-2.0
• These bundles are quantized, repackaged derivatives. No weights were retrained.

Created using Bonsai by Prism ML.

Training data: None was used here. These are post-training quantizations and repackagings of the released Bonsai checkpoint; no additional training, fine-tuning, or calibration data was involved. For the base model's training data, see the upstream Prism ML and Qwen3 model cards.

PII: No dataset was collected, processed, or shipped as part of this conversion, so no personally identifiable information is present in these artifacts beyond whatever the upstream released weights already encode.