USAGE.md

Loading CLEF INT8

Use Linux, an H100/SM90 GPU, and the bundled CuTe runtime. The tested environment is nvcr.io/nvidia/pytorch:25.09-py3 with Transformers 5.12.1, CUTLASS DSL 4.8.0, TVM FFI 0.1.14.post1, Comfy Kitchen 0.2.31, Flash Linear Attention 0.5.2, and causal-conv1d 1.7.0. Install these with CLEF_INSTALL_INT8=1 bash runtime/clef-bootstrap-runtime inside that container. Use the resulting /workspace/clef-venv/bin/python.

The package contains all required weights. No original BF16 checkpoint is needed. Download this repository into a physical directory using local_dir, rather than passing the Hub's symlinked snapshot directory to the strict shard loader.

import sys
from pathlib import Path
import torch
from huggingface_hub import snapshot_download

path = Path(snapshot_download(
    "terminusresearch/cloudflare-clef-int8-convrot", local_dir="./clef-int8"))
sys.path.insert(0, str(path.resolve()))
from clef_int8 import load_release_model
from joint_schema_model import encode_record, collate_records

model, processor = load_release_model(path, device="cuda")
record = {
    "state": {"summary": "The customer confirms service is restored."},
    "questions": {"route": {
        "type": "choice", "instructions": "Choose the handling route.",
        "criteria": {"close": "Resolved issue", "review": "Unresolved issue"},
    }},
}
encoded = encode_record(processor.tokenizer, record, processor=processor)
batch = collate_records([encoded], processor.tokenizer.pad_token_id, torch.device("cuda"))
with torch.inference_mode():
    logits = model(batch)[0]
    probabilities = [value.float().softmax(-1).tolist() for value in logits]
print(probabilities)

Warm the model up before CUDA graph capture. Keep loaded weights immutable. Set fuse_mlp=False to disable paired gate/up epilogue fusion. A Torch integer reference is available with backend="torch", without the reported CuTe performance.

The loader validates both indexes, complete tensor coverage, shard paths, tensor shapes/dtypes, scale values, byte totals, and source fingerprints before returning a model. Original projection matrices are never allocated on the GPU. It streams unchanged tensors into their final storage and recreates the nonpersistent rotary buffers from the included config. Gate/up weights are packed during loading, avoiding duplicate CUDA allocations at first inference. All weights can load offline from this directory.

For Cogs, bake these local weights and runtime into the image; do not download weights in setup() or predict(). The shard target is 1.9 GB; each embedding tensor exceeds that target and occupies its own larger shard without splitting.

Weights and source code derive from Cloudflare/clef@2f3de3dd85f379784083b0814d997ab627200f0c under Apache-2.0. The CuTe GEMM retains NVIDIA's BSD-3-Clause notice. See LICENSE and NOTICE.