File size: 2,884 Bytes
827b404 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 | # Loading CLEF INT8
Use Linux, an H100/SM90 GPU, and the bundled CuTe runtime. The tested environment
is `nvcr.io/nvidia/pytorch:25.09-py3` with Transformers 5.12.1, CUTLASS DSL 4.8.0,
TVM FFI 0.1.14.post1, Comfy Kitchen 0.2.31, Flash Linear Attention 0.5.2,
and causal-conv1d 1.7.0. Install these with
`CLEF_INSTALL_INT8=1 bash runtime/clef-bootstrap-runtime` inside that container.
Use the resulting `/workspace/clef-venv/bin/python`.
The package contains all required weights. No original BF16 checkpoint is needed.
Download this repository into a physical directory using `local_dir`, rather
than passing the Hub's symlinked snapshot directory to the strict shard loader.
```python
import sys
from pathlib import Path
import torch
from huggingface_hub import snapshot_download
path = Path(snapshot_download(
"terminusresearch/cloudflare-clef-int8-convrot", local_dir="./clef-int8"))
sys.path.insert(0, str(path.resolve()))
from clef_int8 import load_release_model
from joint_schema_model import encode_record, collate_records
model, processor = load_release_model(path, device="cuda")
record = {
"state": {"summary": "The customer confirms service is restored."},
"questions": {"route": {
"type": "choice", "instructions": "Choose the handling route.",
"criteria": {"close": "Resolved issue", "review": "Unresolved issue"},
}},
}
encoded = encode_record(processor.tokenizer, record, processor=processor)
batch = collate_records([encoded], processor.tokenizer.pad_token_id, torch.device("cuda"))
with torch.inference_mode():
logits = model(batch)[0]
probabilities = [value.float().softmax(-1).tolist() for value in logits]
print(probabilities)
```
Warm the model up before CUDA graph capture. Keep loaded weights immutable.
Set `fuse_mlp=False` to disable paired gate/up epilogue fusion. A Torch integer
reference is available with `backend="torch"`, without the reported CuTe performance.
The loader validates both indexes, complete tensor coverage, shard paths,
tensor shapes/dtypes, scale values, byte totals, and source fingerprints before
returning a model. Original projection matrices are never allocated on the GPU.
It streams unchanged tensors into their final storage and recreates the
nonpersistent rotary buffers from the included config. Gate/up weights are
packed during loading, avoiding duplicate CUDA allocations at first inference.
All weights can load
offline from this directory.
For Cogs, bake these local weights and runtime into the image; do not download
weights in `setup()` or `predict()`. The shard target is 1.9 GB; each embedding
tensor exceeds that target and occupies its own larger shard without splitting.
Weights and source code derive from
`Cloudflare/clef@2f3de3dd85f379784083b0814d997ab627200f0c` under Apache-2.0.
The CuTe GEMM retains NVIDIA's BSD-3-Clause notice. See `LICENSE` and `NOTICE`.
|