Instructions to use sail/Sailor2-20B-Chat-1203 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use sail/Sailor2-20B-Chat-1203 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="sail/Sailor2-20B-Chat-1203")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("sail/Sailor2-20B-Chat-1203")
model = AutoModelForCausalLM.from_pretrained("sail/Sailor2-20B-Chat-1203", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use sail/Sailor2-20B-Chat-1203 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "sail/Sailor2-20B-Chat-1203"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sail/Sailor2-20B-Chat-1203",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/sail/Sailor2-20B-Chat-1203

SGLang

How to use sail/Sailor2-20B-Chat-1203 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "sail/Sailor2-20B-Chat-1203" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sail/Sailor2-20B-Chat-1203",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "sail/Sailor2-20B-Chat-1203" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sail/Sailor2-20B-Chat-1203",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use sail/Sailor2-20B-Chat-1203 with Docker Model Runner:
```
docker model run hf.co/sail/Sailor2-20B-Chat-1203
```

Add Transformers library and pipeline tag

by nielsr HF Staff - opened Feb 19, 2025

base: refs/heads/main

←

from: refs/pr/3

Discussion Files changed

+12

-10

nielsr

Feb 19, 2025

This PR adds the Transformers library tag to the model card, as well as the pipeline tag, enabling people to more easily discover this model.

Add Transformers library and pipeline tag0413682f

dreamerdeo changed pull request status to closed Feb 20, 2025

nielsr

Feb 20, 2025

Hi @dreamerdeo any reason to close this PR? The library_name:transformers and pipeline_tag:text-generation seem appropriate here

dreamerdeo

Sea AI Lab org Feb 20, 2025

•

edited Feb 20, 2025

@nielsr Thanks! I have updated the library_name:transformers and pipeline_tag:text-generation manually, since the auto-merge seems make the README.md unstructured (like this https://huggingface.co/sail/Sailor2-8B-Chat).

nielsr

Feb 20, 2025

Ok I see, thanks for fixing.

This PR was generated by an LLM so there are still some improvements to be made :D

nielsr

Feb 20, 2025

Btw we have the "new_version" feature if you want to point to a new model: https://huggingface.co/docs/hub/en/model-cards#specifying-a-new-version

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment