Instructions to use ibm-granite/granite-3.0-2b-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ibm-granite/granite-3.0-2b-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ibm-granite/granite-3.0-2b-instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ibm-granite/granite-3.0-2b-instruct")
model = AutoModelForCausalLM.from_pretrained("ibm-granite/granite-3.0-2b-instruct", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ibm-granite/granite-3.0-2b-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ibm-granite/granite-3.0-2b-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-3.0-2b-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ibm-granite/granite-3.0-2b-instruct

SGLang

How to use ibm-granite/granite-3.0-2b-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ibm-granite/granite-3.0-2b-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-3.0-2b-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ibm-granite/granite-3.0-2b-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-3.0-2b-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ibm-granite/granite-3.0-2b-instruct with Docker Model Runner:
```
docker model run hf.co/ibm-granite/granite-3.0-2b-instruct
```

Question for the IBM Granite Team

by rollercoasterX - opened Oct 21, 2024

Discussion

rollercoasterX

Oct 21, 2024

Hello IBM Granite Team,

Thank you for providing this model!

I have a question regarding the training approach you took with this model (granite-3.0-2b-instruct) and the 8B model (granite-3.0-8b-instruct).

I am curious to know how the inclusion of multiple languages affected the model's size and parameter count. Did the addition of many languages significantly increase the model's size, or were you able to find ways to mitigate this effect?

If it did affect the size and parameter count, my next question would be if you considered training future models on a single language and then relying on post-training fine-tuning by community members to adapt the model to other languages?

I'm interested in understanding the trade-offs you made when designing this model, and whether you think a single-language approach with community-driven fine-tuning could be a viable path forward.

Thanks in advance for your time, and I look forward to hearing your thoughts on this!

HuiWuUS

IBM Granite org Oct 22, 2024

Hey! About supporting multiple languages, the change is more on the data that goes into training, less about model size/architecture/parameters!

In terms of choosing base training or finetuning (or both), that is a good question to ask whenever we want to advance models' capabilities or addressing any issues. Usually for a significant extension to the model's scope (like a new programming language, a foreign language), likely the data you might need is of larger scale and it is the best to include them in the base model training. But finetuning on the target tasks/language will definitely help! In summary, given where our model is at now (12 languages), a combination of both approaches would be more effective.

rollercoasterX

Oct 27, 2024

Thank you so much for your reply @HuiWuUS your reply is very insightful and helpful.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment