Instructions to use clibrain/mamba-2.8b-instruct-openhermes with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use clibrain/mamba-2.8b-instruct-openhermes with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="clibrain/mamba-2.8b-instruct-openhermes")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("clibrain/mamba-2.8b-instruct-openhermes", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use clibrain/mamba-2.8b-instruct-openhermes with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "clibrain/mamba-2.8b-instruct-openhermes"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "clibrain/mamba-2.8b-instruct-openhermes",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/clibrain/mamba-2.8b-instruct-openhermes

SGLang

How to use clibrain/mamba-2.8b-instruct-openhermes with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "clibrain/mamba-2.8b-instruct-openhermes" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "clibrain/mamba-2.8b-instruct-openhermes",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "clibrain/mamba-2.8b-instruct-openhermes" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "clibrain/mamba-2.8b-instruct-openhermes",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use clibrain/mamba-2.8b-instruct-openhermes with Docker Model Runner:
```
docker model run hf.co/clibrain/mamba-2.8b-instruct-openhermes
```

mamba-2.8b-instruct-openhermes / README.md

mrm8488

Update README.md

6212a3f over 2 years ago

preview code

Raw

History Blame Contribute Delete

3.63 kB

	---
	license: wtfpl
	datasets:
	- teknium/openhermes
	pipeline_tag: text-generation
	thumbnail: https://huggingface.co/clibrain/mamba-2.8b-instruct-openhermes/resolve/main/mamba_hermes_logo_1.png?download=true
	language:
	- en
	---

	# MAMBA (2.8B) 🐍 fine-tuned on OpenHermes

	<div style="text-align:center;width:250px;height:250px;">
	<img src="https://huggingface.co/clibrain/mamba-2.8b-instruct-openhermes/resolve/main/mamba_hermes_logo_1.png?download=true" alt="mamba-hermes logo"">
	</div>

	Model Card is still WIP!


	## Base model info

	Mamba is a new state space model architecture showing promising performance on information-dense data such as language modeling, where previous subquadratic models fall short of Transformers.
	It is based on the line of progress on [structured state space models](https://github.com/state-spaces/s4),
	with an efficient hardware-aware design and implementation in the spirit of [FlashAttention](https://github.com/Dao-AILab/flash-attention).

	## Dataset info

	The OpenHermes dataset is composed of 242,000 entries of primarily GPT-4 generated data, from open datasets across the AI landscape, including:

	OpenHermes 13B is the first fine tune of the Hermes dataset that has a fully open source dataset!

	OpenHermes was trained on 242,000 entries of primarily GPT-4 generated data, from open datasets across the AI landscape, including:

	- GPTeacher - General Instruct, Roleplay v1, Roleplay v2, and Code Instruct Datasets, by Teknium
	- WizardLM (v1, evol_instruct 70k), by WizardLM Team/nlpxucan
	- Airoboros GPT-4 (v1.0), by JonDurbin
	- Camel-AI's domain expert datasets, by the Camel-AI Team
	- CodeAlpaca, by Sahil2801
	- GPT4-LLM and Unnatural Instructions, by Microsoft
	Filtering included removal of OpenAI refusals, disclaimers, and "As an AI" type examples and more
	The base dataset mix is identical to the original Nous-Hermes', minus the Nous-Instruct and PDACTL datasets which were private datasets.

	## Usage

	```sh
	pip install torch==2.1.0 transformers==4.35.0 causal-conv1d==1.0.0 mamba-ssm==1.0.1
	```

	```py
	import torch
	from transformers import AutoTokenizer, AutoModelForCausalLM
	from mamba_ssm.models.mixer_seq_simple import MambaLMHeadModel

	CHAT_TEMPLATE_ID = "HuggingFaceH4/zephyr-7b-beta"

	device = "cuda:0" if torch.cuda.is_available() else "cpu"
	model_name = "clibrain/mamba-2.8b-instruct-openhermes"

	eos_token = "<\|endoftext\|>"
	tokenizer = AutoTokenizer.from_pretrained(model_name)
	tokenizer.eos_token = eos_token
	tokenizer.pad_token = tokenizer.eos_token
	tokenizer.chat_template = AutoTokenizer.from_pretrained(CHAT_TEMPLATE_ID).chat_template

	model = MambaLMHeadModel.from_pretrained(
	model_name, device=device, dtype=torch.float16)

	messages = []
	prompt = "Tell me 5 sites to visit in Spain"
	messages.append(dict(role="user", content=prompt))

	input_ids = tokenizer.apply_chat_template(
	messages, return_tensors="pt", add_generation_prompt=True
	).to(device)

	out = model.generate(
	input_ids=input_ids,
	max_length=2000,
	temperature=0.9,
	top_p=0.7,
	eos_token_id=tokenizer.eos_token_id,
	)

	decoded = tokenizer.batch_decode(out)
	assistant_message = (
	decoded[0].split("<\|assistant\|>\n")[-1].replace(eos_token, "")
	)

	print(assistant_message)
	```


	## Gradio Demo

	```sh
	git clone https://github.com/mrm8488/mamba-chat.git
	cd mamba-chat

	pip install -r requirements.txt
	pip install -q gradio==4.8.0

	python app.py \
	--model clibrain/mamba-2.8b-instruct-openhermes \
	--share
	```
	## Evaluations

	Coming soon!


	## Acknowledgments

	Thanks to [mamba-chat](https://github.com/havenhq/mamba-chat/tree/main) for heavily inspiring our work