Update README.md to include SGLang instruction

README.md

@@ -1,16 +1,18 @@
 ---
 library_name: vllm
 inference: false
+base_model:
+- mistralai/Mistral-Small-3.1-24B-Base-2503
 extra_gated_description: >-
   If you want to learn more about how we process your personal data, please read
   our <a href="https://mistral.ai/terms/">Privacy Policy</a>.
 tags:
 - mistral-common
-license: other
+license: apache-2.0
 ---
 
-# Devstral 2 123B Instruct 2512
-Devstral is an agentic LLM for software engineering tasks. **Devstral 2** excels at using tools to explore codebases, editing multiple files and power software engineering agents.  
+# Devstral Small 2 24B Instruct 2512
+Devstral is an agentic LLM for software engineering tasks. **Devstral Small 2** excels at using tools to explore codebases, editing multiple files and power software engineering agents.  
 The model achieves remarkable performance on SWE-bench. 
 
 This model is an Instruct model in **FP8**, fine-tuned to follow instructions, making it ideal for chat, agentic and instruction based tasks for SWE use cases.
@@ -18,12 +20,18 @@ This model is an Instruct model in **FP8**, fine-tuned to follow instructions, m
 For enterprises requiring specialized capabilities (increased context, domain-specific knowledge, etc.), we invite companies to [reach out to us](https://mistral.ai/contact).
 
 ## Key Features
-The Devstral 2 Instruct model offers the following capabilities:
+The Devstral Small 2 Instruct model offers the following capabilities:
 - **Agentic Coding**: Devstral is designed to excel at agentic coding tasks, making it a great choice for software engineering agents.
-- **Improved Performance**: Devstral 2 is a step-up compared to its predecessors.
-- **Better Generalization**: Generalises better to diverse prompts and coding environments.
+- **Lightweight**: with its compact size of just 24 billion parameters, Devstral is light enough to run on a single RTX 4090 or a Mac with 32GB RAM, making it an appropriate model for local deployment and on-device use.
+- **Apache 2.0 License**: Open-source license allowing usage and modification for both commercial and non-commercial purposes.
 - **Context Window**: A 256k context window.
 
+Updates compared to [`Devstral Small 1.1`](https://huggingface.co/mistralai/Devstral-Small-2507):
+- **Vision Capabilities**: Enables the model to analyze images and provide insights based on visual content, in addition to text.
+- **Improved Performance**: Devstral Small 2 is a step-up compared to its predecessors.
+- **Attention Softmax Temperature**: Devstral Small 2 uses the same architecture as Ministral 3 using rope-scaling as introduced by Llama 4 and [Scalable-Softmax Is Superior for Attention](https://arxiv.org/abs/2501.19399).
+- **Better Generalization**: Generalises better to diverse prompts and coding environments.
+
 ### Use Cases
 
 AI Code Assistants, Agentic Coding, and Software Engineering Tasks. Leveraging advanced AI capabilities for complex tool integration and deep codebase understanding in coding environments.
@@ -94,15 +102,16 @@ If this is your first time running Vibe, it will:
 
 ### Local Deployment
 
-The model can also be deployed with the following libraries, we advise everyone to use the Mistral AI API if the model is subpar with local serving:
+The model can also be deployed with the following libraries:
 - [`vllm (recommended)`](https://github.com/vllm-project/vllm): See [here](#vllm-recommended)
 - [`transformers`](https://github.com/huggingface/transformers): See [here](#transformers)
 
 We're thankful to the llama.cpp team and their community as well as the LM Studio and Ollama teams that worked hard to make these models also available for their frameworks.
 
 You can now also run Devstral using these (alphabetical ordered) frameworks:
-- [`llama.cpp`](https://github.com/ggml-org/llama.cpp): To use community ones such as [Unsloth's](https://huggingface.co/unsloth/Devstral-2-123B-Instruct-2512-GGUF) or [Bartowski's](https://huggingface.co/bartowski/mistralai_Devstral-2-123B-Instruct-2512-GGUF) make sure to use changes from this [PR](https://github.com/ggml-org/llama.cpp/pull/17945).
-- [`Ollama`](https://ollama.com/): https://ollama.com/library/devstral-2
+- [`llama.cpp`](https://github.com/ggml-org/llama.cpp): To use community ones such as [Unsloth's](https://huggingface.co/unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF) or [Bartowski's](https://huggingface.co/bartowski/mistralai_Devstral-Small-2-24B-Instruct-2512-GGUF) make sure to use changes from this [PR](https://github.com/ggml-org/llama.cpp/pull/17945).
+- [`LM Studio`](https://lmstudio.ai/): https://lmstudio.ai/models/devstral-2
+- [`Ollama`](https://ollama.com/): https://ollama.com/library/devstral-small-2
 
 If you notice subpar performance with local serving, please submit issues to the relevant framework so that it can be fixed and in the meantime we advise to use the Mistral AI API.
 
@@ -131,6 +140,8 @@ docker pull vllm/vllm-openai:nightly
 docker run -it vllm/vllm-openai:nightly
 ```
 
+Alternatively, you can also install `vllm` from latest main by following instructions [here](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/#python-only-build).
+
 > [!Warning]
 > Make sure that your vllm installation includes [this commit](https://github.com/vllm-project/vllm/commit/5c213d2899f5a2d439c8d771a0abc156a5412a2b).
 > If you do not have this commit included, you will get incorrectly parsed tool calls.
@@ -148,7 +159,7 @@ We recommand that you use Devstral in a server/client setting.
 1. Spin up a server:
 
 ```
-vllm serve mistralai/Devstral-2-123B-Instruct-2512 --tool-call-parser mistral --enable-auto-tool-choice --tensor-parallel-size 8
+vllm serve mistralai/Devstral-Small-2-24B-Instruct-2512 --tool-call-parser mistral --enable-auto-tool-choice --tensor-parallel-size 2
 ```
 
 
@@ -163,7 +174,7 @@ from huggingface_hub import hf_hub_download
 url = "http://<your-server-url>:8000/v1/chat/completions"
 headers = {"Content-Type": "application/json", "Authorization": "Bearer token"}
 
-model = "mistralai/Devstral-2-123B-Instruct-2512"
+model = "mistralai/Devstral-Small-2-24B-Instruct-2512"
 
 def load_system_prompt(repo_id: str, filename: str) -> str:
     file_path = hf_hub_download(repo_id=repo_id, filename=filename)
@@ -188,7 +199,7 @@ messages = [
 
 data = {"model": model, "messages": messages, "temperature": 0.15}
 
-# Devstral 2 supports tool calling. If you want to use tools, follow this:
+# Devstral Small 2 supports tool calling. If you want to use tools, follow this:
 # tools = [ # Define tools for vLLM
 #     {
 #         "type": "function",
@@ -215,10 +226,105 @@ print(response.json()["choices"][0]["message"]["content"])
 ```
 </details>
 
+#### SGLang (recommended)
+
+<details>
+<summary>Expand</summary>
+
+We recommend using this model with [SGLang](https://github.com/sgl-project/sglang)
+to implement production-ready inference pipelines (OpenAI-compatible API server).
+
+**_Installation_**
+
+Install SGLang from source (track latest `main` locally):
+
+```
+git clone https://github.com/sgl-project/sglang.git
+cd sglang
+uv pip install -e python
+uv pip install transformers==5.0.0rc # required
+uv pip install nvidia-cudnn-cu12==9.16.0.29 # required for VLM
+```
+
+**_Launch server_**
+
+We recommend that you use Devstral Small 2 in a server/client setting.
+
+1. Spin up a server:
+
+```
+python -m sglang.launch_server --model-path mistralai/Devstral-Small-2-24B-Instruct-2512 --host 0.0.0.0 --port 30000 --tp 2 --tool-call-parser mistral
+```
+
+
+2. To ping the client you can use a simple Python snippet.
+
+```py
+import requests
+import json
+from huggingface_hub import hf_hub_download
+
+
+url = "http://<your-server-url>:30000/v1/chat/completions"
+headers = {"Content-Type": "application/json", "Authorization": "Bearer token"}
+
+model = "mistralai/Devstral-Small-2-24B-Instruct-2512"
+
+def load_system_prompt(repo_id: str, filename: str) -> str:
+    file_path = hf_hub_download(repo_id=repo_id, filename=filename)
+    with open(file_path, "r") as file:
+        system_prompt = file.read()
+    return system_prompt
+
+SYSTEM_PROMPT = load_system_prompt(model, "CHAT_SYSTEM_PROMPT.txt")
+
+messages = [
+    {"role": "system", "content": SYSTEM_PROMPT},
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "text",
+                "text": "<your-command>",
+            },
+        ],
+    },
+]
+
+data = {"model": model, "messages": messages, "temperature": 0.15}
+
+# Devstral Small 2 supports tool calling. If you want to use tools, follow this:
+# tools = [  # Define tools (OpenAI-compatible)
+#     {
+#         "type": "function",
+#         "function": {
+#             "name": "git_clone",
+#             "description": "Clone a git repository",
+#             "parameters": {
+#                 "type": "object",
+#                 "properties": {
+#                     "url": {
+#                         "type": "string",
+#                         "description": "The url of the git repository",
+#                     },
+#                 },
+#                 "required": ["url"],
+#             },
+#         },
+#     }
+# ] 
+# data = {"model": model, "messages": messages, "temperature": 0.15, "tools": tools} # Pass tools to payload.
+
+response = requests.post(url, headers=headers, data=json.dumps(data))
+print(response.json()["choices"][0]["message"]["content"])
+```
+</details>
+
 #### Transformers
 
 <details>
 <summary>Expand</summary
+ 
 
 Make sure to install from main:
 
@@ -229,15 +335,16 @@ uv pip install git+https://github.com/huggingface/transformers
 And run the following code snippet:
 
 ```python
+import torch
 from transformers import (
-    MistralForCausalLM,
+    Mistral3ForConditionalGeneration,
     MistralCommonBackend,
 )
 
-model_id = "mistralai/Devstral-2-123B-Instruct-2512"
+model_id = "mistralai/Devstral-Small-2-24B-Instruct-2512"
 
 tokenizer = MistralCommonBackend.from_pretrained(model_id)
-model = MistralForCausalLM.from_pretrained(model_id, device_map="auto")
+model = Mistral3ForConditionalGeneration.from_pretrained(model_id, device_map="auto")
 
 SP = """You are operating as and within Mistral Vibe, a CLI coding-agent built by Mistral AI and powered by default by the Devstral family of models. It wraps Mistral's Devstral models to enable natural language interaction with a local codebase. Use the available tools when helpful.
 
@@ -400,7 +507,6 @@ print(decoded_output)
 </details>
 
 ## Tests
-
 To help test our model via vLLM or test that other frameworks' implementations are correct, here is a set of prompts you can try with the expected outputs.
 
 1. Call one tool 
@@ -587,8 +693,14 @@ messages = [
       "content": [
         {
           "type": "text",
-          "text": "\n1. 6 + 2 x 3\n2. 3a + 7b - 2a\n19 - (8+2) + 1\n4. 5y (6x - 7).\n\nCompute the results steps by steps for the equations that involve only numbers ? You have to call tools to perform the operations and can do one operation at a time per equation."
+          "text": "Compute the results steps by steps for the equations that involve only numbers displayed in the image. You have to call tools to perform the operations and can do one operation at a time per equation."
         },
+        {
+          "type": "image_url",
+          "image_url": {
+            "url": "https://math-coaching.com/img/fiche/46/expressions-mathematiques.jpg"
+          }
+        }
       ]
     }
 ]
@@ -716,111 +828,111 @@ tools = [
 
 ```json
 [
-  {
-    "role": "assistant",
-    "tool_calls": [
-      {
-        "id": "chatcmpl-tool-84aa136ef835f886",
-        "function": {
-          "arguments": "{\"a\": \"2\", \"b\": \"3\"}",
-          "name": "multiply_number"
-        },
-        "type": "function"
-      }
-    ],
-    "content": "Let's compute the results step by step for the equations that involve only numbers.\n\n### 1. \\( 6 + 2 \\times 3 \\)\n\nFirst, we perform the multiplication \\( 2 \\times 3 \\):\n"
-  },
-  {
-    "role": "tool",
-    "tool_call_id": "chatcmpl-tool-84aa136ef835f886",
-    "name": "multiply_number",
-    "content": "6.0"
-  },
-  {
-    "role": "assistant",
-    "tool_calls": [
-      {
-        "id": "chatcmpl-tool-bedc049b0a248952",
-        "function": {
-          "arguments": "{\"a\": \"6\", \"b\": \"6\"}",
-          "name": "add_number"
-        },
-        "type": "function"
-      }
-    ],
-    "content": "Now, we add the result to 6:\n"
-  },
-  {
-    "role": "tool",
-    "tool_call_id": "chatcmpl-tool-bedc049b0a248952",
-    "name": "add_number",
-    "content": "12.0"
-  },
-  {
-    "role": "assistant",
-    "tool_calls": [
-      {
-        "id": "chatcmpl-tool-ae4558bc0558f4e7",
-        "function": {
-          "arguments": "{\"a\": \"8\", \"b\": \"2\"}",
-          "name": "add_number"
-        },
-        "type": "function"
-      }
-    ],
-    "content": "The result of \\( 6 + 2 \\times 3 \\) is **12**.\n\n---\n\n### 3. \\( 19 - (8 + 2) + 1 \\)\n\nFirst, we compute the expression inside the parentheses \\( 8 + 2 \\):\n"
-  },
-  {
-    "role": "tool",
-    "tool_call_id": "chatcmpl-tool-ae4558bc0558f4e7",
-    "name": "add_number",
-    "content": "10.0"
-  },
-  {
-    "role": "assistant",
-    "tool_calls": [
-      {
-        "id": "chatcmpl-tool-b21f003d298e65df",
-        "function": {
-          "arguments": "{\"a\": \"19\", \"b\": \"10\"}",
-          "name": "substract_number"
-        },
-        "type": "function"
-      }
-    ],
-    "content": "Next, we subtract the result from 19:\n"
-  },
-  {
-    "role": "tool",
-    "tool_call_id": "chatcmpl-tool-b21f003d298e65df",
-    "name": "substract_number",
-    "content": "9.0"
-  },
-  {
-    "role": "assistant",
-    "tool_calls": [
-      {
-        "id": "chatcmpl-tool-bb7628cded899f04",
-        "function": {
-          "arguments": "{\"a\": \"9\", \"b\": \"1\"}",
-          "name": "add_number"
-        },
-        "type": "function"
-      }
-    ],
-    "content": "Finally, we add 1 to the result:\n"
-  },
-  {
-    "role": "tool",
-    "tool_call_id": "chatcmpl-tool-bb7628cded899f04",
-    "name": "add_number",
-    "content": "10.0"
-  },
-  {
-    "role": "assistant",
-    "tool_calls": [],
-    "content": "The result of \\( 19 - (8 + 2) + 1 \\) is **10**.\n\n---\n\n### Summary of Results:\n1. \\( 6 + 2 \\times 3 = 12 \\)\n3. \\( 19 - (8 + 2) + 1 = 10 \\)"
-  }
+    {
+      "role": "assistant",
+      "tool_calls": [
+        {
+          "id": "chatcmpl-tool-ab0c849bd20aaebb",
+          "function": {
+            "arguments": "{\"a\": \"2\", \"b\": \"3\"}",
+            "name": "multiply_number"
+          },
+          "type": "function"
+        }
+      ],
+      "content": "Let's solve the equations step by step. I will start with the first equation: \\(6 + 2 \\times 3\\).\n\n### Equation 1: \\(6 + 2 \\times 3\\)\n\n1. **Multiplication Step**:\n   - Multiply \\(2\\) by \\(3\\).\n   - I will use the `multiply_number` tool for this operation."
+    },
+    {
+      "role": "tool",
+      "tool_call_id": "chatcmpl-tool-ab0c849bd20aaebb",
+      "name": "multiply_number",
+      "content": "6.0"
+    },
+    {
+      "role": "assistant",
+      "tool_calls": [
+        {
+          "id": "chatcmpl-tool-8270f0471244e880",
+          "function": {
+            "arguments": "{\"a\": \"6\", \"b\": \"6\"}",
+            "name": "add_number"
+          },
+          "type": "function"
+        }
+      ],
+      "content": "   - The result of \\(2 \\times 3\\) is \\(6\\).\n\n2. **Addition Step**:\n   - Add \\(6\\) to the result from the previous step (\\(6\\)).\n   - I will use the `add_number` tool for this operation."
+    },
+    {
+      "role": "tool",
+      "tool_call_id": "chatcmpl-tool-8270f0471244e880",
+      "name": "add_number",
+      "content": "12.0"
+    },
+    {
+      "role": "assistant",
+      "tool_calls": [
+        {
+          "id": "chatcmpl-tool-858ecb90f0ec568b",
+          "function": {
+            "arguments": "{\"a\": \"8\", \"b\": \"2\"}",
+            "name": "add_number"
+          },
+          "type": "function"
+        }
+      ],
+      "content": "   - The result of \\(6 + 6\\) is \\(12\\).\n\nSo, the final result for the first equation is \\(12\\).\n\n### Equation 3: \\(19 - (8 + 2) + 1\\)\n\n1. **Parentheses Step**:\n   - Add \\(8\\) and \\(2\\) inside the parentheses.\n   - I will use the `add_number` tool for this operation."
+    },
+    {
+      "role": "tool",
+      "tool_call_id": "chatcmpl-tool-858ecb90f0ec568b",
+      "name": "add_number",
+      "content": "10.0"
+    },
+    {
+      "role": "assistant",
+      "tool_calls": [
+        {
+          "id": "chatcmpl-tool-971da6e6d0c56f0a",
+          "function": {
+            "arguments": "{\"a\": \"19\", \"b\": \"10\"}",
+            "name": "substract_number"
+          },
+          "type": "function"
+        }
+      ],
+      "content": "   - The result of \\(8 + 2\\) is \\(10\\).\n\n2. **Subtraction Step**:\n   - Subtract the result from the previous step (\\(10\\)) from \\(19\\).\n   - I will use the `substract_number` tool for this operation."
+    },
+    {
+      "role": "tool",
+      "tool_call_id": "chatcmpl-tool-971da6e6d0c56f0a",
+      "name": "substract_number",
+      "content": "9.0"
+    },
+    {
+      "role": "assistant",
+      "tool_calls": [
+        {
+          "id": "chatcmpl-tool-9a8fecc42530d395",
+          "function": {
+            "arguments": "{\"a\": \"9\", \"b\": \"1\"}",
+            "name": "add_number"
+          },
+          "type": "function"
+        }
+      ],
+      "content": "   - The result of \\(19 - 10\\) is \\(9\\).\n\n3. **Addition Step**:\n   - Add \\(1\\) to the result from the previous step (\\(9\\)).\n   - I will use the `add_number` tool for this operation."
+    },
+    {
+      "role": "tool",
+      "tool_call_id": "chatcmpl-tool-9a8fecc42530d395",
+      "name": "add_number",
+      "content": "10.0"
+    },
+    {
+      "role": "assistant",
+      "tool_calls": [],
+      "content": "   - The result of \\(9 + 1\\) is \\(10\\).\n\nSo, the final result for the third equation is \\(10\\).\n\n### Summary of Results:\n- **Equation 1**: \\(6 + 2 \\times 3 = 12\\)\n- **Equation 3**: \\(19 - (8 + 2) + 1 = 10\\)"
+    }
 ]
 ```
 
@@ -1334,7 +1446,7 @@ tools = [
 {
     "role": "assistant",
     "tool_calls": [],
-    "content": "I am Devstral-2-123B-Instruct-2512, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris. I power an AI assistant called Le Chat.\n\nToday's date is 2025-12-09."
+    "content": "I am Devstral-Small-2-24B-Instruct-2512, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris. I power an AI assistant called Le Chat.\n\nToday's date is 2025-12-09."
 }
 ```
 
@@ -1584,9 +1696,8 @@ print(json.dumps(messages[origin_messages_len:], indent=2))
 
 </details>
 
-
 ## License
 
-This model is licensed under a [Modified MIT License](https://huggingface.co/mistralai/Devstral-2-123B-Instruct-2512/blob/main/LICENSE).
+This model is licensed under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0.txt).
 
 *You must not use this model in a manner that infringes, misappropriates, or otherwise violates any third party’s rights, including intellectual property rights.*