Japanese Multimodal Models
Collection
Suite of multimodal models focusing on Japan/Japanese-related usage โข 4 items โข Updated โข 12
You need to agree to share your Hugging Face information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
By clicking "Agree", you agree to the License Agreement and acknowledge Stability AI's Privacy Policy.
Log in or Sign Up to review the conditions and access this model content.
Japanese Stable VLM
Please note: for commercial usage of this model, please see https://stability.ai/license
ๅ็จๅฉ็จใซ้ขใใๆฅๆฌ่ชใงใฎๅใๅใใใฏใpartners-jp@stability.ai ใพใงใ้กใ่ดใใพใใ
Model Details
Japanese Stable VLM is a vision-language instruction-following model that enables to generate Japanese descriptions for input images and optionally input texts such as questions.
Usage
import torch
from transformers import AutoTokenizer, AutoModelForVision2Seq, AutoImageProcessor
from PIL import Image
import requests
# helper function to format input prompts
TASK2INSTRUCTION = {
"caption": "็ปๅใ่ฉณ็ดฐใซ่ฟฐในใฆใใ ใใใ",
"tag": "ไธใใใใๅ่ชใไฝฟใฃใฆใ็ปๅใ่ฉณ็ดฐใซ่ฟฐในใฆใใ ใใใ",
"vqa": "ไธใใใใ็ปๅใไธใซใ่ณชๅใซ็ญใใฆใใ ใใใ",
}
def build_prompt(task="caption", input=None, sep="\n\n### "):
assert (
task in TASK2INSTRUCTION
), f"Please choose from {list(TASK2INSTRUCTION.keys())}"
if task in ["tag", "vqa"]:
assert input is not None, "Please fill in `input`!"
if task == "tag" and isinstance(input, list):
input = "ใ".join(input)
else:
assert input is None, f"`{task}` mode doesn't support to input questions"
sys_msg = "ไปฅไธใฏใใฟในใฏใ่ชฌๆใใๆ็คบใจใๆ่ใฎใใๅ
ฅๅใฎ็ตใฟๅใใใงใใ่ฆๆฑใ้ฉๅใซๆบใใๅฟ็ญใๆธใใชใใใ"
p = sys_msg
roles = ["ๆ็คบ", "ๅฟ็ญ"]
instruction = TASK2INSTRUCTION[task]
msgs = [": \n" + instruction, ": \n"]
if input:
roles.insert(1, "ๅ
ฅๅ")
msgs.insert(1, ": \n" + input)
for role, msg in zip(roles, msgs):
p += sep + role + msg
return p
# load model
device = "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModelForVision2Seq.from_pretrained("stabilityai/japanese-stable-vlm", trust_remote_code=True)
processor = AutoImageProcessor.from_pretrained("stabilityai/japanese-stable-vlm")
tokenizer = AutoTokenizer.from_pretrained("stabilityai/japanese-stable-vlm")
model.to(device)
# prepare inputs
url = "https://images.unsplash.com/photo-1582538885592-e70a5d7ab3d3?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=1770&q=80"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
prompt = build_prompt(task="caption")
# prompt = build_prompt(task="tag", input=["ๆฒณๆดฅๆก", "้็ฉบ"])
# prompt = build_prompt(task="vqa", input="ๅญฃ็ฏใฏใใคใงใใ๏ผ")
inputs = processor(images=image, return_tensors="pt")
text_encoding = tokenizer(prompt, add_special_tokens=False, return_tensors="pt")
inputs.update(text_encoding)
# generate
outputs = model.generate(
**inputs.to(device, dtype=model.dtype),
do_sample=False,
num_beams=5,
max_new_tokens=128,
min_length=1,
repetition_penalty=1.5,
)
generated_text = tokenizer.batch_decode(outputs, skip_special_tokens=True)[0].strip()
print(generated_text)
# ๆก่ถใใฎๆฑไบฌในใซใคใใชใผ
Model Details
- Developed by: Stability AI
- Model type: Auto-regressive Vision Language Model
- Language(s): Japanese
- License: STABILITY AI COMMUNITY LICENSE.
Training
This model is a vision-language instruction-following model with the LLaVA 1.5 architecture. It uses stabilityai/japanese-stablelm-instruct-gamma-7b as a language model and openai/clip-vit-large-patch14 as an image encoder. During training, the MLP projection was trained from scratch at the first stage and the language model and the MLP projection were further trained at the second stage.
Training Dataset
The training dataset includes the following public datasets:
- CC12M with captions translated into Japanese
- MS-COCO with STAIR Captions
- Japanese Visual Genome VQA dataset
Use and Limitations
Intended Use
This model is intended to be used by the open-source community in vision-language applications.
Limitations and bias
The training dataset may have contained offensive or inappropriate content even though we applied data filters. We recommend users exercise reasonable caution when using these models in production systems. Do not use the model for any applications that may cause harm or distress to individuals or groups.
How to cite
@misc{JapaneseStableVLM,
url = {[https://huggingface.co/stabilityai/japanese-stable-vlm](https://huggingface.co/stabilityai/japanese-stable-vlm)},
title = {Japanese Stable VLM},
author = {Shing, Makoto and Akiba, Takuya}
}
Contact
- For questions and comments about the model, please join Stable Community Japan.
- For future announcements / information about Stability AI models, research, and events, please follow https://twitter.com/StabilityAI_JP.
- For business and partnership inquiries, please contact partners-jp@stability.ai. ใใธใในใๅๆฅญใซ้ขใใใๅใๅใใใฏsales-jp@stability.aiใซใ้ฃ็ตกใใ ใใใ
- Downloads last month
- 1