Instructions to use ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1

SGLang

How to use ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1 with Docker Model Runner:
```
docker model run hf.co/ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1
```

Bio-Medical-MultiModal-Llama-3-8B-V1 / README.md

SrikanthChellappa

Update README.md

a3dbacd verified over 1 year ago

preview code

Raw History Blame

6.29 kB

---
license: other
library_name: transformers
tags:
- generated_from_trainer
- Healthcare & Lifesciences
- BioMed
- Medical
- Multimodal
- Vision
- Text
- Contact Doctor
- MiniCPM
- Llama 3
base_model: meta-llama/Meta-Llama-3-8B-Instruct
thumbnail: https://contactdoctor.in/images/clogo.png
model-index:
- name: Bio-Medical-MultiModal-Llama-3-8B-V1
  results: []
datasets:
- collaiborateorg/BioMedData
pipeline_tag: image-text-to-text
---
# Bio-Medical-MultiModal-Llama-3-8B-V1

![image/jpeg](https://huggingface.co/static-proxy/cdn-uploads.huggingface.co/production/uploads/653f5b93cd52f288490edc83/zPMUugzfOiwTiRw88jm7T.jpeg)

This model is a fine-tuned Multimodal version of [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) on our custom "BioMedData" text and image datasets.

## Model details

Model Name: Bio-Medical-MultiModal-Llama-3-8B-V1

Base Model: Llama-3-8B-Instruct

Parameter Count: 8 billion

Training Data: Custom high-quality biomedical text and image dataset

Number of Entries in Dataset: 500,000+

Dataset Composition: The dataset comprises of text and image, both synthetic and manually curated samples, ensuring a diverse and comprehensive coverage of biomedical knowledge.

## Model description

Bio-Medical-MultiModal-Llama-3-8B-V1 is a specialized large language model designed for biomedical applications. It is finetuned from the Llama-3-8B-Instruct model using a custom dataset containing over 500,000 diverse entries. These entries include a mix of synthetic and manually curated data, ensuring high quality and broad coverage of biomedical topics.

The model is trained to understand and generate text related to various biomedical fields, making it a valuable tool for researchers, clinicians, and other professionals in the biomedical domain.

## License

This model is licensed under the [Bio-Medical-MultiModal-Llama-3-8B-V1 (Non-Commercial Use Only)](./LICENSE). Please review the terms and conditions before using the model.

## Quick Demo

<video controls autoplay src="https://huggingface.co/static-proxy/cdn-uploads.huggingface.co/production/uploads/653f5b93cd52f288490edc83/RpdFKs3mBY9ZIxvUUWOKc.mp4"></video>

## How to use

import torch

from PIL import Image

from transformers import AutoModel, AutoTokenizer,BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.float16,
)

model = AutoModel.from_pretrained(
	"ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1",
	quantization_config=bnb_config,
	device_map="auto",
	torch_dtype=torch.float16,
	trust_remote_code=True,
	attn_implementation="flash_attention_2",
)

tokenizer = AutoTokenizer.from_pretrained("ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1", trust_remote_code=True)
image = Image.open("Path to Your image").convert('RGB')
question = 'Give the modality, organ, analysis, abnormalities (if any), treatment (if abnormalities are present)?'
msgs = [{'role': 'user', 'content': [image, question]}]
res = model.chat(
    image=image,
    msgs=msgs,
    tokenizer=tokenizer,
    sampling=True,
    temperature=0.95,
    stream=True
)
generated_text = ""

for new_text in res:
    generated_text += new_text
    print(new_text, flush=True, end='')


> Sample Response

The modality is Magnetic Resonance Imaging (MRI), the organ being analyzed is the cervical spine, and there are no abnormalities present in the image.

## Intended uses & limitations

Bio-Medical-MultiModal-Llama-3-8B-V1 is intended for a wide range of applications within the biomedical field, including:

1. Research Support: Assisting researchers in literature review and data extraction from biomedical texts.
2. Clinical Decision Support: Providing information to support clinical decision-making processes.
3. Educational Tool: Serving as a resource for medical students and professionals seeking to expand their knowledge base.

## Limitations and Ethical Considerations

Bio-Medical-MultiModal-Llama-3-8B-V1 performs well in various biomedical NLP tasks, users should be aware of the following limitations:

1. Biases: The model may inherit biases present in the training data. Efforts have been made to curate a balanced dataset, but some biases may persist.
2. Accuracy: The model's responses are based on patterns in the data it has seen and may not always be accurate or up-to-date. Users should verify critical information from reliable sources.
3. Ethical Use: The model should be used responsibly, particularly in clinical settings where the stakes are high. It should complement, not replace, professional judgment and expertise.


## Training and evaluation

Bio-Medical-MultiModal-Llama-3-8B-V1 was trained using NVIDIA H100 GPU's, which provides the computational power necessary for handling large-scale data and model parameters efficiently. Rigorous evaluation protocols have been implemented to benchmark its performance against similar models, ensuring its robustness and reliability in real-world applications.

The model was trained using **MiniCPM**, which allowed us to efficiently handle the multimodal data. MiniCPM provided the ability to process and learn from visual information.

### Contact Information

For further information, inquiries, or issues related to Biomed-LLM, please contact:

Email: info@contactdoctor.in

Website: https://www.contactdoctor.in

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 4
- eval_batch_size: 4
- Number of epochs: 3
- seed: 42
- gradient_accumulation_steps: 4
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.03
- mixed_precision_training: Native AMP

### Framework versions

- PEFT 0.11.0
- Transformers 4.40.2
- Pytorch 2.1.2
- Datasets 2.19.1
- Tokenizers 0.19.1

### Citation

If you use Bio-Medical-MultiModal-Llama-3-8B-V1 in your research or applications, please cite it as follows:

@misc{ContactDoctor_MEDLLM,
  author = ContactDoctor,
  title = {ContactDoctor-Bio-Medical-MultiModal-Llama-3-8B-V1: A High-Performance Biomedical Multimodal LLM},
  year = {2024},
  howpublished = {https://huggingface.co/ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1},
}

	---
	license: other
	library_name: transformers
	tags:
	- generated_from_trainer
	- Healthcare & Lifesciences
	- BioMed
	- Medical
	- Multimodal
	- Vision
	- Text
	- Contact Doctor
	- MiniCPM
	- Llama 3
	base_model: meta-llama/Meta-Llama-3-8B-Instruct
	thumbnail: https://contactdoctor.in/images/clogo.png
	model-index:
	- name: Bio-Medical-MultiModal-Llama-3-8B-V1
	results: []
	datasets:
	- collaiborateorg/BioMedData
	pipeline_tag: image-text-to-text
	---

	# Bio-Medical-MultiModal-Llama-3-8B-V1

	![image/jpeg](https://huggingface.co/static-proxy/cdn-uploads.huggingface.co/production/uploads/653f5b93cd52f288490edc83/zPMUugzfOiwTiRw88jm7T.jpeg)

	This model is a fine-tuned Multimodal version of [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) on our custom "BioMedData" text and image datasets.

	## Model details

	Model Name: Bio-Medical-MultiModal-Llama-3-8B-V1

	Base Model: Llama-3-8B-Instruct

	Parameter Count: 8 billion

	Training Data: Custom high-quality biomedical text and image dataset

	Number of Entries in Dataset: 500,000+

	Dataset Composition: The dataset comprises of text and image, both synthetic and manually curated samples, ensuring a diverse and comprehensive coverage of biomedical knowledge.

	## Model description

	Bio-Medical-MultiModal-Llama-3-8B-V1 is a specialized large language model designed for biomedical applications. It is finetuned from the Llama-3-8B-Instruct model using a custom dataset containing over 500,000 diverse entries. These entries include a mix of synthetic and manually curated data, ensuring high quality and broad coverage of biomedical topics.

	The model is trained to understand and generate text related to various biomedical fields, making it a valuable tool for researchers, clinicians, and other professionals in the biomedical domain.

	## License

	This model is licensed under the [Bio-Medical-MultiModal-Llama-3-8B-V1 (Non-Commercial Use Only)](./LICENSE). Please review the terms and conditions before using the model.

	## Quick Demo

	<video controls autoplay src="https://huggingface.co/static-proxy/cdn-uploads.huggingface.co/production/uploads/653f5b93cd52f288490edc83/RpdFKs3mBY9ZIxvUUWOKc.mp4"></video>

	## How to use

	import torch

	from PIL import Image

	from transformers import AutoModel, AutoTokenizer,BitsAndBytesConfig

	bnb_config = BitsAndBytesConfig(
	load_in_4bit=True,
	bnb_4bit_quant_type="nf4",
	bnb_4bit_use_double_quant=True,
	bnb_4bit_compute_dtype=torch.float16,
	)


	model = AutoModel.from_pretrained(
	"ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1",
	quantization_config=bnb_config,
	device_map="auto",
	torch_dtype=torch.float16,
	trust_remote_code=True,
	attn_implementation="flash_attention_2",
	)


	tokenizer = AutoTokenizer.from_pretrained("ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1", trust_remote_code=True)


	image = Image.open("Path to Your image").convert('RGB')

	question = 'Give the modality, organ, analysis, abnormalities (if any), treatment (if abnormalities are present)?'

	msgs = [{'role': 'user', 'content': [image, question]}]

	res = model.chat(
	image=image,
	msgs=msgs,
	tokenizer=tokenizer,
	sampling=True,
	temperature=0.95,
	stream=True
	)

	generated_text = ""

	for new_text in res:
	generated_text += new_text
	print(new_text, flush=True, end='')


	> Sample Response

	The modality is Magnetic Resonance Imaging (MRI), the organ being analyzed is the cervical spine, and there are no abnormalities present in the image.

	## Intended uses & limitations

	Bio-Medical-MultiModal-Llama-3-8B-V1 is intended for a wide range of applications within the biomedical field, including:

	1. Research Support: Assisting researchers in literature review and data extraction from biomedical texts.
	2. Clinical Decision Support: Providing information to support clinical decision-making processes.
	3. Educational Tool: Serving as a resource for medical students and professionals seeking to expand their knowledge base.

	## Limitations and Ethical Considerations

	Bio-Medical-MultiModal-Llama-3-8B-V1 performs well in various biomedical NLP tasks, users should be aware of the following limitations:

	1. Biases: The model may inherit biases present in the training data. Efforts have been made to curate a balanced dataset, but some biases may persist.
	2. Accuracy: The model's responses are based on patterns in the data it has seen and may not always be accurate or up-to-date. Users should verify critical information from reliable sources.
	3. Ethical Use: The model should be used responsibly, particularly in clinical settings where the stakes are high. It should complement, not replace, professional judgment and expertise.


	## Training and evaluation

	Bio-Medical-MultiModal-Llama-3-8B-V1 was trained using NVIDIA H100 GPU's, which provides the computational power necessary for handling large-scale data and model parameters efficiently. Rigorous evaluation protocols have been implemented to benchmark its performance against similar models, ensuring its robustness and reliability in real-world applications.

	The model was trained using MiniCPM, which allowed us to efficiently handle the multimodal data. MiniCPM provided the ability to process and learn from visual information.

	### Contact Information

	For further information, inquiries, or issues related to Biomed-LLM, please contact:

	Email: info@contactdoctor.in

	Website: https://www.contactdoctor.in

	### Training hyperparameters

	The following hyperparameters were used during training:
	- learning_rate: 0.0002
	- train_batch_size: 4
	- eval_batch_size: 4
	- Number of epochs: 3
	- seed: 42
	- gradient_accumulation_steps: 4
	- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
	- lr_scheduler_type: cosine
	- lr_scheduler_warmup_ratio: 0.03
	- mixed_precision_training: Native AMP

	### Framework versions

	- PEFT 0.11.0
	- Transformers 4.40.2
	- Pytorch 2.1.2
	- Datasets 2.19.1
	- Tokenizers 0.19.1

	### Citation

	If you use Bio-Medical-MultiModal-Llama-3-8B-V1 in your research or applications, please cite it as follows:

	@misc{ContactDoctor_MEDLLM,
	author = ContactDoctor,
	title = {ContactDoctor-Bio-Medical-MultiModal-Llama-3-8B-V1: A High-Performance Biomedical Multimodal LLM},
	year = {2024},
	howpublished = {https://huggingface.co/ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1},
	}