Instructions to use HuggingFaceM4/idefics-9b-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use HuggingFaceM4/idefics-9b-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="HuggingFaceM4/idefics-9b-instruct")

# Load model directly
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("HuggingFaceM4/idefics-9b-instruct")
model = AutoModelForImageTextToText.from_pretrained("HuggingFaceM4/idefics-9b-instruct")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use HuggingFaceM4/idefics-9b-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "HuggingFaceM4/idefics-9b-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceM4/idefics-9b-instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/HuggingFaceM4/idefics-9b-instruct

SGLang

How to use HuggingFaceM4/idefics-9b-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "HuggingFaceM4/idefics-9b-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceM4/idefics-9b-instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "HuggingFaceM4/idefics-9b-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceM4/idefics-9b-instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use HuggingFaceM4/idefics-9b-instruct with Docker Model Runner:
```
docker model run hf.co/HuggingFaceM4/idefics-9b-instruct
```

what are the changes from `normalized=true` to `false` in `special_tokens_map.json`?

by luodian - opened Oct 3, 2023

Discussion

luodian

Oct 3, 2023

•

edited Oct 3, 2023

hi I noticed your configs changed and could I ask why did this change and what could it bring with normalized=false? Thanks!

luodian changed discussion title from what are the changes from `normalized=false` to `true` in `special_tokens_map.json`? to what are the changes from `normalized=true` to `false` in `special_tokens_map.json`? Oct 3, 2023

luodian

Oct 3, 2023

•

edited Oct 3, 2023

running the given example would produce

# normalized=true
User: What is in this image? 
Assistant: This picture depicts Idefix, the dog of Obelix in Asterix and Obelix. Idefix is running on the ground. 
User: And who is that? 
Assistant: That is a cartoon character from the Asterix comics, which is a popular French comic series created by René Goscinny and Albert Uderzo.

# normalized=false
User: What is in this image? 
Assistant: This picture depicts Idefix, the dog of Obelix in Asterix and Obelix. Idefix is running on the ground. 
User: And who is that? 
Assistant: The person in the image is Julius Caesar, a prominent Roman politician and military general in ancient Rome.

VictorSanh

Oct 3, 2023

@HugoLaurencon is the best person to comment!

HugoLaurencon

Oct 3, 2023

Hi @luodian

Can you tell me which version of transformers you are using?

If you are on the main branch and installed the repo from source, there has been a recent big change in tokenizers.

Essentially, if normalized=true, now the special tokens can be split into several sub-tokens, which is not something wanted.
For example, we trained the model using <fake_token_around_image><image><fake_token_around_image>, but here the token <image> could be split into < followed by image>.
In that case, we would not have the token <image>, and we would have no image attention mask or pixel values.
Note that if you are using the example code, we are dealing with these tokens for you in the processor script.

Could you try to see, with your version of transformers, how the prompt is tokenized with both normalized=true and normalized=false?
You need to write

from transformers import AutoProcessor

checkpoint = "HuggingFaceM4/idefics-9b"
processor = AutoProcessor.from_pretrained(checkpoint)
tokenizer = processor.tokenizer

prompt =  "<fake_token_around_image><image><fake_token_around_image>In this picture from Asterix and Obelix, we can see"  # Or a longer prompt
tokens = tokenizer.encode(prompt)
print(tokens)

Are you also using the base model or the instruct one?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment