源文件：llm/pixtral
在您的 Kubernetes 或您自己的云上私密部署 Pixtral#
▄▄▄░░
▄▄▄▄▄█████████░░░░
▄▄▄▄▄▄████████████████████░░░░░
█████████████████████████████░░░░░
▄▄▄▄▄▄█████░░░ █████████████████████████████░░░░░
▄▄▄▄▄██████████████████░░░░░░ ██████████████████████████████░░░░░
▄█████████████████████████████░░░░░░░░██████████████████████████████░░░░░
███████████████████████████████░░░░░░░██████████████████████████████░░░░░
███████████████████████████████░░░░░░░██████████████████████████████░░░░░
███████████████████████████████░░░░░░███████████████████████████████░░░░░
████████████████████████████████░░░░░███████████████████████████████░░░░░
████████████████████████████████░░░░████████████████████████████████░░░░░
█████████████████████████████████░░░████████████████████████████████░░░░░
█████████████████████████████████░░░████████████░███████████████████░░░░░
██████████████████████████████████░█████████████░███████████████████░░░░░
███████████████████░██████████████▄█████████████░███████████████████░░░░░
███████████████████░███████████████████████████░░███████████████████░░░░░
███████████████████░░██████████████████████████░░███████████████████░░░░░
███████████████████░░█████████████████████████░░░███████████████████░░░░░
███████████████████░░░████████████████████████░░░███████████████████░░░░░
███████████████████░░░████████████████████████░░░███████████████████░░░░░
███████████████████░░░░██████████████████████░░░░███████████████████░░░░░
███████████████████░░░░██████████████████████░░░░███████████████████░░░░░
███████████████████░░░░░█████████████████████░░░░███████████████████░░░░░
███████████████████░░░░░████████████████████░░░░░███████████████████░░░░░
███████████████████░░░░░░███████████████████░░░░░███████████████████░░░░░
███████████████████░░░░░░██████████████████░░░░░░███████████████████░░░░░
███████████████████░░░░░░░█████████████████░░░░░░███████████████████░░░░░
███████████████████░░░░░░░█████████████████░░░░░░███████████████████░░░░░
███████████████████░░░░░░░░███████████████░░░░░░░██████████░░░░░░░░░░░░░░
███████████████████░░░░░░░░███████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
███████████████████░░░░░░░░███████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
███████████████████░░░░░░░░░██░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
███████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
██████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ░░░░░░░
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ░░░
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ░░░░░░░░░░░░░░░░░░
░░░░░░░░░░░░░░░░░░░░░░░░░░░░
░░░░░░░░░░░░░░░░░
░░░░░
在 2024 年 9 月 11 日，Mistral 发布了新的 Pixtral 12B，这是他们的第一个多模态模型，支持文本和图像输入。
本指南展示了如何在您自己的云或 Kubernetes 集群上运行和部署此多模态模型。
在任何云或 Kubernetes 上运行 Pixtral#
在您的本地机器上安装 SkyPilot 并检查您的 Kubernetes 和云设置
pip install 'skypilot[all]'
sky check
安装和云设置的详细说明请在此处查看。
在任何云或 Kubernetes 上启动模型
sky launch -c pixtral pixtral.yaml
获取端点并发送请求
ENDPOINT=$(sky status --endpoint 8081 pixtral)
curl http://$ENDPOINT/v1/chat/completions \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer token' \
--data '{
"model": "mistralai/Pixtral-12B-2409",
"messages": [
{
"role": "user",
"content": [
{"type" : "text", "text": "Describe this image in detail please."},
{"type": "image_url", "image_url": {"url": "https://s3.amazonaws.com/cms.ipressroom.com/338/files/201808/5b894ee1a138352221103195_A680%7Ejogging-edit/A680%7Ejogging-edit_hero.jpg"}},
{"type" : "text", "text": "and this one as well."},
{"type": "image_url", "image_url": {"url": "https://www.wolframcloud.com/obj/resourcesystem/images/a0e/a0ee3983-46c6-4c92-b85d-059044639928/6af8cfb971db031b.png"}}
]
}],
"max_tokens": 1024
}' | jq .
在此示例中，我们将两张图片发送给模型并让它进行描述。
示例输出（解析版本）
Sure! Let me describe the images for you.
### Image 1:
This image shows three people jogging outdoors in a lush, green setting. The person on the left is a man wearing a light gray T-shirt and black shorts. He appears to be smiling and is actively running. The person in the middle is a woman with curly hair, dressed in a bright yellow tank top and black shorts. She also looks happy and is running alongside the man. The person on the right is another woman with long, wavy hair, wearing a light pink T-shirt and dark leggings. She is smiling and running as well. The background is filled with dense greenery, suggesting they are in a park or a forest.
### Image 2:
This image features a family of five posing together in a studio setting. The family members are all dressed in matching red outfits. From left to right, the first person is a woman with long blonde hair. Next to her is a young boy with light brown hair. The third person is a man with short dark hair and a mustache, smiling broadly. The fourth person is another young boy with dark hair and a slight smile. The last person is another woman with long blonde hair, mirroring the first woman. They are all laying on the floor, facing forward, with their hands clasped together in front of them. The background is plain white, focusing the attention on the family.
These descriptions should give you a clear picture of the scenes depicted in the images.
原始 JSON
{
"id": "chat-5733a2abfd664a019c7c61e38bb6603c",
"object": "chat.completion",
"created": 1726103777,
"model": "mistralai/Pixtral-12B-2409",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Sure! Let me describe the images for you.\n\n### Image 1:\nThis image shows three people jogging outdoors in a lush, green setting. The person on the left is a man wearing a light gray T-shirt and black shorts. He appears to be smiling and is actively running. The person in the middle is a woman with curly hair, dressed in a bright yellow tank top and black shorts. She also looks happy and is running alongside the man. The person on the right is another woman with long, wavy hair, wearing a light pink T-shirt and dark leggings. She is smiling and running as well. The background is filled with dense greenery, suggesting they are in a park or a forest.\n\n### Image 2:\nThis image features a family of five posing together in a studio setting. The family members are all dressed in matching red outfits. From left to right, the first person is a woman with long blonde hair. Next to her is a young boy with light brown hair. The third person is a man with short dark hair and a mustache, smiling broadly. The fourth person is another young boy with dark hair and a slight smile. The last person is another woman with long blonde hair, mirroring the first woman. They are all laying on the floor, facing forward, with their hands clasped together in front of them. The background is plain white, focusing the attention on the family.\n\nThese descriptions should give you a clear picture of the scenes depicted in the images.",
"tool_calls": []
},
"logprobs": null,
"finish_reason": "stop",
"stop_reason": null
}
],
"usage": {
"prompt_tokens": 4457,
"total_tokens": 4764,
"completion_tokens": 307
},
"prompt_logprobs": null
}
将 Pixtral 端点扩缩容为服务#
使用 SkyServe 启动服务
sky serve up -n pixtral pixtral.yaml
检查服务的状态
sky serve status pixtral
获取端点并发送请求
ENDPOINT=$(sky serve status --endpoint pixtral)
curl http://$ENDPOINT/v1/chat/completions \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer token' \
--data '{
"model": "mistralai/Pixtral-12B-2409",
"messages": [
{
"role": "user",
"content": [
{"type" : "text", "text": "Turn this logo into ASCII art."},
{"type": "image_url", "image_url": {"url": "https://pbs.twimg.com/profile_images/1584596138635632640/HWexMoH5_400x400.jpg"}}
]
}],
"max_tokens": 1024
}' | jq .
示例输出（解析版本）
Here's the logo of SkyPilot converted into ASCII art:
```
______
___// __\\____
/ __ \ __/ __\\
|_\ \_\\___ ____ ___\\
\/ \/ \/ \/
原始 JSON
{
"id": "chat-414fb85491ec42809f54a83845fdd629",
"object": "chat.completion",
"created": 1726109048,
"model": "mistralai/Pixtral-12B-2409",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Here's a simple ASCII representation of the Android logo:\n\n```\n ______\n ___// __\\\\____\n / __ \\ __/ __\\\\\n|_\\ \\_\\\\___ ____ ___\\\\\n \\/ \\/ \\/ \\/\n```",
"tool_calls": []
},
"logprobs": null,
"finish_reason": "stop",
"stop_reason": null
}
],
"usage": {
"prompt_tokens": 660,
"total_tokens": 716,
"completion_tokens": 56
},
"prompt_logprobs": null
}
包含的文件#
pixtral.yaml
envs:
MODEL_NAME: mistralai/Pixtral-12B-2409
HF_TOKEN:
service:
replicas: 2
# An actual request for readiness probe.
readiness_probe:
path: /v1/chat/completions
post_data:
model: $MODEL_NAME
messages:
- role: user
content:
- type: text
text: "Are you alive?"
max_tokens: 1
resources:
accelerators: {L40, L40S, A100, A100-80GB}
cpus: 2+
disk_tier: best
ports: 8081 # Expose to internet traffic.
setup: |
# Requires 0.6.1 for Pixtral support.
pip install vllm==0.6.1
run: |
echo 'Starting vllm api server...'
export OMP_NUM_THREADS=8
vllm serve $MODEL_NAME --tokenizer_mode mistral \
--limit_mm_per_prompt 'image=4' \
--max_num_batched_tokens 16384 \
--max-model-len 10240 \
--port 8081