Transformers documentation
CLI de Serve
CLI de Serve
La CLI transformers serve es una opción ligera para servidores locales o autoalojados. Evita el sobrecargo de tiempo de ejecución y operativo de los motores de inferencia dedicados, como vLLM. Úsela para evaluación, experimentación y despliegues con carga moderada. Funciones como continuous batching aumentan el rendimiento y reducen la latencia.
Para despliegues de producción a gran escala, use vLLM o SGLang con un modelo Transformer como backend. Obtenga más información en la guía de Backends de inferencia.
El comando transformers serve inicia un servidor local compatible con el OpenAI SDK. El servidor funciona con muchas aplicaciones de terceros y admite las API REST siguientes.
/v1/chat/completionspara solicitudes de texto, imagen, audio y video/v1/completionspara completados de texto heredados a partir de un prompt libre/v1/responsesadmite la Responses API/v1/audio/transcriptionspara transcripciones de audio/v1/systemoneresponde preguntas con formato (opción, puntuación, sí/no) con una probabilidad para cada respuesta permitida/v1/modelslista los modelos disponibles para integraciones de terceros/load_modeltransmite el progreso de la carga del modelo mediante SSE
Instale las dependencias de servicio.
pip install transformers[serving]
Ejecuta transformers serve para iniciar un servidor. La dirección predeterminada del servidor es http://localhost:8000.
transformers serve
v1/chat/completions
La API v1/chat/completions se basa en la Chat Completions API. Admite solicitudes de texto, imagen, audio y vídeo para LLMs, VLMs y modelos multimodales. Utilízala con curl, el InferenceClient o el cliente OpenAI.
Completiones basadas en texto
from huggingface_hub import InferenceClient
messages = [{"role": "user", "content": "What is the Transformers library known for?"}]
client = InferenceClient("http://localhost:8000")
result = client.chat_completion(messages, model="Qwen/Qwen2.5-0.5B-Instruct", max_tokens=256)
print(result.choices[0].message.content)El InferenceClient devuelve una cadena impresa.
The Transformers library is primarily known for its ability to create and manipulate large-scale language models [...]
from huggingface_hub import InferenceClient
messages = [{"role": "user", "content": "What is the Transformers library known for?"}]
client = InferenceClient("http://localhost:8000")
stream = client.chat_completion(messages, model="Qwen/Qwen2.5-0.5B-Instruct", max_tokens=256, stream=True)
for chunk in stream:
token = chunk.choices[0].delta.content
if token:
print(token, end="")El InferenceClient devuelve una cadena impresa.
The Transformers library is primarily known for its ability to create and manipulate large-scale language models [...]
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")
completion = client.chat.completions.create(
model="Qwen/Qwen2.5-0.5B-Instruct",
messages=[
{
"role": "user",
"content": "What is the Transformers library known for?"
}
],
)
print(completion.choices[0].message.content)El cliente de OpenAI devuelve una cadena impresa.
The Transformers library is primarily known for its ability to create and manipulate large-scale language models [...]
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")
completion = client.chat.completions.create(
model="Qwen/Qwen2.5-0.5B-Instruct",
messages=[
{
"role": "user",
"content": "What is the Transformers library known for?"
}
],
stream=True,
)
for chunk in completion:
token = chunk.choices[0].delta.content
if token:
print(token, end="")El cliente de OpenAI devuelve una cadena impresa.
The Transformers library is primarily known for its ability to create and manipulate large-scale language models [...]
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-0.5B-Instruct",
"messages": [{"role": "user", "content": "What is the Transformers library known for?"}],
"temperature": 0.9,
"max_tokens": 256
}'El comando devuelve una cadena JSON.
{
"id": "ade3578d-e2c3-47ed-b651-c014991a92f6",
"choices": [
{
"finish_reason": "length",
"index": 0,
"message": {
"content": "The Transformers library is primarily known for its ability to create and manipulate large-scale language models [...]",
"role": "assistant"
}
}
],
"created": 1776184004,
"model": "Qwen/Qwen2.5-0.5B-Instruct@main",
"object": "chat.completion",
"usage": {
"completion_tokens": 256,
"prompt_tokens": 37,
"total_tokens": 293
}
}curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-0.5B-Instruct",
"messages": [{"role": "user", "content": "What is the Transformers library known for?"}],
"temperature": 0.9,
"max_tokens": 256,
"stream": true
}'El comando devuelve una secuencia de fragmentos.
data: {"id":"596d3941-754d-43b4-8773-9bf9940049f5","choices":[{"delta":{"role":"assistant"},"index":0}],"created":1776183469,"model":"Qwen/Qwen2.5-0.5B-Instruct@main","object":"chat.completion.chunk","system_fingerprint":""}
data: {"id":"596d3941-754d-43b4-8773-9bf9940049f5","choices":[{"delta":{"content":"The"},"index":0}],"created":1776183469,"model":"Qwen/Qwen2.5-0.5B-Instruct@main","object":"chat.completion.chunk","system_fingerprint":""}
(...)Complementos basados en imágenes
from huggingface_hub import InferenceClient
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "What's in this image?"},
{
"type": "image_url",
"image_url": {
"url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg",
}
},
],
}
]
client = InferenceClient("http://localhost:8000")
result = client.chat_completion(messages, model="Qwen/Qwen2.5-VL-7B-Instruct", max_tokens=256)
print(result.choices[0].message.content)El InferenceClient devuelve una cadena impresa.
The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon, given the barren, rocky landscape and the dark sky in the background. The astronaut is holding a large egg that has cracked open, revealing a small creature inside. The scene is imaginative and playful, combining elements of space exploration with a whimsical twist involving the egg and the creature.from huggingface_hub import InferenceClient
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "What's in this image?"},
{
"type": "image_url",
"image_url": {
"url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg",
}
},
],
}
]
client = InferenceClient("http://localhost:8000")
stream = client.chat_completion(messages, model="Qwen/Qwen2.5-VL-7B-Instruct", max_tokens=256, stream=True)
for chunk in stream:
token = chunk.choices[0].delta.content
if token:
print(token, end="")El InferenceClient devuelve una cadena impresa.
The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon, given the barren, rocky landscape and the dark sky in the background. The astronaut is holding a large egg that has cracked open, revealing a small creature inside. The scene is imaginative and playful, combining elements of space exploration with a whimsical twist involving the egg and the creature.from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")
completion = client.chat.completions.create(
model="Qwen/Qwen2.5-VL-7B-Instruct",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "What's in this image?"},
{
"type": "image_url",
"image_url": {
"url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg",
}
},
],
}
],
)
print(completion.choices[0].message.content)El cliente de OpenAI devuelve una cadena impresa.
The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon, given the barren, rocky landscape and the dark sky in the background. The astronaut is holding a large egg that has cracked open, revealing a small creature inside. The scene is imaginative and playful, combining elements of space exploration with a whimsical twist involving the egg and the creature.from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")
completion = client.chat.completions.create(
model="Qwen/Qwen2.5-VL-7B-Instruct",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "What's in this image?"},
{
"type": "image_url",
"image_url": {
"url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg",
}
},
],
}
],
stream=True,
)
for chunk in completion:
token = chunk.choices[0].delta.content
if token:
print(token, end="")El cliente de OpenAI devuelve una cadena impresa.
The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon, given the barren, rocky landscape and the dark sky in the background. The astronaut is holding a large egg that has cracked open, revealing a small creature inside. The scene is imaginative and playful, combining elements of space exploration with a whimsical twist involving the egg and the creature.curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-VL-7B-Instruct",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "What's in this image?"
},
{
"type": "image_url",
"image_url": {
"url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg"
}
}
]
}
],
"max_tokens": 256
}'El comando devuelve una cadena JSON.
{
"id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"content": "The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon [...]",
"role": "assistant"
}
}
],
"created": 1753366665,
"model": "Qwen/Qwen2.5-VL-7B-Instruct@main",
"object": "chat.completion",
"usage": {
"completion_tokens": 50,
"prompt_tokens": 120,
"total_tokens": 170
}
}curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-VL-7B-Instruct",
"stream": true,
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "What's in this image?"
},
{
"type": "image_url",
"image_url": {
"url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg"
}
}
]
}
],
"max_tokens": 256
}'El comando devuelve una secuencia de fragmentos.
data: {"id":"f47ac10b-58cc-4372-a567-0e02b2c3d479","choices":[{"delta":{"role":"assistant"},"index":0}],"created":1753366665,"model":"Qwen/Qwen2.5-VL-7B-Instruct@main","object":"chat.completion.chunk","system_fingerprint":""}
data: {"id":"f47ac10b-58cc-4372-a567-0e02b2c3d479","choices":[{"delta":{"content":"The "},"index":0}],"created":1753366701,"model":"Qwen/Qwen2.5-VL-7B-Instruct@main","object":"chat.completion.chunk","system_fingerprint":""}
(...)Complementos basados en audio
Los modelos multimodales como Gemma 4 y Qwen2.5-Omni aceptan entrada de audio a través del tipo de contenido input_audio de OpenAI. Codifica el audio en Base64 y especifica el formato (mp3 o wav).
import base64
from huggingface_hub.utils import httpx
from huggingface_hub import InferenceClient
audio_url = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"
audio_b64 = base64.b64encode(httpx.get(audio_url, follow_redirects=True).content).decode()
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe this audio."},
{"type": "input_audio", "input_audio": {"data": audio_b64, "format": "mp3"}},
],
}
]
client = InferenceClient("http://localhost:8000")
result = client.chat_completion(messages, model="google/gemma-4-E2B-it", max_tokens=256)
print(result.choices[0].message.content)El InferenceClient devuelve una cadena impresa.
This week, I traveled to Chicago to deliver my final farewell address to the nation, following in the tradition of presidents before me. It was an opportunity to say thank you. Whether we've seen eye to eye or rarely agreed at all, my conversations with you, the American people, in living rooms and schools, at farms and on factory floors, at diners, and on distant military outposts, all these conversations are what have kept me honest.import base64
from huggingface_hub.utils import httpx
from huggingface_hub import InferenceClient
audio_url = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"
audio_b64 = base64.b64encode(httpx.get(audio_url, follow_redirects=True).content).decode()
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe this audio."},
{"type": "input_audio", "input_audio": {"data": audio_b64, "format": "mp3"}},
],
}
]
client = InferenceClient("http://localhost:8000")
stream = client.chat_completion(messages, model="google/gemma-4-E2B-it", max_tokens=256, stream=True)
for chunk in stream:
token = chunk.choices[0].delta.content
if token:
print(token, end="")El InferenceClient devuelve una cadena impresa.
This week, I traveled to Chicago to deliver my final farewell address to the nation, following in the tradition of presidents before me. It was an opportunity to say thank you. Whether we've seen eye to eye or rarely agreed at all, my conversations with you, the American people, in living rooms and schools, at farms and on factory floors, at diners, and on distant military outposts, all these conversations are what have kept me honest.import base64
from huggingface_hub.utils import httpx
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")
audio_url = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"
audio_b64 = base64.b64encode(httpx.get(audio_url, follow_redirects=True).content).decode()
completion = client.chat.completions.create(
model="google/gemma-4-E2B-it",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe this audio."},
{"type": "input_audio", "input_audio": {"data": audio_b64, "format": "mp3"}},
],
}
],
)
print(completion.choices[0].message.content)El cliente de OpenAI devuelve una cadena impresa.
This week, I traveled to Chicago to deliver my final farewell address to the nation, following in the tradition of presidents before me. It was an opportunity to say thank you. Whether we've seen eye to eye or rarely agreed at all, my conversations with you, the American people, in living rooms and schools, at farms and on factory floors, at diners, and on distant military outposts, all these conversations are what have kept me honest.import base64
from huggingface_hub.utils import httpx
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")
audio_url = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"
audio_b64 = base64.b64encode(httpx.get(audio_url, follow_redirects=True).content).decode()
completion = client.chat.completions.create(
model="google/gemma-4-E2B-it",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe this audio."},
{"type": "input_audio", "input_audio": {"data": audio_b64, "format": "mp3"}},
],
}
],
stream=True,
)
for chunk in completion:
token = chunk.choices[0].delta.content
if token:
print(token, end="")El cliente de OpenAI devuelve una cadena impresa.
This week, I traveled to Chicago to deliver my final farewell address to the nation, following in the tradition of presidents before me. It was an opportunity to say thank you. Whether we've seen eye to eye or rarely agreed at all, my conversations with you, the American people, in living rooms and schools, at farms and on factory floors, at diners, and on distant military outposts, all these conversations are what have kept me honest.# First, base64-encode the audio file and build the JSON payload
AUDIO_B64=$(curl -sL https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3 | base64 -w 0)
cat <<EOF > /tmp/audio_request.json
{
"model": "google/gemma-4-E2B-it",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe this audio."},
{"type": "input_audio", "input_audio": {"data": "$AUDIO_B64", "format": "mp3"}}
]
}
]
}
EOF
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d @/tmp/audio_request.jsonEl comando devuelve una cadena JSON.
{
"id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"content": "This week, I traveled to Chicago to deliver my final farewell address to the nation [...]",
"role": "assistant"
}
}
],
"created": 1753366665,
"model": "google/gemma-4-E2B-it@main",
"object": "chat.completion",
"usage": {
"completion_tokens": 85,
"prompt_tokens": 200,
"total_tokens": 285
}
}# First, base64-encode the audio file and build the JSON payload
AUDIO_B64=$(curl -sL https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3 | base64 -w 0)
cat <<EOF > /tmp/audio_request.json
{
"model": "google/gemma-4-E2B-it",
"stream": true,
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe this audio."},
{"type": "input_audio", "input_audio": {"data": "$AUDIO_B64", "format": "mp3"}}
]
}
]
}
EOF
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d @/tmp/audio_request.jsonEl comando devuelve una secuencia de fragmentos.
data: {"id":"cb997e1d-98b9-414a-be89-1880288610ef","choices":[{"delta":{"role":"assistant"},"index":0}],"created":1776254856,"model":"google/gemma-4-E2B-it@main","object":"chat.completion.chunk","system_fingerprint":""}
data: {"id":"cb997e1d-98b9-414a-be89-1880288610ef","choices":[{"delta":{"content":"This"},"index":0}],"created":1776254856,"model":"google/gemma-4-E2B-it@main","object":"chat.completion.chunk","system_fingerprint":""}
(...)El tipo de contenido
audio_urles una extensión que no forma parte del estándar de OpenAI y puede cambiar en versiones futuras.
También puede pasar audio por URL con el tipo de contenido audio_url para omitir la codificación base64.
completion = client.chat.completions.create(
model="google/gemma-4-E2B-it",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe this audio."},
{"type": "audio_url", "audio_url": {"url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"}},
],
}
],
)Complementos basados en video
El tipo de contenido
video_urles una extensión que no forma parte del estándar de OpenAI y puede cambiar en versiones futuras.
Use el tipo de contenido video_url para la entrada de video. Si el modelo admite audio (p. ej., Gemma 4, Qwen2.5-Omni), el servidor extrae la pista de audio del video y la procesa junto con los fotogramas visuales.
El procesamiento de vídeo requiere torchcodec. Instálalo con
pip install torchcodec.
from huggingface_hub import InferenceClient
messages = [
{
"role": "user",
"content": [
{"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
{"type": "text", "text": "What is happening in the video and what is the song about?"},
],
}
]
client = InferenceClient("http://localhost:8000")
result = client.chat_completion(messages, model="google/gemma-4-E2B-it", max_tokens=256)
print(result.choices[0].message.content)El InferenceClient devuelve una cadena impresa.
The video captures a live music performance at a music festival or a large concert. There are several musicians on stage, including a central figure playing an acoustic guitar and singing. The foreground is filled with the backs of the audience, indicating a large crowd watching the show. The stage is dramatically lit with bright spotlights and blue and white stage lighting, with haze and smoke creating an immersive atmosphere.
The lyrics of the song are: "I don't care 'bout street, from that fresh street, 'cause there's no problem, another one I want to be, in the storm..."from huggingface_hub import InferenceClient
messages = [
{
"role": "user",
"content": [
{"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
{"type": "text", "text": "What is happening in the video and what is the song about?"},
],
}
]
client = InferenceClient("http://localhost:8000")
stream = client.chat_completion(messages, model="google/gemma-4-E2B-it", max_tokens=256, stream=True)
for chunk in stream:
token = chunk.choices[0].delta.content
if token:
print(token, end="")El InferenceClient devuelve una cadena impresa.
The video captures a live music performance at a music festival or a large concert. There are several musicians on stage, including a central figure playing an acoustic guitar and singing. The foreground is filled with the backs of the audience, indicating a large crowd watching the show. The stage is dramatically lit with bright spotlights and blue and white stage lighting, with haze and smoke creating an immersive atmosphere.
The lyrics of the song are: "I don't care 'bout street, from that fresh street, 'cause there's no problem, another one I want to be, in the storm..."from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")
completion = client.chat.completions.create(
model="google/gemma-4-E2B-it",
messages=[
{
"role": "user",
"content": [
{"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
{"type": "text", "text": "What is happening in the video and what is the song about?"},
],
}
],
max_tokens=256,
)
print(completion.choices[0].message.content)El cliente de OpenAI devuelve una cadena impresa.
The video captures a live music performance at a music festival or a large concert. There are several musicians on stage, including a central figure playing an acoustic guitar and singing. The foreground is filled with the backs of the audience, indicating a large crowd watching the show. The stage is dramatically lit with bright spotlights and blue and white stage lighting, with haze and smoke creating an immersive atmosphere.
The lyrics of the song are: "I don't care 'bout street, from that fresh street, 'cause there's no problem, another one I want to be, in the storm..."from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")
completion = client.chat.completions.create(
model="google/gemma-4-E2B-it",
messages=[
{
"role": "user",
"content": [
{"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
{"type": "text", "text": "What is happening in the video and what is the song about?"},
],
}
],
max_tokens=256,
stream=True,
)
for chunk in completion:
token = chunk.choices[0].delta.content
if token:
print(token, end="")El cliente de OpenAI devuelve una cadena impresa.
The video captures a live music performance at a music festival or a large concert. There are several musicians on stage, including a central figure playing an acoustic guitar and singing. The foreground is filled with the backs of the audience, indicating a large crowd watching the show. The stage is dramatically lit with bright spotlights and blue and white stage lighting, with haze and smoke creating an immersive atmosphere.
The lyrics of the song are: "I don't care 'bout street, from that fresh street, 'cause there's no problem, another one I want to be, in the storm..."curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-4-E2B-it",
"messages": [
{
"role": "user",
"content": [
{"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
{"type": "text", "text": "What is happening in the video and what is the song about?"}
]
}
],
"max_tokens": 256
}'El comando devuelve una cadena JSON.
{
"id": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
"choices": [
{
"finish_reason": "length",
"index": 0,
"message": {
"content": "The video captures a live music performance at a music festival or a large concert [...]",
"role": "assistant"
}
}
],
"created": 1753366665,
"model": "google/gemma-4-E2B-it@main",
"object": "chat.completion",
"usage": {
"completion_tokens": 256,
"prompt_tokens": 350,
"total_tokens": 606
}
}curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-4-E2B-it",
"stream": true,
"messages": [
{
"role": "user",
"content": [
{"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
{"type": "text", "text": "What is happening in the video and what is the song about?"}
]
}
],
"max_tokens": 256
}'El comando devuelve una secuencia de fragmentos.
data: {"id":"cb997e1d-98b9-414a-be89-1880288610ef","choices":[{"delta":{"role":"assistant"},"index":0}],"created":1776254856,"model":"google/gemma-4-E2B-it@main","object":"chat.completion.chunk","system_fingerprint":""}
data: {"id":"cb997e1d-98b9-414a-be89-1880288610ef","choices":[{"delta":{"content":"Based"},"index":0}],"created":1776254856,"model":"google/gemma-4-E2B-it@main","object":"chat.completion.chunk","system_fingerprint":""}
(...)Conversaciones de varios turnos
Para mantener una conversación de varios turnos, incluya el historial completo de la conversación en la lista messages con roles alternos de user y assistant. Al igual que todos los servidores compatibles con OpenAI, la API es sin estado, por lo que cada solicitud debe contener el historial completo de la conversación.
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")
completion = client.chat.completions.create(
model="Qwen/Qwen2.5-0.5B-Instruct",
messages=[
{"role": "user", "content": "What is the capital of France?"},
{"role": "assistant", "content": "The capital of France is Paris."},
{"role": "user", "content": "How many people live there?"},
],
)
print(completion.choices[0].message.content)La pregunta posterior «¿Cuántas personas viven allí?» depende del contexto previo, por lo que el modelo responde sobre París.
As of 2021, the population of Paris is approximately 2.2 million people.v1/completions
La API v1/completions se basa en la API de Completions heredada. A diferencia de /v1/chat/completions, recibe un prompt de texto libre en lugar de mensajes de chat y devuelve el texto generado en choices[].text. Esto resulta útil para modelos base (no instruct) y tareas de completado de texto donde no se necesita una plantilla de chat. También admite suffix para la inserción de texto en el medio.
curl -X POST http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-0.5B",
"prompt": "The capital of France is",
"max_tokens": 20
}'El comando devuelve la siguiente respuesta.
{
"id": "chatcmpl-abc123",
"object": "text_completion",
"created": 1234567890,
"model": "Qwen/Qwen2.5-0.5B@main",
"choices": [
{
"text": " Paris, and the capital of the United States is Washington, D.C.",
"index": 0,
"logprobs": null,
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 5,
"completion_tokens": 16,
"total_tokens": 21
}
}from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")
# Non-streaming
completion = client.completions.create(
model="Qwen/Qwen2.5-0.5B",
prompt="The capital of France is",
max_tokens=20,
)
print(completion.choices[0].text)El cliente de OpenAI devuelve lo siguiente.
Paris, and the capital of the United States is Washington, D.C.
También se admite la transmisión en streaming.
stream = client.completions.create(
model="Qwen/Qwen2.5-0.5B",
prompt="The capital of France is",
max_tokens=20,
stream=True,
)
for chunk in stream:
print(chunk.choices[0].text, end="")v1/responses
La Responses API es el punto de acceso de API más reciente de OpenAI para la generación. Admite interacciones con estado e integra herramientas integradas para ampliar las capacidades de un modelo. OpenAI recomienda usar la Responses API en lugar de la Chat Completions API para nuevos proyectos.
La API v1/responses admite solicitudes de texto, imagen, audio y vídeo a través del comando curl y el cliente OpenAI.
Respuestas basadas en texto
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")
response = client.responses.create(
model="Qwen/Qwen2.5-0.5B-Instruct",
input="Tell me a three sentence bedtime story about a unicorn.",
max_output_tokens=256,
stream=False,
)
print(response.output[0].content[0].text)El cliente de OpenAI devuelve una cadena impresa.
Once upon a time, in a faraway land, there lived a beautiful unicorn named Luna [...]
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")
response = client.responses.create(
model="Qwen/Qwen2.5-0.5B-Instruct",
input="Tell me a three sentence bedtime story about a unicorn.",
max_output_tokens=256,
stream=True,
)
for event in response:
if hasattr(event, "delta") and event.delta:
print(event.delta, end="")El cliente de OpenAI devuelve una cadena impresa.
Once upon a time, in a faraway land, there lived a beautiful unicorn named Luna [...]
curl http://localhost:8000/v1/responses \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-0.5B-Instruct",
"input": "Tell me a three sentence bedtime story about a unicorn.",
"max_output_tokens": 256,
"stream": false
}'El comando devuelve una cadena JSON.
{
"id": "resp_8693af9b-b561-4267-bd1a-22d1b43284cb",
"created_at": 1776258580.7114341,
"model": "Qwen/Qwen2.5-0.5B-Instruct@main",
"object": "response",
"output": [
{
"id": "msg_8693af9b-b561-4267-bd1a-22d1b43284cb",
"content": [
{
"annotations": [],
"text": "Once upon a time, in a faraway land, there lived a beautiful unicorn named Luna [...]",
"type": "output_text"
}
],
"role": "assistant",
"status": "completed",
"type": "message"
}
],
"status": "completed"
}curl http://localhost:8000/v1/responses \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-0.5B-Instruct",
"stream": true,
"input": "Tell me a three sentence bedtime story about a unicorn.",
"max_output_tokens": 256
}'El comando devuelve una secuencia de fragmentos.
data: {"response":{"id":"resp_req_0","created_at":1754059817.783648,"model":"Qwen/Qwen2.5-0.5B-Instruct@main","object":"response","output":[],"status":"queued"},"sequence_number":0,"type":"response.created"}
data: {"content_index":0,"delta":"Once ","item_id":"msg_req_0","output_index":0,"sequence_number":5,"type":"response.output_text.delta"}
data: {"content_index":0,"delta":"upon ","item_id":"msg_req_0","output_index":0,"sequence_number":6,"type":"response.output_text.delta"}
(...)Respuestas basadas en imágenes
La API de Responses también admite entradas de imagen, audio y video.
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")
response = client.responses.create(
model="Qwen/Qwen2.5-VL-7B-Instruct",
input=[
{"type": "input_text", "text": "What's in this image?"},
{
"type": "input_image",
"image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg",
},
],
max_output_tokens=256,
stream=False,
)
print(response.output[0].content[0].text)El cliente de OpenAI devuelve una cadena impresa.
The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon, given the barren, rocky landscape and the dark sky in the background. The astronaut is holding a large egg that has cracked open, revealing a small creature inside. The scene is imaginative and playful, combining elements of space exploration with a whimsical twist involving the egg and the creature.from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")
response = client.responses.create(
model="Qwen/Qwen2.5-VL-7B-Instruct",
input=[
{"type": "input_text", "text": "What's in this image?"},
{
"type": "input_image",
"image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg",
},
],
max_output_tokens=256,
stream=True,
)
for event in response:
if hasattr(event, "delta") and event.delta:
print(event.delta, end="")El cliente de OpenAI devuelve una cadena impresa.
The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon, given the barren, rocky landscape and the dark sky in the background. The astronaut is holding a large egg that has cracked open, revealing a small creature inside. The scene is imaginative and playful, combining elements of space exploration with a whimsical twist involving the egg and the creature.curl http://localhost:8000/v1/responses \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-VL-7B-Instruct",
"input": [
{"type": "input_text", "text": "What'\''s in this image?"},
{"type": "input_image", "image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg"}
],
"max_output_tokens": 256,
"stream": false
}'El comando devuelve una cadena JSON.
{
"id": "resp_f47ac10b-58cc-4372-a567-0e02b2c3d479",
"created_at": 1776258580.7114341,
"model": "Qwen/Qwen2.5-VL-7B-Instruct@main",
"object": "response",
"output": [
{
"id": "msg_f47ac10b-58cc-4372-a567-0e02b2c3d479",
"content": [
{
"annotations": [],
"text": "The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon [...]",
"type": "output_text"
}
],
"role": "assistant",
"status": "completed",
"type": "message"
}
],
"status": "completed"
}curl http://localhost:8000/v1/responses \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-VL-7B-Instruct",
"stream": true,
"input": [
{"type": "input_text", "text": "What'\''s in this image?"},
{"type": "input_image", "image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg"}
],
"max_output_tokens": 256
}'El comando devuelve una secuencia de fragmentos.
data: {"response":{"id":"resp_f47ac10b-58cc-4372-a567-0e02b2c3d479","created_at":1776258580.7114341,"model":"Qwen/Qwen2.5-VL-7B-Instruct@main","object":"response","output":[],"status":"queued"},"sequence_number":0,"type":"response.created"}
data: {"content_index":0,"delta":"The ","item_id":"msg_f47ac10b","output_index":0,"sequence_number":5,"type":"response.output_text.delta"}
(...)Respuestas basadas en audio
import base64
from huggingface_hub.utils import httpx
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")
audio_url = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"
audio_b64 = base64.b64encode(httpx.get(audio_url, follow_redirects=True).content).decode()
response = client.responses.create(
model="google/gemma-4-E2B-it",
input=[
{"type": "input_text", "text": "Transcribe this audio."},
{"type": "input_audio", "input_audio": {"data": audio_b64, "format": "mp3"}},
],
max_output_tokens=256,
stream=False,
)
print(response.output[0].content[0].text)El cliente de OpenAI devuelve una cadena impresa.
This week, I traveled to Chicago to deliver my final farewell address to the nation, following in the tradition of presidents before me. It was an opportunity to say thank you. Whether we've seen eye to eye or rarely agreed at all, my conversations with you, the American people, in living rooms and schools, at farms and on factory floors, at diners, and on distant military outposts, all these conversations are what have kept me honest.import base64
from huggingface_hub.utils import httpx
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")
audio_url = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"
audio_b64 = base64.b64encode(httpx.get(audio_url, follow_redirects=True).content).decode()
response = client.responses.create(
model="google/gemma-4-E2B-it",
input=[
{"type": "input_text", "text": "Transcribe this audio."},
{"type": "input_audio", "input_audio": {"data": audio_b64, "format": "mp3"}},
],
}
],
max_output_tokens=256,
stream=True,
)
for event in response:
if hasattr(event, "delta") and event.delta:
print(event.delta, end="")El cliente de OpenAI devuelve una cadena impresa.
This week, I traveled to Chicago to deliver my final farewell address to the nation, following in the tradition of presidents before me. It was an opportunity to say thank you. Whether we've seen eye to eye or rarely agreed at all, my conversations with you, the American people, in living rooms and schools, at farms and on factory floors, at diners, and on distant military outposts, all these conversations are what have kept me honest.# First, base64-encode the audio file and build the JSON payload
AUDIO_B64=$(curl -sL https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3 | base64 -w 0)
cat <<EOF > /tmp/audio_request.json
{
"model": "google/gemma-4-E2B-it",
"input": [
{"type": "input_text", "text": "Transcribe this audio."},
{"type": "input_audio", "input_audio": {"data": "$AUDIO_B64", "format": "mp3"}}
],
"max_output_tokens": 256,
"stream": false
}
EOF
curl http://localhost:8000/v1/responses \
-H "Content-Type: application/json" \
-d @/tmp/audio_request.jsonEl comando devuelve una cadena JSON.
{
"id": "resp_a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"created_at": 1776258580.7114341,
"model": "google/gemma-4-E2B-it@main",
"object": "response",
"output": [
{
"id": "msg_a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"content": [
{
"annotations": [],
"text": "This week, I traveled to Chicago to deliver my final farewell address to the nation [...]",
"type": "output_text"
}
],
"role": "assistant",
"status": "completed",
"type": "message"
}
],
"status": "completed"
}# First, base64-encode the audio file and build the JSON payload
AUDIO_B64=$(curl -sL https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3 | base64 -w 0)
cat <<EOF > /tmp/audio_request.json
{
"model": "google/gemma-4-E2B-it",
"stream": true,
"input": [
{"type": "input_text", "text": "Transcribe this audio."},
{"type": "input_audio", "input_audio": {"data": "$AUDIO_B64", "format": "mp3"}}
],
"max_output_tokens": 256
}
EOF
curl http://localhost:8000/v1/responses \
-H "Content-Type: application/json" \
-d @/tmp/audio_request.jsonEl comando devuelve una secuencia de fragmentos.
data: {"response":{"id":"resp_a1b2c3d4-e5f6-7890-abcd-ef1234567890","created_at":1776258580.7114341,"model":"google/gemma-4-E2B-it@main","object":"response","output":[],"status":"queued"},"sequence_number":0,"type":"response.created"}
data: {"content_index":0,"delta":"This ","item_id":"msg_a1b2c3d4","output_index":0,"sequence_number":5,"type":"response.output_text.delta"}
(...)El tipo de contenido
audio_urles una extensión que no forma parte del estándar de OpenAI y puede cambiar en versiones futuras.
También puede pasar audio por URL con el tipo de contenido audio_url para omitir la codificación base64.
response = client.responses.create(
model="google/gemma-4-E2B-it",
input=[
{"type": "input_text", "text": "Transcribe this audio."},
{"type": "audio_url", "audio_url": {"url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"}},
],
max_output_tokens=256,
stream=False,
)Respuestas basadas en video
El tipo de contenido
video_urles una extensión que no forma parte del estándar de OpenAI y puede cambiar en versiones futuras.
El procesamiento de vídeo requiere torchcodec. Instálalo con
pip install torchcodec.
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")
response = client.responses.create(
model="google/gemma-4-E2B-it",
input=[
{"type": "input_text", "text": "What is happening in the video and what is the song about?"},
{"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
],
max_output_tokens=256,
stream=False,
)
print(response.output[0].content[0].text)El cliente de OpenAI devuelve una cadena impresa.
The video captures a live music performance at a music festival or a large concert. There are several musicians on stage, including a central figure playing an acoustic guitar and singing. The foreground is filled with the backs of the audience, indicating a large crowd watching the show. The stage is dramatically lit with bright spotlights and blue and white stage lighting, with haze and smoke creating an immersive atmosphere.
The lyrics of the song are: "I don't care 'bout street, from that fresh street, 'cause there's no problem, another one I want to be, in the storm..."from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")
response = client.responses.create(
model="google/gemma-4-E2B-it",
input=[
{"type": "input_text", "text": "What is happening in the video and what is the song about?"},
{"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
],
max_output_tokens=256,
stream=True,
)
for event in response:
if hasattr(event, "delta") and event.delta:
print(event.delta, end="")El cliente de OpenAI devuelve una cadena impresa.
The video captures a live music performance at a music festival or a large concert. There are several musicians on stage, including a central figure playing an acoustic guitar and singing. The foreground is filled with the backs of the audience, indicating a large crowd watching the show. The stage is dramatically lit with bright spotlights and blue and white stage lighting, with haze and smoke creating an immersive atmosphere.
The lyrics of the song are: "I don't care 'bout street, from that fresh street, 'cause there's no problem, another one I want to be, in the storm..."curl http://localhost:8000/v1/responses \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-4-E2B-it",
"input": [
{"type": "input_text", "text": "What is happening in the video and what is the song about?"},
{"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}}
],
"max_output_tokens": 256,
"stream": false
}'El comando devuelve una cadena JSON.
{
"id": "resp_b2c3d4e5-f6a7-8901-bcde-f12345678901",
"created_at": 1776258580.7114341,
"model": "google/gemma-4-E2B-it@main",
"object": "response",
"output": [
{
"id": "msg_b2c3d4e5-f6a7-8901-bcde-f12345678901",
"content": [
{
"annotations": [],
"text": "The video captures a live music performance at a music festival or a large concert [...]",
"type": "output_text"
}
],
"role": "assistant",
"status": "completed",
"type": "message"
}
],
"status": "completed"
}curl http://localhost:8000/v1/responses \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-4-E2B-it",
"stream": true,
"input": [
{"type": "input_text", "text": "What is happening in the video and what is the song about?"},
{"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}}
],
"max_output_tokens": 256
}'El comando devuelve una secuencia de fragmentos.
data: {"response":{"id":"resp_b2c3d4e5-f6a7-8901-bcde-f12345678901","created_at":1776258580.7114341,"model":"google/gemma-4-E2B-it@main","object":"response","output":[],"status":"queued"},"sequence_number":0,"type":"response.created"}
data: {"content_index":0,"delta":"Based ","item_id":"msg_b2c3d4e5","output_index":0,"sequence_number":5,"type":"response.output_text.delta"}
(...)Conversaciones de varios turnos
Para conversaciones de varios turnos, pase una lista de mensajes con claves role en el campo input. Al igual que todos los servidores compatibles con OpenAI, la API es sin estado, por lo que cada solicitud debe contener el historial completo de la conversación.
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")
response = client.responses.create(
model="Qwen/Qwen2.5-0.5B-Instruct",
input=[
{"role": "user", "content": "What is the capital of France?"},
{"role": "assistant", "content": "The capital of France is Paris."},
{"role": "user", "content": "How many people live there?"},
],
max_output_tokens=256,
stream=False,
)
print(response.output[0].content[0].text)La pregunta posterior «¿Cuántas personas viven allí?» depende del contexto previo, por lo que el modelo responde sobre París.
As of 2021, Paris has a population of approximately 2.8 million people.v1/systemone
El punto de acceso v1/systemone responde a preguntas tipadas sobre un state compartido, siguiendo la System One API de TypeSafe. Úsalo para decisiones como enrutar un ticket de soporte, calificar la urgencia o elegir una acción.
Las solicitudes son respondidas por DecisionPipeline alrededor del modelo servido, por lo que el punto de acceso y la pipeline construyen los mismos prompts y devuelven las mismas respuestas. El modelo puntúa las respuestas permitidas sin generar texto, en una sola pasada por delante por prompt. Este punto de acceso admite modelos autoregresivos con una plantilla de chat;
Envía un modelo, el estado a evaluar y preguntas indexadas por nombre:
curl http://localhost:8000/v1/systemone \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-1.5B-Instruct",
"state": "Our integration has been failing for 3 days and we are losing sales!",
"questions": {
"team": {
"type": "choice",
"instructions": "Which team handles this?",
"criteria": {"billing": null, "technical": "Bugs or integration problems"}
},
"urgency": {
"type": "score",
"instructions": "How urgent is this ticket?",
"criteria": ["low", "medium", "high"]
},
"upset": {"type": "noul", "instructions": "Is the customer upset?"}
}
}'Instale el TypeSafe SDK para Node.js 20 o una versión más reciente:
npm install @typesafe-ai/sdk
Establezca baseURL en la raíz del servidor; el SDK añade /v1/systemone. El SDK requiere una clave de API, pero un marcador de posición funciona con el servidor local.
import { choice, noul, score, TypeSafeClient } from "@typesafe-ai/sdk";
const client = new TypeSafeClient({
baseURL: "http://localhost:8000",
apiKey: "<random_string>",
});
const response = await client.systemOne({
model: "Qwen/Qwen2.5-1.5B-Instruct",
state: "Our integration has been failing for 3 days and we are losing sales!",
questions: {
team: choice("Which team handles this?", {
billing: null,
technical: "Bugs or integration problems",
}),
urgency: score("How urgent is this ticket?", ["low", "medium", "high"]),
upset: noul("Is the customer upset?"),
},
});
console.log(response.answers.team.choice);
console.log(response.answers.urgency.score);
console.log(response.answers.upset.noul);El cliente imprime el equipo seleccionado, la puntuación de urgencia y la probabilidad de que el cliente esté molesto. Por ejemplo:
technical 1.6739138662815094 0.851952850818634
Las respuestas usan los mismos nombres de pregunta. Por ejemplo:
{
"model": "Qwen/Qwen2.5-1.5B-Instruct",
"answers": {
"team": {"type": "choice", "choice": "technical", "probabilities": {"billing": 0.000006962256520637311, "technical": 0.999993085861206}, "confidence": 0.9999861717224121},
"urgency": {"type": "score", "score": 1.6739138662815094, "legend": {"0": "low", "1": "medium", "2": "high"}, "probabilities": {"0": 0.0031353600788861513, "1": 0.31981542706489563, "2": 0.6770492196083069}, "confidence": 0.5108707791659981},
"upset": {"type": "noul", "noul": 0.851952850818634}
},
"usage": {"input_tokens": 213, "output_tokens": 0}
}| Tipo de pregunta | criteria | Respuesta |
|---|---|---|
choice | Nombres de opciones mapeados a descripciones, que pueden ser null. | La opción más probable, las probabilidades de las opciones y la confianza. |
score | Descripciones de niveles ordenadas de menor a mayor. | El índice de nivel ponderado por probabilidad, las probabilidades, una leyenda y la confianza. |
noul | Descripciones opcionales para true y false. | La probabilidad de true. |
state, instrucciones opcionales instructions y descripciones pueden contener texto u objetos y matrices JSON. Cada solicitud acepta hasta 64 preguntas, con al menos una opción o nivel de puntuación por pregunta y como máximo una por etiqueta de respuesta (26 con las etiquetas predeterminadas A–Z).
Las preguntas se validan como las de DecisionPipeline, antes de que se cargue el modelo. Una solicitud no válida devuelve un 422 cuyo mensaje enumera cada pregunta no válida, cada una con un ejemplo de una válida, y sugiere coincidencias cercanas para tipos y campos con errores de escritura:
Question 'team': expected a `type` among noul, choice, score, got type 'Choice'. Did you mean 'choice'?
Question 'upset': unknown field 'instruction'. Did you mean 'instructions'? A noul question looks like {"type": "noul", ...}Para los modelos con capacidad de visión, añade assets, como en la API de System One: hasta cuatro imágenes indexadas por un marcador de posición de tu elección (un UUID es lo más seguro). En cualquier lugar donde aparezca un marcador de posición en una cadena de state, en las instructions o en los criteria, el modelo ve la imagen en ese punto. Un asset es {"type": "image", "format": "png", "data_base64": "..."}; no se aceptan URLs de imágenes remotas. El modelo necesita un procesador que maneje imágenes.
{
"model": "Qwen/Qwen2.5-VL-3B-Instruct",
"state": "The customer sent 6f1c0e2a and asks if it is damaged.",
"assets": {"6f1c0e2a": {"type": "image", "format": "png", "data_base64": "iVBORw0..."}},
"questions": {"damaged": {"type": "noul", "instructions": "Is the product in the photo damaged?"}}
}Configuración de decisión
Los modelos de decisión aún no comparten un estándar para almacenar los prompts, las etiquetas de respuesta y la configuración de calibración. Estos varían según la implementación del modelo, por lo que --decision-config permite especificarlos explícitamente.
Por defecto, las preguntas utilizan un prompt de respaldo y la plantilla de chat del modelo con el razonamiento desactivado. Las etiquetas de respuesta son A–Z para opciones y puntuaciones, y Yes/No para valores booleanos. Las probabilidades son el softmax sobre estas etiquetas a temperatura 1.
Pasa --decision-config para sobrescribir estos valores predeterminados para un modelo:
transformers serve openjev/openjev --decision-config decision.json
El archivo JSON admite tres campos opcionales:
| Campo | Significado |
|---|---|
labels | Etiquetas de respuesta por tipo de pregunta, en el orden de las opciones (true, luego false para noul). Cada una debe ser un único token conocido y distinto. |
temperature | Temperaturas positivas por tipo de pregunta, aplicadas como softmax(label_logits / temperature). |
chat_template | Una plantilla Jinja en línea que reemplaza el prompt completo, incluyendo el formato de chat y el prefijo del asistente. |
Por ejemplo, para cambiar las etiquetas y la calibración manteniendo el prompt predeterminado:
{
"labels": {"noul": ["A", "B"]},
"temperature": {"choice": 0.85, "score": 0.85, "noul": 1.5547129}
}Una plantilla personalizada, en este caso para Gemma 3, escribe el prompt completo y <|mark|> donde se lee la respuesta. Las cadenas con imágenes llegan como listas de partes de texto e imagen:
{%- macro text(x) -%}
{%- if x is string or x is none %}{{ x or "" }}{% elif x is mapping %}{{ x|tojson }}{% else -%}
{%- for part in x %}{% if part.type == "image" %}<start_of_image>{% else %}{{ part.text }}{% endif %}{% endfor -%}
{%- endif -%}
{%- endmacro -%}
<start_of_turn>user
{{ text(state) }}
{{ text(question.instructions) }}
{% for option in question.options %}{{ question.labels[loop.index0] }}. {{ option }}: {{ text(question.descriptions[loop.index0]) }}
{% endfor %}<end_of_turn>
<start_of_turn>model
<|mark|>Inclúyalo como chat_template en el archivo --decision-config, o inclúyalo como decision_template.jinja junto a decision.json en un repositorio de modelos.
v1/audio/transcriptions
El punto de acceso v1/audio/transcriptions transcribe audio mediante modelos de voz a texto. Sigue el formato de la API de transcripción de audio.
curl -X POST http://localhost:8000/v1/audio/transcriptions \ -H "Content-Type: multipart/form-data" \ -F "file=@/path/to/audio.wav" \ -F "model=openai/whisper-large-v3"
El comando devuelve la siguiente respuesta.
{
"text": "Transcribed text from the audio file",
}v1/models
El punto de acceso v1/models analiza la caché local de Hugging Face y devuelve una lista de los modelos descargados en el formato compatible con OpenAI. Las herramientas de terceros utilizan este punto de acceso para descubrir los modelos disponibles.
Descarga un modelo antes de ejecutar transformers serve.
transformers download Qwen/Qwen2.5-0.5B-Instruct
Una vez descargado, el modelo aparece en las respuestas de /v1/models.
curl http://localhost:8000/v1/models
El punto final devuelve un objeto JSON con los modelos disponibles.
Carga de modelos
El punto de acceso /load_model precarga un modelo y transmite el progreso mediante Server-Sent Events (SSE). La CLI transformers chat lo utiliza automáticamente para que los usuarios vean el progreso de la descarga y la carga en lugar de un indicador de espera. Úsalo para calentar un modelo antes de enviar solicitudes de inferencia.
Solicitud
curl -N -X POST http://localhost:8000/load_model \
-H "Content-Type: application/json" \
-d '{"model": "Qwen/Qwen2.5-0.5B-Instruct"}'El campo model es un identificador de modelo de Hugging Face, opcionalmente con un sufijo @revision (Qwen/Qwen2.5-0.5B-Instruct@main). Si se omite la revisión, el valor predeterminado es main.
Respuesta
La respuesta es un flujo SSE (Content-Type: text/event-stream). Cada cuadro es un objeto JSON en una línea data:.
data: {"status": "loading", "model": "Qwen/Qwen2.5-0.5B-Instruct@main", "stage": "processor"}Cada evento contiene como mínimo los campos status y model. Los campos adicionales dependen del estado.
| Campo | Presente cuando | Descripción |
|---|---|---|
status | Siempre | loading, ready o error |
model | Siempre | Identificador de modelo canónico model_id@revision |
stage | status == "loading" | Uno de processor, config, download, weights (consulte las etapas a continuación) |
progress | etapas download y weights | Objeto con current y total (entero o null) |
cached | status == "ready" | true si el modelo ya estaba en memoria |
message | status == "error" | Descripción del error |
Etapas
La carga avanza a través de estas etapas en orden. Algunas pueden omitirse (download se omite cuando los archivos ya están en caché local).
| Etapa | ¿Ha habido algún avance? | Descripción |
|---|---|---|
processor | No | Carga del tokenizador/procesador |
config | No | Cargando la configuración del modelo |
download | Sí (bytes) | Descarga de archivos de modelo |
weights | Sí (elementos) | Carga de tensores de pesos en memoria |
El flujo termina con exactamente un evento terminal, ready (éxito) o error (fallo).
Tiempo de espera
transformers serve gestiona las solicitudes para cualquier modelo. Cada modelo se carga bajo demanda y permanece en la memoria de la GPU. Los modelos se descargan automáticamente tras 300 segundos de inactividad para liberar la memoria de la GPU. Establezca --model-timeout en un valor diferente en segundos, o en -1 para deshabilitar la descarga.
transformers serve --model-timeout 400
Cargando ejemplos
Los ejemplos siguientes muestran las respuestas para un modelo recién descargado, un modelo cargado desde su caché local (omite la etapa de descarga) y un modelo ya en memoria.
data: {"status": "loading", "model": "org/model@main", "stage": "processor"}
data: {"status": "loading", "model": "org/model@main", "stage": "config"}
data: {"status": "loading", "model": "org/model@main", "stage": "download", "progress": {"current": 0, "total": 269100000}}
data: {"status": "loading", "model": "org/model@main", "stage": "download", "progress": {"current": 134600000, "total": 269100000}}
data: {"status": "loading", "model": "org/model@main", "stage": "download", "progress": {"current": 269100000, "total": 269100000}}
data: {"status": "loading", "model": "org/model@main", "stage": "weights", "progress": {"current": 1, "total": 272}}
data: {"status": "loading", "model": "org/model@main", "stage": "weights", "progress": {"current": 272, "total": 272}}
data: {"status": "ready", "model": "org/model@main", "cached": false}data: {"status": "loading", "model": "org/model@main", "stage": "processor"}
data: {"status": "loading", "model": "org/model@main", "stage": "config"}
data: {"status": "loading", "model": "org/model@main", "stage": "weights", "progress": {"current": 1, "total": 272}}
data: {"status": "loading", "model": "org/model@main", "stage": "weights", "progress": {"current": 272, "total": 272}}
data: {"status": "ready", "model": "org/model@main", "cached": false}data: {"status": "ready", "model": "org/model@main", "cached": true}Llamada a herramientas
El servidor transformers serve admite la invocación de funciones al estilo de OpenAI. Los modelos entrenados para el uso de herramientas generan llamadas a funciones estructuradas que su aplicación ejecuta.
La invocación de herramientas funciona con cualquier modelo cuyo tokenizador declare tokens de invocación de herramientas. Qwen y Gemma 4 funcionan sin configuración adicional. Abra un issue para solicitar soporte para un modelo específico.
Defina las herramientas como una lista de especificaciones de funciones que siguen el formato de OpenAI.
import json
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="<KEY>")
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather in a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name, e.g. San Francisco"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "temperature unit"
}
},
"required": ["location"]
}
}
}
]Personaliza la generación pasando parámetros de GenerationConfig al argumento extra_body en create.
generation_config = {
"max_new_tokens": 512,
"temperature": 0.7,
"top_p": 0.9,
"top_k": 50,
"do_sample": True,
"repetition_penalty": 1.1,
"no_repeat_ngram_size": 3,
}
response = client.responses.create(
model="Qwen/Qwen2.5-7B-Instruct",
instructions="You are a helpful weather assistant. Use the get_weather tool to answer questions.",
input="What's the weather like in San Francisco?",
tools=tools,
stream=True,
extra_body={"generation_config": json.dumps(generation_config)}
)
for event in response:
print(event)Llamada de herramientas en múltiples turnos
Después de que el modelo devuelva una llamada a una herramienta, ejecute la función localmente y luego envíe el resultado de vuelta en una solicitud de seguimiento para obtener la respuesta final del modelo. El patrón varía ligeramente entre las dos API. Consulte la guía de llamadas a funciones de OpenAI para la especificación completa.
Los ejemplos siguientes reutilizan la lista tools definida anteriormente.
Pasa el resultado de la herramienta como un mensaje con role: "tool" y el tool_call_id correspondiente.
# Model returns a tool call
messages = [{"role": "user", "content": "What's the weather like in San Francisco?"}]
response = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=messages,
tools=tools,
)
assistant_message = response.choices[0].message
# Execute the tool locally
tool_call = assistant_message.tool_calls[0]
result = {"temperature": 22, "condition": "sunny"} # your actual function call here
# Send the tool result back
messages.append(assistant_message)
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": json.dumps(result),
})
final_response = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=messages,
tools=tools,
)
print(final_response.choices[0].message.content)Pasa el resultado de la herramienta como un elemento function_call_output en la lista input de la solicitud de seguimiento.
user_message = {"role": "user", "content": "What's the weather like in San Francisco?"}
response = client.responses.create(
model="Qwen/Qwen2.5-7B-Instruct",
input=[user_message],
tools=tools,
stream=False,
)
tool_call = next(item for item in response.output if item.type == "function_call")
result = {"temperature": 22, "condition": "sunny"}
final_response = client.responses.create(
model="Qwen/Qwen2.5-7B-Instruct",
input=[
user_message,
tool_call.model_dump(exclude_none=True),
{"type": "function_call_output", "call_id": tool_call.call_id, "output": json.dumps(result)},
],
tools=tools,
stream=False,
)
print(final_response.output_text)Razonamiento
Los modelos de razonamiento emiten una cadena de pensamiento oculta antes de la respuesta final. El servidor detecta estos tramos de pensamiento, elimina los delimitadores de la respuesta visible y muestra el texto de razonamiento en un campo dedicado para que los clientes puedan renderizarlo por separado.
- Chat Completions devuelve el razonamiento como
reasoning_contenten el mensaje del asistente. - La API de Responses devuelve el razonamiento como un elemento de salida
reasoningque precede al elementomessage.
La detección de razonamiento se basa en una plantilla de respuesta, que indica al servidor dónde comienza y termina un tramo de razonamiento. El servidor busca una en dos lugares, en orden.
- El atributo
response_templatedel tokenizador, si el modelo lo incluye. - Un respaldo integrado, basado en la arquitectura del modelo, para tokenizadores que aún no incluyen una plantilla.
Los modelos que no tienen ninguno de los dos se sirven sin análisis de la respuesta. El texto decodificado en bruto se devuelve como content y no se muestra ningún campo de razonamiento, incluso si el modelo emite etiquetas <think>...</think>.
Habilitar el razonamiento en el servidor
Use --reasoning para controlar si la plantilla de chat emite tokens de razonamiento.
| Valor | Comportamiento |
|---|---|
auto (predeterminado) | Delegar en el valor predeterminado de la plantilla de chat. |
on | Fuerce el modo de razonamiento activando enable_thinking=True en los kwargs de la plantilla de chat. |
off | Desactive el razonamiento estableciendo enable_thinking=False. |
transformers serve Qwen/Qwen3-1.7B --reasoning on
No todos los modelos admiten razonamiento.
--reasoningsolo funciona cuando la plantilla de chat del modelo gestiona la variableenable_thinking(o cuando su tokenizer declara delimitadores de razonamiento). Para otros modelos, la bandera se ignora silenciosamente.
--reasoning on y --reasoning off funcionan estableciendo la variable de plantilla de chat enable_thinking. Algunas plantillas de chat utilizan un nombre de variable diferente para activar el razonamiento, o aceptan variables adicionales que modifican cómo se renderiza el prompt. Pasa cualquier variable de plantilla de chat directamente con --chat-template-kwargs como un objeto JSON.
transformers serve Qwen/Qwen3-1.7B \
--chat-template-kwargs '{"enable_thinking": true}'Los clientes también pueden enviar chat_template_kwargs en el cuerpo de la solicitud para sobrescribir los valores predeterminados del servidor en una sola solicitud, sin necesidad de reiniciar el servidor.
Chat Completions
Las respuestas no de streaming incluyen reasoning_content junto a content en el mensaje del asistente.
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="<KEY>")
completion = client.chat.completions.create(
model="Qwen/Qwen3-1.7B",
messages=[{"role": "user", "content": "What is 17 * 23?"}],
)
message = completion.choices[0].message
print("reasoning:", message.reasoning_content)
print("answer:", message.content)Streaming emite deltas de reasoning_content antes que deltas de content. Concatene cada campo por separado.
chunks = list(
client.chat.completions.create(
model="Qwen/Qwen3-1.7B",
messages=[{"role": "user", "content": "What is 17 * 23?"}],
stream=True,
)
)
reasoning = "".join(getattr(c.choices[0].delta, "reasoning_content", None) or "" for c in chunks)
content = "".join(c.choices[0].delta.content or "" for c in chunks)curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-1.7B",
"messages": [{"role": "user", "content": "What is 17 * 23?"}],
"chat_template_kwargs": {"enable_thinking": true}
}'La respuesta incluye reasoning_content en el mensaje del asistente.
{
"choices": [
{
"message": {
"role": "assistant",
"content": "17 * 23 = 391.",
"reasoning_content": "Let me compute 17 * 23. 17 * 20 = 340, 17 * 3 = 51, so 340 + 51 = 391."
},
"finish_reason": "stop",
"index": 0
}
]
}API de respuestas
La API de Responses devuelve el razonamiento como un elemento de salida reasoning separado que precede al elemento message.
response = client.responses.create(
model="Qwen/Qwen3-1.7B",
input="What is 17 * 23?",
stream=False,
)
reasoning_item = next(i for i in response.output if i.type == "reasoning")
message_item = next(i for i in response.output if i.type == "message")
print("reasoning:", reasoning_item.content[0].text)
print("answer:", message_item.content[0].text)Streaming abre primero el elemento de razonamiento y emite eventos response.reasoning_text.delta, luego lo cierra y abre el elemento de mensaje para emitir eventos response.output_text.delta.
stream = client.responses.create(
model="Qwen/Qwen3-1.7B",
input="What is 17 * 23?",
stream=True,
)
for event in stream:
if event.type == "response.reasoning_text.delta":
print(event.delta, end="")
elif event.type == "response.output_text.delta":
print(event.delta, end="")Ida y vuelta de varios turnos
Para conversaciones de varios turnos, incluye el texto de razonamiento de la respuesta anterior en la siguiente solicitud. El servidor lo pasa a la plantilla de chat, que puede renderizarlo como parte del turno anterior del asistente para que el modelo vea su propio proceso de pensamiento previo.
Adjunta reasoning_content al mensaje asistente anterior.
first = client.chat.completions.create(
model="Qwen/Qwen3-1.7B",
messages=[{"role": "user", "content": "What is 17 * 23?"}],
).choices[0].message
second = client.chat.completions.create(
model="Qwen/Qwen3-1.7B",
messages=[
{"role": "user", "content": "What is 17 * 23?"},
{
"role": "assistant",
"content": first.content,
"reasoning_content": first.reasoning_content,
},
{"role": "user", "content": "Now multiply that result by 2."},
],
)
print(second.choices[0].message.content)Pasa el elemento reasoning anterior como elemento de entrada antes del mensaje del asistente.
first = client.responses.create(
model="Qwen/Qwen3-1.7B",
input="What is 17 * 23?",
stream=False,
)
reasoning_item = next(i for i in first.output if i.type == "reasoning")
message_item = next(i for i in first.output if i.type == "message")
second = client.responses.create(
model="Qwen/Qwen3-1.7B",
input=[
{"role": "user", "content": "What is 17 * 23?"},
reasoning_item.model_dump(exclude_none=True),
{"role": "assistant", "content": message_item.content[0].text},
{"role": "user", "content": "Now multiply that result by 2."},
],
stream=False,
)
print(second.output_text)Redirección de puertos
El reenvío de puertos le permite servir modelos desde un servidor remoto. Asegúrese de tener acceso SSH al servidor y luego ejecute este comando en su máquina local.
ssh -N -f -L 8000:localhost:8000 your_server_account@your_server_IP -p port_to_ssh_into_your_server
Reproducibilidad
Pase un modelo como argumento posicional para evitar indicios de modelo por solicitud y generar ejecuciones estables y repetibles.
transformers serve Qwen/Qwen2.5-0.5B-Instruct \
--continuous-batching \
--dtype "bfloat16"