briaai/fibo-scene-analyzer-gguf

FIBO Scene Analyzer — GGUF

GGUF files of briaai/fibo-scene-analyzer for llama.cpp, with a Python harness that runs every Scene Analyzer task on llama-server. Made and tested with llama.cpp build 11459.

Start the server

llama-server -m fibo-scene-analyzer-Q4_K_M.gguf --mmproj mmproj-fibo-scene-analyzer-F16.gguf \
    -ngl 99 -fa on -np 4 -c 131072 --image-max-tokens 8192 --chat-template-kwargs '{"enable_thinking": false}'

• -np is the number of slots (requests decoded at once). -c is the total context of all slots: give each slot 32,768 tokens, because a full caption with its image can need that much.
• --image-max-tokens 8192 is the visual-token cap of the model's image processor.
• --chat-template-kwargs matters only for the chat endpoint. Without it, llama-server opens a think block in every chat prompt.

Harness

Put the three files of harness/ on your Python path (they need pillow, pydantic, numpy and, for exact prompt tokens, tokenizers).

from PIL import Image
from scene_analyzer_gguf import DetailLevel, SceneAnalyzerModel, SceneAnalyzerTask, ThinkingEffort

model = SceneAnalyzerModel("http://localhost:8080")
image = Image.open("photo.jpg")

[caption] = model.generate([image], SceneAnalyzerTask.CAPTION, detail_level=DetailLevel.FULL, num_samples=3)
caption.post_process()  # coordinates and colors from tokens to numbers

[objects] = model.generate([image], SceneAnalyzerTask.DETECT, text_inputs=["the red car"])
[text] = model.generate([image], SceneAnalyzerTask.OCR)
[scene] = model.generate([None], SceneAnalyzerTask.EXPAND, text_inputs=["a red fox in fresh snow"], aspect_ratio="1:1")
[reasoned] = model.generate([image], SceneAnalyzerTask.CAPTION, detail_level=DetailLevel.SHORT,
                            thinking_effort=ThinkingEffort.LOW)

Or start and stop the server from Python (the llama-server binary must be on the PATH):

with SceneAnalyzerModel.launch("fibo-scene-analyzer-Q4_K_M.gguf", "mmproj-fibo-scene-analyzer-F16.gguf") as model:
    [caption] = model.generate([image], SceneAnalyzerTask.CAPTION, detail_level=DetailLevel.LONG)

generate() takes one input per image (None for text-only tasks) and returns one answer per input, or None for an input without a valid answer. Inputs run in parallel on the server's slots. Its main arguments:

Argument	Meaning
task	CAPTION, DESCRIBE, DETECT, MASK, KEYPOINTS, OCR, EXPAND, EXPAND_OBJECT, REFINE, EDIT
detail_level	GENERAL, SHORT, LONG, FULL (captions, expansion, refinement, editing)
text_inputs	the target to find, the prompt to expand, or the instruction plus the scene JSON
num_samples	candidates per input; the best valid one is returned
thinking_effort	None for a direct answer, or the effort of a reasoning trace before it
num_iterations	revision rounds that add the objects the model reports as missing
retries	extra attempts for inputs without a valid answer

Constructor options: seed, max_concurrency (default: the server's slots), use_grammar (default on), tokenizer (default briaai/fibo-scene-analyzer; set a local folder, or None to send text).

What the harness does:

• It sends the trained prompt to llama-server's /completion endpoint and samples with temperature 0.6 and top-p 0.97 only. llama.cpp's defaults also apply top-k 40 and min-p 0.05.
• It constrains each answer to the answer schema with its own GBNF grammar. llama.cpp's JSON-schema converter puts the required properties of an object first, which forces the model out of its trained key order and makes it write junk. The harness grammar keeps the schema order, lets optional properties be left out in place, and has no free whitespace.
• It runs thinking in two passes: the trace without the grammar, then the answer under it.
• It sends text-only prompts as the token ids of the original tokenizer. llama.cpp matches overlapping JSON tokens longest-first, so JSON in a prompt can split differently. Prompts with images must be text for llama-server, so edit and revision rounds keep llama.cpp's tokenization.

Without the harness

Use llama-server's chat endpoint: the image first, then the task tags, for example <caption><full> or <detect>the red car, with temperature: 0.6, top_p: 0.97, top_k: 0, min_p: 0. For a reasoning trace, send chat_template_kwargs: {"thinking_effort": "low"}; the trace comes back in reasoning_content. Validate the JSON yourself.