Homelab

Ferret's AI, at home

November 2026

Ferret is self-hosted, and every AI feature in it is optional. In my own house, all of them are on — and every one runs on hardware in a closet: two NVIDIA DGX Sparks for the models, and the NAS for search and documents. No photo, receipt or question goes to an outside AI service. This is how it's put together: what runs where, the Docker setup for each piece, and how Ferret calls it.

Ferret itself is one Docker container on a small app server, behind a reverse proxy, with Apprise beside it for notifications. Nothing below needs Ferret to know about GPUs: it talks plain HTTP to OpenAI-compatible APIs, so any box that can serve the same APIs will do.

What runs where

App server

Ferret — Docker, SQLite, encrypted files
Apprise — reminders and email
Traefik in front (HTTPS)

DGX Spark A

gpt-oss-120b on vLLM — Ask Ferret, documents, the search judge

DGX Spark B

Ollama — qwen3-vl:8b (photos), qwen3-embedding:8b (search)
vLLM — bge-reranker-v2-m3
vLLM — granite-docling-258M (reads pages)

NAS

docling-serve — PDFs and photos of paper → text (CPU)
Qdrant — the vectors for search (CPU)
Ferret's nightly backups

And which Ferret feature uses which:

Feature	Model	Served by	Ferret setting
Reading photos, room scans	Qwen3-VL 8B	Ollama, Spark B	`OLLAMA_URL`
Search by meaning	Qwen3-Embedding 8B	Ollama, Spark B → Qdrant	`OLLAMA_URL`, `QDRANT_URL`
Ordering search results	bge-reranker-v2-m3	vLLM, Spark B	`RERANK_URL`
Reading PDFs and receipts	granite-docling 258M	docling-serve (NAS) → vLLM, Spark B	`DOCLING_URL`, `DOCLING_VLM_URL`
Ask Ferret, warranties and manuals, the judge	gpt-oss-120b	vLLM, Spark A	`CHAT_URL`

The DGX Spark

Each Spark is a GB10 Grace Blackwell box: a 20-core Arm CPU and a Blackwell GPU sharing 128 GB of unified memory (about 121 GB usable), on Ubuntu 24.04. That shared memory is the whole point: a 120-billion-parameter model fits on one desk-sized machine, and "GPU memory" is just memory.

A few things that shaped the setup:

vLLM comes from NVIDIA's NGC containers (nvcr.io/nvidia/vllm), not the upstream image — they're built for Arm and the GB10. Everything else is the normal image.
Every service is a Docker Compose stack, managed with Dockge, with a Dozzle agent on each box for logs. Model weights are cached in one shared Hugging Face folder per host.
vLLM takes 90% of memory by default. That's right for the one big model on Spark A. On Spark B, where several things share the box, each vLLM gets a small slice with --gpu-memory-utilization — a fraction of the whole unified memory, so 0.08 is about 10 GB.

Spark A: gpt-oss-120b on vLLM

One model, the whole machine. OpenAI's open-weight gpt-oss-120b answers Ask Ferret, reads receipts, warranties and manuals (including maintenance schedules) once docling has turned them into text, and is the "judge" in search. With reasoning_effort: "low" an Ask answer takes about 2 seconds; four maintenance tasks out of a test manual took about 12.

spark-a: ai/compose.yaml

services:
  vllm:
    image: nvcr.io/nvidia/vllm:26.09-py3
    container_name: ${NAME}
    restart: unless-stopped
    ipc: host
    ports:
      - 8000:8000
    ulimits:
      memlock: -1
      stack: 67108864
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - VLLM_ENABLE_CUDA_COMPATIBILITY=1
    command: >
      vllm serve ${MODEL} --served-model-name gpt-oss-120b
      --enable-auto-tool-choice --tool-call-parser openai
      --enable-prefix-caching
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

spark-a: ai/.env

NAME=vllm-gpt-oss-120b
MODEL=openai/gpt-oss-120b
HF_TOKEN=hf_…

Prefix caching helps because Ferret's prompts start the same way every time. Tool calling is switched on for the other things in the house that use this model (chat, automations, a Discord help bot); Ferret itself only asks for JSON.

Spark B: Ollama, the reranker and granite-docling

Ollama: the vision and embedding models

spark-b: ai/compose.yaml

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - 11434:11434
    volumes:
      - ./ollama:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=24h      # keep models loaded (the default is 5 minutes)
      - OLLAMA_FLASH_ATTENTION=1
      - OLLAMA_HOST=0.0.0.0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

docker exec ollama ollama pull qwen3-vl:8b
docker exec ollama ollama pull qwen3-embedding:8b

qwen3-vl:8b reads photos: what the thing is, its brand, model and serial number, and any text on it. qwen3-embedding:8b turns items and document passages into 4096-number vectors. The 24-hour keep-alive matters: a cold load on the first search of the morning would blow the search box's time limit.

vLLM: bge-reranker-v2-m3

spark-b: embeddings/compose.yaml

services:
  vllm:
    image: nvcr.io/nvidia/vllm:${VLLM_TAG}
    container_name: vllm-reranker
    restart: unless-stopped
    ipc: host
    ports:
      - ${PORT}:8000
    ulimits:
      memlock: -1
      stack: 67108864
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - VLLM_ENABLE_CUDA_COMPATIBILITY=1
    command: >
      vllm serve ${MODEL} --served-model-name ${SERVED_NAME}
      --gpu-memory-utilization ${MEMORY}
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

spark-b: embeddings/.env

PORT=8002
MODEL=BAAI/bge-reranker-v2-m3
SERVED_NAME=reranker
MEMORY=0.08
VLLM_TAG=25.12.post1-py3

vLLM recognises a cross-encoder and serves it on /v1/rerank. Reranking adds about 80–115 ms to a search.

vLLM: granite-docling-258M

IBM's tiny document model reads a page image and writes it out as structured text. It's the same compose file as the reranker; only the settings differ:

spark-b: docling/.env

PORT=8001
MODEL=ibm-granite/granite-docling-258M
SERVED_NAME=granite-docling
MEMORY=0.08
MODEL_LEN=8192
VLLM_TAG=25.12.post1-py3

…with --max-model-len ${MODEL_LEN} added to the vllm serve line. Spark B also runs a couple of unrelated experiments; with everything loaded it still has room to spare.

The NAS: docling-serve and Qdrant

Neither needs a GPU, and both are happier next to the disks. They share a stack with Open WebUI (which searches the same Qdrant).

nas: compose.yaml (the parts Ferret uses)

services:
  qdrant:
    image: qdrant/qdrant:latest
    container_name: qdrant
    restart: always
    ports:
      - 6333:6333
      - 6334:6334
    volumes:
      - ./qdrant:/qdrant/storage
    environment:
      - QDRANT__SERVICE__API_KEY=${QDRANT_API_KEY}

  docling:
    image: hwdsl2/docling-server:latest
    container_name: docling
    restart: always
    ports:
      - 6331:5001
    volumes:
      - ./docling:/var/lib/docling
    environment:
      - DOCLING_DEVICE=cpu
      - DOCLING_ENABLE_UI=true
      - DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true   # lets the VLM pipeline call granite-docling on Spark B

The setting that's easy to miss: docling-serve refuses to call an outside model unless DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true. Without it the VLM pipeline fails, and the CPU does all the work with docling's standard pipeline.

Docling runs on the NAS's CPU and handles the file — splitting PDFs into pages, assembling the result — while each page is read by granite-docling on Spark B. Qdrant holds one collection per Ferret: the real one, one for development, and one the search evaluation fills and empties.

How Ferret calls them

Ferret's settings

ferret: .env (the AI part)

# Photos and embeddings: Ollama's OpenAI-compatible API
OLLAMA_URL=http://spark-b.lan:11434
VISION_MODEL=qwen3-vl:8b             # the default
EMBED_MODEL=qwen3-embedding:8b       # the default

# Questions and document text: any OpenAI-compatible server
CHAT_URL=http://spark-a.lan:8000/v1
CHAT_MODEL=gpt-oss-120b              # the default
CHAT_REASONING=low                   # the default

# Search
QDRANT_URL=http://nas.lan:6333
QDRANT_API_KEY=…
QDRANT_COLLECTION=ferret
RERANK_URL=http://spark-b.lan:8002   # RERANK_MODEL defaults to "reranker"

# Documents
DOCLING_URL=http://nas.lan:6331
DOCLING_VLM_URL=http://spark-b.lan:8001/v1/chat/completions

Each line switches one feature on; leave it out and that feature quietly isn't there. The startup log lists what's on. Every call happens in a background job with its own lane and retries — a busy GPU or a model still loading just means a retry a little later — except the two that run while someone is waiting on the search box.

Photos

A photo goes to Ollama's /v1/chat/completions with the image and a JSON schema for the answer (response_format: json_schema), so the reply is fields, not prose. What comes back is shown as suggestions to tap — no model output is ever applied by itself.

Ask for a complete transcription explicitly, or Qwen3-VL reads only the first line of a label.
A room scan is two calls per photo: first the list of things, then where each one is. Asking for both at once ran out of tokens.
A whole room is a lot of work for an 8B model: two to four minutes a photo here. It runs in the background while you carry on.

Documents

Ferret sends the file to docling-serve's async API, asking for the VLM pipeline, then long-polls until it's done:

curl -s http://nas.lan:6331/v1/convert/file/async \
  -F files=@manual.pdf -F to_formats=md -F image_export_mode=placeholder \
  -F pipeline=vlm \
  -F 'vlm_pipeline_model_api={"url":"http://spark-b.lan:8001/v1/chat/completions",
       "params":{"model":"granite-docling","max_tokens":6500,"repetition_penalty":1.05,"skip_special_tokens":false},
       "prompt":"Convert this page to docling.","response_format":"doctags","timeout":180,"concurrency":4}'
# → {"task_id": "…"}
curl -s "http://nas.lan:6331/v1/status/poll/$TASK?wait=20"   # until task_status is "success"
curl -s "http://nas.lan:6331/v1/result/$TASK"                # → document.md_content

The synchronous endpoint gives up after two minutes, which a long manual easily passes — hence async. A one-page PDF takes about a minute end to end. The Markdown is stored with the document, split into passages for search, and handed to gpt-oss-120b to suggest the item's details, the warranty, and any maintenance schedule in the manual.

Search

Search is two searches fused into one list, so "the drill" and "something to make holes in the wall" both work:

Keywords — SQLite's full-text search inside Ferret: every word, stemmed, plus substrings for serial numbers.

Meaning — the query is embedded by qwen3-embedding:8b, with an instruction in front (Qwen3-Embedding is instruction-tuned; stored text gets none):

curl -s http://spark-b.lan:11434/v1/embeddings -H 'Content-Type: application/json' -d '{
  "model": "qwen3-embedding:8b",
  "input": ["Instruct: Given a search for something in a home inventory, find the matching items\nQuery: something to make holes in the wall"]
}'

Qdrant returns the nearest passages, always filtered to the household asking.

Rerank — the candidates go to bge-reranker-v2-m3:

curl -s http://spark-b.lan:8002/v1/rerank -H 'Content-Type: application/json' -d '{
  "model": "reranker",
  "query": "warranty for the dishwasher",
  "documents": ["Bosch dishwasher, kitchen…", "Receipt: Home Depot…"]
}'

Fuse the two lists (reciprocal rank fusion) and cut off what's too far away.
Judge — matches nobody's sure of (no keyword hit, a middling embedding score) go to gpt-oss-120b in one yes/no call. The app shows the sure matches at once and adds the judged ones about a second later.

The cut-offs are measured, not guessed: an evaluation of 25 items and 36 queries runs against these exact models. Embeddings and the reranker alone found the right thing first 97% of the time; with the judge, 100%, with no false alarms. One lesson from it: the reranker is excellent at exact text (0.9+ for a serial number buried in a PDF) and nearly blind to intent, so it rescues strong text matches and orders results, but the embedding score decides what's in.

Ask Ferret

A question runs the same search; the best six things — their details, where they are, and their best document passages — become numbered sources, and gpt-oss-120b answers only from those, citing them or saying it doesn't know. About two seconds.

What I'd tell you before you build one

Ask for JSON with a schema, and still check it. Every model here honours response_format: json_schema, but once in a while a reply is cut short. Ferret retries once at a slightly higher temperature and logs it if that fails too.
Changing the embedding model means a new Qdrant collection — the vectors are a different size — and re-measuring the search cut-offs.
Give each Ferret its own collection (and the evaluation its own), even on one Qdrant.
Keep models loaded. A cold start is fine for a background job and too slow for a search box.
Watch the outside services. Ferret records every failed call to each of these (how long it took, why it stopped) in Settings → Server and tells the server admin, which is how I found most of the above.

Don't have two Sparks?

You don't need them. Ferret works with none of this, and each piece is separate: a single GPU with Ollama gives you photos and search by meaning (without CHAT_URL, the vision model answers questions too); docling-serve on any CPU reads documents with its standard pipeline; Qdrant runs anywhere. Start with one, add the rest later.

The full list of settings is in Ferret's help: AI features and installing Ferret. Questions? Ask in the Discord.