vkserve¶
An OpenAI-compatible LLM server that runs several models on any Vulkan GPU, built on compute-ml. Status: prototype. The wheels on PyPI (Linux x86_64 and macOS arm64) include the C engine; other systems build it from source (see Install).
Install¶
On Linux x86_64, pip install vkserve is enough. The wheel includes the C engine. It needs the
system Vulkan loader (libvulkan.so.1) and a GPU driver.
On macOS arm64, pip install vkserve is enough too. The wheel bundles MoltenVK (Vulkan on Metal)
and the Vulkan loader, so it needs no Vulkan SDK. Set VK_DRIVER_FILES to use another driver.
Other systems build the engine from a source checkout:
git clone https://gitlab.com/cardona-oss/vkserve.git
git clone https://gitlab.com/cardona-oss/compute-ml.git
cd vkserve
make engine
make engine needs compute-ml 071d587 (0.4.73) or newer. It stops with a message that names
the missing function when the checkout is older. Update it with git -C ../compute-ml pull.
Or point VKSERVE_ENGINE_LIB at an engine library you built elsewhere.
Run it¶
- Build the C engine. It compiles compute-ml from
../compute-ml.
- Start the server on a model folder or a Hugging Face id. The id is downloaded on first use.
The model id in the API is the last part of the name (Qwen3-0.6B); --name changes it.
Pick the GPU with --device (an index or part of the name). vkserve --list-devices shows
them. Repeat --device to run the model on several GPUs: requests go to the least busy one,
and a GPU that cannot load the model is skipped.
vkserve --list-models shows the models already in the Hugging Face cache, the ones vkserve
can serve first.
Models: Qwen3, Llama 3.2, SmolLM2, Gemma 3 and Gemma 4, as safetensors with a
tokenizer.json. A bare .gguf file is not supported yet. Several models in one server:
repeat --model name=ID,path=FOLDER_OR_HF_ID[,dtype=q8][,slots=4][,max_len=4096].
- Send a request.
curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' \
-d '{"messages":[{"role":"user","content":"Name two planets."}],"max_tokens":30}'
Use it from opencode¶
See examples/opencode: a config, a small agent, and the commands.
Configuration¶
One TOML file, and every key is also a flag. Flags win over the file. See examples/vkserve.toml.
uv run vkserve -t -c vkserve.toml # check the file, load nothing
uv run vkserve -c vkserve.toml # run it
uv run vkserve -c vkserve.toml --port 9000 # a flag overrides the file
uv run vkserve -c vkserve.toml --model qwen --slots 2 # change one model of the file
uv run vkserve --print-config -c vkserve.toml # show every effective setting
[server]keys are plain flags:--host,--port,--log-level,--keep-going,--max-body-bytes,--pid-file,--log-file,--drain-timeout.--model NAMEstarts a model block. The flags after it apply to that block:--path,--dtype,--slots,--max-len,--prefill-chunk,--max-queue,--device(repeat for several devices),--temperature,--top-p,--top-k.--set server.port=9000and--set model.qwen.slots=2reach any key.- An unknown key is an error with a hint:
unknown key 'slot' ... (did you mean 'slots'?).
Run it as a service¶
The master process serves HTTP. Each model on each device runs in its own worker process. A worker that crashes fails its requests with 500 and starts again after 1, 2, 4 ... 30 seconds. The master keeps running.
Send signals with kill, or with -s (it reads pid_file):
| Signal | -s |
What it does |
|---|---|---|
SIGHUP |
reload |
Read the configuration again. A new or changed model loads first, then the old one finishes its requests and stops. A bad file changes nothing. |
SIGTERM, SIGQUIT, SIGINT |
stop, quit |
Stop accepting, wait up to drain_timeout for requests in progress, then exit. A second signal stops at once. |
SIGUSR1 |
reopen |
Reopen log_file (after logrotate moved it). |
host, port, max_body_bytes, pid_file, drain_timeout and keep_going need a restart.
A reload that changes them logs a warning and ignores them.
A systemd unit is in examples/vkserve.service.
Features¶
POST /v1/chat/completionsand/v1/completions, streaming as server-sent events.- Tool calls:
toolsin, OpenAItool_callsout,toolmessages back in. - Several requests in one batched forward pass per step.
- Several GPUs: one worker per device, least-busy routing, failed workers skipped and restarted.
- Operations:
429when the queue is full,X-Request-ID, Prometheus/metricsper device. - Prefix reuse: a request that starts with a cached conversation prefills only the new tokens.
- Attention over long contexts (tested to 20,000 keys).
More: docs/serving.md (design and measurements).