Skip to content

vkserve

An OpenAI-compatible LLM server that runs several models on any Vulkan GPU, built on compute-ml. Status: prototype. The wheels on PyPI (Linux x86_64 and macOS arm64) include the C engine; other systems build it from source (see Install).

Install

On Linux x86_64, pip install vkserve is enough. The wheel includes the C engine. It needs the system Vulkan loader (libvulkan.so.1) and a GPU driver.

On macOS arm64, pip install vkserve is enough too. The wheel bundles MoltenVK (Vulkan on Metal) and the Vulkan loader, so it needs no Vulkan SDK. Set VK_DRIVER_FILES to use another driver.

Other systems build the engine from a source checkout:

git clone https://gitlab.com/cardona-oss/vkserve.git
git clone https://gitlab.com/cardona-oss/compute-ml.git
cd vkserve
make engine

make engine needs compute-ml 071d587 (0.4.73) or newer. It stops with a message that names the missing function when the checkout is older. Update it with git -C ../compute-ml pull.

Or point VKSERVE_ENGINE_LIB at an engine library you built elsewhere.

Run it

  1. Build the C engine. It compiles compute-ml from ../compute-ml.
make engine
  1. Start the server on a model folder or a Hugging Face id. The id is downloaded on first use.
uv run vkserve Qwen/Qwen3-0.6B --dtype q4 --slots 1 --max-len 3000

The model id in the API is the last part of the name (Qwen3-0.6B); --name changes it.

Pick the GPU with --device (an index or part of the name). vkserve --list-devices shows them. Repeat --device to run the model on several GPUs: requests go to the least busy one, and a GPU that cannot load the model is skipped.

uv run vkserve --list-devices
uv run vkserve Qwen/Qwen3-0.6B --device intel --device t2000

vkserve --list-models shows the models already in the Hugging Face cache, the ones vkserve can serve first.

uv run vkserve --list-models

Models: Qwen3, Llama 3.2, SmolLM2, Gemma 3 and Gemma 4, as safetensors with a tokenizer.json. A bare .gguf file is not supported yet. Several models in one server: repeat --model name=ID,path=FOLDER_OR_HF_ID[,dtype=q8][,slots=4][,max_len=4096].

  1. Send a request.
curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' \
    -d '{"messages":[{"role":"user","content":"Name two planets."}],"max_tokens":30}'

Use it from opencode

See examples/opencode: a config, a small agent, and the commands.

Configuration

One TOML file, and every key is also a flag. Flags win over the file. See examples/vkserve.toml.

uv run vkserve -t -c vkserve.toml                 # check the file, load nothing
uv run vkserve -c vkserve.toml                    # run it
uv run vkserve -c vkserve.toml --port 9000        # a flag overrides the file
uv run vkserve -c vkserve.toml --model qwen --slots 2   # change one model of the file
uv run vkserve --print-config -c vkserve.toml     # show every effective setting
  • [server] keys are plain flags: --host, --port, --log-level, --keep-going, --max-body-bytes, --pid-file, --log-file, --drain-timeout.
  • --model NAME starts a model block. The flags after it apply to that block: --path, --dtype, --slots, --max-len, --prefill-chunk, --max-queue, --device (repeat for several devices), --temperature, --top-p, --top-k.
  • --set server.port=9000 and --set model.qwen.slots=2 reach any key.
  • An unknown key is an error with a hint: unknown key 'slot' ... (did you mean 'slots'?).

Run it as a service

The master process serves HTTP. Each model on each device runs in its own worker process. A worker that crashes fails its requests with 500 and starts again after 1, 2, 4 ... 30 seconds. The master keeps running.

Send signals with kill, or with -s (it reads pid_file):

Signal -s What it does
SIGHUP reload Read the configuration again. A new or changed model loads first, then the old one finishes its requests and stops. A bad file changes nothing.
SIGTERM, SIGQUIT, SIGINT stop, quit Stop accepting, wait up to drain_timeout for requests in progress, then exit. A second signal stops at once.
SIGUSR1 reopen Reopen log_file (after logrotate moved it).

host, port, max_body_bytes, pid_file, drain_timeout and keep_going need a restart. A reload that changes them logs a warning and ignores them.

uv run vkserve -s reload -c vkserve.toml

A systemd unit is in examples/vkserve.service.

Features

  • POST /v1/chat/completions and /v1/completions, streaming as server-sent events.
  • Tool calls: tools in, OpenAI tool_calls out, tool messages back in.
  • Several requests in one batched forward pass per step.
  • Several GPUs: one worker per device, least-busy routing, failed workers skipped and restarted.
  • Operations: 429 when the queue is full, X-Request-ID, Prometheus /metrics per device.
  • Prefix reuse: a request that starts with a cached conversation prefills only the new tokens.
  • Attention over long contexts (tested to 20,000 keys).

More: docs/serving.md (design and measurements).