Skip to content

serve: OpenAI-compatible server prototype

A prototype of a vLLM-style server on compute-ml: a multi-sequence engine in C, driven by a Python scheduler behind an OpenAI-compatible HTTP API. It runs on any GPU with a Vulkan driver.

Run it

  1. Build the engine library.
cmake -B build && cmake --build build --target cml_engine
  1. Start the server on a Hugging Face snapshot directory.
cd examples/serve
uv run --no-project --with litestar --with uvicorn --with tokenizers --with jinja2 \
    python server.py <snapshot dir> --dtype q8 --slots 4 --max-len 512
  1. Send a request.
curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' \
    -d '{"messages":[{"role":"user","content":"Name two planets."}],"max_tokens":30}'

Endpoints: POST /v1/completions, POST /v1/chat/completions ("stream": true sends server-sent events), GET /v1/models, GET /health, GET /metrics.

How the work is split

layer file does
KV cache per sequence ../llm/llm.c (cml_llm_kv_*) one cache per sequence, separate from the weights
batched forward ../llm/llm.c (cml_llm_forward_batch) one pass over all sequences, weights read once
engine engine.c slots, one batched forward per step, sampling on the CPU
scheduler engine.py FIFO queue, admission, stop conditions, release, cancel
HTTP server.py tokenizer, chat template, OpenAI JSON, streaming

Python calls cml_engine_step once per step for all active slots. ctypes releases the GIL during the call, so the HTTP server keeps answering while the GPU runs.

In a batched step, the norms, the q/k/v/o projections and the MLP run on all rows at once. Rope, the cache write and attention run per sequence, because each sequence has its own position and cache; cml_copy_range cuts its rows out of the batch and puts them back. A new prompt joins the step with all its tokens next to the one-token decodes.

Check it

test_batch runs three sequences alone and then together (one joins mid-run) and compares the greedy tokens.

cmake --build build --target test_batch
CML_LLM_MODEL=<snapshot dir> ./build/examples/serve/test_batch
3 sequences x 24 tokens: tokens match, max |logit diff| 0.0106

The logits differ by up to 0.011 because 3 rows and 1 row go through different matmul kernels.

Measured

Qwen3-0.6B, Q8_0, Quadro T2000 shared with another process (1.5 GB free), 8 slots of 256 tokens, 64 tokens per request, temperature 0.8, top_p 0.9:

requests at once aggregate tok/s before batching
1 48.9 34.9
2 73.2
4 102.9 39.7
8 124.5

Next, in order

From the published serving work (vLLM, Sarathi-Serve, SGLang, Flash-Decoding), none of which needs CUDA:

  1. Benchmark with a standard tool (vllm bench serve or GuideLLM) and report TTFT (time to first token), ITL (time between tokens) and goodput (requests per second within a latency target).
  2. Paged KV cache (PagedAttention): fixed-size blocks and a block table per sequence, so short requests do not reserve max_len. Now each slot holds max_len tokens (59 MB per slot for Qwen3-0.6B at 256 tokens).
  3. Chunked prefill (Sarathi-Serve): split a long prompt over several steps, so it does not stall the decodes in its step.
  4. Prefix caching (RadixAttention): reuse the KV blocks of a shared system prompt. Needs 2.
  5. Batched attention: one dispatch for all sequences instead of one per sequence per layer.
  6. Sampling on the GPU (src/shaders/sample.comp exists): download one id per sequence instead of the full logits row.
  7. Speculative decoding: helps at batch 1 to 2, little at large batches.

Also missing: stop strings, n > 1, logprobs, Prometheus metrics.