serve: OpenAI-compatible server prototype¶
A prototype of a vLLM-style server on compute-ml: a multi-sequence engine in C, driven by a Python scheduler behind an OpenAI-compatible HTTP API. It runs on any GPU with a Vulkan driver.
Run it¶
- Build the engine library.
- Start the server on a Hugging Face snapshot directory.
cd examples/serve
uv run --no-project --with litestar --with uvicorn --with tokenizers --with jinja2 \
python server.py <snapshot dir> --dtype q8 --slots 4 --max-len 512
- Send a request.
curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' \
-d '{"messages":[{"role":"user","content":"Name two planets."}],"max_tokens":30}'
Endpoints: POST /v1/completions, POST /v1/chat/completions ("stream": true sends
server-sent events), GET /v1/models, GET /health, GET /metrics.
How the work is split¶
| layer | file | does |
|---|---|---|
| KV cache per sequence | ../llm/llm.c (cml_llm_kv_*) |
one cache per sequence, separate from the weights |
| batched forward | ../llm/llm.c (cml_llm_forward_batch) |
one pass over all sequences, weights read once |
| engine | engine.c |
slots, one batched forward per step, sampling on the CPU |
| scheduler | engine.py |
FIFO queue, admission, stop conditions, release, cancel |
| HTTP | server.py |
tokenizer, chat template, OpenAI JSON, streaming |
Python calls cml_engine_step once per step for all active slots. ctypes releases the GIL during
the call, so the HTTP server keeps answering while the GPU runs.
In a batched step, the norms, the q/k/v/o projections and the MLP run on all rows at once. Rope,
the cache write and attention run per sequence, because each sequence has its own position and
cache; cml_copy_range cuts its rows out of the batch and puts them back. A new prompt joins the
step with all its tokens next to the one-token decodes.
Check it¶
test_batch runs three sequences alone and then together (one joins mid-run) and compares the
greedy tokens.
cmake --build build --target test_batch
CML_LLM_MODEL=<snapshot dir> ./build/examples/serve/test_batch
The logits differ by up to 0.011 because 3 rows and 1 row go through different matmul kernels.
Measured¶
Qwen3-0.6B, Q8_0, Quadro T2000 shared with another process (1.5 GB free), 8 slots of 256 tokens, 64 tokens per request, temperature 0.8, top_p 0.9:
| requests at once | aggregate tok/s | before batching |
|---|---|---|
| 1 | 48.9 | 34.9 |
| 2 | 73.2 | |
| 4 | 102.9 | 39.7 |
| 8 | 124.5 |
Next, in order¶
From the published serving work (vLLM, Sarathi-Serve, SGLang, Flash-Decoding), none of which needs CUDA:
- Benchmark with a standard tool (
vllm bench serveor GuideLLM) and report TTFT (time to first token), ITL (time between tokens) and goodput (requests per second within a latency target). - Paged KV cache (PagedAttention): fixed-size blocks and a
block table per sequence, so short requests do not reserve
max_len. Now each slot holdsmax_lentokens (59 MB per slot for Qwen3-0.6B at 256 tokens). - Chunked prefill (Sarathi-Serve): split a long prompt over several steps, so it does not stall the decodes in its step.
- Prefix caching (RadixAttention): reuse the KV blocks of a shared system prompt. Needs 2.
- Batched attention: one dispatch for all sequences instead of one per sequence per layer.
- Sampling on the GPU (
src/shaders/sample.compexists): download one id per sequence instead of the full logits row. - Speculative decoding: helps at batch 1 to 2, little at large batches.
Also missing: stop strings, n > 1, logprobs, Prometheus metrics.