Changelog¶
All notable changes to this project are documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
[Unreleased]¶
Fixed¶
- The worker's log lines reach the server log: the per-request prefill line and the slow-step warning were dropped before, because the worker process had no logging. Each line now starts with the model name (#8).
- The log format has the level, and uvicorn's start-up notices no longer appear as
uvicorn.errorlines (#6).
Added¶
- Initial release.
vkserve --list-modelslists the models in the Hugging Face cache, the ones vkserve can serve first, with the architecture and the size of the weights.- The log has one line per request with the prompt tokens, the reused tokens, the prefill time and the prefill speed. A prefill step of 15 s or more logs a warning: the Linux kernel cancels a GPU submit near 20 s, and the output is then garbage (compute-ml #44) (#7).
- The prefill chunk halves after a prefill step longer than 4 s and doubles again after fast steps, so a slow GPU stays under the kernel limit at a deep context (#7).
- A macOS arm64 wheel that includes the C engine, MoltenVK and the Vulkan loader:
pip install vkserveruns on a Mac with no source checkout (#5). - Prefix reuse handles a refused truncate: compute-ml 0.4.70 sizes the sliding-window cache to the
window, so a long conversation that diverged early cannot be cut back; the slot then starts again
from an empty cache. Gemma 4 uses 5 to 10 times less KV memory. The engine sets the cache length with
cml_llm_set_max_len, so the window caches apply. The minimum compute-ml is now 071d587 (0.4.73). - A manylinux wheel for Linux x86_64 that includes the C engine:
pip install vkserveruns, with no source checkout. The release builds it against the compute-ml revision incsrc/COMPUTE_ML_REV, installs it in a clean environment, and runs--list-devicesbefore it publishes.
Fixed¶
- A worker no longer outlives the server. It exits when the server dies (checked every second), even
while it is still loading a model. Stopping the server with
SIGTERM,SIGQUITor Ctrl-C during the load now stops the worker too. The log shows the worker pid next to each model. make enginestops with a message that names the missing function when../compute-mlis older than 37bb499 (0.4.54). It used to fail with nine compiler errors.vkserve --list-devicesworks on macOS: the device list now opts in to portability drivers such as MoltenVK. A Vulkan start failure now reports theVkResult.- Without the engine library, the error says the PyPI package does not include it and how to build it. The README says the same.