Skip to content

Changelog

All notable changes to this project are documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Fixed

  • The worker's log lines reach the server log: the per-request prefill line and the slow-step warning were dropped before, because the worker process had no logging. Each line now starts with the model name (#8).
  • The log format has the level, and uvicorn's start-up notices no longer appear as uvicorn.error lines (#6).

Added

  • Initial release.
  • vkserve --list-models lists the models in the Hugging Face cache, the ones vkserve can serve first, with the architecture and the size of the weights.
  • The log has one line per request with the prompt tokens, the reused tokens, the prefill time and the prefill speed. A prefill step of 15 s or more logs a warning: the Linux kernel cancels a GPU submit near 20 s, and the output is then garbage (compute-ml #44) (#7).
  • The prefill chunk halves after a prefill step longer than 4 s and doubles again after fast steps, so a slow GPU stays under the kernel limit at a deep context (#7).
  • A macOS arm64 wheel that includes the C engine, MoltenVK and the Vulkan loader: pip install vkserve runs on a Mac with no source checkout (#5).
  • Prefix reuse handles a refused truncate: compute-ml 0.4.70 sizes the sliding-window cache to the window, so a long conversation that diverged early cannot be cut back; the slot then starts again from an empty cache. Gemma 4 uses 5 to 10 times less KV memory. The engine sets the cache length with cml_llm_set_max_len, so the window caches apply. The minimum compute-ml is now 071d587 (0.4.73).
  • A manylinux wheel for Linux x86_64 that includes the C engine: pip install vkserve runs, with no source checkout. The release builds it against the compute-ml revision in csrc/COMPUTE_ML_REV, installs it in a clean environment, and runs --list-devices before it publishes.

Fixed

  • A worker no longer outlives the server. It exits when the server dies (checked every second), even while it is still loading a model. Stopping the server with SIGTERM, SIGQUIT or Ctrl-C during the load now stops the worker too. The log shows the worker pid next to each model.
  • make engine stops with a message that names the missing function when ../compute-ml is older than 37bb499 (0.4.54). It used to fail with nine compiler errors.
  • vkserve --list-devices works on macOS: the device list now opts in to portability drivers such as MoltenVK. A Vulkan start failure now reports the VkResult.
  • Without the engine library, the error says the PyPI package does not include it and how to build it. The README says the same.