pymllm.bench_serve

Benchmark pymllm-server prefill (prompt-processing) and decode (token-gen) throughput, in the style of llama-bench.

Methodology (per prefill size P, repeated --repeat times):

  • PREFILL (pp): send a prompt of ~P tokens with max_tokens=1 and measure pure prefill latency. Prefill time is taken from the server’s own debug_timing block when available (start the server with --server.enable_debug_timing); otherwise it falls back to the client-side TTFT minus one decode step.

  • DECODE (tg): send the same prompt with max_tokens=D (--decode-tokens) and measure steady-state decode throughput from the inter-token arrival timestamps, discarding the first token (which still carries prefill + first-step warmup). This mirrors how llama-bench reports tg.

Results are aggregated over repetitions (mean / std / min / max) and printed as a table, plus an overall summary that is directly comparable to llama.cpp’s pp / tg numbers.

Usage:

# Start the server first (debug timing gives the most accurate prefill): # PYMLLM_GDN_EXTEND_BACKEND=cuda_chunkwise pymllm-server # –server.model_path /path/to/Qwen3.5-2B –server.enable_debug_timing python benchmarks/bench_serve.py python benchmarks/bench_serve.py –prefill-sizes 256,512,1024 –decode-tokens 128 –repeat 5

Classes

StreamResult

SizeStats

Functions

build_prompt(approx_tokens, *[, for_decode])

Build a prompt of roughly approx_tokens tokens (~0.75 tok/word).

stream_request(url, model, prompt, max_tokens, temperature)

server_timed_request(url, model, prompt, max_tokens, ...)

Non-streaming request that returns the server's own debug_timing.

print_report(stats, decode_tokens)

main()

Module Contents

pymllm.bench_serve.build_prompt(approx_tokens, *, for_decode=False)

Build a prompt of roughly approx_tokens tokens (~0.75 tok/word).

Parameters:

  • approx_tokens (int)

  • for_decode (bool)

Return type:

str

class pymllm.bench_serve.StreamResult
prompt_tokens: int
gen_tokens: int
ttft_s: float
token_times: List[float]
server_prefill_ms: float | None = None
server_decode_ms: float | None = None
server_decode_tps: float | None = None
property decode_tps: float | None

Decode throughput, apples-to-apples with llama-bench tg.

Prefers the server-side decode-loop throughput (excludes HTTP / detokenizer / network jitter, like llama-bench’s pure-compute tg); falls back to client-side inter-token gaps when unavailable.

Return type:

Optional[float]

property steady_decode_tps: float | None

Client-side decode t/s from inter-token gaps, discarding token 1.

Return type:

Optional[float]

prefill_tps()
Return type:

Optional[float]

prefill_ms()
Return type:

Optional[float]

pymllm.bench_serve.stream_request(url, model, prompt, max_tokens, temperature)
Parameters:

  • url (str)

  • model (str)

  • prompt (str)

  • max_tokens (int)

  • temperature (float)

Return type:

StreamResult

pymllm.bench_serve.server_timed_request(url, model, prompt, max_tokens, temperature)

Non-streaming request that returns the server’s own debug_timing.

This is the rigorous, llama-bench-aligned path:

  • prefill: experimental_llm_prefill_ms = pure model-forward time for the prompt (no sampling / detokenize / HTTP), matching llama-bench pp.

  • decode : decode_phase_output_tps = server-side decode-loop throughput (no HTTP / network jitter), matching llama-bench tg.

Returns (prompt_tokens, prefill_ms, decode_tps) with None for any field the server did not provide.

Parameters:

  • url (str)

  • model (str)

  • prompt (str)

  • max_tokens (int)

  • temperature (float)

class pymllm.bench_serve.SizeStats
target_tokens: int
prompt_tokens: int = 0
prefill_tps: List[float] = []
decode_tps: List[float] = []
gen_tokens: List[int] = []
server_timing: bool = False
decode_server_timing: bool = False
pymllm.bench_serve.print_report(stats, decode_tokens)
Parameters:

  • stats (List[SizeStats])

  • decode_tokens (int)

pymllm.bench_serve.main()