pymllm.bench_serve
==================

.. py:module:: pymllm.bench_serve

.. autoapi-nested-parse::

   Benchmark pymllm-server prefill (prompt-processing) and decode (token-gen)
   throughput, in the style of ``llama-bench``.

   Methodology (per prefill size P, repeated ``--repeat`` times):
     * PREFILL (pp):  send a prompt of ~P tokens with ``max_tokens=1`` and measure
       pure prefill latency.  Prefill time is taken from the server's own
       ``debug_timing`` block when available (start the server with
       ``--server.enable_debug_timing``); otherwise it falls back to the
       client-side TTFT minus one decode step.
     * DECODE (tg):   send the same prompt with ``max_tokens=D`` (``--decode-tokens``)
       and measure *steady-state* decode throughput from the inter-token arrival
       timestamps, discarding the first token (which still carries prefill +
       first-step warmup).  This mirrors how llama-bench reports tg.

   Results are aggregated over repetitions (mean / std / min / max) and printed as
   a table, plus an overall summary that is directly comparable to llama.cpp's
   ``pp`` / ``tg`` numbers.

   Usage:
       # Start the server first (debug timing gives the most accurate prefill):
       #   PYMLLM_GDN_EXTEND_BACKEND=cuda_chunkwise pymllm-server     #       --server.model_path /path/to/Qwen3.5-2B --server.enable_debug_timing
       python benchmarks/bench_serve.py
       python benchmarks/bench_serve.py --prefill-sizes 256,512,1024 --decode-tokens 128 --repeat 5



Classes
-------

.. autoapisummary::

   pymllm.bench_serve.StreamResult
   pymllm.bench_serve.SizeStats


Functions
---------

.. autoapisummary::

   pymllm.bench_serve.build_prompt
   pymllm.bench_serve.stream_request
   pymllm.bench_serve.server_timed_request
   pymllm.bench_serve.print_report
   pymllm.bench_serve.main


Module Contents
---------------

.. py:function:: build_prompt(approx_tokens, *, for_decode = False)

   Build a prompt of roughly ``approx_tokens`` tokens (~0.75 tok/word).


.. py:class:: StreamResult

   .. py:attribute:: prompt_tokens
      :type:  int


   .. py:attribute:: gen_tokens
      :type:  int


   .. py:attribute:: ttft_s
      :type:  float


   .. py:attribute:: token_times
      :type:  List[float]


   .. py:attribute:: server_prefill_ms
      :type:  Optional[float]
      :value: None



   .. py:attribute:: server_decode_ms
      :type:  Optional[float]
      :value: None



   .. py:attribute:: server_decode_tps
      :type:  Optional[float]
      :value: None



   .. py:property:: decode_tps
      :type: Optional[float]


      Decode throughput, apples-to-apples with llama-bench ``tg``.

      Prefers the server-side decode-loop throughput (excludes HTTP /
      detokenizer / network jitter, like llama-bench's pure-compute tg);
      falls back to client-side inter-token gaps when unavailable.


   .. py:property:: steady_decode_tps
      :type: Optional[float]


      Client-side decode t/s from inter-token gaps, discarding token 1.


   .. py:method:: prefill_tps()


   .. py:method:: prefill_ms()


.. py:function:: stream_request(url, model, prompt, max_tokens, temperature)

.. py:function:: server_timed_request(url, model, prompt, max_tokens, temperature)

   Non-streaming request that returns the server's own ``debug_timing``.

   This is the rigorous, llama-bench-aligned path:
     * prefill: ``experimental_llm_prefill_ms`` = pure model-forward time for
       the prompt (no sampling / detokenize / HTTP), matching llama-bench pp.
     * decode : ``decode_phase_output_tps`` = server-side decode-loop
       throughput (no HTTP / network jitter), matching llama-bench tg.

   Returns ``(prompt_tokens, prefill_ms, decode_tps)`` with ``None`` for any
   field the server did not provide.


.. py:class:: SizeStats

   .. py:attribute:: target_tokens
      :type:  int


   .. py:attribute:: prompt_tokens
      :type:  int
      :value: 0



   .. py:attribute:: prefill_tps
      :type:  List[float]
      :value: []



   .. py:attribute:: decode_tps
      :type:  List[float]
      :value: []



   .. py:attribute:: gen_tokens
      :type:  List[int]
      :value: []



   .. py:attribute:: server_timing
      :type:  bool
      :value: False



   .. py:attribute:: decode_server_timing
      :type:  bool
      :value: False



.. py:function:: print_report(stats, decode_tokens)

.. py:function:: main()