pymllm.server.launch ==================== .. py:module:: pymllm.server.launch .. autoapi-nested-parse:: pymllm HTTP server -- RESTful API entry point. This module implements a FastAPI-based HTTP server that wraps the pymllm :class:`Engine` and exposes OpenAI-compatible and native REST endpoints. Endpoints --------- * ``GET /health`` -- liveness probe * ``GET /v1/models`` -- list served models (OpenAI-compatible) * ``POST /generate`` -- native generate (streaming via SSE) * ``POST /v1/completions`` -- OpenAI-compatible completions * ``POST /v1/chat/completions`` -- OpenAI-compatible chat completions * ``GET /model_info`` -- model metadata * ``GET /server_info`` -- runtime config dump * ``POST /flush_cache`` -- flush internal caches * ``POST /abort_request`` -- cancel a running request Attributes ---------- .. autoapisummary:: pymllm.server.launch.logger pymllm.server.launch.app Classes ------- .. autoapisummary:: pymllm.server.launch.GenerateRequest pymllm.server.launch.ImageUrl pymllm.server.launch.ContentPart pymllm.server.launch.ChatMessage pymllm.server.launch.StreamOptions pymllm.server.launch.ToolFunction pymllm.server.launch.Tool pymllm.server.launch.ChatCompletionRequest pymllm.server.launch.CompletionRequest pymllm.server.launch.AbortRequest Functions --------- .. autoapisummary:: pymllm.server.launch.lifespan pymllm.server.launch.http_exception_handler pymllm.server.launch.health pymllm.server.launch.model_info pymllm.server.launch.server_info pymllm.server.launch.list_models pymllm.server.launch.retrieve_model pymllm.server.launch.generate pymllm.server.launch.openai_completions pymllm.server.launch.openai_chat_completions pymllm.server.launch.flush_cache pymllm.server.launch.abort_request pymllm.server.launch.launch_server pymllm.server.launch.main Module Contents --------------- .. py:data:: logger .. py:class:: GenerateRequest(/, **data) Bases: :py:obj:`pydantic.BaseModel` Body for ``POST /generate``. .. py:attribute:: text :type: Optional[Union[List[str], str]] :value: None .. py:attribute:: input_ids :type: Optional[Union[List[List[int]], List[int]]] :value: None .. py:attribute:: sampling_params :type: Optional[Union[List[Dict[str, Any]], Dict[str, Any]]] :value: None .. py:attribute:: image_data :type: Optional[Any] :value: None .. py:attribute:: audio_data :type: Optional[Any] :value: None .. py:attribute:: video_data :type: Optional[Any] :value: None .. py:attribute:: return_logprob :type: Optional[Union[List[bool], bool]] :value: None .. py:attribute:: logprob_start_len :type: Optional[Union[List[int], int]] :value: None .. py:attribute:: top_logprobs_num :type: Optional[Union[List[int], int]] :value: None .. py:attribute:: lora_path :type: Optional[Union[List[Optional[str]], str]] :value: None .. py:attribute:: session_params :type: Optional[Union[List[Dict[str, Any]], Dict[str, Any]]] :value: None .. py:attribute:: stream :type: bool :value: False .. py:attribute:: rid :type: Optional[Union[List[str], str]] :value: None .. py:attribute:: model_config Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict]. .. py:class:: ImageUrl(/, **data) Bases: :py:obj:`pydantic.BaseModel` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: url :type: str .. py:attribute:: detail :type: Optional[str] :value: 'auto' .. py:class:: ContentPart(/, **data) Bases: :py:obj:`pydantic.BaseModel` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: type :type: str .. py:attribute:: text :type: Optional[str] :value: None .. py:attribute:: image_url :type: Optional[ImageUrl] :value: None .. py:class:: ChatMessage(/, **data) Bases: :py:obj:`pydantic.BaseModel` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: role :type: str .. py:attribute:: content :type: Optional[Union[str, List[ContentPart]]] :value: None .. py:attribute:: name :type: Optional[str] :value: None .. py:attribute:: tool_calls :type: Optional[List[Any]] :value: None .. py:attribute:: tool_call_id :type: Optional[str] :value: None .. py:attribute:: model_config Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict]. .. py:class:: StreamOptions(/, **data) Bases: :py:obj:`pydantic.BaseModel` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: include_usage :type: Optional[bool] :value: False .. py:attribute:: continuous_usage_stats :type: Optional[bool] :value: False .. py:class:: ToolFunction(/, **data) Bases: :py:obj:`pydantic.BaseModel` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: name :type: str .. py:attribute:: description :type: Optional[str] :value: None .. py:attribute:: parameters :type: Optional[Dict[str, Any]] :value: None .. py:class:: Tool(/, **data) Bases: :py:obj:`pydantic.BaseModel` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: type :type: str :value: 'function' .. py:attribute:: function :type: ToolFunction .. py:class:: ChatCompletionRequest(/, **data) Bases: :py:obj:`pydantic.BaseModel` OpenAI ``POST /v1/chat/completions`` body. .. py:attribute:: model :type: str :value: '' .. py:attribute:: messages :type: List[ChatMessage] .. py:attribute:: temperature :type: Optional[float] :value: None .. py:attribute:: top_p :type: Optional[float] :value: None .. py:attribute:: top_k :type: Optional[int] :value: None .. py:attribute:: max_tokens :type: Optional[int] :value: None .. py:attribute:: max_completion_tokens :type: Optional[int] :value: None .. py:attribute:: stream :type: bool :value: False .. py:attribute:: stream_options :type: Optional[StreamOptions] :value: None .. py:attribute:: stop :type: Optional[Union[str, List[str]]] :value: None .. py:attribute:: n :type: int :value: 1 .. py:attribute:: frequency_penalty :type: Optional[float] :value: None .. py:attribute:: presence_penalty :type: Optional[float] :value: None .. py:attribute:: repetition_penalty :type: Optional[float] :value: None .. py:attribute:: seed :type: Optional[int] :value: None .. py:attribute:: logprobs :type: Optional[bool] :value: None .. py:attribute:: top_logprobs :type: Optional[int] :value: None .. py:attribute:: user :type: Optional[str] :value: None .. py:attribute:: tools :type: Optional[List[Tool]] :value: None .. py:attribute:: tool_choice :type: Optional[Union[str, Dict[str, Any]]] :value: None .. py:attribute:: separate_reasoning :type: bool :value: True .. py:attribute:: stream_reasoning :type: bool :value: True .. py:attribute:: chat_template_kwargs :type: Optional[Dict[str, Any]] :value: None .. py:attribute:: model_config Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict]. .. py:class:: CompletionRequest(/, **data) Bases: :py:obj:`pydantic.BaseModel` OpenAI ``POST /v1/completions`` body. .. py:attribute:: model :type: str :value: '' .. py:attribute:: prompt :type: Union[str, List[str]] .. py:attribute:: temperature :type: Optional[float] :value: None .. py:attribute:: top_p :type: Optional[float] :value: None .. py:attribute:: top_k :type: Optional[int] :value: None .. py:attribute:: max_tokens :type: Optional[int] :value: None .. py:attribute:: stream :type: bool :value: False .. py:attribute:: stream_options :type: Optional[StreamOptions] :value: None .. py:attribute:: stop :type: Optional[Union[str, List[str]]] :value: None .. py:attribute:: n :type: int :value: 1 .. py:attribute:: frequency_penalty :type: Optional[float] :value: None .. py:attribute:: presence_penalty :type: Optional[float] :value: None .. py:attribute:: repetition_penalty :type: Optional[float] :value: None .. py:attribute:: seed :type: Optional[int] :value: None .. py:attribute:: echo :type: bool :value: False .. py:attribute:: logprobs :type: Optional[int] :value: None .. py:attribute:: user :type: Optional[str] :value: None .. py:attribute:: model_config Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict]. .. py:class:: AbortRequest(/, **data) Bases: :py:obj:`pydantic.BaseModel` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: rid :type: Optional[str] :value: None .. py:function:: lifespan(app) :async: Startup / shutdown hooks for the FastAPI app. .. py:data:: app .. py:function:: http_exception_handler(request, exc) :async: .. py:function:: health() :async: Liveness / readiness probe. Returns 503 if subprocesses died. .. py:function:: model_info() :async: Return basic model metadata. .. py:function:: server_info() :async: Dump runtime server configuration (sensitive fields redacted). .. py:function:: list_models() :async: OpenAI-compatible model listing. .. py:function:: retrieve_model(model_id) :async: OpenAI-compatible single model retrieval. .. py:function:: generate(obj, request) :async: Native generation endpoint. Supports SSE streaming. .. py:function:: openai_completions(obj, request) :async: OpenAI-compatible text completion endpoint. .. py:function:: openai_chat_completions(obj, request) :async: OpenAI-compatible chat completion endpoint with reasoning & tool-call parsing. .. py:function:: flush_cache() :async: Cache flush (not yet implemented). .. py:function:: abort_request(obj) :async: Abort a running request by rid. .. py:function:: launch_server() Launch the pymllm Engine then start the uvicorn HTTP server. It first boots all engine subprocesses (tokenizer, scheduler, model-runner, detokenizer) and then hands off to uvicorn to serve HTTP traffic. .. py:function:: main() CLI entry point.