pymllm.engine.io_struct¶
Classes¶
Module Contents¶
- class pymllm.engine.io_struct.BaseReq¶
- rid: str | List[str] | None = None¶
- regenerate_rid()¶
- Return type:
Union[str, List[str]]
- class pymllm.engine.io_struct.BaseBatchReq¶
- rids: List[str]¶
- regenerate_rids()¶
- Return type:
List[str]
- class pymllm.engine.io_struct.GenerateReqInput¶
Bases:
BaseReq- text: List[str] | str | None = None¶
- input_ids: List[List[int]] | List[int] | None = None¶
- sampling_params: List[Dict[str, Any]] | Dict[str, Any] | None = None¶
- return_logprob: List[bool] | bool | None = None¶
- logprob_start_len: List[int] | int | None = None¶
- top_logprobs_num: List[int] | int | None = None¶
- stream: bool = False¶
- image_data: Any | None = None¶
- video_data: Any | None = None¶
- audio_data: Any | None = None¶
- lora_path: List[str | None] | str | None = None¶
- session_params: List[Dict[str, Any]] | Dict[str, Any] | None = None¶
- extra_options: Dict[str, Any]¶
- is_single: bool = True¶
- batch_size: int = 1¶
- normalize_batch_and_arguments()¶
- Return type:
None
- __getitem__(i)¶
- Parameters:
i (int)
- Return type:
- to_request_dict()¶
- Return type:
Dict[str, Any]
- class pymllm.engine.io_struct.TokenizedGenerateReqInput¶
Bases:
BaseReq- input_text: str = ''¶
- input_ids: List[int] = []¶
- mm_inputs: Dict[str, Any] | None = None¶
- sampling_params: Dict[str, Any]¶
- stream: bool = False¶
- return_logprob: bool = False¶
- logprob_start_len: int = -1¶
- top_logprobs_num: int = 0¶
- lora_path: str | None = None¶
- session_params: Dict[str, Any] | None = None¶
- class pymllm.engine.io_struct.BatchTokenizedGenerateReqInput¶
Bases:
BaseBatchReq- reqs: List[TokenizedGenerateReqInput]¶
- __len__()¶
- Return type:
int
- __getitem__(i)¶
- Parameters:
i (int)
- Return type:
- __iter__()¶
- Return type:
Iterator[TokenizedGenerateReqInput]
- class pymllm.engine.io_struct.BatchTokenIDOutput¶
Bases:
BaseBatchReq- finished_reasons: List[str | None]¶
- decode_ids: List[int]¶
- read_offsets: List[int]¶
- output_ids: List[int] | None¶
- skip_special_tokens: List[bool]¶
- prompt_tokens: List[int]¶
- completion_tokens: List[int]¶
- input_token_logprobs_val: List[float] = []¶
- input_token_logprobs_idx: List[int] = []¶
- output_token_logprobs_val: List[float] = []¶
- output_token_logprobs_idx: List[int] = []¶
- input_top_logprobs_val: List[List[float]] = []¶
- input_top_logprobs_idx: List[List[int]] = []¶
- output_top_logprobs_val: List[List[float]] = []¶
- output_top_logprobs_idx: List[List[int]] = []¶
- class pymllm.engine.io_struct.BatchStrOutput¶
Bases:
BaseBatchReq- finished_reasons: List[str | None]¶
- output_strs: List[str]¶
- output_ids: List[int] | None¶
- prompt_tokens: List[int]¶
- completion_tokens: List[int]¶
- input_token_logprobs_val: List[float] = []¶
- input_token_logprobs_idx: List[int] = []¶
- output_token_logprobs_val: List[float] = []¶
- output_token_logprobs_idx: List[int] = []¶
- input_top_logprobs_val: List[List[float]] = []¶
- input_top_logprobs_idx: List[List[int]] = []¶
- output_top_logprobs_val: List[List[float]] = []¶
- output_top_logprobs_idx: List[List[int]] = []¶