pymllm.models.qwen3_5¶

Inference-only Qwen3.5 model for pymllm.

Implements the hybrid attention architecture: - Full attention layers (standard transformer with RoPE + output gate) - GDN linear attention layers (Gated Delta Network, O(n) complexity)

Layers alternate: linear, attention, linear, attention, … based on full_attention_interval in the config.

Supports: - Dense (non-MoE) variant - Vision-Language (multimodal) via inheritance from Qwen3VL

Adapted from sglang’s qwen3_5.py.

Attributes¶

logger

Classes¶

`Qwen3_5FullAttention`	Standard multi-head attention with RoPE, QK-norm, and optional output gate.
`Qwen3_5AttentionDecoderLayer`	Decoder layer with full attention + MLP.
`Qwen3_5LinearDecoderLayer`	Decoder layer with GDN linear attention + MLP.
`Qwen3_5ForCausalLM`	Qwen3.5 causal language model with hybrid attention.
`Qwen3_5VisionRotaryEmbedding`	Simple rotary embedding used inside the vision encoder.
`Qwen3_5VisionMLP`
`Qwen3_5VisionPatchEmbed`	3-D conv patch embedding (same structure as Qwen3VL).
`Qwen3_5VisionPatchMerger`	Merge `spatial_merge_size²` adjacent patches into one token.
`Qwen3_5VisionAttention`	Multi-head attention for the vision encoder.
`Qwen3_5VisionBlock`	Transformer block for the vision encoder.
`Qwen3_5VisionModel`	Qwen3.5 native vision encoder.
`Qwen3_5ForConditionalGeneration`	Qwen3.5 multimodal model (text + vision).

Functions¶

get_rope_index_qwen3_5(input_ids, image_grid_thw, ...)

Compute M-RoPE 3-D position IDs for a single Qwen3.5 sequence.

Module Contents¶

pymllm.models.qwen3_5.logger¶

class pymllm.models.qwen3_5.Qwen3_5FullAttention(config, layer_id, quant_config=None, prefix='')¶

Bases: torch.nn.Module

Standard multi-head attention with RoPE, QK-norm, and optional output gate.

Parameters:

layer_id (int)
prefix (str)

hidden_size¶

num_heads¶

num_kv_heads¶

head_dim¶

q_size¶

kv_size¶

scaling¶

layer_id¶

attn_output_gate¶

q_proj¶

k_proj¶

v_proj¶

o_proj¶

q_norm¶

k_norm¶

partial_rotary_factor¶

rope_theta¶

rotary_dim¶

attn¶

forward(positions, hidden_states, forward_batch)¶

Parameters:

positions (torch.Tensor)
hidden_states (torch.Tensor)
forward_batch (Any)

Return type:

torch.Tensor

class pymllm.models.qwen3_5.Qwen3_5AttentionDecoderLayer(config, layer_id, quant_config=None, prefix='')¶

Bases: torch.nn.Module

Decoder layer with full attention + MLP.

Parameters:

layer_id (int)
prefix (str)

self_attn¶

mlp¶

input_layernorm¶

post_attention_layernorm¶

forward(positions, hidden_states, residual, forward_batch)¶

Parameters:

positions (torch.Tensor)
hidden_states (torch.Tensor)
residual (Optional[torch.Tensor])
forward_batch (Any)

class pymllm.models.qwen3_5.Qwen3_5LinearDecoderLayer(config, layer_id, gdn_layer_idx=0, quant_config=None, prefix='')¶

Bases: torch.nn.Module

Decoder layer with GDN linear attention + MLP.

Parameters:

layer_id (int)
gdn_layer_idx (int)
prefix (str)

linear_attn¶

mlp¶

input_layernorm¶

post_attention_layernorm¶

forward(positions, hidden_states, residual, forward_batch)¶

Parameters:

positions (torch.Tensor)
hidden_states (torch.Tensor)
residual (Optional[torch.Tensor])
forward_batch (Any)

class pymllm.models.qwen3_5.Qwen3_5ForCausalLM(config, quant_config=None)¶

Bases: torch.nn.Module

Qwen3.5 causal language model with hybrid attention.

Alternates between full attention and GDN linear attention layers. Dense (non-MoE) variant.

config¶

quant_config = None¶

hidden_size¶

vocab_size¶

embed_tokens¶

layer_types¶

layers¶

full_attn_layer_ids¶

num_gdn_layers = 0¶

norm¶

forward(input_ids, positions, forward_batch, input_embeds=None)¶

Parameters:

input_ids (torch.Tensor)
positions (torch.Tensor)
forward_batch (Any)
input_embeds (Optional[torch.Tensor])

Return type:

torch.Tensor

load_weights(weights)¶

Load HuggingFace checkpoint weights with name remapping.

Parameters:: weights (Iterable[Tuple[str, torch.Tensor]])

class pymllm.models.qwen3_5.Qwen3_5VisionRotaryEmbedding(dim, theta=10000.0)¶

Bases: torch.nn.Module

Simple rotary embedding used inside the vision encoder.

Parameters:

dim (int)
theta (float)

dim¶

forward(seqlen)¶

Parameters:: seqlen (int)
Return type:: torch.Tensor

class pymllm.models.qwen3_5.Qwen3_5VisionMLP(hidden_size, intermediate_size, hidden_act='gelu_pytorch_tanh')¶

Bases: torch.nn.Module

Parameters:

hidden_size (int)
intermediate_size (int)
hidden_act (str)

linear_fc1¶

linear_fc2¶

forward(x)¶

Parameters:: x (torch.Tensor)
Return type:: torch.Tensor

class pymllm.models.qwen3_5.Qwen3_5VisionPatchEmbed(patch_size, temporal_patch_size, in_channels, embed_dim)¶

Bases: torch.nn.Module

3-D conv patch embedding (same structure as Qwen3VL).

Parameters:

patch_size (int)
temporal_patch_size (int)
in_channels (int)
embed_dim (int)

patch_size¶

temporal_patch_size¶

in_channels¶

embed_dim¶

proj¶

forward(x)¶

Parameters:: x (torch.Tensor)
Return type:: torch.Tensor

class pymllm.models.qwen3_5.Qwen3_5VisionPatchMerger(hidden_size, spatial_merge_size, out_hidden_size)¶

Bases: torch.nn.Module

Merge spatial_merge_size² adjacent patches into one token.

Unlike Qwen3VL there is no deepstack — a single merger outputs out_hidden_size features.

Parameters:

hidden_size (int)
spatial_merge_size (int)
out_hidden_size (int)

merged_dim¶

norm¶

linear_fc1¶

act_fn¶

linear_fc2¶

forward(x)¶

Parameters:: x (torch.Tensor)
Return type:: torch.Tensor

class pymllm.models.qwen3_5.Qwen3_5VisionAttention(hidden_size, num_heads)¶

Bases: torch.nn.Module

Multi-head attention for the vision encoder.

Processes each image/frame separately via cu_seqlens boundaries.

Parameters:

hidden_size (int)
num_heads (int)

hidden_size¶

num_heads¶

head_dim¶

qkv¶

proj¶

forward(hidden_states, cu_seqlens, cos, sin)¶

Parameters:

hidden_states (torch.Tensor)
cu_seqlens (torch.Tensor)
cos (torch.Tensor)
sin (torch.Tensor)

Return type:

torch.Tensor

class pymllm.models.qwen3_5.Qwen3_5VisionBlock(hidden_size, num_heads, intermediate_size, hidden_act)¶

Bases: torch.nn.Module

Transformer block for the vision encoder.

Parameters:

hidden_size (int)
num_heads (int)
intermediate_size (int)
hidden_act (str)

norm1¶

norm2¶

attn¶

mlp¶

forward(x, cu_seqlens, cos, sin)¶

Parameters:

x (torch.Tensor)
cu_seqlens (torch.Tensor)
cos (torch.Tensor)
sin (torch.Tensor)

Return type:

torch.Tensor

class pymllm.models.qwen3_5.Qwen3_5VisionModel(depth=27, hidden_size=1152, hidden_act='gelu_pytorch_tanh', intermediate_size=4304, num_heads=16, in_channels=3, patch_size=16, spatial_merge_size=2, temporal_patch_size=2, out_hidden_size=3584, num_position_embeddings=2304)¶

Bases: torch.nn.Module

Qwen3.5 native vision encoder.

Key differences from Qwen3VL’s Qwen3VLVisionModel:

No deepstack — single merger produces out_hidden_size features.
Learned absolute position embedding (nn.Embedding) with bilinear interpolation to arbitrary resolutions.
2-D rotary position embedding (row, col only — no temporal dim in the rotary frequencies).
Returns a list of per-image tensors instead of a single concatenated tensor with deepstack channels.

Parameters:

depth (int)
hidden_size (int)
hidden_act (str)
intermediate_size (int)
num_heads (int)
in_channels (int)
patch_size (int)
spatial_merge_size (int)
temporal_patch_size (int)
out_hidden_size (int)
num_position_embeddings (int)

spatial_merge_size = 2¶

patch_size = 16¶

hidden_size = 1152¶

out_hidden_size = 3584¶

patch_embed¶

pos_embed¶

num_grid_per_side = 0¶

rotary_pos_emb¶

blocks¶

merger¶

forward(pixel_values, grid_thw)¶

Run vision encoder.

Returns a list of per-image tensors, each of shape [num_merged_tokens_i, out_hidden_size].

Parameters:

pixel_values (torch.Tensor)
grid_thw (torch.Tensor)

Return type:

List[torch.Tensor]

pymllm.models.qwen3_5.get_rope_index_qwen3_5(input_ids, image_grid_thw, image_token_id, vision_start_token_id, spatial_merge_size, start_pos=0)¶

Compute M-RoPE 3-D position IDs for a single Qwen3.5 sequence.

Compatible with pymllm’s [3, T] M-RoPE interface.

Key differences from Qwen3VL’s get_rope_index:

Position advance for vision tokens is max(H, W) // spatial_merge_size (Qwen3VL uses max(T, H, W)).
Temporal position for image tokens is constant (= current_pos).

Parameters:

start_pos (int) – Starting M-RoPE position counter. Must equal extend_prefix_lens so that positions are computed correctly when part of the prompt is already cached (radix-cache prefix hit). Returns delta adjusted to subtract start_pos so that the decode formula pos = (seq_len - 1) + delta remains correct.
input_ids (torch.Tensor)
image_grid_thw (Optional[torch.Tensor])
image_token_id (int)
vision_start_token_id (int)
spatial_merge_size (int)

Return type:

Tuple[torch.Tensor, int]

class pymllm.models.qwen3_5.Qwen3_5ForConditionalGeneration(config, quant_config=None)¶

Bases: torch.nn.Module

Qwen3.5 multimodal model (text + vision).

config¶

quant_config = None¶

spatial_merge_size¶

model¶

num_gdn_layers = 0¶

full_attn_layer_ids¶

lm_head¶

image_token_id¶

video_token_id¶

vision_start_token_id¶

forward(input_ids, positions, forward_batch, input_embeds=None, pixel_values=None, image_grid_thw=None)¶

Parameters:

input_ids (torch.Tensor)
positions (torch.Tensor)
forward_batch (Any)
input_embeds (Optional[torch.Tensor])
pixel_values (Optional[torch.Tensor])
image_grid_thw (Optional[torch.Tensor])

Return type:

torch.Tensor

load_weights(weights)¶

Load HuggingFace checkpoint weights (visual + language).

Parameters:: weights (Iterable[Tuple[str, torch.Tensor]])