Skip to content
mllm

Quantize model

You can quantize the mllm model to an int4 model on your own.

Before quantizing the mllm model, please follow the model conversion tutorial to obtain your mllm float model, or download it from here. We only support two quantization modes: Q4_0 and Q4_K.

Terminal window
cd bin
./quantize model.mllm model_q4_0.mllm Q4_K