NaViL: Rethinking Scaling Properties of Native MLLMs

Abstract

Compositional training has been the de-facto paradigm in existing Multimodal Large Language Models (MLLMs), where pre-trained vision encoders are connected with pre-trained LLMs through continuous multimodal pre-training. However, the multimodal scaling property of this paradigm remains difficult to explore due to the separated training.

In this paper, we focus on the native training of MLLMs in an end-to-end manner and systematically study its design space and scaling property under a practical setting, i.e., data constraint. Through careful study of various choices in MLLM, we obtain the optimal meta-architecture that best balances performance and training cost. After that, we further explore the scaling properties of the native MLLM and indicate the positively correlated scaling relationship between visual encoders and LLMs.

Based on these findings, we propose a native MLLM called NaViL, combined with a simple and cost-effective recipe. Experimental results on 14 multimodal benchmarks confirm the competitive performance of NaViL against existing MLLMs. Besides that, our findings and results provide in-depth insights for the future study of native MLLMs.

Core Insights

We conducted a systematic study on the design and scaling properties of native MLLMs, leading to five key conclusions that guided the design of NaViL:

1. LLM Initialization is Crucial

Initializing the model from a pre-trained LLM significantly accelerates the convergence of multimodal training. Its performance is generally superior to training from scratch, even with a large amount of multimodal data.

2. MoE Architecture is Effective

The Mixture-of-Experts (MoE) architecture can significantly enhance the model's ability to process heterogeneous data and improve overall performance without increasing inference costs (activated parameters). We found that introducing modality-specific experts for both attention and feed-forward networks (FFN) yields the best results.

3. Flexibility of Visual Encoder Architecture

For a given parameter budget, the performance of the visual encoder is nearly optimal across a wide range of depth and width configurations. Shallower encoders converge faster in the early stages of training, while deeper encoders perform slightly better with more data.

4. Asymmetric Scaling Effects

Scaling up the LLM consistently improves multimodal performance, following traditional language model scaling laws. However, the benefits of scaling the visual encoder diminish, with its performance ceiling being constrained by the LLM's capacity.

5. Joint Scaling Law for Vision and Language

Our research reveals for the first time that the optimal scale of the visual encoder is directly proportional to the scale of the LLM on a logarithmic scale. This implies that they should be scaled jointly and highlights the sub-optimality of existing compositional MLLMs that pair a fixed-size visual encoder with LLMs of different sizes.

NaViL Architecture

Based on the insights above, we built NaViL. It is a native, MoE-based MLLM that can be trained end-to-end and natively supports images of arbitrary resolutions.

Visual Encoder: Responsible for the initial extraction of visual information.
MLP Connector: Projects visual features into the LLM's feature space.
MoE-extended LLM: Contains modality-specific attention (MHA-MMoE) and feed-forward networks (FFN-MMoE) to fuse visual and text information more effectively.
Visual Multi-scale Packing: Further enhances model performance during inference by processing image inputs at multiple scales.

Performance

We conducted a comprehensive evaluation of NaViL on 14 mainstream multimodal benchmarks, covering general capabilities, visual question answering, OCR, chart, and document understanding. With comparable parameter sizes, NaViL-2B and NaViL-9B surpass all existing native MLLMs in average performance and achieve a level comparable to top-tier compositional MLLMs (e.g., InternVL-2.5, Qwen2.5-VL).

Model	#A-Param	Avg	MMVet	MMMU	MMB	MME	MathVista	OCR-Bench	TextVQA	DocVQA	AI2D	ChartQA	InfoVQA
Compositional MLLMs
Qwen2.5-VL	8.2B	80.2	67.1	58.6	83.5	2347	68.2	864	84.9	95.7	83.9	87.3	82.6
InternVL-2.5	8.1B	77.3	62.8	56.0	84.6	2344	64.4	822	79.1	91.9	84.5	84.8	75.7
Native MLLMs
EVEv2	7B	62.3	45.0	39.3	66.3	1709	60.0*	702	71.1	77.4*	74.8	73.9	45.8*
SAIL	7B	63.7	46.3	38.6*	70.1	1719	57.0	783	77.1	78.4*	76.7	69.7*	47.3*
NaViL-2B (ours)	2.4B	68.8	78.3	41.8	71.2	1822	50.0	796	76.9	85.4	74.6	78.0	56.0
NaViL-9B (ours)	9.2B	77.0	79.6	54.7	76.5	2225	66.7	837	77.2	90.6	82.4	85.4	70.2

* denotes results tested locally using VLMEvalKit and OpenCompass.
The average score is computed by normalizing each metric to a range of 0-100.

Qualitative Analysis

By visualizing attention maps, we found that a sufficiently large visual encoder (following our joint scaling law) helps the model focus on global information in shallower layers and promotes earlier interaction between visual and text features, which explains the performance improvement.

Top: Using a 150M visual encoder; Bottom: Using a 1.2B visual encoder. The latter exhibits stronger global attention and cross-modal interaction even in shallow layers (Layer 1).

Citation

@article{tian2025navil,
  title={NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints},
  author={Tian, Changyao and Li, Hao and Luo, Gen and Zhu, Xizhou and Su, Weijie and Deng, Hanming and Zhu, Jinguo and Shao, Jie and Zhu, Ziran and Liu, Yunpeng and Lu, Lewei and Wang, Wenhai and Li, Hongsheng and Dai, Jifeng},
  journal={arXiv preprint},
  year={2025}
}

Acknowledgement

This website is adapted from Nerfies, licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints

NeurIPS 2025