Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training

Gen Luo*, Xue Yang*, Wenhan Dou*, Zhaokai Wang*, Jifeng Dai, Yu Qiao, Xizhou Zhu
▶ OpenGVLab, Shanghai AI Laboratory ▶ Tsinghua University ▶ Shanghai Jiao Tong University
arXiv Code & Model

Abstract

The rapid advancement of Large Language Models (LLMs) has led to an influx of efforts to extend their capabilities to multimodal tasks. Among them, growing attention has been focused on monolithic Multimodal Large Language Models (MLLMs) that integrate visual encoding and language decoding into a single LLM. Despite the structural simplicity and deployment-friendliness, training a monolithic MLLM with promising performance still remains challenging. In particular, the popular approaches adopt continuous pre-training to extend a pre-trained LLM to a monolithic MLLM, which suffers from catastrophic forgetting and leads to performance degeneration. In this paper, we aim to overcome this limitation from the perspective of delta tuning. Specifically, our core idea is to embed visual parameters into a pre-trained LLM, thereby incrementally learning visual knowledge from massive data via delta tuning, i.e., freezing the LLM when optimizing the visual parameters. Based on this principle, we present Mono-InternVL, a novel monolithic MLLM that seamlessly integrates a set of visual experts via a multimodal mixture-of-experts structure. Moreover, we propose an innovative pre-training strategy to maximize the visual capability of Mono-InternVL, namely Endogenous Visual Pre-training (EViP). In particular, EViP is designed as a progressive learning process for visual experts, which aims to fully exploit the visual knowledge from noisy data to high-quality data. To validate our approach, we conduct extensive experiments on 16 benchmarks. Experimental results not only validate the superior performance of Mono-InternVL compared to the state-of-the-art MLLM on 6 multimodal benchmarks, e.g., +113 points over InternVL-1.5 on OCRBench, but also confirm its better deployment efficiency, with first token latency reduced by up to 67%.

The Monolithic Architecture

Mono-InternVL consists of tokenizers and a multimodal mixture-of-experts structure.

  • (1) Visual and textual embeddings. Compared to modular MLLMs, Mono-InternVL directly patchifies images to input visual sequences using a lightweight module. However, as the GPU computation is fully utilized, the speed difference becomes negligible.
  • (2) Multimodal mixture-of-experts structure. The key principle of Mono-InternVL is to embed visual experts into a pre-trained LLM. In this case, Mono-InternVL can not only facilitate the visual pre-training with the pre-trained LLM knowledge, but also significantly mitigates the catastrophic forgetting issue during pre-training.

Endogenous Visual Pre-training

Endogenous Visual Pre-training (EViP) aims to maximize the benefits of Mono-InternVL from visual experts through pre-training on massive noisy and synthetic data. Unlike existing methods, we formulate EViP from the perspective of delta tuning, in which most of the LLM parameters are frozen to preserve its pre-trained knowledge. EViP is designed as a progressive learning process and consists of three sub-stages, namely concept learning, semantic learning and alignment learning. For different sub-stages, we use carefully partitioned data to achieve coarse-to-fine visual learning.

  • Concept learning. Concept learning aims to encourage the model to learn fundamental visual concepts, such as object categories or basic shapes. Therefore, we first pre-train Mono-InternVL with about 922 million noisy samples, which are sampled from Laion-2B and Coyo-700M. In this sub-stage, Mono-InternVL employs a simple prompt to perform generative learning, i.e., "provide a one-sentence caption for the image". Meanwhile, we constrain the maximum number of image patches of the visual tokenizer to 1,280 for training efficiency. To ensure that the foundational language capabilities are preserved while enabling visual specialization, the entire LLM is kept frozen during concept learning, and only the patch embedding and visual experts are optimized.
  • Semantic learning. After concept learning, Mono-InternVL is able to understand basic concepts in the image, but organizing this information to produce reasonable descriptions remains challenging. To achieve a higher-level visual understanding, we utilize the pre-trained InternVL-8B to produce short captions for 258 million images. Compared to the original noisy captions, synthetic captions typically depict complex visual knowledge, such as relationship and world knowledge, etc., while containing less noisy information unrelated to the image, e.g., time of shooting, and the photographer. In this sub-stage, we adopt the same optimization strategy as concept learning, except that the maximum number of image patches is increased to 1,792.
  • Alignment learning. To meet the visual requirements of downstream tasks, we further perform alignment learning on Mono-InternVL. Our alignment data is sampled from the pre-training data of InternVL-1.5, including 143 million samples of image captioning, detection and optical character recognition (OCR). In particular, captioning data, detection data and OCR data account for about 53.9%, 5.2% and 40.9% of the total, respectively. In this sub-stage, we utilize the task-specific prompts from InternVL-1.5 for the generative learning, and increase the maximum number of image patches to 3,328. Compared to previous sub-stages, the multi-head attention layers are additionally optimized to achieve better vision-language alignment.

Performance

Examples

Citation


  @article{luo2024mono,
    title={Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training},
    author={Luo, Gen and Yang, Xue and Dou, Wenhan and Wang, Zhaokai and Dai, Jifeng and Qiao, Yu and Zhu, Xizhou},
    journal={arXiv preprint arXiv:2410.08202},
    year={2024}
  }

  @article{chen2024far,
    title={How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites},
    author={Chen, Zhe and Wang, Weiyun and Tian, Hao and Ye, Shenglong and Gao, Zhangwei and Cui, Erfei and Tong, Wenwen and Hu, Kongzhi and Luo, Jiapeng and Ma, Zheng and others},
    journal={arXiv preprint arXiv:2404.16821},
    year={2024}
  }

  @inproceedings{chen2024internvl,
    title={Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks},
    author={Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and Zhong, Muyan and Zhang, Qinglong and Zhu, Xizhou and Lu, Lewei and others},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    pages={24185--24198},
    year={2024}
  }

Acknowledgement

This website is adapted from Nerfies, licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.