Title: Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding
https://arxiv.org/pdf/2306.02858.pdf
Authors: Hang Zhang, Xin Li and Lidong Bing from DAMO Academy, Alibaba Group
Word Count: Approximately 2200
Read Time: Around 5-7 minutes
Source Code: The authors have open-sourced the entire codebase for pre-training and fine-tuning as well as the model weights at https://github.com/DAMO-NLP-SG/Video-LLaMA
Video-LLaMA is an audio-visual language model that aims to empower large language models with the ability to understand visual and auditory content in videos. It has two branches:
Vision-Language Branch: Uses a pre-trained image encoder for video frames and a Video Q-Former to generate visual query tokens that are compatible with the LLM's text embeddings.
Audio-Language Branch: Uses a pre-trained ImageBind audio encoder and an Audio Q-Former to generate audio query tokens that align with the LLM's embeddings.
Video-LLaMA is trained in a multi-branch fashion:
The vision components are first pre-trained on video caption datasets to learn video-text correspondence.
They are then fine-tuned on instruction-following datasets to gain vision-instruction ability.
The audio components leverage the shared embedding space from ImageBind and are trained on visual datasets due to lack of audio data.
The model demonstrates the ability to perceive and comprehend video content, generating meaningful responses grounded in visual and audio information.
In summary, Video-LLaMA shows potential as a prototype for audio-visual AI assistants but has limitations like limited perception capacity and handling of long videos.
This model demonstrates how large language models can be extended with multimodal capabilities through a modular approach, leveraging pre-trained vision and audio encoders. With further improvements, such text-to-video understanding models could enable various applications like video summarization, visual dialogue systems, etc.
