Enhancing Vision-Language Models: The Role of LLMs in Augmenting Performance and Reasoning

Abstract

Vision-Language Models (VLMs) have demonstrated remarkable capabilities in complex visio-linguistic tasks. An extensive body of work has explored how prompting techniques and fine-tuning methods can be used to enhance their performance. However, modern multimodal LLMs still struggle with tasks that require complex reasoning, external knowledge, and human-aligned responses. In this work, we investigate the limitations of large-scale, multimodal models in handling open-ended tasks that demand external knowledge and commonsense reasoning. Focusing on the Stanford Image Paragraph Captioning and OK-VQA datasets, we find that although these models demonstrate substantial cognitive, linguistic, and reasoning abilities, their performance deteriorates when managing complex tasks simultaneously while adhering to specific response formats. Our analysis reveals that state-of-the-art multimodal models surpass existing datasets in paragraph generation but continue to face challenges in generating high-quality paragraphs. Similarly, they continue to struggle with knowledge-based, open-ended benchmarks such as OK-VQA. To boost their performance in the latter, we employ a collaborative framework comprising three models: the Scout, an LVLM that takes an image as input and describes it in a paragraph; the Analyser, an LLM that generates an initial answer to the question based on the image description; and the Resolver, an LLM that extracts and formats the final answer based on a set of predefined rules. Our framework yields improved performance over the single-agent baseline, indicating the effectiveness of a collaborative approach.