BloomWise: Enhancing problem-solving capabilities of LLMs using Bloom’s-Taxonomy-inspired prompts

Abstract

The limited ability of Large Language Models (LLMs) in mathematics—a skill critical for solving complex problems—has garnered significant interest from the research community. Many approaches have employed in-context learning to improve LLMs’ performance in such tasks. The most prominent of these focus on encouraging LLMs, through prompts, to approach problems gradually by developing their reasoning in textual form (Chain of Thought) or solving the problem using code (Program of Thought). However, the highest accuracy is achieved by methods that integrate multiple approaches and select the appropriate one for each case, such as the X of Thought (XoT). In this thesis, we propose BloomWise, a new, Bloom’s- Taxonomy-inspired prompting technique aimed at improving LLMs’ performance in solving mathematical problems. BloomWise encourages models to approach problems initially with simple, and, if necessary, progressively higher cognitive skills. Through extensive experiments on various datasets and models, we demonstrate the effectiveness of the method. Additionally, we present variations of the approach, highlight the usefulness of each component through extensive ablation studies, and conduct an in-depth analysis of the results, focusing on the effectiveness of each cognitive skill in the taxonomy, both by dataset and by model. We draw conclusions both about our method and the capabilities of LLMs. Regarding our method, it achieves accuracy comparable to, and sometimes better than, the methods it was compared against. Specifically, the performance of BloomWise is similar to XoT and better than CoT and PoT, while the significantly higher accuracy in the Oracle setting highlights the method’s potential. As for the LLMs, the method offers valuable insights into the cognitive skills each LLM demonstrates, as well as the skills required for solving various types of mathematical problems, thus enhancing interpretability. Some of the key observations are as follows: across all models, the highest accuracy was achieved at the "Analyzing" and "Understanding" stages, and, while difficult problems achieve better performance at higher taxonomy stages, the reverse does not hold true.