📊 GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models

The paper investigates the mathematical reasoning abilities of large language models (LLMs). The authors created a new benchmark, GSM-Symbolic, to test LLMs' performance in a more reliable way. The results show that LLMs perform poorly and inconsistently across different versions of the same question, indicating a fragility in their reasoning abilities. Additionally, the models are sensitive to irrelevant information, suggesting they may be relying on pattern matching rather than true logical reasoning. The study concludes that LLMs still have significant limitations in performing genuine mathematical reasoning and emphasizes the need for further research to develop more robust and logical models.

📎 Link to paper