Artificial Intelligence (AI) has come a long way, with language models (LMs) like GPT-3 showcasing impressive abilities. But how much of these abilities are due to genuine reasoning, and how much is simply reciting learned patterns? A recent paper titled ‘Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks’ delves into this question.
The researchers behind the paper propose a unique way to evaluate these AI models. They use what they call ‘counterfactual’ tasks. These are tasks that deviate from the norm, changing the conditions or rules under which these tasks are performed. The reasoning procedure for these tasks remains the same under the new conditions, but the specific input-output mappings are changed. This approach helps to determine whether the skills these models exhibit are general and transferable, or specialized to specific tasks seen during pretraining.
The study involves 11 tasks, including arithmetic, programming, basic syntactic reasoning, natural language reasoning with first-order logic, spatial reasoning, drawing, music, and chess. For each task, the authors create counterfactual variants.
The results are fascinating. While current LMs may possess abstract task-solving skills to a degree, they often also rely on narrow, non-transferable procedures for task-solving. This suggests that the impressive performance of LMs across a wide range of tasks is not solely due to their abstract reasoning skills. Instead, they also heavily rely on recognizing and recalling specific tasks seen frequently during pre-training.
The authors also highlight several surprising relations between model behavior on default and counterfactual tasks, including correlations between default and counterfactual performance, varying effectiveness of zero-shot chain-of-thought prompting, and interactions between task- and instance-level frequency effects.
In layman’s terms, this means that while AI models like GPT-3 are indeed impressive, their abilities are not as generalized as we might think. They are excellent at recognizing and recalling tasks they’ve seen before, but when presented with a new variant of a task, their performance can vary.
This study provides valuable insights into the capabilities and limitations of language models, suggesting the need for a more careful interpretation of language model performance that teases apart these aspects of behavior. It’s a reminder that while AI has come a long way, there’s still much we don’t understand about how these models learn and reason.