(71) Generative AI characteristics (2)
By Onno Hansen-Staszyński | Last Updated: 11 October 2025
The following is a second fragment from a draft version of an academic paper I’m currently working on regarding text-based generative artificial intelligence (GAI)/ large language models (LLMs).
Characteristic 2: pattern reproduction, not reasoning
Wolfram (2023) found that ChatGPT is „just saying things that “sound right” based on what things “sounded like” in its training material”. GAIs have the capacity to build upon their training data, but only up to a certain extent. When confronted with high-complexity tasks, GAIs collapse[1]. (Shojaee, 2025) Even adding nonsense texts to prompts seriously undermines GAIs’ performance. (Rajeev et al., 2025)
Zhao et al. (2025) investigated „if CoT reasoning reflects a structured inductive bias learned from in-distribution data, allowing the model to conditionally generate reasoning paths that approximate those seen during training.”[2] They conclude: “Empirical findings consistently demonstrate that CoT reasoning effectively reproduces reasoning patterns closely aligned with training distributions but suffers significant degradation when faced with distributional deviations. Such observations reveal the inherent brittleness and superficiality of current CoT reasoning capabilities.”
This means that GAIs fail to develop generalizable reasoning capacities (Shojaee, 2025; Pearl, 2018), lack the robustness and generality of human analogy-making (Lewis & Mitchell, 2024), and have limited (Wang et al., 2025) or no cognitive comprehension (Bishop, 2021). Schroeder et al. (2025) conclude that GAI models behave differently to humans and are conceptually distinct from human minds. They do not reach human-like and -level of cognition (Van Rooij et al., 2024).
Just as the first characteristic necessitates a ‘zero-trust’ policy toward GAI outputs due to their indifference towards truth, the second characteristic demands a zero-trust stance toward their cognitive depth. Users must verify GAI outputs not only for factual accuracy but also for applicability, using human reasoning to evaluate whether the responses are suitable, relevant, and effective for the specific problem or context at hand.
Literature
· Bishop, J. (2021). Artificial Intelligence Is Stupid and Causal Reasoning Will Not Fix It. Front. Psychol. https://doi.org/10.3389/fpsyg.2020.513474
· Lewis, M. & Mitchell, M. (2024). Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in Large Language Models. ArXiv. https://doi.org/10.48550/arXiv.2402.08955
· Pearl. J. (2018). Theoretical Impediments to Machine Learning With Seven Sparks from the Causal Revolution. ArXiv. https://arxiv.org/abs/1801.04016
· Rajeev, M. et al. (2025) Cats Confuse Reasoning LLM: Query Agnostic Adversarial Triggers for Reasoning Models. ArXiv. https://arxiv.org/pdf/2503.01781v1
· Schroeder, S. et al (2025) Large Language Models Do Not Simulate Human Psychology. ArXiv. https://doi.org/10.48550/arXiv.2508.06950
· Shojaee, P. (2025). The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity. ArXiv. https://arxiv.org/abs/2506.06941
· Van Rooij, I. et al. (2024). Reclaiming AI as a Theoretical Tool for Cognitive Science_. Comput Brain Behav 7_, 616–636. https://doi.org/10.1007/s42113-024-00217-5
· Wang, Y. et al. (2025). Drivel-ology: Challenging LLMs with Interpreting Nonsense with Depth. ArXiv. https://arxiv.org/abs/2509.03867
· Wolfram, S. (2023). What Is ChatGPT Doing … and Why Does It Work? stephenwolfram.com. https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/
· Zhao, C. et al. (2025). Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens. ArXiv. https://arxiv.org/abs/2508.01191
Footnotes
[1] As do more specialized Large Reasoning Models (LRMs).
[2] Chain-of-Thought (CoT) entails complex reasoning through intermediate reasoning steps.
Subscribe now &
Get the latest updates
Subscribe