Research Philosophy
Core Philosophy
The whole program in one loop: read the model with cognitive science, and return the insight to model design.
Why This Philosophy
I have always wondered how intelligence works, and from there I also came to wonder how the consciousness and self of the “I” who uses that intelligence work. Along the way, I came across a few books, by Dennett, Hofstadter, Kim, and others, that left a deep mark on me.
I think consciousness and the self are not a soul or spirit sitting separately inside the body, but a special kind of thought pattern (a way in which thinking is put together). The philosopher Daniel Dennett (1991) saw the self not as a separate entity but as a story a system keeps telling about itself (a self-description). Douglas Hofstadter (2007) described the “I” as a pattern that arises from a loop of symbols in the brain that point back at themselves.
I think thought patterns can be tested through behavior, which lets us compare models and people. This is possible if thought patterns show themselves in behavior, and that is my own assumption. Alan Turing (1950) replaced the vague question of whether machines think with a test: can a machine converse in a way that cannot be told apart from a person? That opened a path to testing intelligence through behavior. Jaegwon Kim (2005) held that a mind defined by what it does, as beliefs and desires are, can be explained by the function that does it. That is as far as their ideas support me.
This view rests on an assumption and has limits. I assume that a thought pattern is determined not by what it is made of but by how it is put together, and my own research cannot test that assumption. Kim set aside the felt quality of sensation as something function cannot explain, and Francisco Varela and colleagues (1991) argued that mind arises not from computation inside the brain but from the activity of a living body exchanging with its environment. If Varela is right, my assumption shakes. I do not know whether the feel of sensation can be explained as a thought pattern either. So I look only at how much models and people share in thinking that shows itself in speech and judgment, and I leave sensory experience and the activity of the body out of scope.
The self shows itself in speaking and judging about oneself. It is too large a topic to take on at once, so I first carry thinking that shows itself in speech and judgment from people to AI, and then use what I learn from models to ask about human thinking again. The two directions of my motto are these two steps.
Daniel C. Dennett
Consciousness Explained (1991)
Douglas Hofstadter
I Am a Strange Loop (2007)
Jaegwon Kim
Physicalism, or Something Near Enough (2005)
Varela, Thompson, and Rosch
The Embodied Mind (1991)
Four of the books behind this view: the self as a story, the self as a loop, mind as function, and mind as the activity of a living body.
Earlier Research
To find out what thinking is, and how human thinking differs from the way current language models think, in my earlier research I split thinking into three areas, reasoning, understanding, and decision-making, and examined the behavior of large language models (models like ChatGPT). I found that in every area, models were excessively influenced by their learned data or by the prompt, a pitfall. In all three areas, the models were drawn less by content, such as rules, understanding, or reasons, than by form on the surface: familiar examples, the look of answer options, and the conditions they were given. What the three share, I came to think, is that the model lacks the ability to notice and correct that pull on its own, which is metacognition.
Reasoning
The model was pulled toward familiar solutions rather than solving by understanding the rule, and it seemed unable to check that on its own. ARC is a collection of puzzles in which you find a visual rule from a few examples. The model mostly could not solve newly made problems built on the same rule, even for tasks it had solved once, and even for the same problem, performance changed a great deal depending on whether it was given only the DSL (a program-like description of the rule) or the DSL plus a human’s natural-language explanation (Lee et al., TIST 2025, which took the Language of Thought Hypothesis, a view of human reasoning as the composition of rules, as its frame for evaluation). If it had understood the rule, it should be able to solve the problem even when the look or the form of the input changes, so this looks like a first sign that the model cannot check for itself whether its solution came from the rule. It is where I started to think metacognition is needed.
The Language of Thought lens: three properties of human reasoning (left) against how LLMs hold up on each (right).
Understanding
The models were pulled by the look of the answer options rather than by understanding, and they seemed unable to check that on their own. To see where understanding breaks down, I turned ARC puzzles into multiple choice (MC-LARC, Lee et al., EMNLP Findings 2024), and the models found the correct answer through formal differences among the options alone, such as picking the wording that repeats most across the options or eliminating options that contradict themselves. They scored far above chance even when shown only the options without the puzzle image. When I reduced these tricks, accuracy fell and the models fell far behind people. The model seems unable to tell whether it chose by understanding or by a trick, and this too points to the need for metacognition, the ability to check one’s own judgment.
Generation hides where a model goes wrong; a contrastive multiple-choice format brings the specific mistake into view.
Decision-Making
Given freedom of choice, the model was pulled toward risk and toward irrational choices known from people, and it seemed unable to stop that flow on its own. In a slot machine designed to lose money on average, a model whose bet and goal were fixed usually stopped after a single loss, but when it chose its own bet and goal, it kept going, raising bets and goals to win back lost money, and went bankrupt more often. Even for the same model, risk changed a great deal depending on whether the prompt told it to set a goal (Lee et al., NeurIPS 2026). This pull may come from learning on human-written text, but I could not confirm it. That the model could not change course even as losses piled up suggests it cannot check its own behavior, and here too metacognition is needed.
In the model’s explanations of its own decisions (its self-description), phrases about winning back losses and about reading streaks also increased. But I could not confirm whether those explanations are the real reasons, and I take up that problem below in “Current Research.”
Testing whether a model can drift into gambling-addiction-like behavior.
Attempts and Limits
Both attempts to reduce this pull helped, but both depended on a check from outside the model. One aimed at reasoning: to reduce the pull toward familiar solutions, I injected random vectors (bundles of numbers) into the input so the model produces different solutions (Kim & Lee et al., NeurIPS 2026). When solutions are more varied, a correct one is more likely to appear.
The other was a framework in which an AI agent, one that acts on its own to solve problems, looks back at its own attempt when it gets stuck and rewrites its solving tips (a memo of how to approach problems). In this framework too, whether the reflection was right was judged from outside.
In both attempts, checking whether an answer was right and deciding whether to accept a revised tip was done by people or by external programs. The noticing was done for the model, from outside. That made me wonder whether the model itself knows where it went wrong. I call this ability metacognition, the ability to examine and fix what one is doing, and I formed the hypothesis that this may be what models lack. If the self is seen as a story one tells about oneself, then examining oneself is part of how that story gets made.
One disease, three prescriptions, and where they converge.
Where I Am Heading
Current Research
The three questions of the current research: can a model notice where it may be wrong, correct it, and does its self-description guide its behavior? Together they aim at metacognition inside the model.
To test this hypothesis, I am working on checking for metacognition inside models. The goal is for a model to notice and correct, on its own, the pull seen in the three areas, and I split this into three questions. Of the three questions below, the first two ask whether a model can notice and fix its own errors without outside verification, and the third asks whether its self-description guides its behavior.
The first is whether a model notices where it may be wrong. I look at whether signals inside the model (values such as how low the probability of its answer is) show where it may go wrong. If they do, the next step is to make the model use those signals itself.
The second is whether a model can fix an error it has noticed. To fix it, I believe the model has to know its own state. But a self-description, what the model says about itself, may not be the real reason but an explanation added to fit the answer it gave. So I split this into two steps. First, I find the parts inside the model that seem to have influenced the answer, compare them with the self-description, and also check that the parts I found are right. If the description captures the real reason, I then test whether the model can fix its error with that description, and if they diverge, I map how they diverge.
The third is a different question from the first two: does a self-description lead to self-awareness? I test in models whether the self-story that Dennett described actually guides behavior, and I call this self-awareness. I rewrite the model’s self-description and watch whether its behavior changes with it. But behavior can change after such an edit simply because the model followed a new instruction. Telling those two apart is the hardest part of this question.
Long-Term Vision
My longer-term goal is to understand how intelligence, and the consciousness and self within it, work. I cannot yet answer that question, and the three questions are a first look at the part that shows itself in speech and judgment.
Going from AI back to people is work that lies ahead. Psychology has found that people sometimes add reasons after making a judgment (Nisbett and Wilson, 1977, is the best known). If I can find the conditions under which a model’s self-description and its actual internal parts diverge, I can compare them with this research and ask again how people check their own judgments. But models and people may err in different ways, so I first have to check whether this comparison holds. That comparison is my next goal.
Related Publications
ACM TIST, 2025
Analyzed how language models reason on ARC, using the Language of Thought Hypothesis as the frame.
EMNLP Findings, 2024
Proposed MC-LARC, a multiple-choice benchmark for measuring how well language models understand.
NeurIPS 2026
In slot-machine experiments, language models showed risk-taking that resembles gambling addiction.
NeurIPS 2026
Injecting random vectors into the input makes a language model's solutions more varied.