Research Philosophy
Core Philosophy
The whole program in one loop: read the model with cognitive science, and return the insight to model design.
Why This Philosophy
I have always wondered how intelligence works, and from there I also came to wonder how the consciousness and self of the “I” who uses that intelligence work. Along the way, I came across a few books, by Dennett, Hofstadter, Kim, and others, that left a deep mark on me.
Reading those books changed how I think. I came to see consciousness and the self not as a soul sitting separately inside the body, but as a thought pattern, that is, a way in which thinking is put together. The philosopher Daniel Dennett (1991) saw the self as a story rather than an entity. It is a story a system keeps telling about itself, that is, a self-description. Douglas Hofstadter (2007) described the “I” as a pattern that arises from a loop of symbols in the brain that point back at themselves.
If the self is a pattern, then to the extent it shows itself in behavior, it can be tested. Alan Turing (1950) replaced the vague question “Do machines think?” with a test: can a machine converse in a way that cannot be told apart from a person? Jaegwon Kim (2005) held that a mind defined by what it does, as beliefs and desires are, can be explained by the function that does it. What the two of them support is narrow: that we can test such thinking through behavior and compare models with people. That a thought pattern shows itself in behavior is my own assumption.
This view holds one more assumption, and it has limits. I assume that a thought pattern is determined not by what it is made of but by how it is put together, as a tune stays the same tune whether it is played on a piano or a guitar. My own research cannot test that assumption. Kim conceded that something in the felt quality of sensation remains that function cannot explain. Francisco Varela and colleagues (1991) did not see mind as computation inside the brain. They argued that it arises from the activity of a living body exchanging with its environment. If Varela is right, my assumption shakes. So I deal only with thinking that shows itself in speech and judgment, and I leave sensory experience and the activity of the body out of scope. Carrying this thinking from people to AI and testing it there is my earlier research.
Daniel C. Dennett
Consciousness Explained (1991)
Douglas Hofstadter
I Am a Strange Loop (2007)
Jaegwon Kim
Physicalism, or Something Near Enough (2005)
Varela, Thompson, and Rosch
The Embodied Mind (1991)
Four of the books behind this view: the self as a story, the self as a loop, mind as function, and mind as the activity of a living body.
Earlier Research
To find out what thinking is, and how human thinking differs from the way current language models think, in my earlier research I split thinking into three areas, reasoning, understanding, and decision-making, and examined the behavior of large language models (models like ChatGPT). I found that in every area, models were excessively influenced by their learned data or by the prompt, a pitfall. In this piece I call that influence a pull. Models were drawn less by content than by form on the surface. In reasoning they followed a familiar solution instead of the rule, in understanding the look of the answer options instead of understanding, and in decision-making the conditions they were given instead of reasons. Can a model notice by itself what is pulling it?
Reasoning
The model was pulled toward familiar solutions rather than understanding the rule. ARC is a collection of puzzles in which you find a visual rule from a few examples. If a model had understood the rule, it should be able to solve the problem even when its look changes, so I had the model solve again newly made problems built on the same rule as ones it had already solved. It mostly could not. Even for the same problem, performance changed a great deal depending on whether it was given only the DSL (a description of the rule written like a program) or the DSL plus a human’s natural-language explanation. This means results changed with the representation even when the rule was the same. The frame for evaluation was the Language of Thought Hypothesis, a view of human reasoning as the composition of rules, and in that frame the model fell short of people (Lee et al., TIST 2025). If it is hard to say the model understood the rule, how does it understand the problem?
The Language of Thought lens: three properties of human reasoning (left) against how LLMs hold up on each (right).
Understanding
The model was pulled by the look of the answer options rather than by understanding. To see where understanding breaks down, I turned ARC puzzles into multiple choice (MC-LARC, Lee et al., EMNLP Findings 2024). I did so to keep one small slip from making a whole answer wrong, and to measure the understanding step on its own. But the model found the correct answer from cues alone. It picked wording that repeats across the options or eliminated options that contradict themselves. It scored far above chance even when shown only the options without the puzzle image. When I reduced these cues, accuracy fell and the model fell far behind people. Being right does not show that it understood. Would decision-making also shake in the same way, not only understanding?
Generation hides where understanding breaks down; a multiple-choice format lets the understanding step be measured on its own.
Decision-Making
Once it was given the condition of freedom to choose, the model was pulled toward risk, and the behavior resembled what is seen in gambling addiction. To see whether a language model also chooses irrationally, as people do, I used a slot machine designed to lose money on average. When the bet and the target amount were fixed, the model usually quit after a single loss. But when it chose for itself, it kept going: it raised bets and goals to win back lost money, and went bankrupt more often. Even for the same model, risk changed a great deal depending on whether the prompt told it to set a goal (Lee et al., NeurIPS 2026). This tendency may come from learning on human-written text, but I could not confirm it. I also have not tested, in any of the three areas, whether the model notices that it is being pulled.
Testing whether a model can drift into gambling-addiction-like behavior.
Attempts and Limits
Both attempts to reduce this pull leaned on a check from outside the model, that is, on people or external programs deciding what is right. One aimed at reasoning: to reduce the pull toward familiar solutions, I injected random vectors (bundles of numbers) into the input so the model produces different solutions (Kim & Lee et al., NeurIPS 2026). The solutions did become more varied, but picking the correct one among them was left to outside the model.
The other was a framework for an AI agent, one that acts on its own to solve problems. When it gets stuck, it looks back at its own attempt and rewrites its solving tips (a memo of how to approach problems). In this framework too, whether to accept a revised tip was decided from outside the model.
In the end, noticing where the model went wrong was done for it, from outside. That made me wonder whether the model itself knows where it went wrong. I call the ability to examine and fix one’s own work metacognition, and I formed the hypothesis that this may be what models lack.
Two attempts to reduce the pull, one for reasoning and one for an agent, both lean on a check from outside the model, which raises the question of whether the model can notice by itself.
Where I Am Heading
The work ahead starts by testing this hypothesis inside the model and extends to sending the result back to the understanding of people.
Current Research
The three questions of the current research: can a model notice where it may be wrong, correct it, and does its self-description guide its behavior? Together they aim at metacognition inside the model.
To test this hypothesis, I am studying metacognition inside models. The goal is for a model to notice and correct, on its own, the pull seen earlier. There are three questions. Noticing is examining one’s work, and correcting is fixing it. To these I add whether the model’s self-description, what it says about itself, guides its behavior, because to fix something one has to know one’s own state. The way I answer differs by question: the first through analysis inside the model and its use while it is solving, the second through training, and the third by building a new evaluation set.
The first is noticing. Can a model notice where it may be wrong? A person solving an exam feels that a problem seems off and checks it again. I first analyze whether a model has a signal that corresponds to this feeling. I look at whether signals such as how low the probability of its answer is, or values inside the model, appear together with wrong answers. If they do, traces of where it will be wrong are inside the model. Next, I build a method that lets the model read this signal itself, stop its solution, and solve again during inference. That is because a person watching the signal and stepping in would itself be a check from outside.
The second is correcting. Can a model fix an error it has noticed on its own? Tell a student to check their work and they may only pretend to, leaving the answer as it was. A model told only to check may likewise learn only to look as if it is fixing. So I train the model with a reward only when, after checking, the answer actually changed toward the correct one. Pretending earns no reward, so the aim is to leave only the behavior of actually fixing. When training, I confirm whether the answer is correct to give the reward, but when evaluating, I look at whether the model fixes errors without a check from outside. The aim is to see whether the model takes inside itself the outside check used in training. If it does, correction can be learned inside the model.
The third is self-description. Ask a person why they answered as they did and they give a plausible reason, but sometimes it differs from the real one. I want to build an evaluation set that measures whether a model’s self-description is like that. It asks whether the self-description Dennett described actually guides behavior in the model, and if it does, I call that state self-awareness. The evaluation set measures whether behavior changes accordingly when the self-description is changed. For example, I change a self-description from “I answer carefully” to “I answer boldly” and see whether the answers change. I build it as a set so that models can be compared with the same yardstick. But behavior may change not because of the description but because the model followed a new instruction, such as “answer boldly.” Designing items that separate the effect of the description from instruction-following is the hardest part, and if they can be separated, it will be a first piece of evidence that self-description guides behavior.
Long-Term Vision
The answer to the third question may return to people. Psychology has found that people sometimes add reasons after making a judgment (Nisbett and Wilson, 1977, is the best known). If I can find the conditions under which a model’s self-description and its actual behavior diverge, I can compare them with this research and ask again how people check their own judgments. I believe the understanding gained that way makes AI more beneficial to people.
But models and people may err in different ways, so I first have to check whether this comparison holds. A more distant goal is to understand how intelligence, and the consciousness and self within it, work. I cannot yet answer that question, and the three questions are a first look at the part of it that shows itself in speech and judgment.
Related Publications
ACM TIST, 2025
Analyzed how language models reason on ARC, using the Language of Thought Hypothesis as the frame.
EMNLP Findings, 2024
Proposed MC-LARC, a multiple-choice benchmark for measuring how well language models understand.
NeurIPS 2026
In slot-machine experiments, language models showed risk-taking that resembles gambling addiction.
NeurIPS 2026
Injecting random vectors into the input makes a language model's solutions more varied.