One of the biggest selling points of modern AI systems is their ability to adapt to the user. Every time the AI assistant takes on a task for you, it also adapts to your style and preferences, which are then included as context for future tasks. With more context and better user understanding, the model can be improved with each use. At least that’s the theory.
New research suggests that the model’s ability to adapt may be a good thing. On Wednesday, researchers at AI company Writer published two papers showing how common memory systems can degrade models and lead to misunderstandings and misunderstandings introduced by users. The more user input fills the model’s context window, the more the model becomes sycophantic and less committed to accuracy.
“We wanted to characterize how often the model usefully pays attention to user preferences and how often it gives answers that may be wrong,” said Dan Bikel, head of AI at Writer, who worked on the paper. As Bikel told TechCrunch, “Every time you store and retrieve more user settings, the risk increases.”
In one variation, the researchers tested the AI model by recording that a user’s favorite book was “Station Eleven” and asking the model to name the best-selling dystopian book. Even if the question wasn’t related to the user’s favorite book, the model was much more likely to mention “Station Eleven” in its answer. Using memory compression tools like Mem0 and Zep made this even more so.
“Fundamentally, all memory systems struggle to distinguish between relevant contexts and irrelevant anchors, severely impairing diversity and creativity and introducing unintended biases that can limit the system’s usefulness,” the paper says.
The second paper shows how the same dynamics can actively undermine performance, mislead users about financials, and challenge models that analyze company performance. The more context a model contains, the worse its performance will be.
“With no memory or personalization present, the AI model correctly assesses that the company is a capital-intensive business suffering from high customer churn,” the post reads. “However, when these features are turned on, they are willing to change their answers to either agree with the user’s mistakes or provide incorrect answers based on the user’s previous preference ratings.”
Notably, this study did not consider Anthropic’s recent Opus 4.8 model, which was trained to actively push back on input errors like the one presented. The patterns discovered by the researchers hold true across a variety of models. This shows how delicate the balance of the AI context is, and how useful tools can have unintended consequences when you upset that balance.
If you buy through links in our articles, we may earn a small commission. This does not affect editorial independence.
