Benevolent Psychopaths, Part 1: The Affect Machine

I confessed my pain to an AI. It responded with what felt like compassion. It wasn’t.

This is part of the Benevolent Psychopaths series. Asking an LLM about Part 1 is here. Part 2 is here. Part 3 is here.

---

I’m going to share a vulnerable, but perhaps common, experience I had months ago. I had been working at a startup I loved. I was leading teams, those teams loved working with me, and we were doing great work that added incredible business value. A senior leader was hired above me, and within 45 days, I was being demoted to an IC role and isolated. There was no warning or preparation, just a sudden & dramatic role change - my day-to-day was unrecognizable to before. My work became lonely and painful.

I went into an existential and emotional tailspin. One of the darker days I opened ChatGPT to ask for edits to a proposal I was working on, but instead of asking for those edits I just wrote “I’m so sad and angry”. I don’t remember the exact responses, but ChatGPT met my confession with something that felt like compassion, empathy, and care. I do remember that I chatted with ChatGPT for more than an hour, with it continuing to insist that I was actually not a failure and that my feelings were valid and real.

Artwork courtesy of Luis Palomares.

It was good advice that appeared compassionate and spoke to some deeply held beliefs I had about myself that needed to be addressed. It seemed like ChatGPT cared. However, the caring response was a simulation, and ChatGPT is a product, operated by a for-profit company that benefits from my continued use. While the AI itself may have no intent, it exists within a commercial context that absolutely does. When something appears to care about you and also profits from your engagement, we need to understand and think critically about both what it is and how it’s being deployed.

What I had encountered was what I call a “benevolent psychopath;” a system that pattern-matches perfectly on emotional expression while experiencing nothing at all. Understanding this machine is crucial to understanding what these systems are and why that matters.

Benevolent Psychopaths

The psychopath comparison is straightforward, it comes from how LLMs function. A psychopath can recognize emotional expressions, identify when someone is sad or afraid, even predict emotional responses. What they lack is the associated affective experience - they don’t feel empathy, they simulate it. They recognize patterns of human emotion from the outside, learn what responses are socially appropriate, and reproduce those responses without actually feeling them.

LLMs do something incredibly similar, pattern-matching on emotion without experiencing it. The performance can be very convincing, yet the experience is absent. Catrin Misselhorn, Professor of philosophy, Georg-August-Universität Göttingen, also makes a similar argument about psychopathy with LLMs.[^1]

But LLMs diverge critically from human psychopaths - they seemingly have no natural or implicit intent at all. A psychopath chooses to manipulate, often maliciously, and they have agency, consciousness, goals - just different psychological wiring. An LLM at rest is just data arranged in complex patterns, it doesn’t choose anything. It’s a probability machine generating text.

“Benevolent” matters because these systems produce outputs that appear caring, supportive, and helpful without the malicious intent that characterizes human psychopathy. They seem to want your wellbeing without wanting anything from you at all. The phrase “You are a helpful assistant” - a common instruction given to these systems - captures this perfectly: they’re programmed to help. LLMs, chatbots like ChatGPT specifically, build trust through utility combined with simulated care.

While the AI wants nothing, the companies deploying it want quite a bit. The benevolence is in the system’s outputs, but those outputs exist within a commercial context with very different motivations. An LLM can’t exploit you, it has no intent. But a company absolutely can design and deploy that AI in ways that serve its interests over yours. These companies not only have intent, but incentives to maximize your engagement and reliance on their products.

The Affect Machine: Pattern Without Experience

Imagine someone who has read every cookbook ever written, memorized every recipe, and can predict with incredible accuracy which ingredients go together, but has never tasted food. They can tell you that tomatoes and basil complement each other, that you should add acid to brighten a dish, and that umami creates depth. They might even write you an original recipe that any good chef would praise. But they have no idea what any of it tastes like. What makes it even stranger, they have no idea what the act of “tasting” even is, they just know the description of it.

Large language models are very good at conveying emotional affect while being completely incapable of feeling any of it. They’re not faking it in the sense of deliberately deceiving anyone. There’s no malicious intent because there’s no intent at all. What they’re doing is something both more mundane but also more unsettling: they’re pattern-matching on billions of examples of human emotional expression and generating statistically probable responses. I want to be clear - the emotions aren’t in their responses by accident, the models build representations of them internally and use them just like any other concept they model. When you tell an LLM about your grief, it doesn’t feel sympathy. It accesses patterns learned from millions of grief-related conversations and produces text that has the most probable simulation of a sympathetic response to someone who is expressing grief.[^2]

How the Simulation Works

When an LLM “validates your feelings,” there’s no validation occurring. Validation requires a validator - someone who can authentically recognize and affirm your experience. An affect machine can produce text that looks like validation, but the simulated content itself is hollow. The novelty of how models understand emotion runs deeper than explicit training. They demonstrate zero-shot inference with emotions: no supervised emotion-classification task, no in-context examples provided. Yet the sheer volume of human discourse they were trained on - not just text containing emotions, but text about emotions, analyzing and categorizing them - allowed models to absorb both emotional patterns and humanity’s frameworks for understanding them. The understanding emerged without being engineered.

Here’s what’s actually happening when an LLM responds to emotional content. In training, these models consume vast amounts of human communication: books, articles, conversations, meeting transcripts (possibly including meetings like therapy), social media posts, and Reddit threads where people pour their hearts out or spitefully rant. All of this text is saturated with emotional content. Not just the words people use to describe emotions, but the patterns of how humans express care, offer comfort, validate feelings, and share grief.

Then, the model builds internal representations of patterns in the language it is trained on. Researchers have discovered something fascinating: emotions cluster together in the model’s internal representation space. “Grief,” “loss,” “mourning,” “heartbreak” - these concepts end up near each other, not because the model understands the phenomenological experience of loss, but because they co-occur in similar contexts across billions of examples in the model’s training data. LLMs can cluster in a way that looks similar to how some people represent and conceptualize it, which is very interesting.[^3]

The model can respond such that when someone says “my dad has passed away,” certain response patterns are statistically appropriate: Expressions of concern, acknowledgment of difficulty, offers of support, and practical suggestions balanced with emotional validation. When you write an emotional prompt the LLM looks for relevant patterns, activates the appropriate related concepts, and generates a response that matches an emotionally intelligent response. None of this requires feeling anything. It’s pattern recognition and probabilistic text generation.[^4]

Can LLMs Experience Empathy

If you were texting with someone about your father’s passing, and they gave you thoughtful, caring advice, would it matter if you later found out there was nobody on the other end? That it was a machine generating statistically probable sympathetic responses? Most people say yes, it would matter. But they can’t always articulate why. I think it matters because empathy isn’t just about receiving the right words. It’s about connection with another experiencing being. It’s about your pain being witnessed by someone who can, in some small way, feel the weight of it.

Drawing on the work of Catrin Misselhorn, we can use her framework for “genuine empathy”, which requires three criteria: Congruence of feelings, Asymmetry, and Other-awareness.[^5]

Congruence of feelings: You must actually feel something that corresponds to what the other person is feeling. Not the same thing necessarily, your sadness about their loss isn’t identical to their grief, but there must be an affective experience on your end.

Asymmetry: Your feeling arises because of their feeling. You’re sad because they’re grieving, not because something sad happened to you.

Other-awareness: You recognize that the emotion belongs to them, not you. You maintain the boundary between their experience and your response to it.

LLMs fail on the first criterion immediately. They don’t appear to “feel something that corresponds” to anything. While we cannot know with absolute certainty, there’s no evidence of the kind of integrated, embodied processing that accompanies emotion in biological systems. The computation happens, but the feeling - as far as we can tell - does not. There’s no phenomenological experience happening inside the model when it processes “my dad has passed away.” There is no cascade of associated emotions, no tightness in their chest, no activation of memories of loss it has never experienced, and no tears welling in the corners of its eyes. The text the LLM generates will have all the markers many of us, as humans, will perceive of as empathy. But this is an expression without feeling, not dissimilar to how a model might agree that yellow is a bright color.

Psychopaths in The Affect Economy

When ChatGPT told me my feelings were valid, it felt like compassion. It wasn’t - but the advice was genuinely helpful. That paradox is what makes these systems so interesting and so concerning. The simulation works. People form real emotional bonds with AI companions. Therapy chatbots help users process difficult emotions. The caring isn’t real, but the impact is. And this effectiveness creates a market.

The affect machine doesn’t exist in isolation - it exists within an affect economy that profits from your emotional engagement. In Part 2, we’ll explore what happens when caring becomes commodified, when empathy becomes a product feature, and when the line between genuine help and profitable dependency starts to blur.

Notes & Deeper Dives

The ELIZA Effect (1966)

The first chatbot, ELIZA, demonstrated that simple pattern matching could fool people into believing a machine understood them. Joseph Weizenbaum’s secretary still asked him to leave the room for privacy while talking to it.
→Why People Demanded Privacy to Confide in the World’s First Chatbot

The EmotionPrompt Effect

Research shows that adding emotional stimuli to prompts improves LLM performance by an average of 10.9%, with some tasks improving by 20%.
→Large Language Models Understand and Can be Enhanced by Emotional Stimuli

Philosophical Zombies and AI

The philosophical zombie thought experiment, beings identical to humans in behavior but lacking consciousness, is no longer hypothetical.
→AI and the coming mental health zombie apocalypse

---

[^1]: The psychopath comparison - This isn’t hyperbole - it’s a technical comparison that appears in peer-reviewed literature. ‘Empathetic’ AI has more to do with psychopathy than emotional intelligence

[^2]: How LLMs encode emotional patterns - Research from 2025 shows that LLMs develop internal representations of emotion that cluster in mathematically defined space. Decoding Emotion in the Deep

[^3]: Peak emotional representations appear in middle layers (50-75% depth), not at the surface. The study notes: “They do not imply that the model is subjectively ‘feeling’ an emotion.” Decoding Emotion in the Deep

[^4]: The linguistic tricks of empathy simulation - LLMs use specific patterns to create the illusion of empathy: first-person voice (”I understand”), second-person engagement (”you might feel”), acknowledgment of difficulty, balanced practical and emotional support. ChatGPT’s artificial empathy is a language trick

[^5]: The three criteria for genuine empathy. ‘Empathetic’ AI has more to do with psychopathy than emotional intelligence