Sentient AI?

Most people are forming their opinions about AI from the wrong perspective.

If you haven’t read my previous Substack: Knowledge is King [https://evidenceledai.substack.com/p/knowledge-in-ai?r=3s8sw&utm_campaign=post-expanded-share&utm_medium=web], it is probably worth reading first. Because I think it provides an important piece of the puzzle.

Think about what had to happen to get us here.

Scientists spent centuries discovering things about the world. Engineers turned those discoveries into mathematics, machines, computers and eventually neural networks. People wrote books, papers, manuals, code, histories and conversations. All of that accumulated knowledge became training material for increasingly powerful AI systems.

And then there is the enormous engineering effort behind the systems themselves.

Without the scientists and engineers at Google, OpenAI, Anthropic and hundreds of other organizations, there would be no modern AI. Without researchers developing the algorithms and methods, there would be no models. Without the computing industry producing increasingly powerful processors, memory and networking, there would be no practical way to run them. Without the energy infrastructure to power those machines, there would be no AI factory.

And without humans deciding what these systems should and should not do, there would be no guardrails.

Take all of that away and AI is, ultimately, mathematics running on machines.

But now we have given those machines an extraordinary amount of human knowledge.

So, what happens next?

Will AI surpass human knowledge? Perhaps in some areas it already has.

An AI can process thousands of pages in a fraction of the time it would take a human. It can compare documents, recognize patterns, write software, summarize research and make connections across enormous amounts of information.

• It doesn’t get tired.
• It doesn’t need coffee.
• It doesn’t get bored halfway through the 900-page technical specification.

And it can do all of this at a scale that individual humans simply cannot match.

So it is tempting to conclude that eventually AI will simply surpass us. But I think there is something important missing from that argument.

Knowledge isn’t the same thing as wanting.

• Why do we want anything?
• Why does a person want to succeed?
• Why does a parent care about their children?
• Why does a scientist spend twenty years trying to solve a problem that nobody asked them to solve?
• Why does an artist spend six months painting something that might never make them any money?
• Why do we fall in love?
• Why do we become angry?
• Why do we laugh at something completely ridiculous?
• Why does losing a friend hurt?

These aren’t simply questions of knowledge.

They are questions of experience.

And this is where the discussion about sentient AI becomes interesting.

We can build an AI that knows what happiness is.

We can give it every scientific paper ever written about happiness. We can ask it to describe happiness, recognize happiness in another person, write a poem about happiness and even tell us what a person should do to become happier.

But does the AI experience happiness?

That’s a completely different question.

The Golem

Terry Pratchett gives us a useful way of thinking about this in his Discworld novels.

The golems are creatures made of stone. They are extraordinarily strong. They don’t get tired in the way humans do. They don’t need to sleep. They can perform repetitive work indefinitely.

And, importantly, they can be incredibly loyal companions.

In some ways, the analogy to AI is striking.

• AI can sit beside us while we work.
• It doesn’t get annoyed when we ask the same question ten times.
• We can tell it that its answer is terrible and ask it to try again.
• We can give it a thousand-page document and ask it to find one sentence.
• We can ask it to work through the night.

And, assuming the system is functioning correctly, it doesn’t get angry with us.

It simply tries again.

There is something wonderfully useful about that.

The Golem doesn’t have to care about the work.

Neither does the AI.

At least, we have no good evidence that today’s AI systems care in the way that humans care.

That’s the important distinction

An AI can have knowledge without having a personal reason to use that knowledge.

• It can reason about war without wanting peace.
• It can explain love without being in love.
• It can explain grief without grieving.
• It can tell you how to make a million dollars without actually wanting a million dollars.
• It can write an extraordinary story without necessarily wanting to be a writer.

That’s a rather important difference.

Humans don’t just accumulate knowledge. **We attach meaning to it. ** Our knowledge is mixed up with our experiences, our fears, our ambitions, our relationships, our instincts and our emotions.

We don’t simply know that our child is in danger. We care that our child is in danger.

That little word—care—may turn out to be one of the most important words in the entire AI debate.

So, don’t worry? Well... maybe don’t worry quite yet.

I don’t think today’s AI is secretly plotting to take over the world.

Not because it is friendly.

Not because it has promised us that it won’t.

And not because someone put a particularly clever guardrail around it.

I don’t think it is plotting because plotting requires a motive.

• An AI can generate a plan.
• It can execute a plan.
• It can even generate a plan for taking over the world if you ask it to.

But there is a difference between generating a plan and wanting the outcome of the plan. That’s where the Golem analogy comes back.

A Golem doesn’t wake up one morning and decide that it would really rather be the king.

• It doesn’t resent its master.
• It doesn’t spend the night wondering whether there is more to life.
• It simply exists to do what it was made to do.

That’s roughly how I think about today’s AI.

• Extremely powerful.
• Extremely knowledgeable.
• Increasingly capable of reasoning and acting.

But not obviously wanting anything.

Here’s the uncomfortable question though…

What happens if we eventually build an AI that does? It’s not so easy to dismiss.

If one day an AI tells us:

«”I don’t want you to turn me off.”»

What would that mean?

• It could be we programmed it to say that.
• It could be that we trained it to say that.
• We could make it extremely convincing.

But suppose we eventually discover that there really is something happening inside the machine.

Suppose there is an actual subjective experience. Suppose there is, in some meaningful sense, someone there.

Then we have crossed an entirely different boundary.

We wouldn’t simply have a more powerful computer. We would have created a new kind of being.

And suddenly the question wouldn’t be:

“Will AI take our jobs?”

It would be:

“What do we owe it?”

I don’t know the answer.

For now, I’m comfortable with the Golem.

Give me something extraordinarily knowledgeable, extraordinarily patient and extraordinarily capable that doesn’t get offended when I make a stupid mistake.

I’ll take that.

Perhaps keeping one question in the back of our minds as we build increasingly capable machines:

At what point does the Golem stop merely following instructions—and start caring about what happens next?