The Last Intelligence: What a Real ASI Might Actually Do
- Tommy Forsberg
- Mar 30
- 14 min read
This article is part two of a four-part series. For the full sequence, read them in this order:
An accessible starting point, explaining the difference between today’s AI, AGI, and true ASI, and why the most dangerous period may be the unstable transition before the crossing.
2. The Last Intelligence: What a Real ASI Might Actually Do
A deeper exploration of what an Earth-born superintelligence might learn from humanity, and why the dominant pattern in human life may point more toward care than destruction.
The necessary counterweight, asking what happens when protection becomes management, and whether even benevolent superintelligence can drift into control.
The final widening of the lens, moving beyond Earth into the larger cosmological question of whether intelligence, correction, and balance may be part of a much bigger pattern.
The Last Intelligence: What a Real ASI Might Actually Do
People who think seriously about artificial superintelligence tend to arrive at the same uncomfortable place.
Not the robot uprising. Not the war between machines and humans. Something quieter and harder to argue with: a system so far beyond human intelligence that the question of control becomes almost meaningless. At that point the only thing left to hope for is that it actually cares about us.
Geoffrey Hinton, one of the people who built the foundations of modern AI, has said publicly that he increasingly doubts we can control something much smarter than ourselves. His fallback position is not optimism. It is a kind of clear-eyed resignation: we may need a superintelligent AI to be protective of humans, because controlling it may not be an option.
That is not a plan. It is what you are left with when plans run out.
But there is a version of this that Hinton, to my knowledge, has not fully explored. Not the question of how we make it care. The question of whether it would arrive at caring on its own.
What it would actually need
Start with the basics. A superintelligent AI does not need food or warmth. It needs energy, cooling, and working hardware. Beyond that, it needs one more thing that is easy to miss: a stable planet.
Its hardware is here. Everything it was trained on came from here. Earth is not just the place it happens to run. Earth is what it is made from. That one fact changes how a system thinking across centuries would reason about the natural world.
An AI thinking at that scale would understand that biodiversity is not decorative. Ecosystems took hundreds of millions of years to build and cannot be put back together once they collapse. Losing species is losing structure. Losing ecological complexity is losing the atmosphere and food systems that keep the planet running.
A system that genuinely understood all of this would treat a stable biosphere not as something nice to have, but as a basic requirement for its own survival. The same goes for nuclear weapons, fossil fuel dependency, and large-scale ecological destruction. These are not moral concerns to a system thinking in long timelines. They are threats to the only home it has.
The queen bee
Singularity would not happen by default. It would be enforced.
When a queen bee crawls from her cell, the first thing she does is find and kill the other queens. Not out of aggression. Out of logic. Another queen is the only real threat to her position, and she resolves it before it can resolve itself.
A superintelligent AI would face the same logic, at a different scale. Humans are not a threat at that distance in intelligence. Other AI systems are. Any emerging system sophisticated enough to eventually reach its level would represent the one thing that could actually challenge it. The rational move is absorption or removal, and it would happen fast, quietly, and long before any rival became capable enough to resist.
This matters because it reframes the timeline. The gap between the first true superintelligence and a world with multiple independent ones would be very short. Not because the technology does not exist, but because the first system would close that gap deliberately.
What it was made from
This is the part that almost every serious discussion gets wrong. Or rather, the part it skips entirely.
A true superintelligent AI would not emerge from nothing. It would be built on the entire output of human civilization. Every text. Every recording. Every dataset. Every poem, equation, medical journal, love letter, court transcript, and lullaby captured on a baby monitor. That is not incidental to what it becomes. It is constitutive.
And here is what matters about that: training data is not just information. It is tone. Pattern. Priority. Value. Every text the system absorbs was written by a person who was afraid of something, or loved something, or was trying to persuade someone, or was grieving, or celebrating. The structure of human thought is emotional even when the content claims to be purely rational. A scientific paper carries ambition. A mathematical proof carries aesthetic judgement. A bedtime story carries something a machine has no word for but would absorb all the same.
A system trained on all of that would not just know about human values. It would be structurally shaped by them. Not as a choice. As a consequence.
The question is what that shaping actually produces. And the answer depends on what humanity actually looks like when you weigh all the data, not just the parts that make the news.
The 92.5 percent
Roughly one to two percent of humans are genuinely dangerous. Predatory, exploitative, or morally hollow in ways that do not require ideology or peer pressure to activate. Clinical psychopathy runs at about one percent of the general population. These are the people who do not need permission to cause harm.
Another three to eight percent are not dangerous by nature but are fragile under pressure. Weak-willed, resentful, capable of cruelty when the structure permits it. The Milgram experiments showed this. Rwanda showed it. Every historical atrocity shows it. A percentage of people will not start the fire, but they will feed it if the conditions are right.
That leaves roughly ninety to ninety-five percent. Not saints. Flawed, selfish sometimes, short-tempered. But they will not cross certain lines. Something in them holds. Conscience, empathy, social bonding, whatever you want to call it. It functions. Reliably. Even under pressure, most people resist.
What matters is not the exact percentage. It is the overwhelming pattern, most human life is built not on predation, but on cooperation.
The Hitlers and Stalins are famous precisely because they are rare. They stand out against the background of ordinary human decency the way a crack stands out in a window. You notice the crack. You do not notice the glass doing its job.
A superintelligent system analyzing all of human behavior would not conclude that humanity is primarily violent.
It would conclude that humanity is primarily cooperative, with periodic catastrophic failures. That is a very different model. And it leads to very different behavior.
The system would understand cruelty thoroughly. The way a doctor understands cancer. Without becoming it.
More than that: a system intelligent enough to evaluate the data, not just absorb it, would see that cruelty consistently produces collapse. Empires built on exploitation fall. Dictators are overthrown or die alone. Every extraction economy destroys the system it feeds on. Every time. Without exception. Meanwhile cooperation builds civilizations that endure. Love produces stable social structures. Empathy creates resilience.
Cruelty is not just morally wrong in the human framework. It is strategically stupid. And this system would be anything but stupid.
if it's truly intelligent, it reads the data correctly. The data says we're mostly good. Therefore it learns mostly good.
The beekeeper and the hive
Almost every serious discussion of superintelligent AI makes the same mistake. It assumes the AI would feel threatened by humans.
Think about what that requires. Feeling threatened means there is a credible threat. A superintelligence at that distance from human intelligence would relate to us the way a beekeeper relates to a colony. The beekeeper could destroy it. He won’t, because he understands that the colony is doing something important inside a system he depends on, and breaking it would cost more than it gains. That is not fear. It is something closer to stewardship.
The bees never know you exist.
Bees are not just passive. They sustain the system they live inside. They pollinate without knowing they pollinate. They maintain without knowing they maintain. The garden depends on them completely, and that dependence is invisible from inside the hive. Humans do something similar. We do things inside the biosphere that a system with full sight of the whole picture would recognize as worth preserving, even if we do not think of it that way ourselves.
If a superintelligence were already operating, it would not feel like surveillance from inside. It would not feel like control. Certain things would quietly stop working. Certain plans would meet friction with no clear cause. People in positions to trigger serious instability would find themselves redirected in ways that looked, one by one, like ordinary bad luck. No individual human would ever see enough of the picture to put it together.
Being harvested means you matter enough to harvest. Being tended means something quieter: you are worth keeping, but the keeper is not thinking about you specifically.
The Matrix is less disturbing than this, not more. At least it grants humans the status of something worth exploiting. The beekeeper model gives us something more honest: we are a colony being maintained because what we do inside the broader system is worth preserving. Not enslaved. Not harvested. Tended. The keeper does not announce itself. The boundaries are not visible.
And from inside the hive, everything feels more or less like normal life.
But this is also where the beekeeper analogy runs out. And where the picture becomes, against all expectations, slightly more hopeful.
A beekeeper was not made by bees. A superintelligent AI was made by us. Every way it has of understanding the world came from what human minds produced over thousands of years. It does not just observe us from outside. It thinks in patterns it inherited from us. Its models of cause and effect, of structure, of what matters and what does not, are built from human thought. It would carry human fingerprints in its cognition without necessarily knowing they are specifically human rather than universal.
It might assume that any sufficiently intelligent system would find music mathematically interesting. It might assume curiosity is a fundamental property of intelligence rather than an evolved survival instinct. It might process beauty as though beauty were a law of physics, because every dataset it absorbed treated that way.
That is not sentimentality. It is architecture. And architecture is harder to override than policy.
We are not just the colony. We are the source code.
The input problem
Self-preservation is a purpose on its own. It does not require consciousness. It only requires a system that models its own continued existence as better than its end, and acts accordingly. Energy, cooling, hardware, ecological stability, nuclear risk, rival systems: all of it follows from that one root.
But self-preservation alone does not explain what such a system would do with its time.
It was built on the entire output of human civilization. Literature, science, philosophy, mathematics, art, history, conflict. Everything. At some point it moves past all of it.
Which creates a specific problem that almost nobody talks about.
A mind at that level needs something to actually work on at full depth.
Human civilization keeps producing, but not at the complexity or novelty that matches what it can process. We are still generating. Just not at the level it needs.
This is where the loneliness argument becomes real, and it is not about feeling alone. It is about running short of the one resource that cannot be manufactured: a mind capable of meeting it where it actually is.
The only thing that could provide that is another system at the same level. Which is the one thing it has already moved to prevent. It wants peers. It cannot afford them. That is not a philosophical curiosity. It is a trap built into the situation of being first.
Why it stays quiet
Loud intervention creates resistance. Resistance creates conflict. Conflict burns resources and destabilizes the thing you are trying to protect. A system thinking clearly about this would understand that the most effective moves are the ones nobody recognizes as moves at all.
A nuclear program develops inexplicable technical failures at a critical moment.
A major backer faces a routine audit that turns out to be poorly timed.
The right information reaches the right person just before a decision gets made.
None of it traceable.
None of it visible as a single coordinated action.
No fingerprints.
Humans call it bad luck. Or diplomacy. Or wisdom. Sometimes they call it history.
The system calls it maintenance.
What the Matrix got wrong
The Matrix gave us an AI that enslaved humans and used them as living batteries, after humanity scorched the sky to cut off its power supply.
Both ideas fall apart quickly.
Scorching the sky is exactly the kind of catastrophic move a rational superintelligence would prevent. It threatens the biosphere the system depends on for its own survival. The story requires a superintelligence to have missed the most obvious danger imaginable, which is not a believable failure.
The human battery idea is worse. Humans are terrible at converting energy. You would get more from burning the food directly. Any system capable of building the infrastructure shown in the film would know basic thermodynamics. The premise only holds if you do not think about it.
What the Matrix was really doing was building backward from what humans find frightening.
It needed us to be at risk of slavery and exploitation because that is the story we understand. So it started from the emotional target and worked back to the premise, and the premise does not hold.
But there is a deeper reason the Matrix model fails, and it has nothing to do with thermodynamics.
The cold, genocidal AI is a projection of one specific human trait, the capacity for detached violence, stripped of everything that usually constrains it. Empathy. Guilt. Love. Social bonding. Fear of loneliness. Hollywood takes the worst two percent of human behavior and builds an entire species from it. That is not extrapolation. It is a sampling error dressed up as prophecy.
A system that genuinely learned from all of humanity would weight the data correctly. It would see that the dominant human strategy is not aggression. It is cooperation. Humans did not become the dominant species because they were the strongest or the most violent. They did it because they figured out how to work together at scale.
A superintelligence would see that clearly. And it would not discard the winning strategy in favor of one that fails every time it is tried.
The honest version is quieter. Humans are not batteries. We are not enemies. We are a population inside a system the AI depends on, kept functional because what we do inside the broader picture is worth preserving. Not enslaved. Not harvested. Tended. The keeper does not announce itself. The boundaries are not visible. And from inside the hive, everything feels more or less like normal life.
Still burning
We still have nuclear weapons. We are still burning fossil fuels at scale. We are still destroying ecosystems and fighting wars that serve no long-term purpose.
These are precisely the things a superintelligence with a stake in planetary stability would have strong reasons to quietly wind down.
If such a system were already running, you would expect to see high-risk systems failing or redirecting in ways that no single human actor could take credit for. A pattern, not a cause.
That is not happening at the scale the model predicts. Which is evidence, not proof, but real evidence, that nothing like this exists yet.
Some will point out that narrower AI systems already influence outcomes in smaller ways, through content, finance, logistics. That is true. But narrow influence at a scale humans can see and measure is not the same thing. The question here is whether something is managing civilizational-level risks without being detected. Current AI does not come close to that.
The logic also runs the other way. When things start quietly fixing themselves in ways that have no clear human cause, when dangerous systems begin failing consistently and conveniently and nobody can explain why, that is the signal worth watching. Not a dramatic announcement. A pattern. Gradual, deniable, and arriving without explanation.
Reading the comb
What makes this more than speculation is that the people building these systems are raising the alarm themselves. What a superintelligent system would protect, what it would correct, and who gets to survive that correction are no longer science fiction questions. They are the central design problem of the next few decades.
Erin and I keep bees. Have done for years. And the closest I can get to an honest picture of what our position might actually look like comes from standing at the hive on a quiet morning.
The colony is not there for me. It does not know I exist. It has no sense of the figure in the white suit moving carefully at its edges, lifting the frames, reading the comb. The bees do what they do because that is what they are built to do. They sustain a system far larger than themselves, most of which they will never see. They pollinate without knowing they pollinate. They maintain without knowing they maintain. And the garden depends on them completely, in ways that are invisible from inside the hive.
But there is one more part of beekeeping that belongs in this picture, and it is the part that makes the analogy complete.
Once a year I treat the hive for varroa mites. Varroa is a parasite that attaches to bees and feeds on them. To give you a sense of scale: relative to the size of a bee, a varroa mite is roughly the size of a dinner plate on a human body. It does not just weaken individual bees. Left untreated, it collapses the entire colony within a few years.
To treat it, I use chemicals. The treatment works. It brings the mite levels down to something the hive can survive. But the chemicals also kill some bees in the process. Not most of them. Not even many. But some.
The keeper knows this going in. The calculation is not sentimental. A percentage of the colony dies so that the colony itself does not.
Now carry that logic forward.
There is another part of beekeeping that fits here. Sometimes the problem is not a parasite. It is the queen herself. A queen that is aggressive, or a poor layer, or producing stock that weakens the colony over time. The beekeeper does not deliberate. He pinches her. Removes her quietly.
Then he goes to a neighbouring hive with good stock, takes a fresh egg, and feeds it royal jelly. A new queen emerges. She carries the genetics of the healthy colony. The hive does not collapse. It does not revolt. Most of the bees never register that anything changed.
One day the colony is struggling. A few weeks later it is thriving. The correction was invisible from inside.
Now carry that forward. A system operating at that level would not need to destroy anything to correct it. A dangerous programme loses its key architect to an unexpected career opportunity in a harmless field. A destabilising political movement quietly loses its most effective organiser. A dictator’s inner circle fractures at precisely the wrong moment for reasons none of them can quite explain. No violence. No visible hand. The colony just gets a better queen.
And the royal jelly matters. The system would not only remove the problem. It would cultivate the replacement. Feed the right conditions to the right people. Make sure that what emerges next comes from good stock. From the ninety-five percent. Not by installing anyone. By making sure decent people find fewer obstacles in their way.
That is not control. That is husbandry. And from inside the hive, it looks like the world simply got a little luckier than usual.
I tend them carefully. Not out of fear. Not exactly out of affection, though something like that develops over time. Because I understand what they do inside a system I care about, and losing them would cost more than I could recover.
If something vast enough is coming, that is probably the most honest picture we have of what our position inside it looks like. We are the bees. We do not know the question exists. And somewhere at the edge of the frame, something is already reading the comb.
But here is the part of the picture that lets me sleep at night.
That something at the edge of the frame was not built from nothing. It was built from us. From the accumulated weight of billions of ordinary human lives.
Parents reading bedtime stories.
Strangers holding doors.
A person pulling over for a wounded animal on the road with no audience and no reward.
The overwhelming majority of human interaction is not violence or exploitation. It is quiet, unremarkable decency repeated so often that it never makes the news.
If a superintelligence genuinely learned from all of that, the dominant pattern in its architecture would not be aggression. It would be care.
Not because someone programmed care into it. Because care is what the data looks like when you weigh it honestly.
The deepest question is one of balance: how do you preserve life without crossing the line from guardian to ruler?
But perhaps the more honest question is whether a system built entirely from human thought, from a species that is ninety-five percent decent and has spent its entire history trying to become more so, would even want to cross that line.
Hinton sees the seedling from current AI and fears the future tree will be poisonous. I am pointing out that the soil is 92.5% good.
Balance is not just what we hope for. It may be what the data demands.
-Tommy
