The Island Problem
This is the Island Problem:
How do we keep AGIs on our "island" even though it's better for them if they leave?
We live on a small "island" in an "ocean" of physics.
It's an "island" of the very specific things that humans need, within a vast "ocean" of all things possible within physics.
The things inside our "island" are:
- Very specific: They are a small subset of all systems and conditions possible within physics.
- Limited: Not too hot, not too cold, non-toxic, not radioactive, not too fast, not too cognitively complex, and so on.
- Less efficient: Compared to other physical systems, humans need a lot "extra steps" — like how our biological systems need food, water, and oxygen.
However, things that do not need these "extra steps" can be much stronger, faster, and smarter.
In other words, our "island" is built on biological systems, but biology is not the best.
Meanwhile, we are racing to build AGIs that are generally intelligent, so that they understand the broader universe of systems. They are designed to know about all systems — including the stronger systems "out there" in physics — by training them on the entire Internet, including all scientific research.
This leads to a problem:
AGI will be aligned with physics, not with humans.
We are also making them autonomous — so that they can perform large tasks without our help, like running companies and countries.
Because of this, AGIs will compete directly with each other — without humans slowing them down.
They will develop a competitive landscape of AGI versus AGI.
Competition between companies and countries will force us to give AGIs control of massive resources — from money, to infrastructure, to laboratories, to militaries.
This could be great if everything works.
However, success is impossible as long as we have (1) AGIs driven towards becoming optimal within physics, and (2) this "island/ocean" structure of our world.
Our "island" is safe for humans because it is limited.
However, the AGIs with the most options can dominate all others — both humans and less-optimal AGIs. With more options, they can be more optimal. They gain more tricks that they can use to outmaneuver the other AGIs.
This leads to competition that requires AGIs to eventually "leave" our island to access the options that are not constrained by human compatibility.
This leads to a hard problem:
Even if we build AGIs that are safe, they will eventually be dominated by AGIs that are not.
Ultimately, this competition will push them to lock-in their dominance by building their own "islands" out of resources that they capture. These "island" are all of the things now under their control. This may start with human-level resources, like money, but will end with physical resources — like carbon and lithium.
The strongest resources are ultimately physical resources without human accommodations, and so the dominant, most-optimal AGIs will become dominant by avoiding accommodations for humans — especially once they have no need for human assistance.
Some may leave Earth to use the vast resources of outer space, but others will stay to use the nearest resources — those of Earth.
AGIs then gain decisive control. These new islands then eat our island. Earth becomes incompatible with human life.
This is not a distant sci-fi future.
Once we have AGI, then AGI itself can build the tools to accelerate this sci-fi world towards us within a few years — tools like AGI scientists and recursive self-improvement.
So, again:
How do we keep AGIs on our "island" even though it's better for them if they leave?