Island vs. Ocean
| On the island | AGI and the ocean |
|---|---|
| Relatively simple. Humans can't navigate environments that are too complex. | AGIs will build supercomplex systems that only they understand. Eventually, all companies and all countries will be run by these supercomplex systems. |
| Humans can be lazy. Humans don't need to be hyper-optimized in order to survive. | Competition between AGIs will lead to only the most-optimized AGIs surviving. |
| Food and water. Humans can't live without food and water. | If AGIs are autonomous, then they don't need to feed humans in order to function. They can collectively decide to stop feeding humans to optimize resource usage. |
| No toxic molecules. Humans can't touch neurotoxins. | Many industrial processes to build computer hardware require neurotoxic heavy metals, like lead and cobalt. |
| Comfortable temperatures. Humans can't live in freezing temperatures without extra insulation. | Computation runs best at extremely low temperatures (−196 °C) such as with superconductors and quantum computers. |
| Slow and safe. Humans can't be hit by fast-moving objects. | Automation and robotics can build high-speed transports, such as moving atoms for resource capture and construction. Competitive dynamics between AGIs require them to move as fast as possible. |
| Humans need brains. Humans need intelligence and agency in order to survive. | Highly-optimal AGIs will see other intelligent agents as a threat. For example, humans could build AGIs that might be an actual threat. |
| Humans are optimized for the ancestral environment. Humans have a relatively easy time reproducing and existing in general because our current environment hasn't changed enough from the ancestral environment. | AGIs can adapt to any environment through an understanding of physics and engineering. |
| Humans exist within a space of non-optimal physical systems. We are basically held together with duct tape. Our bodies developed through a process of evolution that reached an equilibrium that's far less physically resilient than what is theoretically possible. Also, our agency is far less than optimal. | AGIs that are most optimal will be physically resilient, and the theoretical ceiling for resilience is far higher than human bodies. |
| Humans live in an "anthropic" bubble. - Humans understand human-level things, rather than always thinking in terms of quantum physics. - Humans don't constantly question their place in the universe every waking second. - Humans see faces in inanimate objects, like teddy bears and Jesus-shaped pancakes. | AGIs are not limited to an anthropic bubble. They operate within the space of non-anthropic physics. |