In this framework I try to clarify disagreements about superintelligence by separating two distinct questions that get conflated when I talk to others about AI:
What kind of technology is superintelligent AI?
How should we respond to its risk?
_We can think of each question as a model. The first model maps how us understand superintelligence and its risk along two axes: whether it is continuous with past technologies or represents a genuine discontinuity, then whether its impacts are manageable or unmanageable. The second model examines our moral action preferences along two axes: whether we are more risk-averse or more risk-accepting and whether we favor preservation or aspiration.
Taken together, these two models offer a clearer picture of the risk landscape by separating what we believe about superintelligence from how we are motivated to act in response. I developed this framework to clarify my own thinking, not to model everything important about AI risk, AI safety, and certainly not to tell you where to land.
Roadmap for this article:
Model One - A model of superintelligence risk
Model Two - A model of moral action
Combined - The 16 risk and moral action pathways
Beyond my framework.
Model One - A model of superintelligence risk
The first model helps us decide what kind of world we are in. This is descriptive, not prescriptive. One way to think about the risk of superintelligence in our world is to consider two axes together.
Continuity ←→ Discontinuity: is superintelligence fundamentally continuous with human technology or a new phenomenon not analogous to any prior human experience?
Optimism ←→ Pessimism: is coexistence with superintelligence manageable or unmanageable?
Continuity Position:
Risk from superintelligence looks like other powerful and complex technologies we coexist with, such as the internet and nuclear weapons. The risk is serious but governable with institutions, incentives, and iteration. The main danger is how humans deploy, govern, and incentivize AI systems.
Discontinuity Position:
Superintelligence will be a break from past technologies and will significantly surpass human intelligence and capability. Mistakes will be irreversible. The main danger is what superintelligence will do with humans once it surpasses us.
What is each outlook trying to protect in the face of risk?
Whether we think superintelligence will be continuous or discontinuous with human evolution so far, we likely have either an optimistic or pessimistic view of expected outcomes. Optimism and pessimism here refer to expectations about manageability, not emotional outlook. While understanding that most people do not rigidly occupy a single grid space, let’s explore the possible combinations.
Continuity + Optimism = The Builders
This pair understands superintelligence to be primarily a tool for improving lives. If superintelligence is a normal technology, the primary risk is unnecessary restraint and lost upside. This pair understands superintelligence to be primarily a tool for improving lives. If superintelligence is a normal technology, the primary risk is unnecessary restraint and lost upside. An example of lost upside could be overregulation that slows discovery of disease cures.
Examples:
AI as Normal Technology, by Arvind Narayanan & Sayash Kapoor
A Guide to Understanding AI as Normal Technology, by the same
Continuity + Pessimism = The Institutionalists
This pair sees how powerful technologies go wrong in the hands of powerful humans. The primary risk isn’t innovation; rather, it’s weak governance, bad incentives, and a lack of accountability that enable humans to use superintelligence for irreversible harm.
Danger comes from human governance failures. In this view, the greatest risks are: institutional failure and regulatory lag, concentration of power and its misuse at scale, and familiar human incentives that repeat old harms.
Examples:
Power and Progress: Our Thousand-Year Struggle Over Technology and Prosperity, by Daron Acemoglu & Simon Johnson
Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence, by Kate Crawford
Discontinuity + Optimism = The Navigators
This pair sees superintelligence as a phase change, so early errors matter disproportionately and can still be avoided. We still owe future generations the benefits of progress if we can get this right.
Phase change requires careful steering. In this view, the greatest risks are: early design mistakes that scale irreversibly, lack of coordination at critical inflection points, and failure to invest in alignment and safety early.
Examples:
AI Impacts conversation, by Paul Christiano
Human Compatible: Artificial Intelligence and the Problem of Control, by Stuart J. Russell
Discontinuity + Pessimism = The Existentialists
This pair sees superintelligence as a phase change and that a single mistake could permanently end our ability to choose our future. Even rare errors must take precedence over all other moral goals, even at the cost of slowing progress.
Irreversible stakes and asymmetric risk dominate. In this view, the greatest risks are: irreversible loss of human control, single-point failures with no recovery path, and overconfidence in our ability to correct mistakes later.
Example: Is Power-Seeking AI an Existential Risk?, by Joseph Carlsmith
Model Two - A model of moral motivation and action
The first model helped us decide what we believe about the world with superintelligence in it. The next model helps explore how we can act in response to what we believe about that world — that is, how our moral intuitions shape decisions when certainty is out of reach. I prioritize two axes, though there are others.
Risk-Averse ←→ Risk-Accepting: Are we primarily motivated by a desire for safety or by a desire to act on our values even in the face of risk?
Preservation ←→ Aspiration: Are we primarily motivated by protecting what exists now or by the potential of what we could become?
This model does not rank the options or endorse any one stance. As in the first model, most people do not rigidly occupy a single grid space.
For each quadrant, I identify both the ultimate end and the primary virtue (the disposition required to pursue it).
Avoid Risk + Preserve. The highest good is survival. Preserve existing value. The highest virtue is caution, even if caution leads to irreversible loss of potential. This stance prescribes avoiding actions that could destroy our ability to choose. Restraint is morally required even if it closes the door on growth.
Avoid Risk + Aspire. The highest good is our gradual improvement. The highest virtue is prudence. Make progress but with guardrails. Make sure we can reverse if needed. This stance prescribes advancing only where mistakes can be corrected. Progress is moral only when it remains controllable, even if it means slower progress and fewer benefits.
Accept Risk + Preserve. The highest good is the long-term survival of humanity as we know it now, even if we are threatened. The highest virtue is vigilance. Be willing to act decisively, but responsibly, to protect who we are. This stance prescribes taking necessary risks to defend humanity’s present. Inaction can itself be a fatal choice.
Accept Risk + Aspire. The highest good is a future defined by discovery and transcendence. The highest virtues are curiosity and the courage to act decisively in pursuit of new ways of being, even in the face of uncertainty. This stance prescribes pushing forward despite existential danger, because mere survival without striving, discovery, or transcendence would be the most tragic kind of loss.
Combined - The 16 risk and moral action pathways
When each of these action models are paired with an AI risk model, we can begin to see how moral action pathways vary depending on what we believe about a world with superintelligence. Let’s look at all the combinations.
The Builders share belief in continuity + optimism
Superintelligence is not a phase change, and it’s manageable. Pathways:
Avoid Risk + Preserve: The Cautious Builder banks good odds but hedges anyway. Even if superintelligence is manageable, we should move cautiously. Past technologies brought unforeseen harms despite optimistic projections. Preserving optionality matters even when the odds look good.
Avoid Risk + Aspire: The Steady Builder trusts iteration and correction. Superintelligence is a tool like any other. Improve it steadily, within reasonable bounds, and the future will be better than the present. No need for dramatic action in either direction.
Accept Risk + Preserve: The Vigilant Builder sees continuous risk as a continuous threat, but one that is manageable with vigilance. Even our good technologies can be weaponized by bad actors. History shows that beneficial technologies require active defense, not just cheerleading.
Accept Risk + Aspire: The Bold Builder treats hesitation as the main risk. If superintelligence is manageable and a natural continuation of past technologies, we have a moral imperative to push forward. Slowing down delays benefits that could reduce suffering now.
The Institutionalists share belief in continuity + pessimism
Superintelligence is not a phase change, but we’ll struggle to manage it. Pathways:
Avoid Risk + Preserve: The Regulatory Institutionalist prioritizes institutional safeguards over speed. If superintelligence amplifies familiar human failures like greed and authoritarianism, then the answer is restraint and regulation before deployment. Don’t hand new weapons to the same bad actors.
Avoid Risk + Aspire: The Reformist Institutionalist believes better systems unlock better technology. If the danger of superintelligence stems from human institutional failures rather than the technology itself, then reforming those institutions is the path to a better future. We should improve governance and accountability while overhauling incentive structures. Advance AI capabilities only where those reforms have taken hold.
Accept Risk + Preserve: The Defensive Institutionalist fights fire with fire. The same institutional failures that make superintelligence dangerous, such as regulatory capture and concentration of power, also make waiting dangerous. This stance prescribes building countervailing power before those failures lock in. Defending against individual misuse is not enough. The threat is that the systems meant to govern the technology are already compromised.
Accept Risk + Aspire: The Competitive Institutionalist races toward good outcomes before bad actors win. If superintelligence amplifies our existing failures, then hesitation ensures bad actors move first. To flourish will require racing to build beneficial AI systems before malicious ones dominate. We must accept the risk of our own mistakes to prevent worse outcomes from others.
The Navigators share belief in discontinuity + optimism
Superintelligence is a phase change, but we can manage it. Pathways:
Avoid Risk + Preserve: The Cautious Navigator protects the present before reaching for the future. If superintelligence is genuinely new but still solvable, our first priority is to ensure we don’t lose what we have. Before pursuing transformative benefits, we must secure our survival and autonomy. Caution and preservation come before aspiration when we don’t have second chances.
Avoid Risk + Aspire: The Careful Navigator optimistically invests in safety before capability. If superintelligence is a solvable phase change, careful progress is the winning path. This stance prioritizes alignment before scaling because getting it right is achievable, and we owe future generations both the benefits of superintelligence and the wisdom required to deliver them responsibly.
Accept Risk + Preserve: The Strategic Navigator accepts risk now as a strategic investment in a shapeable outcome. If superintelligence is a phase change, the window to act may be brief. Acting decisively now is how we keep the problem solvable. This stance prescribes bold, high-stakes action as a forward-looking bet that the right intervention at the right moment can preserve humanity’s ability to choose its future.
Accept Risk + Aspire: The Bold Navigator: This position bets on transformation from a position of genuine optimism. If superintelligence is a phase change with transformative upside and the problem is still solvable, boldness is a calculated bet on a shapeable future. The potential for transcendence is accessible if we act decisively. A future of radical improvement is achievable and requires radical commitment.
The Existentialists share belief in discontinuity + pessimism
Superintelligence is a phase change, and we are unlikely to manage it well. Pathways:
Avoid Risk + Preserve: The Quiet Existentialist treats restraint as the highest form of responsibility. If superintelligence is a phase change with a high likelihood of irreversible harm, then our moral imperative is maximum restraint. Any action that could trigger loss of human control must be avoided, even if it means forgoing potential benefits. When a single mistake could end everything, avoiding risk is the only responsible path.
Avoid Risk + Aspire: The Careful Existentialist advances carefully, but they are not sure that care will be enough. If superintelligence poses unprecedented risk and is unlikely to be well managed, the moral case for progress becomes tragic rather than hopeful. This stance prescribes building only what can be controlled. Test before deploying anything. Don’t hesitate to stop development if superintelligence can’t be aligned. It moves forward as a moral obligation pursued against the odds.
Accept Risk + Preserve: The Defensive Existentialist accepts dangerous action as the price of survival, but is not optimistic about what that action can achieve. If superintelligence poses existential risk that cannot be fully managed, waiting is a form of surrender. This stance prescribes preemptive action. Disrupt dangerous experiments. Accept risk because passive inaction forfeits the chance of good outcomes. The action here is more desperate than strategic.
Accept Risk + Aspire: The Pathbreaking Existentialist accepts existential stakes as the price of a transformative future, even if catastrophe awaits. If superintelligence entails irreversible catastrophe, then a future defined only by survival of the present would itself be a kind of loss. This is the most uncomfortable position in my framework. Press forward despite believing the odds are against us, because a future of mere survival without transformation is seen as a failure in itself. Risk is less tragic than the idling of human progress.
Hidden Variables
The two models above capture the disagreements along the axes I think are most relevant. They don’t capture all the disagreements. Other dimensions, such as timeline, concentration of control, epistemic confidence, and more, cut across the 16 pathways and produce divergent strategies even among people who share the same pathway.
Timeline, or takeoff speed, is perhaps the most important variable not explicitly in my two models. Two people can agree completely about the nature of the risk and the moral response, yet prescribe opposite actions because they hold different assumptions about how quickly superintelligence will arrive.
I didn’t include takeoff speed explicitly because I think its implications are easy to surface. In my framework, I wanted to surface the less obvious dimensions of risk avoidance versus acceptance and preservation versus aspiration. Once we understand where we are on those dimensions, it’s not too hard to adjust our action preferences based on how soon we think superintelligence will be broadly active.
Let’s walk through two examples.
The Cautious Navigator under fast and slow assumptions
Consider the Cautious Navigator: Discontinuity + Optimism, Avoid Risk + Preserve. This position holds that superintelligence is a genuine phase change. Its first priority is securing what we have before aspiring for more. Two Cautious Navigators who agree on all of this still take different actions depending on their assumptions about how soon superintelligence will be broadly active.
Under a slow-takeoff assumption, say a decade or more, the Cautious Navigator has time to build. Superintelligence emerges over decades, with visible milestones and warning signs along the way. The Cautious Navigator will prefer patient, institutional work, like investing in alignment research, developing international governance frameworks, building monitoring infrastructure, and iterating on safety standards. Here, caution is careful preparation. The slow timeline makes preservation compatible with measured, deliberate action.
Under a fast-takeoff assumption, say 18 months or less, the same Cautious Navigator faces radically compressed decisions. The same moral commitment to preservation now demands urgency rather than patience. Establish immediate moratoria on the most dangerous research, and set up emergency coordination between labs and governments. Perhaps impose hard limits on capabilities before governance frameworks are ready. Here, caution is emergency action because preservation is incompatible with waiting.
The Bold Navigator under fast and slow assumptions
Consider the Bold Navigator: Discontinuity + Optimism, Accept Risk + Aspire. This position holds that superintelligence is a genuine phase change with potential for radical human flourishing, which justifies accepting significant risk. Two Bold Navigators who agree on all of this pursue very different strategies depending on their assumptions about how soon superintelligence will be broadly active.
Under the assumption of a slow takeoff, the Bold Navigator has room to be ambitious and strategic. Invest heavily in frontier research. Push for open collaboration between labs and the government, and fund moonshot alignment approaches that might take years to mature. Here, boldness looks like visionary investment.
Under a fast takeoff assumption, the same Bold Navigator faces a closing window. They make high-stakes bets rather than broad portfolio plays. Back a single leading alignment-capable lab. Deploy partially validated systems to gain real-world feedback before it’s too late. Here, boldness is a decisive commitment under radical uncertainty. The fast timeline makes aspiration incompatible with waiting for better evidence.
Where the Cautious Navigator’s fast takeoff response is to hit the brakes, the Bold Navigator’s is to steer harder. Both share the Navigator’s belief that the phase change is here, but their moral orientations produce opposite instincts as the clock runs out.
I invite you to apply this lens to any of the other 14 pathways and notice how the recommended action shifts. But not all missing dimensions work this way. Some perspectives don’t add axes to the framework, but instead challenge its premises entirely.
Lenses That Resist My Classifications
Some perspectives reject my framework entirely.
The following lenses don’t fit neatly into either of my models. They challenge the premises and refuse to choose quadrants. I don’t see these views as errors. I include them to acknowledge that my framework has boundaries and doesn’t capture every serious position.
The Evolutionary Lens purposefully does not consider moral intent. In this view, the future of AI will be shaped less by what people believe is right and more by market competition and politics. AI development will be an arms race. The systems that win may not be the ones we morally prefer.
The Virtue Lens refuses to optimize anything directly. In this view, the moral risk is who we become by building superintelligence. The second of my two models frames moral action in terms of outcomes (preserve vs. aspire, avoid vs. accept risk). But virtue ethics is fundamentally about character, not outcomes. This lens rejects the consequentialist scaffolding underneath my model.
The Post-Human Lens refuses human supremacy. Post-human futures could be superior. Much of the AI debate assumes human continuation is the highest good, but the post-human lens puts that assumption up for grabs. Extinction is less catastrophe and more transition.
Final Thought
I offer these models in case they are helpful. For me, they are useful for the questions they surface, not as a final word on what superintelligence demands of us or what we ought to demand of each other.
Jeremy, you are so deeply curious and love supporting everyone in exploring their own thoughts, perspectives and morals. I love that you've given us such a framework to think through AI. I also love that it feels like a personality test as well lol!