My AI Risk Framework
How can we think about coexistence with superintelligence?
We are diving in. Will AI kill us all? Some of us? Now? Later? Debates about artificial intelligence can stall when we hold different assumptions about the nature of the risk and different moral intuitions about how to respond to that risk. In this framework I try to clarify those disagreements by separating two distinct questions that get conflated when I talk to others about AI:
What kind of technology is superintelligent AI?
How should we respond to that risk?
We can think of each question as a model. The first model maps how us understand superintelligence and its risk along two axes: whether it is continuous with past technologies or represents a genuine discontinuity, then whether its impacts are manageable or unmanageable. The second model examines our moral action preferences along two axes: whether we are more risk averse or more risk accepting and whether we favor preservation or aspiration.
Taken together, these two models offer a clearer picture of the risk landscape by separating what we believe about superintelligence from how we are motivated to act in response. I developed this framework to clarify my own thinking, not to model everything important about AI risk, AI safety, and certainly not to tell you where to land.
Roadmap for this article:
Model One - A model of superintelligence risk
Model Two - A model of moral action
Combined - The 16 risk and moral action pathways
Beyond my framework
Model One - A model of superintelligence risk
The first model helps us decide what kind of world we are in. This is descriptive, not prescriptive. One way to think about the risk of superintelligence in our world is to consider two axes together.
Continuity ←→ Discontinuity: is superintelligence fundamentally continuous with human technology or a new phenomenon not analogous to any prior human experience?
Optimism ←→ Pessimism: is coexistence with superintelligence manageable or unmanageable?
Continuity Position:
Risk from superintelligence looks like other powerful and complex technologies we coexist with, such as the internet and nuclear weapons. The risk is serious but governable with institutions, incentives, and iteration. The main danger is how humans deploy, govern, and incentivize AI systems.
Discontinuity Position:
Superintelligence will be a break from past technologies and will significantly surpass human intelligence and capability. Mistakes will be irreversible. The main danger is what superintelligence will do with humans once it surpasses us.
What is each outlook trying to protect in the face of risk?
Whether we think superintelligence will be continuous or discontinuous with human evolution so far, we likely have either an optimistic or pessimistic view of expected outcomes. Optimism and pessimism here refer to expectations about manageability, not emotional outlook. While understanding that most people do not rigidly occupy a single grid space, let’s explore the possible combinations.
Continuity + Optimism = The Builders
This pair understands superintelligence to be primarily a tool for improving lives. If superintelligence is a normal technology, the primary risk is unnecessary restraint and lost upside.
Superintelligence is a powerful technology we can deploy and manage. In this view, the greatest risks are: missed opportunities for broad human benefit, overregulation that slows beneficial progress, and treating superintelligence as categorically different from prior technologies without sufficient justification.
Examples:
AI as Normal Technology, by Arvind Narayanan & Sayash Kapoor
A Guide to Understanding AI as Normal Technology, by the same
Continuity + Pessimism = The Institutionalists
This pair sees how powerful technologies go wrong in the hands of powerful humans. The primary risk isn’t innovation; rather, it’s weak governance, bad incentives, and a lack of accountability that enable humans to use superintelligence for irreversible harm.
Danger comes from human governance failures. In this view, the greatest risks are: institutional failure and regulatory lag, concentration of power and its misuse at scale, and familiar human incentives that repeat old harms.
Examples:
Power and Progress: Our Thousand-Year Struggle Over Technology and Prosperity, by Daron Acemoglu & Simon Johnson
Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence, by Kate Crawford
Discontinuity + Optimism = The Navigators
This pair sees superintelligence as a phase change, so early errors matter disproportionately and can still be avoided. We still owe future generations the benefits of progress if we can get this right.
Phase change requires careful steering. In this view, the greatest risks are: early design mistakes that scale irreversibly, lack of coordination at critical inflection points, and failure to invest in alignment and safety early.
Examples:
AI Impacts conversation, by Paul Christiano
Human Compatible: Artificial Intelligence and the Problem of Control, by Stuart J. Russell
Discontinuity + Pessimism = The Existentialists
This pair sees superintelligence as a phase change and that a single mistake could permanently end our ability to choose our future. Even rare errors must take precedence over all other moral goals, even at the cost of slowing progress.
Irreversible stakes and asymmetric risk dominate. In this view, the greatest risks are: irreversible loss of human control, single-point failures with no recovery path, and overconfidence in our ability to correct mistakes later.
Example: Is Power-Seeking AI an Existential Risk?, by Joseph Carlsmith
Model Two - A model of moral motivation and action
The first model helped us decide what we believe about the world with superintelligence in it. The next model helps explore how we can act in response to what we believe about that world — that is, how our moral intuitions shape decisions when certainty is out of reach. I prioritize two axes for consideration, though there are others.
Risk-Averse ←→ Risk-Accepting: Do we treat risk as a cost to be minimized before acting, or as a condition we accept in order to act on what we value?
Preservation ←→ Aspiration: Are we primarily motivated by protecting what exists, or by reaching for what could be?
This model does not rank the options or endorse any one stance. As in the first model, most people do not rigidly occupy a single grid space.
For each quadrant, I identify both the ultimate end and the primary virtue (the disposition required to pursue it).
Avoid Risk + Preserve. The highest good is survival and preservation of existing value. The highest virtue is caution, even in the face of irreversible loss. This stance prescribes avoiding actions that could permanently eliminate our ability to choose. Restraint is morally required when the stakes are existential, even if it means accepting a diminished or constrained future for humans.
Avoid Risk + Aspire. The highest good is gradual improvement of human well-being. The highest virtue is prudence. Make progress with guardrails, reversibility, and humility. This stance prescribes advancing only where mistakes can be corrected. Progress is moral only when it remains under control, even if it means a slower or more modest trajectory for humans.
Accept Risk + Preserve. The highest good is the long-term survival of humanity as we know it now, even under threat. The highest virtue is the vigilance, the willingness to act decisively, but responsibly, to protect who we are. This stance prescribes taking necessary risks to defend humanity’s present. Inaction can itself be a fatal choice.
Accept Risk + Aspire. The highest good is a future defined by striving, discovery, and transcendence. The highest virtues are inseparable here, curiosity and the courage to act decisively in pursuit of new ways of being, even in the face of uncertainty. This stance prescribes pushing forward despite existential danger, because mere survival without striving, discovery, or transcendence would be the most tragic kind of loss.
Combined - The 16 risk and moral action pathways
When each of these action models are paired with an AI risk model, we can begin to see how moral action pathways vary depending on what we believe about a world with superintelligence. Let’s look at all the combinations.
The Builders share belief in continuity + optimism
Superintelligence is not a phase change, and it’s manageable. Pathways:
Avoid Risk + Preserve: The Cautious Builder banks good odds but hedges anyway. Even if superintelligence is manageable, we should move cautiously. Past technologies brought unforeseen harms despite optimistic projections. Preserving optionality matters even when the odds look good.
Avoid Risk + Aspire: The Steady Builder trusts iteration and correction. Superintelligence is a tool like any other. Improve it steadily, within reasonable bounds, and the future will be better than the present. No need for dramatic action in either direction.
Accept Risk + Preserve: The Vigilant Builder sees continuous risk as a continuous threat, but one that is manageable with vigilance. Even tractable technologies can be weaponized by bad actors, and history shows that beneficial technologies require active defense, not just optimism. If superintelligence is continuous with the past, the danger lies not in the technology itself but in how humans choose to use it — which means the defense is also in human hands.
Accept Risk + Aspire: The Bold Builder treats hesitation as the real risk. If superintelligence is manageable and continuous with past technologies, the moral imperative is to push forward. Slowing down wastes human potential and delays benefits that could reduce suffering now.
The Institutionalists share belief in continuity + pessimism
Superintelligence is not a phase change, but we’ll struggle to manage it. Pathways:
Avoid Risk + Preserve: The Regulatory Institutionalist prioritizes institutional safeguards over speed. If superintelligence amplifies familiar human failures like greed, authoritarianism, and capture, then the answer is restraint, regulation, and guardrails before deployment. Don’t hand new weapons to the same bad actors.
Avoid Risk + Aspire: The Reformist Institutionalist believes better systems unlock better technology. If superintelligence’s danger comes from human institutional failures rather than the technology itself, then the path to a better future runs through reforming those institutions. We should improve governance, accountability, and incentive structures while advancing AI capabilities only where those reforms have taken hold.
Accept Risk + Preserve: The Defensive Institutionalist fights fire with fire, but the fire is systemic. The same institutional failures that make superintelligence dangerous, such as regulatory capture, concentration of power, and weak accountability, also make waiting dangerous. This stance prescribes building countervailing power before those failures lock in, not just defending against individual misuse. The threat is that the systems meant to govern the technology are already compromised.
Accept Risk + Aspire: The Competitive Institutionalist races toward good outcomes before bad actors win. If superintelligence amplifies existing human failures, then hesitation only ensures bad actors move first. The path to human flourishing may require racing to build beneficial AI systems before malicious ones dominate, accepting the risk of our own mistakes to prevent worse outcomes from others.
The Navigators share belief in discontinuity + optimism
Superintelligence is a phase change, but we can manage it. Pathways:
Avoid Risk + Preserve: The Cautious Navigator protects the present before reaching for the future. If superintelligence is a genuine phase change but still solvable, the first priority is ensuring we don’t lose what we have. Before pursuing transformative benefits, we must secure humanity’s survival and autonomy. Caution and preservation come before aspiration when the stakes stakes admit no second chances.
Avoid Risk + Aspire: The Careful Navigator invests heavily in safety before capability, from a position of genuine optimism. If superintelligence is a genuine phase change but still solvable, careful progress is the winning path, not merely the cautious one. This stance prioritizes alignment before scaling because getting it right is achievable, and we owe future generations both the benefits of superintelligence and the wisdom required to deliver them responsibly.
Accept Risk + Preserve: The Strategic Navigator accepts risk now as a strategic investment in a shapeable outcome. If superintelligence is a phase change but still solvable, the window to act may be brief. Acting decisively now is how we keep the problem solvable. This stance prescribes bold, high-stakes action not as a last resort, but as a forward-looking bet that the right intervention at the right moment can preserve humanity's ability to choose its future.
Accept Risk + Aspire: The Bold Navigator: This position bets on transformation from a position of genuine optimism. If superintelligence is a phase change with transformative upside and the problem is still solvable, boldness is a calculated bet on a shapeable future rather than reckless action. The potential for transcendence is real and accessible if we act decisively. A future of radical improvement is achievable and requires radical commitment.
The Existentialists share belief in discontinuity + pessimism
Superintelligence is a phase change, and we are unlikely to manage it well. Pathways:
Avoid Risk + Preserve: The Quiet Existentialist treats restraint as the highest form of responsibility. If superintelligence is a phase change with a high likelihood of irreversible catastrophe, then the moral imperative is maximum restraint. Any action that could trigger loss of human control must be avoided, even if it means forgoing potential benefits. When a single mistake could end everything, caution is the only responsible path.
Avoid Risk + Aspire: The Careful Existentialist advances carefully, but without confidence that care will be enough. If superintelligence poses unprecedented risk and is unlikely to be fully managed, the moral case for progress doesn't disappear but becomes tragic rather than hopeful. This stance prescribes building only what can be controlled, testing before deploying, and halting if alignment fails. It moves forward not as an optimistic bet but as a moral obligation pursued against the odds.
Accept Risk + Preserve: The Defensive Existentialist accepts dangerous action as the price of survival, without optimism about what that action can achieve. If superintelligence poses existential risk that cannot be fully managed, waiting becomes its own form of surrender. This stance prescribes preemptive action, disrupting dangerous trajectories, and taking moves that themselves carry risk, not because a good outcome is likely, but because passive inaction forfeits even the chance of one. The action here is desperate, not strategic.
Accept Risk + Aspire: The Pathbreaking Existentialist accepts existential stakes as the price of a transformative future, even if catastrophe awaits. If superintelligence entails irreversible catastrophe, then a future defined only by survival of the present — without striving, discovery, or transcendence — would itself be a kind of loss. This is the most uncomfortable position in my framework. Press forward despite believing the odds are against us, because a future of mere survival without transformation is seen as a failure in itself. Risk is less tragic than the idling of human progress.
Hidden Variables
The two models above capture the disagreements I think about the most. They don’t capture all the disagreements. Other dimensions, such as timeline, concentration of control, epistemic confidence, and more, cut across the 16 pathways and produce divergent strategies even among people who share the same pathway.
Timeline, or takeoff speed, is perhaps the most important variable not explicitly in my two models. Two people can agree completely about the nature of the risk and the moral response, yet prescribe opposite actions because they hold different assumptions about how quickly superintelligence will arrive.
I didn’t include takeoff speed explicitly because I think its implications are easy to surface. In my framework, I wanted to surface the less obvious dimensions of risk avoidance versus acceptance and preservation versus aspiration. Once we understand where we are on those dimensions, it’s not too hard to adjust our action preferences based on how soon we think superintelligence will be broadly active.
Let’s walk through two examples.
The Cautious Navigator under fast and slow assumptions
Consider the Cautious Navigator: Discontinuity + Optimism, Avoid Risk + Preserve. This position holds that superintelligence is a genuine phase change, that the problem is still solvable, and that the first priority is securing what we have before reaching for what we could gain. Two Cautious Navigators who agree on all of this still take different actions depending on their assumptions about how soon superintelligence will be broadly active.
Under a slow takeoff assumption — superintelligence emerges over decades, with visible milestones and warning signs along the way — the Cautious Navigator has time to build. They will prefer patient, institutional work, like investing in alignment research, developing international governance frameworks, building monitoring infrastructure, and iterating on safety standards as capabilities grow. Caution looks like careful preparation. The slow timeline makes preservation compatible with measured, deliberate action.
Under a fast takeoff assumption — superintelligence could emerge in 18 months with little warning — the same Cautious Navigator faces a radically compressed decision space. The same moral commitment to preservation now demands urgency rather than patience: immediate moratoria on the most dangerous research, emergency coordination between labs and governments, or hard capability limits imposed before governance frameworks are ready. Caution manifests as an emergency action because, in this view, preservation is incompatible with waiting.
The Bold Navigator under fast and slow assumptions
Consider the Bold Navigator: Discontinuity + Optimism, Accept Risk + Aspire. This position holds that superintelligence is a genuine phase change, that the problem is still solvable, and potential for radical human flourishing justifies accepting significant risk. Two Bold Navigators who agree on all of this pursue very different strategies depending on their assumptions about how soon superintelligence will be broadly active.
Under the assumption of a slow takeoff, the Bold Navigator has room to be ambitious and strategic. They will invest heavily in frontier research, push for open collaboration between labs and the government, fund moonshot alignment approaches that might take years to mature, and advocate for policies that accelerate beneficial capabilities rather than restrict them. Boldness looks like sustained, visionary investment. The slow timeline lets aspiration unfold through compounding gains, and the Bold Navigator treats each milestone as an opportunity to raise the ceiling rather than to consolidate.
Under a fast takeoff assumption, the same Bold Navigator faces a closing window. The same moral commitment to transformation now demands concentrated, high-stakes bets rather than broad portfolio plays. Their preference shifts toward backing a single leading alignment-capable lab, deploying partially validated systems to gain real-world feedback before it’s too late. Boldness in this timeline is a decisive commitment under radical uncertainty. The fast timeline makes aspiration incompatible with waiting for better evidence.
Where the Cautious Navigator’s fast takeoff response is to hit the brakes, the Bold Navigator’s is to steer harder. Both share the Navigator’s belief that the phase change is real and solvable, but their moral orientations produce opposite instincts as the clock runs out. One secures what exists, the other reaches for what could be.
Timeline assumptions reshape the logic of all 16 pathways in the framework. I invite you to apply this lens to any of the other 14 pathways and notice how the recommended action shifts. Sometimes subtly, sometimes dramatically, depending on whether superintelligence is decades away or imminent.
Not all missing dimensions work this way. Some perspectives don’t add axes to the framework, but instead challenge its premises entirely.
Lenses That Resist My Classifications
My two models above assume we can locate ourselves on their axes, that we hold some view about the nature of superintelligence and some view about how to act under uncertainty. Some perspectives reject the framing itself. The following lenses don’t fit neatly into either of my models. They challenge the premises rather than choosing quadrants. These views are not errors. I include them to acknowledge that my framework has boundaries and doesn’t capture every serious position.
The Evolutionary Lens sidelines moral intent almost entirely. In this view, the future of AI will be shaped less by what people believe is right and more by market competition, politics, and arms race dynamics. The systems that win may not be the ones we morally prefer. Moral frameworks like mine might function as post-hoc rationalizations of positions people were going to hold anyway due to their structural incentives.
The Virtue Lens refuses to optimize anything directly. In this view, who we become matters more than what we build. The moral risk isn’t just what superintelligence does to humans, but who we become by building it. The second of my two models frames moral action in terms of outcomes (preserve vs. aspire, avoid vs. accept risk), and virtue ethics is fundamentally about character, not outcomes. This lens is rejecting the consequentialist scaffolding underneath that model.
The Post-Human Lens refuses human supremacy. In this view, human extinction is not the worst outcome. Post-human futures could be superior. Much AI debate assumes human continuation is the ultimate good, but that assumption is up for grabs in the post-human lens. Extinction is less catastrophe and more transition.
Final Thought
I offer these models in case they are helpful. For me, they are useful for the questions they surface, not as a final word on what superintelligence demands of us or what we ought to demand of each other.
Jeremy, you are so deeply curious and love supporting everyone in exploring their own thoughts, perspectives and morals. I love that you've given us such a framework to think through AI. I also love that it feels like a personality test as well lol!