Artificial intelligence (AI) can already write software, solve mathematical problems and produce engineering designs. But when it comes to the economically valuable implementation of its output, a person still has to take responsibility by signing off on it.

This may turn out to be one of the most important distinctions between an AI assistant and an AI professional.

An Economic Superintelligence Revolution could begin when institutions are able and willing to accept AI output without human approval.

Responsibility and Moral AI

Large Language Models typically show more or less the same warning: “The model can make errors.” In other words, it does not take responsibility for its mistakes. If you use it for important work, you are taking that responsibility.

This limits how much work AI can actually replace, giving rise to the question: what would catapult AI beyond being an assistant?

One possibility is that the developer of the AI takes responsibility, but this could put the whole company at risk. Another is that an AI could have a legal personality of its own. This does not necessarily mean treating AI as a human; a company, for example, has a legal personality too. But AI could eventually require a different legal category.

This economic question is related to a much older philosophical debate about moral agency: whether an AI system can distinguish right from wrong and be held responsible for its actions.

Giving AI moral agency is a step further than legal personality, as it could pave the way for a human-like legal status. In that case, AI could sign documents, take responsibility, and defend people in court. Moreover, the existence of a moral AI entity could shift some responsibility away from the company and onto the AI itself.

This is different from the legal personality of a company, where a human always signs off on contracts.

Economic Superintelligence

If AI acquires such moral and legal agency, the economic consequences could be profound.

The Economic Superintelligence Revolution would potentially begin when an AI can legally sign its work. This would mean the AI sells a finished product rather than advice.

This shift could affect several professional areas.

Some examples from the engineering, legal, and medical fields include AI giving formal legal advice and even defending a person in court; AI creating a production drawing that is used directly by a factory; AI delivering complete software; and AI giving a diagnosis.

AI Insurance

Legal authority to sign is economically useful if the output is sufficiently reliable. Such a sign-off requires a degree of reliability, which depends on the cost of an error. For a small task, large errors may be acceptable. For an important task, such as a health-related one, almost no errors are acceptable. The question is: can the AI company pay for all its errors and still make a profit?

This leads to the question of warranty-backed AI. Munich Re started to provide such services as early as 2018. Armilla AI, backed by Lloyd’s of London, is a more recent example. Its warranty pays out if model accuracy decreases below a certain threshold.

Insurability of such products depends on the ability to measure and control these risks. This is where formal verification becomes important. Formal verification is, and increasingly will be, at the centre of AI services. In mathematics, proof checkers can now verify whether proofs are correct at a superhuman speed. Similarly, software can be formally verified, and simulations in a physics-based world model can check engineering designs.

The Transition

The economic revolution would occur when the full process requires little human work and reaches an accuracy level that is insurable. Around this threshold, a small gain in reliability and/or scalability can ignite the Economic Superintelligence Revolution.

This could create a difficult transition period for labour markets. During this period, can the process of creative destruction come to the rescue?

Policymakers face an important question: Can people still work and produce economically meaningful output? Can people continue to live in a dignified way despite job losses?

For these reasons, from a labour-market perspective, the most consequential question about AI may be whether and when it gets legal entity status, rather than when it surpasses a certain IQ level.

Thus, the discussion of AI morality has implications far beyond its philosophical origins.