When Satya Nadella speaks about the future of artificial intelligence, we should listen carefully. We should also remember who is speaking.
As Microsoft’s CEO, Nadella is not a neutral observer of the AI transformation. Microsoft has invested heavily in OpenAI, operates one of the world’s largest cloud platforms and benefits from an economy in which organisations consume increasing amounts of computing capacity. His argument that the future of AI lies in trusted ecosystems, interchangeable models and enterprise-controlled data aligns closely with Microsoft’s commercial position.
That does not make his argument wrong.
In fact, dismissing an idea simply because the person presenting it benefits from its acceptance would be a form of motivated reasoning. The more responsible approach is to separate the commercial narrative from the underlying strategic insight. When we do this, Nadella’s interview with Fareed Zakaria raises one of the most important questions confronting organisations today: Who retains the knowledge, judgement and economic value when intelligence becomes something we purchase as a service?
The real test of AI is not technological
Nadella argues that the enormous investment flowing into chips, data centres and AI models will only be justified if it produces widespread economic growth. The test is not whether a model can write code, produce poetry or pass another benchmark. The test is whether small businesses, large organisations and public institutions become meaningfully more productive.
This is an important shift in perspective. AI adoption is often measured through activity: the number of licences purchased, employees trained, prompts submitted or processes automated. These measures tell us that the technology is being used. They do not tell us whether the organisation is becoming more capable.
A company can increase AI usage while weakening its own judgement. It can accelerate output while reducing the depth of its decisions. It can produce more information while gradually losing the institutional knowledge required to recognise when that information is wrong.
In this sense, adoption and capability are not the same thing.
The reverse information paradox
Nadella describes what he calls the “reverse information paradox”. When an organisation asks an AI system a question, it does not merely receive an answer. It may also reveal something about itself.
Its questions disclose what it considers important. Its prompts reveal how it frames problems. Its corrections show what it regards as acceptable. Over time, these interactions contain traces of the organisation’s priorities, reasoning processes, professional knowledge and decision patterns.
Nadella refers to this accumulated knowledge as “token capital”.
The concept is useful because it expands the conversation beyond conventional data privacy. A client list or financial record is clearly recognised as valuable data. Less obvious is the value embedded in thousands of AI-supported conversations through which employees explain problems, test alternatives, correct mistakes and refine decisions.
That information may represent the organisation’s thinking architecture.
If the AI provider retains this knowledge while the organisation merely receives the immediate output, the organisation may be paying twice. It pays financially for access to the system, and it pays again by contributing part of its institutional intelligence.
Nadella therefore argues that organisations should separate their own context, memory and operating framework from any individual AI model. Models should be interchangeable. Organisational knowledge should not be.
This position clearly benefits Microsoft. A model-independent ecosystem creates commercial opportunities for cloud and enterprise-platform providers. Yet the strategic warning remains valid: organisations that cannot retain their own knowledge may gradually surrender their ability to think independently.
Adoption can conceal cognitive surrender
This concern connects directly with my work in behavioural economics, neuroscience and the development of the Behavioural Adoption Risk Index, or BARI.
My interest is not limited to whether people accept AI. I am interested in what happens to human judgement after they accept it.
Traditional adoption models tend to ask whether people find a technology useful, easy to use or socially acceptable. These remain relevant questions, but generative AI introduces a deeper behavioural problem. A system can be readily adopted precisely because it reduces effort, removes friction and provides fluent answers.
Human beings often use fluency as a shortcut for truth. When an answer is clear, confident and immediately available, it feels easier to accept. Under pressure, the temptation becomes even stronger. Attention narrows, cognitive effort becomes more expensive, and challenging a plausible answer requires energy that the user may no longer have.
This creates the possibility of cognitive surrender. The person remains formally responsible for the decision but no longer contributes sufficient independent thought to it.
That is why BARI examines more than usage. It considers whether people retain the cognitive capacity to work with AI, whether trust is appropriately calibrated, whether employees still challenge questionable output, whether responsibility remains clear and whether the institution provides the conditions required for safe human-AI collaboration.
The danger is not simply that AI could make a mistake. The greater danger is that the surrounding human system may lose its ability to detect, question and correct that mistake.
The human contribution is changing
Nadella describes the emerging pattern of work as “macro delegation and micro steering”. The human defines the intention, delegates much of the execution to AI and then steers the result through judgement, correction and contextual understanding.
This is a powerful description of how work may change. It also contains an assumption that should not go unexamined. It assumes that people will continue to steer.
Behavioural science gives us reason to be cautious. When a system repeatedly produces useful answers, trust becomes habitual. As familiarity grows, checking may decline. The person may gradually move from active oversight to passive approval without consciously recognising the change.
The future value of human work may therefore lie not only in creativity. It may lie in discernment: the ability to frame the right problem, recognise what is missing, identify when an answer is inappropriate and accept responsibility for the final decision.
These qualities will not survive automatically. They must be practised, measured and supported by the organisation.
Human agency requires architecture
Nadella argues that society will accept AI that enables people to achieve more but reject technology that removes human dignity and agency. I agree with the principle, but agency cannot be protected through intention alone.
An organisation cannot simply declare that “a human remains in the loop” and assume that meaningful control has been preserved. A person may click the final approval button while lacking the time, expertise, psychological safety or confidence required to challenge the system.
Human oversight is only real when the person has the capacity and authority to disagree.
This requires an organisational architecture that protects decision depth. Employees need to understand the limitations of AI, know when escalation is required and feel safe questioning an output that appears authoritative. Accountability must be clear, especially when decisions are distributed across people, models and automated agents.
This reflects a principle that also runs through my work in behavioural financial literacy: knowledge alone does not guarantee effective behaviour. Decisions are shaped by pressure, attention, emotion and environment. If we want people to behave responsibly, we must design systems that make responsible behaviour possible under real-world conditions.
The same is true of AI governance. Training people to use AI is insufficient if the surrounding environment rewards speed, discourages challenge and treats every hesitation as inefficiency.
The question organisations should be asking
The strategic AI question is no longer simply, “Which model should we use?”
A more important set of questions is emerging:
Are we becoming more capable, or merely more dependent?
Does our organisational knowledge remain with us?
Are employees using AI to extend their judgement, or to avoid exercising it?
Can they recognise when the system is wrong?
Do they still have permission to challenge it?
And if the AI system disappeared tomorrow, would the organisation still understand how its own decisions were made?
Nadella’s warning should not be accepted uncritically. Microsoft has much to gain from the ecosystem he describes. But commercial interest does not invalidate strategic insight. It simply requires us to examine the argument with greater discipline.
The deepest risk of AI may not be that machines become capable of thinking. It may be that organisations become progressively less capable of thinking without them.
The goal should therefore not be to keep humans ceremonially “in the loop”. It should be to build organisations in which human judgement, institutional knowledge and the capacity to challenge remain alive.
AI should expand human agency. Whether it actually does so will depend less on the intelligence of the model than on the behavioural and institutional architecture we build around it.