1. The Paradox of Progress
A few years ago, the dream of truly “agentic” AI, meaning autonomous systems that reason, plan, and act with minimal human input, felt like science fiction. Today, that future seems tangible. AI agents can browse, book, schedule, code, analyze, and even debate. They can plan vacations or design production workflows.
In the Episode 167 of the AI After Work (AIAW) Podcast, Luka Crnkovic-Friis, Distinguished Engineer and Head of AI/ML at King (Microsoft), explores exactly this frontier. As he reminds us, there is a world of difference between what is possible in a demo and what works inside a business process.
In his words, AI’s biggest limitation is not in capability; it is in fit. Deploying reasoning agents in enterprise operations exposes a host of bottlenecks that theory alone does not predict: non-human error modes, cognitive trust gaps, and the psychology of adoption among those whose jobs are being “augmented.”
The conversation reveals an uncomfortable truth: agentic AI systems are crossing the usability threshold but have not yet crossed the reliability one.
2. When AI Fails Differently
Crnkovic-Friis begins with a simple story: trying to book a family trip using OpenAI’s new Atlas browser, which combines reasoning models with autonomous web navigation. “It’s solid,” he says, “but slow. It struggles with dropdowns, image-based elements, things trivial for humans.”This anecdote underscores a key point: AI does not fail like we do. Where humans make subtle, contextual mistakes, large language models (LLMs) can make glaring, almost comical ones, missing basic details while handling complex reasoning flawlessly.
In enterprise workflows, this asymmetry matters. A single “stupid” output can destroy trust in a system that is otherwise 90 percent correct. In human cognition, errors cluster around nuance; in LLMs, they appear random and absolute. As Crnkovic-Friis puts it, “The failure mode of LLMs is vastly different from that of humans. That makes evaluation and trust much harder.”
This is the Moravec Paradox in corporate form: tasks easy for humans, such as recognizing UI patterns or understanding intent, remain hard for machines, while the reverse, such as processing vast data or synthesizing patterns, is trivial for them.
3. Why Trust Is the True Bottleneck
Even in 2025, trust, not compute, is the scarce currency of AI transformation. A 2024 MIT Sloan study found that only 14 percent of enterprises report “high trust” in AI outputs used for decision support, even when accuracy rates exceed 85 percent.
This erosion of confidence is not technical; it is psychological. When a domain expert reads an AI-generated report and spots an obvious factual slip, the natural instinct is to distrust the entire output. Humans do not calibrate trust; they toggle it.
Crnkovic-Friis calls this the “psychological cliff” of AI reliability. “Once people see one basic error,” he explains, “they think, if it can’t even do this, how could it possibly do anything more advanced?” The result is paradoxical. The better AI gets, the less tolerant humans become of its mistakes.
4. Change Management, Not Just Model Management
Beyond accuracy, Crnkovic-Friis highlights a deeper challenge: how humans feel about being augmented.
Many AI leaders, himself included, come from technical backgrounds. For them, the rise of agentic systems feels liberating. They can automate tedious work and rediscover hands-on creation. But for developers and engineers who cherish the craft itself, AI feels invasive. “They love coding,” he says. “It’s their art. And now something is doing it, but not in the way they would.”
This cultural divide mirrors what McKinsey calls the AI persona gap: between “augmenters,” who see AI as empowerment, and “guardians,” who see it as encroachment. Both groups exist in every enterprise, and both must be addressed deliberately through differentiated adoption strategies. Henrik Göthberg , the podcast co-host, frames it succinctly: “You need a persona strategy. How you evangelize AI to a proud coder is not how you do it to an executive who just discovered superpowers.”
This reframes AI transformation as a matter of identity, not efficiency. The technology works, but people must want it to work with them.
5. The Disappearing Boundary Between “Tech” and “Non-Tech”
Crnkovic-Friis predicts that AI will erase the divide between technical and non-technical work. “Anything that’s information-based,” he argues, whether art, analysis, or operations, will become accessible through universal interfaces. In that world, distinctions like “developer” or “creative” blur. Artists can generate code. Engineers can design visuals. The future professional is a generalist orchestrating tools and agents across domains.
This “generalist renaissance” echoes research from Stanford’s Human-Centered AI Institute, which shows that interdisciplinary roles combining creativity, analysis, and coordination have grown 2.5 times faster than specialized technical roles in AI-intensive firms since 2022. Yet this shift demands new kinds of leadership. As Crnkovic-Friis notes, “You’ll either be the manager of a series of AIs or be managed by them.”
Leadership itself becomes a meta-skill: the ability to delegate, orchestrate, and align human and synthetic agents toward shared goals.
6. Beyond Next-Token Prediction: Toward System 2 AI
Crnkovic-Friis draws an important distinction between “System 1” and “System 2” AI, borrowing from Daniel Kahneman’s model of human cognition.
System 1 is fast, intuitive, and automatic, the way LLMs work today, predicting one token at a time without deliberation. System 2 is slow, reflective, and goal-driven, the realm of reasoning models and reinforcement learning.
The transition from System 1 to System 2 AI marks a profound evolution. Reinforcement Learning with Human Feedback (RLHF) gave us the chat interfaces that made models usable. But as Crnkovic-Friis observes, it also “lobotomized” them, aligning behavior at the cost of raw intelligence. The next step, reasoning models like OpenAI’s O-series, add a layer of reflection: the ability to generate chains of thought and optimize for correct answers rather than just plausible ones.
Yet even here, he warns, the breakthrough may be overstated. “In many cases,” he says, “reinforcement learning is just undoing the damage from alignment; it restores capability we already had.” Anders Arpteg, co-host at the podcast agrees.
Still, the direction is clear: from reaction to reasoning, from autocomplete to deliberation.
7. The Engineering Frontier: Scaling versus Smarter Scaling
Underneath the philosophical debates lies a more pragmatic truth: building modern AI systems is an engineering war.
“The algorithms aren’t the bottleneck,” Crnkovic-Friis explains. “It’s about making them run efficiently on GPUs.”
At King and Microsoft, his focus is on optimizing inference, reducing compute, compressing key-value caches, and lowering latency without sacrificing quality. These improvements do not grab headlines, but they make AI viable at enterprise scale.
He describes new approaches like Principal Component Analysis (PCA) compression in transformer caches, which can “cut memory usage drastically while keeping performance nearly intact.” Such optimizations, often invisible to end users, are what will make agentic AI systems affordable for broad deployment.
“We’re past the algorithmic breakthrough phase,” he says. “Now it’s an industrial engineering problem.”
8. Reinforcement Learning and the “Absolute Zero” Experiment
The conversation turns to a fascinating experiment from China, the Absolute Zero model, inspired by AlphaZero. Instead of relying on human feedback, it trained itself through self-play, predicting not only answers but also the questions.
Crnkovic-Friis finds the approach “conceptually brilliant,” though he doubts it yet scales to state-of-the-art models. Still, the principle, self-generating improvement through interaction, could redefine how AI learns. If RLHF made models human-aligned, and reasoning models made them reflective, self-play could make them self-evolving.
That said, he cautions that the real frontier is not algorithmic but economic: “When a single training run costs hundreds of millions of dollars, it kills innovation. We need to lower that barrier before breakthroughs can flourish.”
9. The Collapse of Cost and the Rise of Hybrid Models
To reduce those barriers, AI research is turning to mixture-of-experts architectures, systems that activate only relevant subnetworks for a given task. This selective activation mimics the human brain’s efficiency, using specialized modules instead of one monolithic engine.
The result is smaller, faster, more context-aware systems that retain depth where needed. As Crnkovic-Friis notes, “Everything is going hybrid. You’ll see architectures that plug in different modules for reasoning, perception, retrieval. It’s not one model anymore, it’s a federation.”
This modular future aligns with current enterprise adoption trends: companies increasingly deploy model ecosystems, specialized LLMs, reasoning agents, and retrieval systems orchestrated through workflows, rather than chasing a single “general” AI.
10. The Human Factor: Cognitive Trust Gaps
For all the technical progress, Crnkovic-Friis keeps circling back to one theme: human psychology.
When an AI agent fails at something trivial, people lose trust instantly. When it succeeds spectacularly, they take it for granted. The asymmetry is cognitive, not rational. He recounts a case where an AI autonomously completed a corporate training course, scoring 100 percent, but took as long as a human would, fumbling through navigation like a digital intern. “It’s not intelligence that’s the problem,” he says. “It’s sensor input. The system sees the interface as compressed images; it doesn’t really see.”
This perception bottleneck underscores a deeper lesson: until AI perceives the world as richly as humans do, it cannot reliably act within it. The frontier of reasoning must meet the frontier of perception before agentic AI can truly integrate into business processes.
11. The Changing Nature of Work
As roles evolve, the future workplace may look less like a pyramid and more like a network of humans and agents collaborating dynamically.
Crnkovic-Friis envisions a continuum where humans move up abstraction levels, managing, delegating, and orchestrating, while AI handles execution across domains. He cites Andrej Karpathy’s view that AI’s evolution mirrors software’s climb up abstraction layers: from assembly to high-level languages, from manual memory management to automated reasoning. The implication is clear: success in the AI age depends less on what you know and more on how fast you can move up abstraction levels.
“If your passion is writing TypeScript pages,” he cautions, “that’s not a great long-term outlook. You’ll need to evolve, or risk being replaced by the tools themselves.”
12. Ethics, Governance, and the Fast-Takeoff Question
As agentic systems grow more capable, ethical governance becomes existential. Autonomous models operating across business functions raise questions of accountability, auditability, and bias propagation.
Microsoft, Anthropic, and others are embedding ethical review into their development cycles, but regulation still lags. The EU’s AI Act, passed in 2025, focuses primarily on risk classification and data provenance, not on reasoning autonomy or multi-agent coordination. Crnkovic-Friis warns of “fast-takeoff” scenarios, where agents rapidly compound capabilities beyond human oversight. The practical risk today is not runaway AGI; it is runaway complexity, systems becoming so intertwined that no one can explain their decisions.
The remedy, he argues, is transparency by design. “We need architectures that remain legible, where we can trace why an agent did what it did, even as the system scales.”
13. A World of “Skills”
One of the most intriguing developments he highlights is Anthropic’s Claude Skills, modular, portable skill packages encoded as simple Markdown files containing examples, metadata, and style rules.
Crnkovic-Friis tested the system himself. “I packaged an internal workflow that connects our analytics systems through Claude Code. Now I can reuse it across projects like a plug-and-play skill.” It may seem minor, but it represents a paradigm shift. AI workflows are becoming composable. Skills can be traded, versioned, and shared, hinting at a future marketplace of intelligence.
If today’s AI apps are like software in the 1990s, skills are the APIs of the next decade, reusable modules that encode organizational knowledge.
14. The Economics of Intelligence
When asked about OpenAI’s controversial move toward a for-profit structure, Crnkovic-Friis is pragmatic.
“There was no practical way forward without it,” he says. “The costs are astronomical. The US GDP uptick last quarter came largely from data center investment. That’s the scale we’re talking about.” In other words, intelligence now has a balance sheet. The infrastructure of cognition—GPUs, energy, cooling, and data—has become the new industrial base.
This economic reality forces a recalibration of ambition. Innovation must coexist with sustainability. Efficiency, not just capability, will define the next wave of AI leadership.
15. What Enterprises Should Learn from This
So what should enterprise leaders take from all this?
First, expect non-linear progress. AI may solve one bottleneck while exposing another. Today’s failure modes are perceptual and contextual; tomorrow’s may be motivational or ethical.
Second, treat AI transformation as a human-change project. Different personas, creators, coders, and managers, require different pathways to adoption. Empathy is as important as engineering.
Third, embrace modularity. The future is not one giant model; it is an ecosystem of specialized agents orchestrated for specific outcomes.
And finally, cultivate trust through transparency. Show employees how and why AI reaches its conclusions. The fastest way to lose adoption is to hide uncertainty.
16. The Frontier Is Still Human
For all his technical expertise, Crnkovic-Friis ends where he began: with humility.
“We’re not at the point where AI can replace human judgment,” he says. “We’re at the point where it challenges us to be better at using it.”
The next generation of enterprise AI will not just automate work; it will redefine what work is. As systems learn to reason, humans must learn to orchestrate.
In the end, agentic AI may not remove humans from the loop; it may finally teach us what the loop is for.
This article was enhanced with the help of AI tools, drawing on the podcast transcript and complementary online research. To go deeper into the source material, listen to the full episode and make your own learnings.
Full episode on: www.aiawpodcast.com
Resources:
- “Building Enterprise AI Maturity” — MIT CISR (Weill, Woerner & Sebastian) https://cisr.mit.edu/publication/2024_1201_EnterpriseAIMaturityModel_WeillWoernerSebastian cisr.mit.edu
- “When Humans and AI Work Best Together” — MIT Sloan School of Management / Center for Collective Intelligence (Malone et al.) https://mitsloan.mit.edu/ideas-made-to-matter/when-humans-and-ai-work-best-together-and-when-each-better-alone MIT Sloan
- “The State of AI: How organizations are rewiring to capture value” — McKinsey & Company Global Survey (March 2025) https://www.mckinsey.com/~/media/mckinsey/business%20functions/quantumblack/our%20insights/the%20state%20of%20ai/2025/the-state-of-ai-how-organizations-are-rewiring-to-capture-value_final.pdf McKinsey & Company
- “Seizing the Agentic AI Advantage” — McKinsey (June 13 2025) https://www.mckinsey.com/capabilities/quantumblack/our-insights/seizing-the-agentic-ai-advantage McKinsey & Company
- “Claude Skills: Customize AI for your workflows” — Anthropic (October 2025) https://www.anthropic.com/news/skills
