Somewhere in the last year I stopped being surprised by the dashboard. A leadership team brings me in, and before the second coffee someone proudly turns a screen around. Seats deployed, up and to the right. Prompts per week, climbing. Licence utilisation, green. The mood in the room is the mood of a job going well. So I ask the one question the dashboard cannot answer: has anything the organisation actually does got measurably better since these numbers started climbing? And the room goes quiet in a particular way — the quiet of people realising they have been measuring the thermometer instead of the temperature.
That quiet is the subject of this piece. The metrics most organisations use to track their AI programme are not merely incomplete — they point in the wrong direction, and confidently. Seat counts, prompt volumes and licence utilisation flatter you. They measure adoption, and adoption is the one thing you do not need to prove.
Adoption already happened. Stop counting it.
MIT’s Project NANDA, in The GenAI Divide, found that only about 40% of companies hold official large-language-model subscriptions, while in over 90% of companies employees already use AI tools on their own. Client data from Astu Labs, the company I founded, puts a floor under it: across roughly thirty Nordic organisations surveyed between 2025 and 2026, shadow AI was running inside every single one, and 24% of people admitted, to a deliberately self-incriminating question, to using a tool their employer never sanctioned.
So when your dashboard shows adoption rising, it is telling you something you already knew and cannot take credit for: the appetite arrived on its own, bottom up, before any programme. Counting it more precisely moves you no closer to value. It just simulates progress on the one axis where progress was never the problem.
There is a second, quieter reason the usage curve deceives. The research has a name for it: the novelty effect. Interest and use rise when a new technology arrives, then fall away over the following weeks and months unless the use becomes intrinsically useful. A seat-count line that climbs beautifully in month one is at least as likely to be measuring curiosity as capability; read it as proof of transformation and you are planning your next investment on a number about to bend back down.
The individual moved. The organisation did not.
Even when the usage is real and sustained, it measures the wrong unit. The sharpest evidence I know of is a randomised controlled trial across 66 companies and more than 7,000 knowledge workers: giving people Microsoft 365 Copilot saved them around two hours of email a week, a genuine individual time saving. But the structure of their work did not change. The number and composition of their tasks stayed the same. The authors are explicit that deeper change would require redesigning processes and reallocating responsibilities.
Sit with what that means for your dashboard. You can post a real per-person productivity gain, celebrate it, scale it, and have moved the operating model not one millimetre. The tool lifts the individual; it does not, on its own, touch how the work is organised, and that is the only thing that shows up in the results the board cares about. A metric that captures the first and misses the second is not a small error. It is the whole error, dressed as success.
This is why MIT located the binding constraint not in infrastructure, regulation or talent, but in learning — the organisation’s ability to retain feedback, adapt to context and improve over time. Learning is a property of the operating model, not of the licence. And not one of the vanity metrics (seats, prompts, logins) can see it.
Measure the work, not the tool
So what should sit on the screen instead? The principle is simple to say and uncomfortable to adopt: measure whether the organisation is getting better at the work, not whether it is using the tool.
Put concretely: if your programme’s headline KPIs are active users, licence utilisation and prompts per week, you are measuring adoption. If they are cycle time, error and rework rates, time-to-fix, and how quickly a new hire reaches target performance, you are measuring whether the work itself is getting better. The first set tells you the tool is being used; the second tells you whether that use is turning into an advantage.
The strongest external signal for where to point comes from McKinsey. Of twenty-five organisational attributes they tested, the redesign of workflows had the biggest effect on a company’s ability to see EBIT impact from generative AI, yet only 21% said they had fundamentally redesigned even some workflows. (Self-reported and correlational, so treat it as direction, not proof.) The lesson is direct: if workflow redesign is where the value lives, redesigned workflows are what you instrument. Track cycle time, rework and quality, and whether feedback from each run re-enters and improves the next, not how many people logged in near them.
BCG’s AI Value Gap -survey points the same way: only about 5% of companies extract value from AI at scale, while roughly 60% see none worth the name. (Consultant survey data with a commercial interest, so hold it lightly.) The 5% are not distinguished by better tools (everyone buys from the same shelf) but by the operating model built around them, which is exactly what seat counts cannot detect.
There is even a learning metric hiding in the productivity research. When Brynjolfsson, Li and Raymond followed customer-service agents, AI raised productivity by around 15% on average, but most of all for the least experienced, for whom the gain reached roughly a third. A programme that is genuinely working should therefore show capability spreading to where it compounds: the newest people getting good faster. A seat count is blind to that. A time-to-competence number is not.
I put it to leadership teams as a pair. The euro tells you whether it paid off. The human metric tells you why. You need both, and most dashboards carry only a weak proxy for the first and nothing at all for the second, which is why they cannot tell you what to do next.
The honest counter: learning is soft, the CFO wants a number
The fair objection is that all of this sounds unmeasurable next to a clean utilisation percentage. The CFO wants a hard figure by Friday, and “is the organisation learning” is not that.
Two answers. First, learning is more measurable than its reputation suggests: it is just harder than counting logins. Cycle time on a redesigned process, rework and error rates falling month over month, time-to-competence for new joiners, whether a feedback loop demonstrably changes the next iteration: these are operational, auditable numbers, not vibes. This is the Check and Act half of the discipline behind ISO/IEC 42001, the half most programmes skip, and the half that separates a pilot from an operating model. Second, and more bluntly: choosing the easy wrong number over the harder right one is a decision, not a constraint — looking for your keys under the streetlight because that is where the light is. The keys are not there.
None of this makes usage data worthless. Early on it is a floor — you cannot learn from a tool nobody touches. But usage is a threshold to clear, not a target to chase, and the moment you treat clearing the floor as winning the game, the dashboard starts lying to you in green.
Start here this quarter
1. Demote the vanity dashboard. Seats, prompts and licence utilisation become a floor check you glance at, not the scoreboard you report. If they are the headline of your AI review, change the headline first.
2. Instrument one workflow redesign end to end. Choose a single process you are going to rebuild around AI, and instrument it from the start: cycle time, rework, quality, and whether feedback re-enters the loop. One workflow you measure through its redesign teaches you more than ten counted pilots.
3. Track a learning-rate metric. Is the organisation getting better at this work over time: error rates down, time-to-competence shortening, capability reaching the least experienced? If nothing improves month over month, you have adoption without transformation, which is the divide, precisely located.
4. Put a euro figure and a human figure side by side. On every initiative, report both. The euro says whether it paid off; the human metric says why, and only the second tells you where to invest next.
The dashboard was never the point
The 95% of programmes that show no measurable return are not, for the most part, under-measuring. They are measuring the wrong thing with real precision. A screen full of green adoption numbers can sit, comfortably and indefinitely, on top of an organisation that has learned nothing it did not already know a year ago.
Change what you count, and the operating-model gap stops being invisible — it becomes the most useful number you have. The tool will happily tell you it is being used. Whether that use ever becomes an advantage is decided one layer down, in how the work is organised and how fast the people doing it learn. Measure the tool and you will manage the tool. Measure the learning, and you finally start managing the thing that pays.
About the author
For nearly three decades, Lenni Laukkanen has helped organisations develop and grow through technology. He argues that AI is not a technology question but a leadership one: it rarely fails as a tool — it exposes ways of operating built for a world that no longer exists. As founder of Astu Labs, trusted advisor to executive teams and keynote speaker, he helps leaders navigate the organisational changes required to capture value from AI. A runner, cyclist and adventurer, he holds that organisations, like athletes, change one deliberate step at a time. More at lennilaukkanen.fi.
*The views and opinions expressed by the author do not necessarily state or reflect the views or positions of Hyperight.com or any entities they represent.
