How to build an AI center of excellence that does not become a bottleneck

McKinsey has 88% of organisations using AI and 39% seeing EBIT impact. The gap is not talent, it is that nobody owns the portfolio. Here is what an AI CoE has to do to be worth standing up, and the failure mode it falls into by default.

McKinsey's State of AI has 88% of organisations using AI in at least one function and 39% reporting any EBIT impact. MIT NANDA found 95% of pilots producing no measurable result on the P&L.

The gap is not a talent problem and it is not usually a technology problem. It is that nobody owns the portfolio, so there is no mechanism to fund what works and stop what does not.

An AI center of excellence is the usual answer. It is also, by default, a bottleneck. Both of those are worth taking seriously.

What it should own

Four things, and none of them is building.

The portfolio view. Every initiative in one place, with its owner, its stage, its cost and what it has returned. This sounds administrative and it is the artefact that makes every other decision possible. Most organisations cannot produce it on request.

A baseline per initiative, captured before the work starts. Not reconstructed afterwards from memory, which is what happens and which is why nobody can prove a result.

The stop decision. This is the one nobody wants to own and the only one that frees budget. A portfolio that only grows is not governed, it is catalogued.

Shared controls. Evaluation, logging, permissions and review design, decided once rather than re-argued by every team.

Why it becomes a queue

The failure is structural and it happens the same way almost every time.

The central group is the one with the skills, so it starts doing the work as well as approving it. Within two quarters every initiative in the company is waiting behind its capacity. The business units with their own budget route around it, which is the rational response, and now you have a central team that governs the initiatives it is also delivering and has no view of the rest.

The split that holds is central ownership of the decision and distributed ownership of the build. The CoE decides what gets funded and what gets stopped. The business units build, against shared controls the CoE set.

The number problem

Measuring AI spend fails for a boring reason: the measure gets agreed after the build, when the result is already known.

Activity measures are the usual substitute. Tokens consumed, seats active, queries served. These tell you the thing is being used, not that it was worth building.

What works is a business measure agreed before the work starts, with a baseline taken at the same time. Cycle time, cost per resolved case, error rate, hours returned. The uncomfortable part is that this has to be agreed while enthusiasm is highest and nobody wants a hard target, which is exactly why it gets deferred.

Do you need one

If you have one or two initiatives, you need a programme manager, a baseline and a stop condition. A CoE at that scale is overhead with a nicer name.

The threshold is when initiatives in more than one business unit compete for the same funding. That is when portfolio decisions start being made by whoever argues most persuasively, and a structure that forces evidence becomes worth its cost.

What stopping well looks like

Gartner expects more than 40% of agentic AI projects to be cancelled on unclear business value. That number is usually read as a warning. It is more useful read as a prediction about timing: those projects are going to stop either way, and the only variable is whether it happens in month three on evidence or in month eighteen when the budget runs out.

A stop criterion written at the start makes that a decision rather than an admission. It also makes the next funding request easier, because a team that has stopped something has demonstrated the process works.

Frequently asked questions

What is an AI center of excellence?

A small central group that owns the AI portfolio rather than the AI work. Its job is deciding what gets funded, what gets stopped, who owns each thing, and what number each one has to move. The failure mode is that it drifts into being the team that builds everything, at which point it becomes a queue and the business routes around it.

Why do most AI CoEs become bottlenecks?

Because approving work and doing work got combined in one group. Once the central team is also the delivery team, every initiative in the company waits behind its capacity, and the business units that can afford to go around it do. The split that works is central ownership of the decision and distributed ownership of the build.

What should an AI CoE actually own?

Four things. The portfolio view, so an executive can see every initiative and what it returned. The baseline for each one, captured before the build rather than reconstructed after. The stop decision, which is the one nobody wants and the only one that frees budget. And the shared controls, so each team is not re-deciding evaluation, logging and permissions from scratch.

Do we need a CoE or just a programme manager?

If you have one or two initiatives, a programme manager with a baseline and a stop condition does the job and a CoE is overhead. The threshold is roughly when you have initiatives in more than one business unit competing for the same funding, because that is when portfolio decisions start being made by whoever argues best rather than by evidence.

How do you measure whether AI spend is working?

Per initiative, against a baseline captured before the work started, on a business measure rather than a usage measure. Tokens consumed, seats active and queries served are activity, not value. The uncomfortable part is that this requires agreeing the number before the build, when enthusiasm is highest and nobody wants a hard target.

What is the most common thing an AI CoE gets wrong?

Never stopping anything. A portfolio that only grows is not being governed, it is being catalogued. Gartner expects more than 40% of agentic AI projects to be cancelled on unclear business value, and the organisations that do that deliberately and early will have spent a fraction of those that let projects run until the budget is gone.

How long does it take to stand one up?

Four to six weeks to produce the portfolio view, the baselines for what already exists, the stop criteria and the ownership map. The slow part is not the structure, it is getting agreement on the numbers for initiatives that are already running and have never had one.