Anyone can ship an AI demo. Shipping AI that survives Monday morning, a traffic spike and a budget review is a different job.

AWS reviewed the AI work we have put into production, and allOps now holds the AWS AI Competency in both Generative AI Consulting Services and Agentic AI Consulting Services. The AI Competency validates partners across those two categories against standards for security, reliability and operational excellence.

What it certifies is narrow and worth stating plainly. Not that we can build generative and agentic systems. That we have put them in production for real customers, and that they held.

The validation runs on customer deployments, so here is what ours looked like

For WellPet.ai, we cut normalized inference cost by 37% and took recommendation relevance from 80% to 94%.

For TrueStay, the move to Amazon Bedrock cut AI spend by up to 90% while the product scaled from 4,000 to 16,000 users.

For Lighthouse, an AI-native CRM for VCs and their agents, consolidating the AI data pipeline onto AWS took 22% off the monthly bill.

Three different problems, and the same pattern in all of them. The interesting number is not the model benchmark.

It is the cost curve and whether the thing still works at four times the users. All three are written up in full on our case studies page, with more coming over the next few weeks and months.

Why so few make it beyond an AI PoC

AWS states the position on its own competency page: fewer than 40% of organizations are past the experimental phase with AI.

The most-cited figure is bleaker. MIT’s The GenAI Divide: State of AI in Business 2025 reviewed 300 public deployments and found that 95% of enterprise pilots delivered no measurable P&L impact, with only about 5% of custom systems reaching production with real value.

That number has drawn methodological criticism and deserves the caveat, but the shape of the finding is consistent. There’s high adoption, but low transformation. The gap was not model quality and not regulation. It was integration.

The model is rarely the hard part

Model choice is the visible decision, so it absorbs most of the conversation. It is almost never what kills the project. The data underneath it is.

Pipelines that hold at production volume rather than on the sample the demo ran against. Governance, so you can say where an answer came from and who was allowed to see the input.

Evaluation that keeps working after launch, because relevance drifts and nobody notices until a customer does. And a cost model that still makes sense when the POC meets real traffic, which is where inference economics turn from a rounding error into the second-largest line on the bill.

That is where AI projects quietly stall. It is also the half we spend most of our time on, and the reason the numbers above are cost and reliability figures rather than benchmark scores.

Generative and agentic AI Competency, and why both

The two categories cover different failure modes.

Generative AI, in AWS’s framing, applies foundation models to specific use cases, whether vertical work like document analysis and product descriptions or horizontal work like service desks, knowledge management and onboarding.

Agentic AI raises the stakes, because those are production-grade systems that reason, plan, use tools, collaborate and act with minimal supervision.

An agent that acts autonomously is a system that can be wrong at speed and at scale. Holding both categories means the guardrails, evaluation and cost controls were validated for the autonomous case, not only for the case where a human reads every output before it goes anywhere.

Thanks to the team, and to the customers who let us build the difficult parts with them.

Talk to an AWS Hero about which of your workloads are actually suited to AI.