Plenty of AI rollouts are working. The tools get used. The pilots ship. The teams like them. The harder question, the one most measurement programs aren't built to answer, is whether any of it has changed the business.
Activity is just activity. You measure weekly active usage (WAU) and it looks great, but quality, efficiency, and cost aren't moving.
To be clear: activity is great for learning how to use a tool, and learning by doing is valuable. But activity alone doesn't produce measurable business outcomes.
A developer using AI assisted coding tools for 4 hours a day isn't automatically shipping faster, especially when the slowest part of the workflow is a manual, cumbersome code review.
A support team member using agentic chat for 3 hours a day isn't closing tickets any faster if the longest delay is caused by misrouted ticket assignments. An AI agent that automatically routed tickets would have been a better investment.
In both scenarios, the activity wasn't aligned with a business outcome. AI rewards intentional, strategic choices about where to apply it.
The chain you're missing
An outcome-focused approach is where return on investment starts to show up. But because AI is so flexible, knowing where to start and what to measure isn't obvious.
Think about it in three layers:
- Business goal: The outcome that matters. Faster releases. Happier customers. Lower costs. This is the destination.
- Drivers: The levers that influence that goal. For faster releases, drivers might include development speed, quality, and team efficiency. These are the areas where change actually happens.
- Indicators: The measurable signals that show whether the drivers are moving, such as cycle time, defect rate, deployment frequency, and throughput.
Many AI adoption initiatives skip straight to tool deployment, bypassing this chain completely. Teams pick a tool first instead of a driver. They measure usage when they should be measuring an indicator tied to a real driver.
The result: AI everywhere, outcomes nowhere.
Where do teams go wrong?
They optimize the wrong driver. A team focuses on "incident response time" and invests in AI generated root cause analysis, but that analysis runs after the incident has closed, so it doesn’t improve incident response time. The driver they targeted wasn't tied to their goals.
They measure inputs, not indicators. "Our developers use AI 3 hours a day" tells you nothing about whether cycle time dropped. Measure the thing that's actually connected to the outcome.
They don't isolate causality. When teams make many changes at once, they can't tell which one moved the outcome. A drop in defect rate is fantastic, but was it caused by the AI testing tools, the new hire, or the refactored architecture? Without a clear hypothesis and baseline, you can't know what to double down on.
They deploy AI to individuals, not workflows. AI unlocks individual productivity, but doesn't automatically unlock team throughput. If handoffs, approval gates, or deployment pipelines are the constraints, giving every developer an AI assistant doesn't change the system. Many systems are optimized for people, not agents.
How to fix it
Before your next AI investment, trace the chain backward:
- Name the business goal explicitly. Instead of saying "we want to improve engineering," try “we want to reduce time-to-production for new features by 30%.” If you can't say it in a single sentence with a number attached, it's not a goal yet — it's a wish.
- Identify the top two or three drivers. What actually controls that goal in your context? Talk to your team leads. Look at your data. Where is the time going? Where do defects originate? The bottleneck is usually obvious once you ask the right people.
- Define indicators before you deploy. Decide what you'll measure to know if a driver is improving, then set the baseline before going live with the tool. Without a baseline, you'll never know if anything moved.
- Pick the AI application that targets a specific driver. Don't choose tools based on popularity or a compelling demo. Choose the one that addresses the bottleneck.
- Review the indicators on a regular cadence. Monthly at minimum. If the indicator isn't moving after a few weeks, pivot. The tool may not be hitting the driver, or the driver may not be the real lever. Either way, change course.
Imperfect metrics are better than no metrics at all
Some drivers are genuinely hard to measure. Team efficiency, knowledge quality, and decision speed don't always produce clean metrics.
Start with rough proxies like rework percentage, time in review, or number of escalations per sprint. A proxy that's imperfect but tracked is better than not having any metrics at all. You can iterate and improve on it as you learn more.
Final thought
If your AI investments aren't moving the needle, don't add more tools. Draw the chain. Find the gap. Then deploy with precision.
Goal → drivers → indicators. That's the framework. Everything else is just noise.
Next steps
AI guide for enterprise leaders: Building the right approach
Download our guide for enterprise leaders to learn how to prepare your C-suite, executive leadership, and development teams for what AI can do today — and will do in the near future — to accelerate software development.
Frequently asked questions
Key takeaways
- High AI usage rates, completed pilots, and positive team sentiment all measure activity. They don't prove AI is driving business outcomes.
- To connect AI investments to results, work backward from the business goal you want, to the drivers that move it, to the indicators that show progress.
- Before deploying any AI tool, set a numerical goal, target the actual bottleneck, and baseline your indicators. Review on a cadence and pivot if nothing moves.