A platform you can't see is a platform you can't run — and a platform whose costs you can't attribute is one that finance will eventually shut down. This part turns the telemetry the gateway already emits into three things every enterprise AI platform needs: visibility into what's happening, alerts before things break, and a chargeback model that makes shared spend fair.
The problem: invisible usage, unaccountable spend
In the sprawl world, every team's AI spend lands in a different subscription under a different cost center with no token-level detail. Nobody can answer the three questions leadership actually asks: How much are we spending on AI? Which initiative is driving it? Is any of it failing or being abused? Worse, when a spoke app degrades, there's no trace that connects "the user saw an error" to "Azure OpenAI returned a 429 in region X." The hub fixes both by funneling every call through one gateway that emits structured telemetry.
The telemetry backbone
We deployed the pieces back in Part 3: a single Log Analytics workspace and an Application Insights resource, shared by the gateway and every spoke. Because all traffic flows through APIM, that one integration point captures the entire platform's request stream — no per-team instrumentation project required.
Per-spoke usage you can actually see
Remember the azure-openai-emit-token-metric policy from Part 4, dimensioned by spoke? That one line is what makes this view possible. Prompt and completion tokens land in Application Insights tagged with the spoke that spent them, so "who is using the platform, and how much" becomes a chart instead of a guess:
Tokens consumed this month, by spoke — the raw material for both capacity planning and chargeback.
The query behind a chart like this is a few lines of KQL against the metrics the gateway emitted:
customMetrics
| where name == "Total Tokens"
| where timestamp > ago(30d)
| extend spoke = tostring(customDimensions["spoke"])
| summarize TotalTokens = sum(valueSum) by spoke, bin(timestamp, 1d)
| render timechartDistributed tracing: from symptom to root cause
When a spoke reports a failure, you need to follow one request across every hop. Because the platform shares an Application Insights resource and APIM propagates correlation IDs, a single trace links the spoke call → the gateway policy decisions → the Azure OpenAI backend response. The "user saw an error" event and the "backend returned 429 in East US" event sit on the same timeline.
Dashboards and workbooks
On top of the shared data you build Azure Monitor workbooks that the whole organization shares: a platform-health view (latency, error rate, cache-hit ratio, throttling) and a cost view (tokens and estimated spend by spoke and model). One source of truth replaces a dozen team-built spreadsheets.
Proactive alerts
Visibility is reactive; alerts are proactive. A handful of rules catch the failure modes this architecture is prone to before users feel them:
- Quota pressure — a spoke approaching its token-limit ceiling (time to raise its limit or coach the team).
- Error-rate spike — 429/5xx responses crossing a threshold (a region under stress or a misbehaving spoke).
- Latency regression — P95 response time climbing (often a sign the cache-hit ratio dropped).
- Cost anomaly — daily token spend for a spoke jumping above its trend (a runaway loop or an abuse pattern).
FinOps: from shared bill to fair chargeback
This is where observability pays for itself politically. Because every token is attributed to a spoke, the shared platform bill can be split back to the teams that drove it — real showback or chargeback. That single capability changes the conversation: instead of "the AI platform is expensive," leadership sees "the Customer Bot delivers X at Y cost per conversation." Cost becomes a per-initiative metric tied to value, and teams gain a built-in incentive to use caching and right-size their prompts.
What this layer bought us
| Capability | Problem it solved |
|---|---|
| Per-spoke token metrics | Invisible, un-attributable usage |
| Distributed tracing | Slow, blind root-cause analysis |
| Shared dashboards/workbooks | Fragmented, inconsistent views |
| Proactive alerts | Learning about issues from users |
| Chargeback / showback | Unfair, unaccountable shared spend |
References
- Azure Monitor overview
- Emit token metrics from API Management
- Azure Monitor workbooks
- Log Analytics & KQL
- FinOps on Azure
What's next
We've designed every layer of the platform — network, services, gateway, identity, and observability. The final question is how to build it repeatably and run it as a product. In Part 7 we bring it home with Infrastructure as Code, a dev→staging→prod CI/CD pipeline, and the operating model that decides who owns what — the difference between a clever architecture and a platform teams actually adopt.