We Put Two AI Coding Assistants Through a Government Risk Assessment. The Evidence Surprised Me.

Most AI coding assistant evaluations end the same way: a developer plays with both for a week and says “I liked this one.”

That is a perfectly good way to pick a text editor. It is not a way to decide whether OFFICIAL-classified source code gets sent through a third-party model on behalf of a government agency.

Over the past few weeks I ran a governed proof of concept for a public-sector client, comparing two agentic coding assistants head to head: OpenAI’s Codex CLI running against Azure OpenAI, and Anthropic’s Claude Code running against Claude models hosted in Microsoft Foundry.

The deliverable was never an application. It was infrastructure as code, governance configuration, telemetry, a measurement harness, and an evidence pack that a risk owner could actually sit down and read.

Here’s what we set out to answer, what we found, and what I now think matters when an organisation like this asks the question underneath all the other questions: can our developers use one of these?

The Ground Rules (or: How Not to Run a Bake-Off)

Strip the brief back and there was really only one question: can developers use an AI coding assistant through the agency’s own Azure subscription without surrendering control of identity, model choice, telemetry, cost or evidence?

That framing rules out the usual approach of installing both tools and forming an opinion. So before touching anything, we wrote down the rules:

• Plan before build. Discovery was read-only. Nothing got deployed until a written plan was approved.
• Everything as code. Bicep only. Every change previewed with a what-if diff before it was applied. No portal clicking, no imperative CLI.
• Never touch a credential. The agent doing the build used my signed-in identity and never saw a key. No personal access tokens, no service principal secrets. None.
• Measure, don’t judge. Token and cost figures had to come from Azure Monitor and Log Analytics, never from the tool’s own transcript. The verdict belongs to the agency, not to the consultant.
• Say plainly which controls are enforced and which are advisory. This one looked like a formality. It turned out to be the most valuable rule of the lot.

Each tool got its own fully isolated resource group: its own Log Analytics workspace, Key Vault, managed identity, budget and alerts. Azure Policy was assigned to deny any model deployment with auto-upgrade enabled and any untagged resource.

Then five representative tasks from the agency’s real work (a Fabric pipeline change, a Bicep module, a Python transform with tests, a bug fix and a documentation task) ran through four model arms. Twenty scored runs in total.

A footnote on method, because I know some of you will enjoy this: Claude Code was the agent that built most of the harness, under supervision, using my cached Azure session. I ran the login; the agent ran the what-if. “Your identity, my hands.” It worked well enough that I would use the pattern again tomorrow.

What We Found

1. The rate limiter decided the outcome. Not the model.

This is the one I did not see coming.

Both platforms enforce tokens-per-minute limits, but they count differently. Azure OpenAI applies its limit to the whole prompt it estimates at admission time, and cache hits still count. Codex CLI resends its entire conversation on every tool round trip. Put those two facts together and a single developer on a 250K TPM deployment got throttled in an ordinary interactive session. With 96 percent cache hits.

Claude in Foundry applies its limit only to uncached input tokens. The heaviest scored Claude run pushed just under half a million cache-read tokens through a 40K TPM deployment, six times smaller than the Codex one, and never saw a throttle.

Scale that to five concurrent developers in a large monorepo and the Codex path needs a custom capacity allocation above Microsoft’s highest published tier. The Claude path is a published default for the right subscription type.

Here’s the part to sit with: a bigger quota request cannot change how the limiter counts. That is a structural difference, and it outweighed everything else in the final recommendation.

2. Quality was a tie on everything a test could check.

All four arms passed every deterministic acceptance check on the first four tasks. Clean sweep.

The only separator was the documentation task, the one that needed a human reviewer. One model described a behaviour the code did not have. The other three documented the code faithfully, and both Claude arms found a genuine latent defect that nobody had planted.

If your evaluation only scores tasks with automated checks, you will never see this.

3. Cost figures need their caveats stapled on.

Per scored task, Claude cost two to four times what Codex did. Sounds decisive. It isn’t.

Claude Opus 5’s rate card is lower than the top Codex model’s on every line. The gap was token volume: cache writes were 38 percent of the Opus total and are not even captured on the Azure OpenAI side, output volume was higher, and background work ran at Opus rates because no cheaper model was deployed.

Comparing raw “input tokens” across the two platforms is also wrong, because Anthropic’s metric excludes cached tokens and OpenAI’s includes them. We documented every one of these before quoting a dollar figure. I would treat any cross-vendor cost comparison that does not do the same with real suspicion.

4. Most governance is advisory unless you do the boring part.

The server side held up. Azure Policy denials: real and proven. Entra-only authentication with keys refused: real and proven. Metadata-only telemetry: proven across every captured record. Session, tool and token metrics only, never a prompt, a file or a response.

The client side was a different story. Claude Code’s managed settings file only binds when endpoint management puts it on a non-admin device. Codex’s config file is user-writable, so it is guidance. Repository instruction files are instructions, not controls. Azure budgets alert and never block consumption.

We wrote all of that down in a controls matrix with an enforcement level against each row. The risk owner told us it was the most useful document in the pack. Not the architecture, not the benchmarks. The table that said which rules were actually rules.

5. The monitoring found its own gap.

Mid bake-off, 1.95 million billed tokens went through one deployment in twenty minutes and the token-spike alert stayed silent.

The metric it watched excluded cache reads, which are roughly 90 percent of agentic volume. We only caught it because we checked fired-alert history against the run log. Same-day fix, same-day evidence file.

Monitoring you have not tested against real traffic is a hypothesis.

6. Residency is a decision, not a setting.

Neither tool keeps inference in Australia. Neither trains on your code. Everything between those two sentences is where the real decision lives, and it got its own chapter in the evidence pack. It gets its own section below.

7. The brief was wrong in places. That is normal.

The brief said Codex CLI had no native OpenTelemetry export. It does. It named model versions that did not match what the region actually offered.

A discovery phase exists to find these things and table them, not to work around them quietly and hope nobody asks.

Data Sovereignty and Training Use, Side by Side

“Is it in Azure?” turns out to be the wrong question. It is the question everyone asks first, and answering “yes” tells you almost nothing.

The useful questions are five: where does the data sit, where does inference run, who is the processor of the prompt content, what leaves the platform, and can you switch off whatever retention remains? Here is how the two Azure routes answered them, from the vendors’ own published statements as read in early September 2026. Not from a sales deck. From the pages.

Codex CLI on Azure OpenAI (Data Zone Standard, Australia East)

• At rest: Australia East, including Microsoft’s abuse-monitoring store.
• Inference: the Asia-Pacific data zone. Data Zone is the most restrictive SKU these models offer, and it means “somewhere in APAC”, not “Australia”. A Global deployment would be worse.
• Processor: Microsoft. Prompt content never leaves Azure and, in Microsoft’s words, “Models sold by Azure do NOT interact with any services operated by” OpenAI.
• What remains: Microsoft’s abuse monitoring may store flagged prompts and completions for human review by Microsoft staff, in the resource geography. An approved modified-abuse-monitoring exemption switches that store and review off. It is a Limited Access application, and we did not apply for it in the PoC.
• Training: Microsoft states that prompts, completions and embeddings are not available to OpenAI, not used by model providers to improve their models, and not used to train any foundation model without your instruction. That is the default, not an opt-out.

Claude Code on Microsoft Foundry (Hosted on Azure, Global Standard, East US 2)

• At rest: East US 2. There is no Australian or Asia-Pacific region or data zone for Claude in Foundry. None.
• Inference: Azure infrastructure, but Global Standard does not pin the geography beyond “inside Azure”. A US Data Zone SKU exists that would pin it to the United States. Our quota request for it was auto-denied. Either way, never Australia.
• Processor: Anthropic, as an independent processor for Microsoft. Anthropic’s data terms and Data Processing Addendum govern the prompt content, not Microsoft’s. This is the line most people miss.
• What leaves Azure: two things. Usage metadata, and content that Anthropic’s automated safety systems flag, which goes to Anthropic Trust and Safety for exceptions-only review and can be retained for up to two years. Anthropic’s pages do not say where that flagged content is stored.
• Training: Anthropic states that conversation content is not retained by default and that retained data is never used for model training without express permission. Opus 5 and Sonnet 5 are not Anthropic “Covered Models”, so their mandatory 30-day retention does not apply. Zero data retention is contractual, and whether it can be granted on a Hosted on Azure deployment was still unconfirmed when the PoC closed.
• The client: Claude Code in Foundry mode sends no telemetry, error reports or feedback to Anthropic by default. The only telemetry in our setup went to the agency’s own Log Analytics, content redacted.

Direct with the vendor, if Azure billing is not a hard constraint

The proposal also costed going direct, because someone always asks.

Anthropic’s API lets you choose US or global inference per request, offers zero data retention per organisation, and has the same no-training default. OpenAI now offers Australian data at rest for eligible API and Enterprise customers, with inference in the United States, and its API and enterprise plans do not train on customer data by default.

Both routes make the vendor the sole processor of your source code by design. Neither was tested in the PoC. Those are published positions, not evidence, and the pack labels them that way.

What was true on both Azure routes

• Nothing at the platform layer trains on your data. Both vendors say so in writing and neither needs an opt-out ticket. If someone tells you otherwise, ask them for the page.
• The agency’s own logs were metadata only. Ninety days, in the agency’s tenant. Session, tool and token counts. No prompts, files or responses. Proven across every captured record.
• The workstation is the store everyone forgets. Codex keeps a local history file until it is deleted. Claude Code keeps plaintext transcripts for 30 days by default, adjustable through managed settings. If “not retained” matters to you, a managed-device policy has to say so, because the default does not.

The honest one-line summary we gave the risk owner: Codex on Azure OpenAI keeps everything with Microsoft, at rest in Australia, inference in APAC. Claude on Foundry keeps prompts in Azure in the United States, with Anthropic as processor and a narrow safety-flagged exception that can leave the platform. Neither trains on your code. Neither runs inference in Australia.

Which of those is acceptable for OFFICIAL work is the decision. It is not a technical one, and no amount of engineering changes it.

The Classification Problem Nobody Draws on the Architecture Diagram

Here is the question that comes up in every one of these conversations, usually from the security team, usually about ten minutes in: what exactly goes down the wire?

The honest answer is: not the file you are editing. The agent sends whatever it read to get the job done. The config it opened to understand the connection. The .env it found next to the script. The stack trace with the connection string in it. The parameter file with the tenant ID, the subscription ID and three workspace GUIDs. The test fixture someone built from a production extract in 2021 and never replaced.

So the classification of the transmission is not the classification of the repository. It is the classification of the most sensitive token in the context window. A repo can be OFFICIAL on the label and carry material inside it that would never be approved for a third-party model on its own. The model does not know the difference. It sees tokens.

That reframes the guardrail question. It is not “is this code allowed to go to the model?” It is “what is in the context, and what stops the wrong thing getting in?”

Same rule as the rest of the pack: enforced versus advisory, and be honest about which is which.

Enforced, because you control the path:

• Have nothing to leak. Managed identity and Entra-only authentication with keys disabled means there is no key in any file for an agent to find. This is the strongest control on the list and it costs nothing at the agent layer, because it was done at the platform layer.
• Parameterise every identifier. Tenant IDs, subscription IDs, workspace and capacity GUIDs are not secrets, but together they are a map of your estate. They belong in parameter files and environment resolved at deploy time, never as literals in source. The agent should only ever see the placeholder.
• Put a gateway you own between the CLI and the model. Both tools take a custom endpoint, which is how they were pointed at Azure OpenAI and Foundry in the first place. That endpoint can be a front door that pattern-matches the outbound request for connection strings, key formats, JWTs, GUIDs and high-entropy strings, and redacts or refuses before anything leaves your network. Log the count of redactions, never the content, and the telemetry stays metadata-only.

Advisory, and say so:

• Agent-side deny rules. Claude Code can be told never to read .env, *.pem or a secrets/ path. That is a real control on a managed non-admin device and a suggestion everywhere else, for exactly the reason in finding four. .gitignore is not a control at all; an ignored file is still a readable file.
• Commit-time scanning. Push protection and pre-commit secret scanners are excellent at keeping secrets out of history. They run after the agent has already read the working tree. Useful, necessary, and not the layer that protects the wire.
• The provider’s data handling terms. Zero retention and no training are contractual promises. Worth having, worth reading, and not something your Azure Policy can prove.

Two last points on this, because they catch people out.

Agents read terminal output. A failed connection prints the string it failed with. If your CI masks secrets in build logs, your developer shell needs the same treatment when an agent is watching it.

And metadata-only telemetry, which we proved and were proud of, cuts both ways. You cannot audit prompt content after the fact, because you deliberately did not keep it. Prevention has to sit upstream of the wire, because there is nothing downstream to catch it with.

The Verdict, and How It Got There

The final recommendation was not the tool I expected to recommend when I started, and it was not the harness the client’s operator preferred.

It was the route where the capacity story was published rather than negotiated, and where every control the proof of concept had proven carried over unchanged. The evidence pack says so in those words, with the trade-off stated and the stop rule attached.

Three things I would tell anyone about to run one of these:

• Test the limiter before you test the model. Run a real agentic session, not a benchmark prompt, and watch what the platform counts.
• Write down what is enforced and what is merely configured. Then test the enforced ones with a second identity. “Demonstrated on the consultant’s laptop” is not “proven denied.”
• Keep the verdict out of the measurement. The apparatus is yours to build. The decision about what is acceptable for OFFICIAL work belongs to the people accountable for it.

The tools will change by next quarter. The method will not.

I lead data and platform engagements at DW Data. If you are working through a similar assessment, I am always happy to compare notes.