AI Has Largely Solved Software Engineering. It Hasn't Solved Data.

Here is something I did not expect to be saying in 2026. Across recent engagements building out Azure and Microsoft Fabric data platforms, the platform side has been almost boring. Infrastructure as code, CI/CD, workspace provisioning, environment promotion, security baselines: smooth, predictable, well understood. Describe what you want, let the tooling do the typing, and it works the way the documentation says it will. I wrote a while back asking whether CI/CD for Fabric was finally there. It is. That part of the job has become a solved problem, and AI has made it more solved.

The data engineering side has been the opposite, every time. It consumes a disproportionate amount of the time, the thinking, and the senior people. Not because the code was hard to write. The code was easy to write. It was hard to know what the code should be.

That gap is what this article is about, and I want to be upfront: I am a data architect, and the conclusion is going to sound self-serving. I think it is also true.

What AI Actually Solved

Let us be honest about how far things have come. For most software engineering work, the context an AI agent needs is already in the repository. The types are in the code. The tests describe the behaviour. The infrastructure is declared in Bicep or Terraform. The pipeline is YAML. Give a capable agent the repo and a clear intent and it will produce something that compiles, deploys and passes, and it will do so faster than a team of people ever could.

I use this every day. I build my own tools this way. The velocity is real and I have no interest in pretending otherwise.

But notice what made it possible: the full context of the problem was visible to the agent. Everything it needed to know had been written down somewhere it could read.

The Extra Dimension Data Has

Data has a dimension that code does not. The context that matters is mostly not in the repository. It is in people, in history, and in decisions nobody ever recorded.

Some examples from the last decade or so, all lightly disguised, all real:

A status column where the value 9 means “cancelled” — except for rows created before a system migration, where it meant “on hold”. Nobody wrote that down. Two people in the business know it, and one of them is retiring.
A customer key that was reused after an ERP upgrade, so the same identifier points at two different organisations depending on the year.
A “net sales” figure that Finance quietly excludes one store format from, for a reason that made sense in 2017 and has been baked into every report since.
A nightly load that everyone on the team knows to rerun on Monday mornings, because a source system does something odd over the weekend. The rerun has become tribal ritual rather than documented behaviour.
Three tables named almost the same thing, of which one is authoritative, one is a stale copy someone forgot to delete, and one is a workaround built during an outage that a critical report now depends on.

None of that is visible to an agent reading a schema. An agent will write you a beautiful, well-tested PySpark notebook that faithfully reproduces the wrong answer, and it will do it with complete confidence. The problem is not intelligence. The problem is that the information does not exist anywhere the agent can see.

Greenfield Is Engineering. Legacy Is Archaeology.

This is what makes legacy uplifts and extensions so different from greenfield builds. A greenfield platform is mostly a software engineering problem, and AI is superb at those. A legacy migration is an archaeology problem. The value is in recovering the decisions, the exceptions and the human behaviour that shaped the data, then deciding which of those to carry forward and which to finally put down.

Some of that recovery work can be made machine-readable — that is exactly why we care about semantic layers and about field-level lineage, so that meaning and provenance stop living in people’s heads. But someone has to extract that meaning from the organisation first, and that part is stubbornly human. It is the same argument I made in From BI to HI to AI: the human intelligence step is not a stage you graduate out of, it is the input everything downstream depends on.

Why the Fundamentals Matter More Now, Not Less

You might expect AI to make data modelling and ETL discipline less important. I have found the reverse.

When the cost of writing code drops to near zero, the cost of writing the wrong code does not drop at all. It goes up, because you can now produce wrong pipelines at industrial scale. A well-structured model — clear grain, agreed definitions, honest handling of history — is what stops that. It is the contract the agents build against.

The same applies to ETL practice. Idempotent loads, explicit lineage, validation at the boundaries, reconciliation back to source, quarantining bad records rather than silently dropping them. None of this is new. What is new is that these are now the things a human has to insist on, because they encode judgement rather than syntax. An agent will happily skip them if you let it, and everything will still look green.

Good data modelling was always about making decisions explicit. That turns out to be exactly what AI needs from us.

What to Look For When You Hire

If you are bringing in a data architect, a data engineer or a consultant to help with a platform build or a legacy uplift, the obvious skills are table stakes. They should know the platform, they should be able to model, and they should be able to read and write the code even if they are no longer the one typing most of it.

Here is what I think separates the good from the great, based on the people I have watched succeed and the ones I have watched struggle:

They are current. Not “attended a conference” current. Hands-on with the latest patterns and tooling, because the right answer in 2026 is often different from the right answer in 2022, and someone still solving today’s problems with yesterday’s patterns will build you something that is legacy on the day it goes live. Ask them what they have built in the last six months. Ask what changed their mind recently.
They talk to the business, and they listen. The context described above lives with business heads, analysts, and the people who have been running month-end for fifteen years. A great architect finds those people early, asks open questions, and treats the answers as requirements rather than anecdotes.
They create a safe place for the existing team to tell the truth. This one is underrated and it matters enormously. Every team carries skeletons: the shortcut taken under deadline pressure, the table that should have been retired, the decision that turned out badly. Engineers are often reluctant to volunteer these, because the person who admits to the bad decision tends to get ridiculed for it. A great architect makes it clear, early and genuinely, that they are there to factor those things in, not to assign blame. Teams that open up give you the map. Teams that stay quiet let you rediscover every landmine the hard way, at three times the cost.
They think laterally. There are usually five ways to solve any given data problem, and four of them are perfectly defensible. The great ones are not the ones who know all five. They are the ones who can see which path is still the right one in three years, when volumes have doubled, the source system has been replaced and the team that built it has moved on. Choosing the best long-term path over the fastest short-term one is the whole job.
They can lead. Nobody solves a platform migration alone. If an architect can bring the team along, explain why the hard way is the right way, and sell the destination convincingly enough that people want to get there, the project gains a momentum no amount of individual brilliance can replace. Look for someone the team wants to follow, not just someone who is usually right.

The Honest Takeaway

AI has taken most of the typing out of my job, and I am grateful for it. The trade is that what remains is almost entirely the hard part: understanding what the data means, why it looks the way it does, and what the organisation needs it to become.

That is not going to be automated by reading a schema. It requires someone who has been in the trenches, who has made some of those bad decisions themselves and learned from them, and who is still curious enough to be building with the newest tools this week.

If that sounds self-serving coming from a data architect, fair enough. I would simply invite you to look at where the time is actually going on your own platform work right now. I suspect it is going to the same place it has been going on every build I have seen lately.

If that is the gap you are staring at, I would genuinely enjoy talking it through — it is precisely the shape of work we do as forward deployed engineers. Get in touch and tell me what you are trying to ship, or just compare notes if you are seeing something different.