Concept Drift vs Data Drift in Coding Agent Evaluation

Coding agents don't perform the same way tomorrow as they do today, even if nobody touches their code. Two separate forces cause this, and each one needs a different fix: data drift and concept drift. Mix them up, and an engineering team ends up fixing the wrong problem while the real one keeps quietly getting worse.

What data drift and concept drift actually mean, and why the difference is not academic

Data drift is when the inputs change shape. The codebase, the task descriptions, the API calls look different than they used to. The relationship between input and correct output hasn't moved; the world just started sending different questions.

Concept drift is the opposite problem. The inputs can stay exactly the same, but what counts as the right answer changes anyway. The mapping from question to correct answer shifted under your feet.

Both show up as the same thing on a dashboard: performance going down, which is why they get lumped together under the term "model drift." A team that builds a single dashboard for both is often behind, because the two tend to need different detection methods and different fixes, and a single number rarely tells you which one you're looking at.

You can catch data drift largely by watching what comes in. No labels needed. Track the shape of the inputs and notice when it moves. Concept drift needs labels; someone has to confirm the output is still right by today's standards, and in coding, getting those labels is slow, because that someone has to actually read the code.

Get the diagnosis wrong and the fix often fails in a specific way. If the real problem is data drift, the answer is exposing the agent to the new inputs and retraining or re-tuning on what's actually showing up now. If the real problem is concept drift, the answer is revising what "correct" even means: updating the rubric, maybe changing the agent's goal entirely. Apply the data-drift fix to a concept-drift problem, and the agent gets better at handling inputs while still being judged by a standard that no longer applies. Apply the concept-drift fix to a data-drift problem, and the rubric changes while the agent is still misaligned with what's actually coming in the door.

A peer-reviewed study found temporal degradation across most of the models it tested. Getting worse over time is a common case, not the exception. That alone makes telling these two failure modes apart a basic requirement, not a nice-to-have.

How data drift shows up specifically in coding agent environments

Coding agents don't take in one tidy stream of data. Their inputs span natural language task descriptions, repo structure, file contents, dependency manifests, tool schemas, and API docs. Any one of those can shift out from under the agent, and each shift looks different.

A few concrete versions worth knowing by name:

• Codebase drift. The repo grows, gets refactored, adopts new patterns. Whatever the agent "knew" about that codebase's structure stops matching what's actually there now.
• Prompt drift. What people ask for changes. New frameworks, new jargon, new edge cases nobody tested for. This shows up as movement in embedding space: the cluster of "typical requests" physically shifts position.
• Tool schema drift. An API the agent calls changes its signature, silently. The tool it depends on now expects something different, and calls start failing.
• Upstream model drift. If an agent calls out to a third-party model API, the provider can update that model without telling anyone. The system that got built and tested is not the system running right now.
• Retrieval drift. In RAG-based agents, the knowledge base, the embeddings, and the search index can each drift on their own schedule. The question stays the same; the context the agent sees to answer it does not.

Much of this gets missed by tools built for old-school tabular data drift. Catching prompt drift means watching embedding space move. Catching tool schema drift means diffing schemas and running integration tests. For most teams running agents today, this isn't a future risk to plan for. It's the water they're already swimming in.

How concept drift operates in coding agent evaluation — and why it is harder to catch

Concept drift in a coding agent looks like this: the gap between "passes its tests" and "is actually good code" tends to widen over time, even though nothing about the test suite changed.

The clearest version shows up across multiple editing turns. Ask an agent to build something, then extend it, then extend it again, and a pattern tends to emerge: verbose constructions pile up, logic gets duplicated, structural shortcuts accumulate. Each new edit builds on top of the last one's mess instead of cleaning it up. Meanwhile, unit test pass rates can hold steady or even climb, because the tests were never checking for maintainability to begin with.

This kind of decay — verbosity and structural erosion accumulating across multi-turn code extension tasks — is precisely what standard single-shot benchmarks are not designed to surface.

Why call this concept drift instead of just "bad code"? Because the label "correct" hasn't moved. What's moved is the meaning underneath that label: readable, maintainable, coherent code, which the evaluation never measured in the first place. The evaluation and the actual goal quietly came apart.

A second version worth naming: a team's own standards shift. New style guide, new testing philosophy, new architectural pattern. What the team means by "correct" changes even though the benchmark sitting on the shelf hasn't been touched. The agent keeps satisfying the old bar while missing the new one, and often nobody notices, because the score still looks fine.

A third version, quieter still: knowledge drift. The model's training reflects a world that has since moved on. An answer that was correct when the model was trained can become wrong later, for reasons that have nothing to do with the inputs it's seeing now.

All three share the same detection problem: someone or something has to judge whether the output meets today's quality bar, and that judgment is slow, costly, and rides on human reasoning that doesn't fit neatly into an automated metric.

Why standard benchmarks struggle to catch either drift type

Most coding agent evaluation still works the same way: run the agent against a fixed set of held-out tasks, check if the final answer is right, report a number. That number tells you one thing: how the agent did on a set of tasks somebody picked in advance.

It tells you little about whether that set still looks like what's happening in production, whether the agent looped before landing on an answer, whether it recovered from a bad tool call, or whether the code it wrote survives three more rounds of editing.

Against data drift, a static benchmark is a snapshot. It captures the input mix at the moment it was built. As real-world tasks drift away from that snapshot, the benchmark score becomes a worse predictor of what's actually happening, because retrievers, judges, and agents can all behave differently under a shift the benchmark was never built to reflect.

Against concept drift, the problem is different but just as structural. The most widely used coding benchmarks score single-shot answers against a fixed spec. They don't touch multi-turn sessions, so they can't see architectural decay building up. And their idea of "good code" doesn't move as the field's idea of good code moves.

A benchmark score that only checks unit test pass rates is often measuring the wrong thing. A benchmark harness can be satisfied by code that would not hold up under real scrutiny, because the harness was only ever checking what it was built to check. The harness gets satisfied; the actual task doesn't. That's the benchmark's own version of concept drift: the measurement and the thing it's supposed to measure have quietly split apart.

There's a second-order version of this too. When agent outputs get scored by another model acting as judge, and that judge model gets updated by its provider, scores can shift even though the agent under evaluation hasn't changed at all. Comparing scores across time without tracking which judge model produced them is comparing apples to a slightly different apple.

Put together: a single benchmark score is a point estimate on a target moving in two directions at once. It can go stale from data drift and stale from concept drift, at the same time, and the number itself gives no hint which one is happening.

Detecting data drift and concept drift differently because they require different signals

Data drift you can often catch without labels. Concept drift, most of the time, you can't. If a team only has one alert for "the agent got worse," that alert is likely too blunt to act on.

For data drift, a few concrete moves:

• Embed incoming task descriptions and track how that embedding cluster moves over time. When the center shifts, the inputs have changed.
• Watch tool call patterns. A change in which tools get called, how often, in what order, is a sign the agent is hitting inputs its evaluation never covered.
• Track API schema versions directly and flag any mismatch between what the agent was tested against and what it's calling right now.
• For RAG agents, version the knowledge base and the embeddings on purpose. Retrieval drift is easy to miss unless someone is tracking versions deliberately.
• Treat provider model updates as a dependency to watch. Run a canary evaluation against every upstream model update before it touches production traffic.

For concept drift, the moves lean harder on human judgment:

• Instrument multi-turn sessions specifically. Track verbosity, structural complexity, and the ratio of tests passed to actual quality across editing cycles, not just at the finish line.
• Put a sample of agent outputs in front of humans on a regular basis, judged against current team standards. This is slow and it costs real hours, but it's one of the few ways to catch drift that no automated metric encodes.
• Pin down which version of the judge model produced every score, and only compare scores generated by the same version.
• Treat a new style guide or a new architectural standard as a drift event in its own right. Update the rubric and re-baseline scores when the criteria change.

Recent research frames this whole space as "Agent Drift," split into four types: Goal Drift, Context Drift, Reasoning Drift, and Collaboration Drift. Each one maps back to either data or concept drift depending on what actually moved, and the taxonomy earns its keep mainly as a decision aid: it tells you which detection signal to reach for once you can name what's shifting.

The practical setup: wire both signal types into one observability pipeline, alongside existing traces and evaluation gates. One alert system, one source of truth, even if the two halves of it run on different clocks.

Designing evaluations that stay valid as both drift types evolve

An evaluation isn't a permanent scorecard. It's a versioned artifact, valid for a specific slice of inputs and a specific definition of correct, and both of those should be written down somewhere, not left implicit in someone's head.

For data drift:

• Build task sets from real production inputs on a rolling basis, not from one benchmark-construction pass that never gets revisited. The held-out set should track the current mix of inputs.
• Break the evaluation set into slices (codebase size, language, task length) so drift in one slice shows up before it gets buried in an averaged score.
• Run integration tests against live tool and API schemas, not versions pinned and frozen in time, or run both and track the gap between them.

For concept drift:

• Keep pass/fail correctness separate from quality scoring. Measure them independently so a test-passing score and an architectural-quality score can visibly diverge, instead of blending into one number that hides the real story.
• Put multi-turn extension tasks in the core suite. Single-shot benchmarks tend to be blind to the kind of decay that builds up over several rounds of editing; only tasks that make an agent extend its own prior work will expose that.
• Schedule regular criteria reviews. At real project milestones, ask whether the rubric still matches what the team actually considers good code. Treat this as routine maintenance, not an audit nobody gets around to.
• When criteria change, rerun the suite under the new criteria before retiring the old scores. That gives a clean cutover point instead of an invisible jump where old and new scores get compared as if they meant the same thing.

A bigger version of this problem is coming as agents take on longer jobs: planning, building, testing, deploying across a whole feature cycle. Evaluation has to cover the whole path the agent took, not just whether it landed on the right answer at the end. Looking at which steps failed and which recoveries worked is one of the few ways to catch drift in reasoning and tool use over a long session.

On the judge-model problem specifically: pin the evaluator version in production evals, run the old and new judge side by side when a provider updates its model, and report which evaluator version produced any given score as a matter of course.

What a drift-aware evaluation strategy looks like in practice for engineering teams

Drift isn't an incident. It's a background condition, running much of the time, and an evaluation strategy needs to be built around that from day one, not bolted on after a score drops.

Two separate playbooks make sense here, one for each type.

For data drift: find which slice of the input mix moved, expand or resample the evaluation set to cover it, update retrieval indexes or tool schema dependencies as needed, and re-run evaluation before routing new traffic patterns to the agent.

For concept drift: pull together a criteria review, update the rubric to reflect current standards, re-baseline old scores under the new criteria, and go back through recent agent output looking for degradation that slipped past the old rubric unnoticed.

A few starting priorities for teams building this out:

• Log every agent session with enough detail to reconstruct what actually happened: tool calls, intermediate outputs, recovery attempts, not just the final result.
• Version everything that can drift: the model, the judge model, the prompt, the tool schema, the retrieval index, the rubric itself.
• Set alerts on task embeddings so a real shift in input distribution triggers a human look before it shows up as a drop in outcome metrics.

For agents working across a full development cycle (writing code, running tests, opening pull requests, responding to CI failures), drift monitoring belongs in the same observability layer as build and deployment signals, not a separate ML monitoring tool bolted on the side.

Concept drift detection at the quality level still comes down largely to human judgment, and that's not a weakness to engineer around. No automated system can fully decide whether a team's evolving sense of "good code" has been met. Any evaluation strategy that assumes otherwise is likely measuring the wrong thing with a lot of precision.