The Only Real Test of AI Skill Is What You've Actually Shipped

Everyone is trying to figure out who actually knows how to use AI, and nobody has a good answer yet.

There's been a surge of interest in "AI skills" — prompt engineering courses, multi-agent design patterns, tutorials on how to get Claude to write better code. You could quiz someone on all of it: can they break a problem into subtasks? Do they know how to write a system prompt? Can they chain together a planning agent, a dev agent, and a review agent? Do they know what loop engineering means? These are real things worth knowing. But they're scattered signals. Knowing the vocabulary of agentic AI and actually building something real with it are completely different.

I've started to think the most honest signal is simpler: have they shipped something real with it? And by real, I mean something that breaks in inconvenient ways at inconvenient times.

I've been building live projects with agentic AI — Canadeal (automated deal aggregation across FastAPI, Next.js, MongoDB, Playwright), T&T Price Tracker, Flashfood Deal Finder, GoLink, an AI Resume Builder. These aren't demos. They have users. They break in production and I have to fix them. And through building all of this, one thing has become clear: the problems that actually test your understanding of agentic AI only appear when the project gets big.

Things that only go wrong when the project gets big

The first thing I kept running into was context length. Models have limits on how much they can hold in their working memory at once — Claude's context window is roughly 200,000 tokens, which sounds enormous until you realize a moderately sized FastAPI project can hit that ceiling and still leave out half the codebase. When that happens, the model doesn't flag a warning. It just silently works with what it has.

The failure mode I hit on Canadeal: I asked the agent to fix a bug in the deal aggregation route. It produced a plausible patch. What it didn't have in context was the authentication middleware sitting upstream — a JWT check that was the actual source of the error. The agent fixed the wrong thing, confidently. The bug persisted. I lost an hour before I realized the model had been reasoning about half the call chain.

And then there's hallucination — not the dramatic version where the AI makes up citations, but the quieter kind that actually costs you time. An agent writes a Playwright scraper that calls page.wait_for_selector('.product-price'). Looks reasonable. But the site uses the class name .price-tag, not .product-price. The agent saw .product-price somewhere else in the codebase, pattern-matched to it, used it confidently. The scraper doesn't crash — wait_for_selector times out, the exception gets swallowed by a try/except the agent also wrote, and the function returns None. No stack trace. Just silent data loss downstream. Or: the agent imports from pymongo.collection import BulkWriteOperation, which doesn't exist in PyMongo 4.x — removed in a major version bump. Passes linting. Fails in production.

The one that actually cost me the most time was error accumulation. Agents in loops compound small mistakes. Here's a condensed version of how this played out in a pipeline I built: Step 1 (planner) defines the scraper output with price as a string example — {'price': '$12.99'}. Step 2 (implementation) returns price as a string with the dollar sign, because that's what the planner showed. Step 3 (normalization) was told to expect a float, tries float(item['price']), raises ValueError, gets caught by a bare except, continues with price set to 0.0. Step 4 (deduplication) runs fine on the normalized data. Step 5 (database write) inserts clean-looking records with price: 0.0 for everything. Pipeline exits with status: success. The failure only surfaced when users reported every deal was listed as free. Root cause: a type assumption in step 1 that no downstream agent was equipped to catch.

Treating the AI as part of the system instead of a shortcut

None of this is fatal. But I had to change how I was thinking about the AI's role before things started clicking. Less "give it the problem and see what happens", more "scope it like you'd scope any other component."

Before I hand anything to an agent, I'm thinking about scope. What does this piece need to know? What is it not responsible for? If I'm building a new scraper for Canadeal, I define the output shape before writing a single line — concretely, a Pydantic model:

class DealResult(BaseModel):
    title: str
    price: float
    source_url: HttpUrl
    scraped_at: datetime
    category: Optional[str] = None
    error: Optional[str] = None

That schema is the interface contract. The agent's job is to produce objects that satisfy it. The scraper doesn't write to the database, doesn't deduplicate, doesn't send notifications. It transforms a URL into a DealResult or an error. That's it. When I hand that to an agent, it has a bounded problem with a verifiable output. I can test whether what came back is valid without reading the implementation.

And then incremental verification: after the scraper agent delivers code, I don't immediately hand it to the next agent. I run it against one known URL and assert the output — is price a float, not a string? Is source_url a valid HTTPS URL, not a relative path? Is scraped_at a real datetime object, not an ISO string? Five lines of validation that would have caught the dollar-sign pipeline bug above. Only once that passes does the next agent build on top of it.

Modular design isn't a new idea. But it matters more when you're delegating to an AI, because the agent needs a bounded problem to work well. If the scope is fuzzy, the output will be too. Software engineering best practices don't become less relevant because you're using AI — they become more important, because you're now coordinating between multiple agents that can't read each other's minds.

What to actually look for

If I'm trying to figure out whether someone actually knows what they're doing with agentic AI, I'd want to see what they've shipped. Not their prompts. Not their system design notes. The thing they actually finished.

More specifically: do they have schema files — Pydantic models, TypeScript interfaces, explicit data contracts at the boundaries between components? That's evidence of someone who planned scope before prompting. Is there error handling that isn't just try/except pass — retry logic with backoff, partial failure recovery, structured error objects? Does the commit history show iteration on the integration points, not just the features — commits that say "fix scraper output schema" or "add validation step before DB write"? Is there proof of production: environment configs, deployment files, database migration scripts? Demo repos almost never have these. Real projects do.

A person who has shipped a real project with agentic AI has implicitly solved a coordination problem: they've figured out how to get a powerful but limited tool to contribute meaningfully to something bigger than what that tool can hold in its head. That requires a software engineering mindset. It requires thinking in systems.

None of this is a permanent solution either — the tooling changes, the models change, and half of what I figured out six months ago is probably wrong now. But the underlying shift — treating the AI as a component you design around rather than a tool you prompt at — that part has held up.