When NOT to Build an AI Agent (Do This Instead)

Published August 2026.

TL;DR: Most AI agent ideas are wrong tools for the job. Before I write any agent code I check for five disqualifying signals — an unstable process, low frequency, no pass/fail test, a simpler tool that already works, or an irreversible failure mode I can’t gate in time. If any one of them is true, I don’t build. I go down a ladder of cheaper alternatives instead, and only come back to a custom agent if nothing on that ladder holds up.

[Operator’s read] I run 30+ production agents across a consulting brand and Pickleland, a pickleball facility in Pflugerville, TX. I’ve killed at least as many agent ideas as I’ve shipped, and almost none of them died because the idea was bad — they died because an agent was the wrong tool for that particular job. This post is the filter I run before “should I build this” turns into “how do I build this.”

Table of contents

Open Table of contents

• The default answer is no
• Five signals an agent is the wrong tool
• The ladder I climb before building
• The two-week shadow test
• A rule I apply after killing an idea
• FAQ
  • Isn’t this just an argument against automation?
  • What if the task will clearly grow in volume later?
  • How do I know if a no-code tool step is “good enough” versus needing custom code?
  • Does this apply to internal tools differently than customer-facing ones?
  • What’s the most common reason you kill an agent idea?

The default answer is no

The ROI framework I use tells you whether an automation pays back its build and maintenance cost. That’s the right second question. The first question is simpler and gets skipped constantly: does this need to be an agent at all?

“Agent” has become the default label for anything involving an LLM, the same way “app” became the default label for anything involving a screen fifteen years ago. Not everything that touches a model needs a standing, autonomous, tool-calling system watching for triggers and taking actions on its own. A lot of what people call “building an agent” is really “writing a really good prompt and running it by hand,” and that’s not a failure mode — it’s often the correct final state.

I treat “build a custom agent” as the most expensive option on a ladder of options, not the first rung. Before I reach for it, I check whether the task disqualifies itself.

Five signals an agent is the wrong tool

Any one of these, on its own, is usually enough to stop me.

1. The process isn’t stable yet. If the workflow has changed twice in the last month because the business itself is still figuring out what it wants, an agent locks in today’s version of a process that’s about to change again. You’ll rewrite the prompt, the tool schema, and the eval set every time the process shifts — which means you’re maintaining an agent instead of running a business. Run it manually until it holds still for a quarter, then automate the settled version.

2. It runs too rarely to pay back. A task that happens twice a year doesn’t accumulate enough executions to justify build time, testing time, and an eval set, no matter how well it would perform once built. Low-frequency, high-effort-to-build is close to the worst quadrant for automation — you pay the full build cost and collect almost none of the savings.

3. You can’t write a pass/fail test for it. If you can’t describe, in advance, what a correct output looks like well enough to check it programmatically, you can’t build an eval harness for it — and an agent you can’t evaluate is an agent you’re flying blind on. Tasks that are pure taste (“does this sound like me”) or pure judgment with no consistent rubric behind it stay manual, or stay human-reviewed every time, which defeats the point of automating them.

4. A simpler tool already does the job. Before scoping an agent, ask what a spreadsheet formula, a Zapier/Make/n8n workflow with a single LLM step, or a saved prompt would get you. If the honest answer is “90% of the way there,” the last 10% rarely justifies standing up an agent with its own infrastructure, monitoring, and maintenance tax. I’ve scoped agents for tasks a filter view and a recurring calendar reminder would have solved just as well.

5. The failure mode is irreversible and you don’t have time to build the gate properly. Some actions — a mass email send, a refund, a public post — can’t be undone. Human-in-the-loop gates exist for exactly this, but a rushed gate that nobody actually reviews is worse than no automation at all: it creates the appearance of oversight without the substance. If you don’t have the time to build and staff the gate correctly, that’s a signal to slow down, not a reason to skip the gate.

If none of the five apply — the process is stable, it runs often enough, you can define correct, no simpler tool covers it, and the failure mode is either reversible or properly gated — then it’s worth running the ROI math on it.

The ladder I climb before building

When a task fails one of the five checks — or before I’ve even gotten that far — I go down this list in order, and I stop at the first rung that actually solves the problem.

1. Just ask the model directly. No wrapper, no tool calls, no standing infrastructure. Open Claude, paste the context, ask the question, use the answer. This handles more one-off and occasional tasks than people expect, because the “agent” instinct kicks in even for things that only need to happen once.

2. A saved prompt or project instructions. If the same kind of request comes up repeatedly but each instance still needs a human to gather the inputs and review the output, save the prompt as a template — project instructions, a custom instruction set, a snippet — rather than automating the trigger. You get the consistency benefit of an agent without the infrastructure.

3. A no-code automation tool with a single LLM step. For tasks that genuinely need a trigger (a new form submission, a new row in a sheet) but the logic itself is simple, a workflow tool with one model call in the middle is dramatically cheaper to build and maintain than custom code. I reach for this before custom infrastructure whenever the trigger is standard and the volume is low-to-moderate.

4. A template run manually. Some processes benefit more from a checklist than from automation, because the value is in a human thinking through each step, not in speed. Don’t automate away the thinking on tasks where the thinking is the point.

5. Outsourcing. For anything with real ambiguity or judgment where you don’t have time to build and maintain an eval set, a person — a VA, a specialist, a productized service provider — is often faster to get running and easier to correct mid-course than an agent you’re still tuning.

6. Only then: a custom agent. If you’ve gone down the ladder and nothing on it holds — the trigger needs real judgment under load, the volume is too high for manual or outsourced handling, and it clears the ROI math — that’s when a purpose-built agent with its own reliability stack earns its build cost.

The two-week shadow test

For anything sitting on the edge — passes the five checks but I’m still not confident — I run a two-week shadow test before committing to a build. I do the task myself, using the model as a copilot rather than an autonomous system: same prompt I’d eventually give the agent, same inputs, but I read every output before it goes anywhere.

Two things come out of that test. First, whether the model is actually good at the task at the quality bar I need — if I’m rewriting half its output by hand, the task isn’t ready to automate regardless of everything else. Second, a real eval set: two weeks of inputs and the outputs I judged correct are exactly what an eval harness needs, and I’ve usually collected it for free by the time I decide to build.

The shadow test also surfaces edge cases before they’re in production. It’s much cheaper to discover during a manual trial that 15% of inputs need special handling than to discover it from a customer complaint after the agent shipped.

A rule I apply after killing an idea

Killing an agent idea isn’t the same as killing the underlying problem. If a task disqualifies itself for now — the process is still shifting, the volume is too low — I write down why and set a rough re-check point (usually tied to a specific trigger: “re-check once bookings pass 50/week,” not just a date). Agent ideas that get killed once and never revisited quietly turn into permanent manual work that nobody remembers evaluating twice.

The inverse discipline matters just as much: an idea that clears the five checks and the ROI math doesn’t automatically get built today. It goes in the same queue as everything else, ranked against the automations already proven to pay back. Passing the filter earns a task a place in line, not an exemption from prioritization.

FAQ

Isn’t this just an argument against automation?

No — it’s an argument against defaulting to the most expensive form of automation. Most of the alternatives on the ladder above are still automation; they’re just lighter-weight. I run dozens of production agents. The point isn’t to avoid building; it’s to stop skipping straight to “build a custom agent” when a saved prompt or a no-code workflow gets you the same outcome for a fraction of the build and maintenance cost.

What if the task will clearly grow in volume later?

That’s a legitimate reason to build ahead of the current numbers — I cover this exception in the ROI framework. It doesn’t override the five signals above, though. If the process is still unstable or you can’t define a correct output yet, growing volume just means you’ll be maintaining a broken agent at a larger scale. Fix instability and testability first; scale is a reason to build sooner once those are solved, not a reason to skip them.

How do I know if a no-code tool step is “good enough” versus needing custom code?

Try it first and measure it against your eval set, even an informal one. No-code LLM steps handle single-purpose, single-input tasks well. They start to strain once you need multi-step tool use, persistent state across runs, or conditional logic that the tool’s builder can’t express cleanly. If you hit that wall, that’s real signal to move to custom infrastructure — not a reason to start there.

Does this apply to internal tools differently than customer-facing ones?

The five signals apply the same way, but the stakes differ. An internal tool with an unstable process just wastes your own team’s time when it breaks. A customer-facing one with an unstable process erodes trust with people who didn’t sign up to be your eval set. I hold customer-facing automations to a stricter version of signal five in particular — the bar for “properly gated” is higher when a stranger, not a colleague, is on the other end of a mistake.

What’s the most common reason you kill an agent idea?

Signal three — no clean pass/fail test. It’s the easiest one to miss during scoping because the task feels well-defined right up until you try to write down, in advance, what a correct output actually looks like. If I can’t do that in a sentence or two, I know the agent is going to be un-evaluable, which means un-improvable, which means it doesn’t get built yet.