Is This Worth Building?

Overview

The Question Nobody Wants to Ask

There is a certain kind of product meeting that happens dozens of times each week inside technology companies. Someone — a VP, a founder, a well-meaning senior engineer — says some version of the following: "We should add AI to this." Maybe they saw a competitor demo. Maybe they read a press release. Maybe they just got back from a conference. The idea sounds exciting, it gets into the roadmap, and three months later a team is deep in the weeds of a feature that was never properly stress-tested at the idea level.

The most valuable skill a product manager can develop right now is the ability to stop that meeting — respectfully, rigorously, with data — and ask the question that saves companies millions of dollars and months of engineering time: Does this problem actually need AI?

This is not skepticism for its own sake. AI-powered features can create genuine, defensible, transformative value for users and organizations. But AI is also expensive to build, hard to maintain, difficult to test, and prone to failure modes that traditional software never encounters. Every hour you spend validating whether AI is the right tool is an hour that protects your team from building the wrong thing.

This chapter gives you the framework to answer that question before you commit.

Two Very Different Things AI Can Do

The first conceptual mistake PMs make when evaluating AI features is treating "AI" as a single category of capability. It is not. For purposes of product decision-making, there are two fundamentally different things an AI-powered feature can do, and they require completely different kinds of justification.

Automation uses AI to do something a human already does, faster, cheaper, or at greater scale. The task is understood, the output is defined, and the value comes from removing cost or friction. An AI that reads invoices and extracts line items is automating a known task. An AI that categorizes support tickets is automating a known task. The signal that you are looking at an automation use case: you could describe a clear rubric for what "correct" looks like.

Transformation uses AI to do something that wasn't possible before — or that would only have been possible for the most resourceful, best-staffed organizations. An AI that synthesizes a year of customer feedback into a prioritized list of product themes isn't just automating reading — it's enabling a qualitative capability that most teams simply didn't have. An AI that predicts which accounts are likely to churn next month before any human signals appear is surfacing information that didn't previously exist in actionable form.

The distinction matters because automation use cases live and die on economics and accuracy. Transformation use cases live and die on whether users trust and act on what the AI surfaces. Both are worth building under the right conditions — but the way you validate them, the metrics you track, and the risks you manage are completely different.

This is the first principle of Signal Capture as applied to product management: before you can capture the right signal from a market or user base, you need to know what kind of value you are actually trying to generate.

The "Unlimited Humans" Test

One of the most clarifying questions you can ask about an AI feature idea is this:

If you had unlimited, infinitely patient, highly skilled humans available at zero cost, could they do this job instead?

If the answer is yes — and the primary motivation for AI is cost or speed — you are looking at an automation use case. The AI is replacing human labor. That's a legitimate and often valuable thing to build, but it means your success criteria, your risk framework, and your ROI calculation all need to be anchored to that economic logic. You need to know the cost per unit of work today, the cost per unit with AI, and the accuracy threshold below which the automation creates more problems than it solves.

If the answer is no — if the task requires synthesizing more information than any human team could process, making connections across datasets that no individual could hold in their head, or operating at a speed that makes human intervention impossible — then you are looking at a potential transformation use case. The value isn't labor substitution; it's capability creation.

If the answer is "sort of, but it would take too long" — pay close attention. This is the gray zone where many of the most valuable AI features live. A human could review every customer session recording to identify friction patterns, but you have ten thousand sessions per week. A human could read every competitor's documentation to spot positioning gaps, but there are forty competitors updating weekly. The AI doesn't do something impossible; it does something possible-but-impractical and makes it operationally viable. This is automation at scale, and it requires a different framing than pure transformation.

The Signal Capture Test: 5 Questions

Before any AI feature idea advances from conversation to discovery, run it through these five questions. They constitute what the LegacyForward framework calls the Signal Capture test, because you cannot build a useful product if you have not correctly identified the signal you are trying to capture.

Question 1: What is the specific user pain, and how does the user experience it today?

Vague answers to this question are disqualifying. "Users find it hard to get insights" is not a specific pain. "Users spend 4 hours each week manually pulling data from three systems to build a report they use for exactly one decision" is a specific pain. The more precisely you can describe the current-state experience — including how long it takes, how often it fails, what workarounds people use, and what they do wrong — the more clearly you can evaluate whether AI changes that experience or merely adds complexity to it.

Question 2: What does "the AI worked" look like in concrete terms?

If you cannot describe success in observable, measurable terms before you build, you will not be able to evaluate it after you build. "The recommendation is useful" is not a success criterion. "The user takes an action based on the recommendation within 24 hours, and that action correlates with improved outcome X" is a success criterion. Precision here is your friend. Vagueness is a warning sign.

Question 3: What's the cost of being wrong?

Every AI-powered feature has failure modes. The question isn't whether the AI will sometimes be wrong — it will — but what happens when it is. If an AI recommendation leads a user to make an irreversible decision that harms them, that's a catastrophic failure mode. If an AI classification is wrong 8% of the time but users can easily correct it, that's a manageable failure mode. The acceptable error rate for an AI feature is not a technical parameter; it's a product decision that should be driven by this cost analysis.

Question 4: What data does this require, and do you have it?

AI features are not built from logic. They are built from data. This question has two parts that PMs often conflate. First: what kind of data does the AI need to do its job? Second: does that data exist, is it accessible, is it of sufficient quality, and is your organization permitted to use it? Many promising AI feature ideas die here, not because the concept is wrong, but because the data foundation does not exist or cannot be assembled at realistic cost and timeline.

Question 5: What's the simplest non-AI solution, and why isn't it good enough?

This is the question most PMs skip because it feels like it undermines the excitement of the AI idea. It doesn't. Forcing yourself to articulate the best possible non-AI approach — a rule engine, a simple algorithm, a better UI, a workflow change — and then clearly explaining why it falls short of the user need is the most rigorous justification you can provide for choosing AI. If you can't articulate why the non-AI solution is insufficient, you haven't yet made the case for AI.

Decision Table: AI vs. Traditional Software by Feature Type

The following table is a practical tool for early-stage feature evaluation. It maps common feature types to the most appropriate technical approach, with signal indicators for when AI genuinely adds value versus when it adds complexity without proportionate benefit.

The key column is the last one. Warning signs don't mean "never build this" — they mean "pause and make sure you've honestly stress-tested the need before committing."

Transformation vs. Automation: The Signal Capture Frame

The Signal Capture principle from the LegacyForward framework is, at its core, about identifying which signals from your market and users actually matter, and then building your product architecture around those signals rather than around what is technically interesting. Applied to the build-or-not decision, it asks: are you chasing a genuine signal from users, or are you chasing excitement about the technology itself?

Here is how this plays out in practice. A genuine signal looks like this: users are consistently taking a long and painful workaround to accomplish something, and when you describe an AI-powered version that removes that workaround, their eyes light up. Or: a high-value user segment is not using your product for a category of work they're currently doing elsewhere, and the reason is a capability gap that AI can plausibly fill.

A false signal looks like this: no user has ever complained about the problem you are solving. Or the problem exists, but users are satisfied with their current solution and do not feel the pain acutely. Or the feature sounds impressive in a demo but does not integrate naturally into the workflow where the decision actually gets made.

The Signal Capture test is not passed by validating that the AI can do the thing. It is passed by validating that the thing, done well, will change user behavior in a way that creates measurable business value.

The Anti-Pattern: A Solution Looking for a Problem

Every experienced PM has seen this pattern: a team falls in love with a technology and then works backward to find a problem it can solve. In the AI era, this anti-pattern is especially dangerous because the technology is genuinely impressive, demos beautifully, and generates real organizational excitement.

The tells are subtle but consistent. Teams in this pattern spend more time describing what the AI does than what the user problem is. They measure success by model metrics (accuracy, latency, benchmark scores) rather than user outcomes. They frame competitive risk as "we will fall behind if we do not have AI" rather than "users will switch to a competitor that solves X better." And when you ask "what does a user do differently because this feature exists?" the answer is vague.

The solution-looking-for-a-problem pattern is not just wasteful. It is actively harmful. It consumes engineering capacity that could go to genuine user needs, creates technical debt in the form of features nobody uses, and trains the organization to evaluate ideas by how impressive they sound rather than by what problem they solve.

The antidote is disciplined application of the five Signal Capture questions above, consistently, before any feature gets resources.

Putting It Together: A Pre-Discovery Checklist

Before an AI feature idea moves from your backlog into active discovery work, it should clear this checklist. Each "no" is a blocker that needs to be resolved, either by doing more research, reframing the feature, or removing it from consideration.

A feature that clears all seven checkpoints isn't guaranteed to succeed — but it has earned the right to move into discovery. A feature that can't clear the first two checkpoints hasn't been defined well enough to evaluate.

The Organizational Pressure Question

One final reality that PMs need to navigate honestly: sometimes the pressure to build an AI feature comes not from user demand but from organizational politics. A new executive wants AI in the roadmap. A board presentation needs to show AI investment. A competitor launched something and the company wants a response. These are real pressures, and pretending they don't exist isn't useful.

The right response to organizational pressure is not to abandon the Signal Capture test — it's to apply it transparently and share the results. If the test reveals that a genuine user need exists and AI is the right solution, you have a clear path forward. If the test reveals that the feature is primarily driven by competitive anxiety rather than user need, that's important information that leadership needs to hear, framed constructively: "Here's what we'd need to be true for this feature to create real user value, and here's what we need to validate before we commit."

The most valuable thing a PM can do in an AI-first product environment is be the person who asks the right questions early — before the costs are committed, before the team is locked in, and before the feature becomes someone's organizational identity. That's not being a skeptic. That's being the professional who ensures that when your team builds something, it's worth building.

Summary

The first question for any AI feature is not "can we build this?" It is "should we?" Answering that question requires understanding whether you are pursuing automation or transformation, applying the five Signal Capture questions, honestly comparing AI to the best non-AI alternative, and using the decision table to match feature types to the right technical approach.

The Unlimited Humans test cuts through technical excitement to the underlying value question. The pre-discovery checklist ensures you've done the homework before committing resources. And the awareness of the solution-looking-for-a-problem anti-pattern keeps you grounded when organizational pressure pushes toward AI for its own sake.

Every chapter that follows assumes you have cleared this gate. Not because perfect certainty is achievable at the idea stage, it never is, but because disciplined early filtering is what separates product teams that build the right things from product teams that build impressive demos that users never adopt.