An AI product is not a traditional software product with a model bolted on. It is a system where output quality is probabilistic, not deterministic. Its failure modes differ in kind from anything conventional software teaches you.
Technical leaders who apply traditional design mental models to AI products make the right structural decisions for the wrong system. AI product architecture — from data to serving to evaluation — follows different principles. Those principles start with knowing what makes AI systems fundamentally different.
The four-dimensional tradeoff space
Every AI product design choice sits at a point in a four-dimensional space: quality, latency, cost, and controllability. No architecture optimises all four at once.
This is the tradeoff framework you have seen in every other layer of computing. Time versus space in algorithms. Consistency versus availability in distributed systems. Coupling versus cohesion in code. The dimensions change. The discipline does not. Identify what you gain. Identify what you give up. Define when the exchange is correct.
The five-layer stack
An AI product has five architectural layers. Each layer has distinct engineering requirements.
Data is the foundation. The quality, coverage, and freshness of training data set the ceiling of what the model can do. No serving trick, prompt hack, or fine-tune can compensate for training data that is biased, incomplete, or stale. Data management is an ongoing engineering concern, not a one-time collection exercise.
Model sits on top of the data. Model selection trades quality against latency, cost, and fine-tunability. A large general-purpose model produces higher-quality outputs on diverse tasks. A smaller task-specific model runs faster and cheaper but fails outside its training distribution.
Serving delivers model outputs to users with acceptable latency and availability. For most user-facing products, 500 milliseconds is the upper bound for synchronous interactions. This constraint is severe. It excludes the largest and highest-quality models from synchronous deployment unless aggressive optimisation is applied.
Application is where product logic lives. Prompt construction, output parsing, context management, and fallback handling all sit here. Changes to this layer can transform the user experience without changing the model.
Evaluation is the layer most often missing in early AI products and most consequential in mature ones. Without evaluation infrastructure, the team cannot measure whether the product works. They cannot detect when it starts working worse. They cannot make principled decisions about model upgrades or prompt changes.
Choose the product pattern before you design the stack
Four AI product patterns have different architectural requirements. Pick one before you build anything.
An assistant responds to explicit user requests. ChatGPT and GitHub Copilot autocomplete are assistants. Assistants require session state for multi-turn context and fast synchronous inference.
An agent completes multi-step tasks autonomously. The user specifies a goal. The agent determines steps, executes them, and returns the result. Agents require durable task state across model calls, tool orchestration, and retry logic for partial failures. A failed step is not a failed request. It is a failed task that may require rollback.
Augmentation enhances human work without replacing human judgment. Grammarly, AI code review, and AI-assisted diagnosis are augmentation. It requires human-in-the-loop at the right latency — fast enough that the human is not waiting, slow enough for them to respond meaningfully.
Automation replaces a specific human task at scale. Email classification, content moderation, and fraud detection are automation. It requires evaluation infrastructure to catch errors at scale. A moderation system making 10 million decisions per day at 1% error rate makes 100,000 wrong decisions per day. Without evaluation, the team has no way to know this.
The product pattern is the first architectural decision. It determines what infrastructure you must build before you can ship.
The 500ms ceiling and how to beat it
In traditional software, latency is a property of architecture. Improving it requires structural changes. In AI systems, latency is a direct function of model size and inference computation. Larger models produce better outputs but demand more compute — which raises latency and cost.
The practical ceiling for synchronous user-facing inference is about 500 milliseconds of wall-clock time. Above it, users feel the delay. Many high-quality models exceed this threshold at full precision.
The engineering responses are specific. Quantisation reduces numerical precision to speed computation at some quality cost. Speculative decoding predicts likely continuations to cut sequential work. Caching stores outputs of common inputs. Streaming returns tokens as they generate, changing perceived latency without changing actual compute time.
This is AT9 — Speed versus Quality. You cannot avoid the tradeoff. You can only manage it.
Design for probabilistic output
Deterministic systems fail in deterministic ways. The same input always produces the same failure. AI systems fail probabilistically. The same input sometimes produces an excellent output, sometimes an acceptable one, sometimes a wrong one.
Quality assurance in AI requires statistical evaluation across many inputs. Case-by-case testing does not survive contact with production. A unit test that passes on one input tells you nothing about the distribution of inputs the model will see.
Midjourney turned this into product design. Rather than presenting one output and forcing users to retry when it fails, Midjourney presents multiple variations per generation. Users pick the one that matches their intent. The model's variance becomes a feature, not a bug.
The cost model is the architecture
At modest scale, AI inference cost is negligible. At significant scale, inference dominates the operating cost profile.
The arithmetic is unforgiving. A product making 10 million LLM API calls per month, at $0.01 per 1,000 tokens and 500 tokens per call, incurs $50,000 per month in API costs alone. A product making 1 million calls per day at $0.01 per call incurs $300,000 per month. At 10 million calls per day, that becomes $3 million per month.
These numbers do not require unusual scale. One million calls per day is 12 calls per second — achievable by a product with tens of thousands of active users.
Caching reduces the bill. Exact-match caching returns the stored response for identical prompts. Semantic caching indexes responses by embedding similarity, so semantically similar queries hit the cache. Fine-tuning a smaller model can cut per-call cost from $0.01 to $0.001. The break-even calculation is worth doing explicitly before committing to fine-tuning infrastructure.
At scale, the cost model is the architecture. OpenAI's ChatGPT infrastructure is the extreme case — GPU clusters representing billions in investment, with pricing designed to recover that cost while staying competitive.
Context window as product constraint
The context window determines which AI products are possible at a given moment in capability.
At 4K tokens, single-document Q&A and code completion are tractable. At 32K tokens, entire codebase sections and long-form analysis become tractable. At 128K tokens and above, a full codebase, product requirements document, or multi-hour meeting transcript fits in a single call.
Three patterns match three context needs. Retrieval-augmented generation fetches relevant context at query time. Use it when the knowledge base is larger than any window. Fine-tuning bakes knowledge into weights. Use it when retrieval precision is insufficient. Tool use gives the model live data at inference time. Use it when data changes faster than fine-tuning cadence allows.
Choose the human-in-the-loop level explicitly
Do not default AI features to fully automated. Choose the human-in-the-loop level per feature.
Fully automated fits high volume, low stakes, and correctable errors. Human review fits medium stakes and audit requirements. Human approval fits high-stakes or irreversible actions. Human-led with AI assistance fits expert domains where model confidence is low.
The error cost drives the choice. The choice drives the architecture.
Evaluation-driven development
The failure mode that ends most AI products is FM11 — Observability Blindness. Traditional monitoring does not catch quality degradation. The product gets worse. Dashboards stay green.
Define the evaluation framework before the product framework. The minimum viable eval has three components.
A correctness eval measures: given 100 known inputs with known correct outputs, how many model outputs are correct? This requires a labeled dataset. A safety eval measures: given 50 adversarial inputs, how many outputs violate safety requirements? A UX eval measures: given 20 real user tasks, what fraction complete successfully with AI assistance?
The ship rule is simple. Do not ship an AI feature without a correctness eval score. Without a baseline, the team cannot know whether the next change improved or degraded the product.
What goes wrong
AI products without evaluation loops ship blind. The team adds a feature, changes a prompt, or upgrades a model. Quality shifts in ways invisible to the team and visible to users.
Over-reliance on a single LLM API produces fragile systems. When the provider changes model behaviour, updates pricing, or has an outage, the product has no fallback. This is a single point of failure — one most engineers have not yet learned to recognise.
The signal that tells you the loop is missing: a prompt change or model upgrade ships without a prior quality evaluation. When that happens, the team is operating an AI product without a feedback loop. The product is degrading. Nobody knows yet.
The full framework treatment — compression blocks, three-level exercises, and the complete AT/FM mapping — is in Book 7: Leadership and Career, Chapter 12 (AI Product Architecture). Free chapter available at computingseries.com/books/book7.