When AI Was Still a Feature
A 2020 AI design framework, revisited
A few years ago when I was working on clinical trials software, our team spent a lot of time thinking about how to design AI into products responsibly. How do you introduce a model to users who don’t trust it yet? What do you do when it’s wrong? When does confidence become a liability?
A lot of product teams were asking the same questions. AI was something you deliberately added to a product — a decision that required justification, documentation, a framework.
I've been revisiting those questions a lot as the ground beneath continues to shift. The same user interface, six years apart, tells most of the story.
What the guide assumed
The framework our team worked from — adapted largely from Google’s PAIR Guidebook and translated into something our teams could apply — opened with a deceptively simple question: will AI add value here?
The guide offered a checklist. AI probably makes sense when you’re making content recommendations, predicting future events, or surfacing things that would be hard to find otherwise. It probably doesn’t make sense when you need a fully predictable interface, when full transparency is required — or when minimizing errors is the top priority.
Pause on that last one... In 2020, especially in clinical trials, “don’t use AI where error minimization matters most” was a reasonable thing to write. It assumed a world where AI was optional enough that you could decide not to use it in high-stakes situations. You evaluated the problem, assessed whether the model’s error rate was acceptable given the consequences, and chose accordingly.
The second organizing question was automate or augment? Automate tasks that are difficult, unpleasant, or dangerous — ones where experts can agree on the correct answer. Augment tasks that carry social value, that people enjoy, or that require judgment and creativity. It’s a helpful distinction, but assumes you’re the one deciding.
The question that subtly disappeared
Somewhere between 2021 and 2023, “should we use AI here?” stopped being a real question for most product teams.
The answer was increasingly being made upstream — by the platform, the vendor contract, the infrastructure already purchased. AI stopped arriving as a feature request. It arrived as a capability already present, and the design work became figuring out what to do with what was already there.
By 2025, McKinsey found that 88% of organizations had adopted AI in at least one business function, up from roughly 50% when frameworks like this one were being written. The deliberateness of the deployment model didn’t altogether disappear, but it did migrate to a layer most designers can’t see. Today it lives in the foundation models embedded in design tools, the AI features bundled into SaaS contracts, the vendor SDKs that arrive with capabilities already switched on. By the time a designer touches the work, the decision has usually already been made.
This matters because almost everything in frameworks like the PAIR Guidebook was organized around that decision point. The checklists, the stakeholder communication guides, the opt-out patterns for users — all of it assumed a team had made a considered choice to introduce AI into a specific experience. When AI exists by default, that assumption folds. It isn’t about whether to use it; it’s what you're accountable for now that it's there.
The durable parts
Four areas of the original framework have aged well. None of them were about process.
The reward function. The guide defined this as the formula an AI model uses to determine right from wrong — and drew a useful distinction between precision (fewer outputs, more reliable) and recall (more outputs, at the cost of including some the user won’t want). The design implications differ in ways that matter: a high-recall recommendation is a different experience from a high-precision one, and neither is inherently correct.
The legibility of the answer is what’s changed. Teams building AI features in 2020 typically had some visibility into what their model was optimizing for. Teams inheriting AI capabilities embedded in third-party infrastructure often don’t. The question of “what is this actually rewarding?” is more urgent than ever, and much harder to answer.
Mental models and co-learning. The guide argued for staged onboarding, realistic expectations, and designing for the relationship between user behavior and model output to evolve over time. It was written against a backdrop of what researchers were calling “algorithm aversion,” a well-documented tendency for users to distrust automated systems even when they outperform human judgment.
What it didn’t anticipate was the inverse. A 2021 study in npj Digital Medicine found that clinicians shown AI diagnostic recommendations showed measurably reduced critical evaluation of results, even in cases where the AI was wrong. The co-learning problem was flipped on its head; designing for appropriate skepticism is now as important as designing for appropriate trust.
Error taxonomy. This section has aged better than almost anything else, and it’s underused in most AI design conversations today. The guide distinguished between several failure categories — user errors, system errors, context errors — but the most interesting was what it called background errors: situations in which the system isn’t working correctly, but neither the user nor the system registers a failure.
Background errors are invisible by definition. They don’t trigger support tickets. They don’t surface in usability testing. They produce outcomes that look like user decisions but are actually model failures — search results that are wrong but not obviously wrong, confidence scores that are high but miscalibrated, recommendations that subtly narrow what a user considers without anyone noticing. When AI is ambient, background errors are everywhere.
Trust calibration. The guide argued that the goal of AI design isn’t maximum user trust — it’s appropriate trust. High enough that users benefit from the outputs, low enough that they apply judgment when needed. It identified specific tools: partial explanations, progressive disclosure, confidence displays, and counterfactuals — explaining why the AI didn’t make a certain prediction, not just why it did.
The framing remains exactly right. What’s shifted is the difficulty of the problem. In 2020, a designer could reason about trust in a specific AI feature. Now users are calibrating trust across dozens of AI-assisted surfaces simultaneously, often without knowing which parts of their experience are AI-generated and which aren’t. Research from Nielsen Norman Group has consistently found that users have poor mental models of where AI begins and ends in complex interfaces — a problem that compounds as AI becomes more embedded. Trust calibration has scaled from a feature-level concern to a systemic one.
There’s a counterargument to consider. Cynthia Rudin, a Duke professor whose work on interpretable machine learning has been widely cited, argues that the quest to explain black box models is itself the wrong framing — that the goal should be building models that are inherently interpretable in the first place, particularly in high-stakes domains like healthcare. It’s a harder ask, but it reframes the design problem entirely.
The questions, still
What is this system actually optimizing for, and does that match what users need? What does the user think it can do? Are they right? How will that change over time? What kinds of wrong is it most likely to be, and what do those failures cost? At what point does trust in this system become a liability?
These weren’t specific to 2020’s version of AI, and they’re not really even specific to AI. They’re foundational questions you ask when you’re designing a system that makes consequential decisions on behalf of people who can’t fully see inside it.
What’s changed is who’s asking, and when. In 2020, these were questions you brought to a decision. You asked them before committing to a direction, as part of evaluating whether to proceed. Now they’re questions you ask after the fact, about systems already running, already shaping behavior, already trusted by users who didn’t opt in.
That’s a different kind of design work. You didn’t author the thing. You received it. And in most cases, so did your users. The forward move isn’t to pretend otherwise. Treat the inheritance seriously. Ask these questions anyway, even without the leverage you would’ve had if you were there at the beginning.
Do what you can with what you can see, stay honest about what you can't, and keep asking what the system is actually doing to and for the people inside it.
On By Default explores how design choices shape behavior and agency. Written by Moira Gill, a product designer focused on health, finance, and the systems that influence how people decide.