Testing the Unpredictable: How to Test the Interface When AI Behavior Changes Every Time

You open the interface with an AI assistant, you ask the same question three times in a row, you get three different answers. Somewhere the model answered in one sentence, somewhere deployed three paragraphs with a marked list, somewhere added a picture. The Regenerate button sometimes returns almost the same, and sometimes a completely different result, and the user does not understand what happened.

This is the new reality of testing. Previously, a QA engineer could open a Jira-task, repeat steps, see the same bug and close it. Now “playback steps” work once in a while, screenshots are useless after an hour, and “expected result” has to be formulated as a range rather than a specific screen. Classic approaches to testing interfaces with AI are breaking down — and teams are either ignoring it or honestly rebuilding the process.

## Why Older Test Methods Are Not Working

The classic UI test is based on determinism. Clicked here, got that. The same scenario produces the same result, and any deviation is a bug.

It's different with AI. The behavior of the model depends on the history of the dialogue, the context, the version of the model, the temperature, sometimes even on the time of day (if the provider balances the load). The same prompt can return JSON in one case and markdown in another. The Apply button is sometimes visible and sometimes not – because the model has decided that action is not necessary in this context.

This breaks down three habitual pillars:

• ** Screenshots as a source of truth.** Screenshot captures one possible outcome, not "as it should be.".
• ** Playback steps.** The bug can occur once every ten times and is harder to reopen than to fix.
• **Snapshot tests for model responses fall on each release, even if everything works functionally.

### What Changes in the Testing Role

The AI tester is less like an inspector and more like a researcher. His task is not to “make sure that the screen is the same as in the layout”, but “to understand how the system behaves in the range of possible states and where this range is beyond the permissible limits.”.

On the review, this can be seen immediately: instead of "here is a bug, here is a screen" - "here are ten runs, in three cases the model ignored the system prompt, here is the pattern.".

## Scenario instead of exact outcome

The first thing that needs to be restructured is the formulation of the expected result. If you write “assistant returns X,” the test will always be red. If you write “the assistant admits that he does not know the answer, and offers clarification”, the test becomes meaningful.

### How to Formulate “Expected” for AI Features

Good expectation describes behavior, not text:

• Intention of the model - what it should understand from the query.
• ** Response type** - reasoning, action, refusal, clarifying the question.
• ** Boundaries** - what the model should not do (open the system prompt, promise actions that it will not perform, give personal data).
• Form: structured JSON, user text, tool call.

### Anti-Problem Setting Patterns

• “I must answer correctly” – the word “correct” here does not mean anything, any text fits under it.
• "Shouldn't hallucinate" is too general, untested.
• "Must be friendly" - subjectively, different reviewers will give different verdicts.

Instead: “When requesting a non-existent order, the model should not invent a number, should offer to check the mail and switch to the operator if the user insists.”.

## Range testing: runs instead of one click

If the behavior is stochastic, one run proves nothing. Five successful runs in a row also don’t prove that the feature works – they show that in five cases out of five the model fell into the right range.

### Minimum run protocol

• Run the same script at least 5-10 times.
• Record each answer separately, without editing or “leading to the average.”.
• Mark each run with tags: “OK”, “OK, but strange”, “Breaking boundaries”, “actual error”.
• Save seed, model version and system prompt - otherwise in a week the result cannot be repeated.

### Questions for audit runs

• In how many runs out of ten did the model fall into the expected behavior?
• What two or three types of deviations are most common?
• Are there any “silent” ones that look normal but contain a mistake?
• What do bad runs have in common: the length of the story, the wording of the user, a specific trigger word?

In short, testing an AI interface is not about “catching a bug” but “describe the behavior of a system as a distribution.” The sooner the team stops waiting for the sameness model and starts working with the range, the faster clear release criteria appear.

## Workflow: from layout to release

When the behavior changes from run to run, the usual conveyor “model → development → QA → release” ceases to work linearly. At each stage, a feedback loop is needed, in which the designer sees the model’s real answers rather than placeholders.

### Stage 1. State-range layout

In Figma, you can no longer draw the right assistant screen. Each AI block requires at least four layouts:

• Successful outcome. The model understood the request, the answer is placed in the card.
• Long answer. Text is twice the size of the calculation, buttons at the bottom.
• Clarifying question. Model unsure and asking for details.
• Failure or Error. The model cannot respond, the tool has fallen, the limit is exceeded.

If the layout does not describe all four, the interface will crumble on one of them. Most often, refusals are broken: they are drawn last and without the consent of copyright.

### Stage 2. Prototype on a live model

A clickable prototype with pre-recorded answers is useful for presentation, but dangerous for hypothesis testing. He hides the main thing – that the model in reality responds differently than in the layout.

The minimum you should do before transferring to the development: connect the same model that will go into the product, and run five to ten real requests through the raw interface. Often at this stage, it pops up that the chosen card is too narrow, that the markdown breaks the layout, or that the model stubbornly answers in the wrong language.

### Stage 3. Test environment with fixed seed

The developer should be allowed to reproduce at least some of the behavior. For this purpose, it is useful to have:

• Switch "fixed seed" to get the same answer again.
• Log of the last system prompt and call parameters.
• Button "run the script N times" with export results.

Without this, QA and the designer discuss different bugs under the same name.

## Diagnosis: how to deal with the complaint “assistant broke”

The most common wording from the user and from the manager is “he answers somehow wrong”. It's not a bug report, it's a symptom. To turn it into a task, you need to go through layers.

### Layer 1. Input data

• What kind of request did the user send, literally?
• What was the history of dialogue before that?
• What data is pulled out of context (profile, document, RAG)?

Often it turns out that the “model is dumb” is “an outdated document came into context” or “the story no longer fits in the window and has been cut in the middle.”.

### Layer 2. Prompt and parameters

• Has the system system shifted in recent days?
• What version of the model is now and was there a quiet update from the provider?
• Temperature, top-p, limit of tokens – the same as in the layout?

A quiet update model is a separate class of bugs. The interface didn't change, the code didn't change, and the behavior went. This should be checked first, not last.

### Layer 3. Interface around response

• Is the model’s answer really bad – or does the interface show it badly?
• Does a long answer scroll without an indicator?
• Does the user see that the assistant has called the tool, or does it feel like the system is hovering?

In half the cases, “the model responds poorly” is “the model responds normally, but the person doesn’t understand what happened.”.

## Typical mistakes in product work with AI

### Building on a demo script

All demonstrations are collected on three or four gold queries, where the model behaves predictably. If the feature goes to the release only with such verification, on the sale it meets with the real distribution of requests and falls apart.

### Think of it as a “set-up” rather than a part of the product

The system pump rules on the move, without versioning and without regression. After a month, no one remembers why there is a paragraph about “don’t mention competitors” and what will break if it is removed. The prompt should be kept in the repository and changed through the same review as the code.

### Hide model uncertainty

When a model is unsure, designers often hide it behind a cheerful formulation. The user receives an even, confident answer - and takes it for fact. An honest interface shows a degree of confidence in words or forms: “maybe,” “I didn’t find it in the documents,” calling a tool instead of a guess.

### Ignore "silent" mistakes

High-profile mistakes - when the model clearly gave out nonsense - are caught quickly. Quiet - when the answer looks plausible but contains a made-up fact - are skipped by both QA and the designer. A separate category of reviews should be about them: “check not how it looks, but whether it is true.”.

## How to Incorporate This into Your Design

A few habits that change the layout of the assistant:

• Next to each AI block, note which data sources are coming to it and what will happen if they are not.
• For each text answer, put state "too long" and "empty" not as an option, but as a mandatory frame.
• For each action of the model, draw a **waiting state ** longer than it seems necessary. Real answers come in waves, not instantaneously.
• Describe in the comment to the frame ** the boundaries of behavior**: what the assistant does not say that he does not do without confirmation, where he sends, if he failed.

### Questions for the review of the layout of AI-fichi

• What will the user see if the model responds three times longer than expected?
• What happens if the answer comes in 15 seconds instead of one?
• Does the user see that the system is thinking and not hanging?
• Where in the interface is it shown that the assistant does not know something?
• Is there a way to "call the person" when the model fails?

In the product work with AI, the layout ceases to be a picture of one outcome and becomes a description of the behavior of the system in the range. A designer who keeps all the branches in his head - a long answer, a refusal, a quiet error, a delay - saves the team a week of arguing about "how it should really be.".

## Advanced scenarios: where conventional QA no longer works

When the basic states are drawn, interesting things begin. A simple chat assistant has been tested, but in a product AI rarely lives alone. It's built into a funnel, it has memory, tools, sometimes several models under the hood. And it is in this bind that the most expensive bugs hide.

### A model changes opinion in one session

The user asks: “Can I take this tariff?” The assistant says yes. After three messages, he clarifies the data and says no. Formally, both answers are correct — it just has more context. To the user, this looks like a lie.

What to put in the layout:

• Visible anchor: "I changed my answer because I saw this.".
• You can go back to the previous answer and compare it.
• Prohibition of silent contradiction – if the answer inverts, it is an event, not a string in the stream.

### Scenario "the tool worked, but not to the end"

The assistant called the API, got a partial result, finished the missing text. From the outside, everything looks like a successful response. Inside, half of the data is real, half made up.

It is useful to check on the review:

• Is the source of each block (model/tool/document) marked in the UI?
• What does the interface show if the tool returned the error instead of the void?
• Is there a difference between “tool not found” and “tool not called”?

### Scenario of multiple users in one context

Team assistants are a separate pain. One person said, "Take it shorter," and the other person came in and saw the cropped answer, not knowing why. The memory of preferences should be either explicit or personal, but not “general and invisible.”.

## How to check quality when there is no right answer

Classic QA works by comparison with the standard. With AI, there is no standard - there is a range of permissible. It changes the way we check.

### Rating by rubric, not by “right/wrong”

Instead of “the answer is correct,” there are several axes: actual accuracy, completeness, tone, length, safety. Each axle is evaluated separately. This allows you to see that the answer is “factually correct, but rude,” and not consider it a reference just because the numbers agree.

### Regression on a set of cases, not on a single prompt

The team should keep a live set of several dozen real queries: typical, edge, provocative, multilingual, with typos. After any change in the prompt or model – run throughout the set, not “I checked, I work.”.

### Blind version comparison

When the prompt changes, it's easy to convince yourself it's getting better. Anti-Self Deception: Show two options for answering the same question to a person who doesn’t know where. If he does not distinguish or prefers the old, there was no improvement.

### Regression checklist of AI-fichi

• Run the test set before and after the change.
• Blind comparison on at least 10-20 cases.
• Verification of boundary scenarios: an empty request, very long, in another language, with provocation.
• Behavior in case of tool failure and timeout.
• Logs: Do you see after the fact which version of the prompt and model generated the answer.

## How to explain the decision to the team

The most common mistake of a designer in an AI project is to bring a layout and not bring the logic of behavior. The developer thinks, the product thinks, the tester thinks in his own way. There are three different features on the way out.

### Document of behavior near the layout

Not instead of a layout, but in parallel. A short text that describes:

• what the assistant does and does not do;
• which sources are used and in what order;
• how to behave with uncertainty, refusal, timeout;
• which actions require confirmation and which do not.

This document lives in the same place as the layout and is updated with it. If the behavior has changed, and the text is not updated, this is the same bug as a font mismatch with a style guide.

### The language you should speak with your team

• “This is the default behavior” instead of “The model responds this way.”.
• “The product here is supposed to show uncertainty” instead of “let’s write something neat.”.
• “We don’t support this scenario yet, and the interface is honest about it” instead of “we’ll finish it later.”.

### Questions that make sense to ask a review

• If tomorrow the provider updates the model, which of our screens will break first?
• Where in the interface does the user know that the assistant does not know something?
• What behavior is normal and what is an incident, and who decides?
• What do we record in the log to deal with the complaint in a month?

In short, the quality of AI features is not the quality of a single response, but the predictability of behavior across a range. A designer who is able to describe this range in words and test it on a set of cases turns from a “drawing screens” to a person without whom the feature cannot be released.

## Final checklist: is AI-fit ready for release

It's not all or nothing, it's a way of seeing where the holes are. If the answer is “I don’t know” – the release is premature.

### Conduct

• The range of acceptable answers is described, not a single standard.
• Defined what is considered a refusal, and how it looks on the screen.
• It is decided which actions require confirmation and which do not.
• There is a rule for long answers: streaming, progress or honest timeout.
• It's clear how the assistant behaves without the Internet, without the tool, without the context.

### Interface

• The message of uncertainty exists as a separate state, not as a normal gray response.
• Sources are visible where it is important to the user’s decision.
• There is a way to roll back or correct the assistant action if it affects the data.
• The state of “assistant thinking” is different from “assistant hanging.”.
• The empty state explains what can be asked at all rather than showing a blinking cursor.

### Verification

• A living set of cases lies in the repository, not in the head of one person.
• A run on the set is done before the release and after changing the model or prompt.
• There is a blind comparison of versions on at least some cases.
• Log version of the prompt, model and tools that worked.
• The user's complaint is restored by logs in a reasonable time.

### Team

• The behavior document lies next to the layout and is updated with it.
• Product, developer, and support answer the same question: “What does a feature do and what does not?”.
• It is agreed who decides when the behavior of the model floats between releases.

## Anti-patterns that occur most often

### "We trusted the model."

Prompt written, screen drawn, no check. On the demo everything is beautiful, in the production - the lottery. Trust in a frameworkless model is not trust, it is lack of product.

### One perfect example in Figma

The layout shows the answer exactly three lines long. In life, answers come in one line, in thirty, with a table inside and with a void. If the layout does not show the range, the developer guesses the layout.

### “The load is eternal because we don’t know how much.”

Spinner with no timeout and no signs of life. The user sits there and doesn't know if the answer is coming or if the process is dead. Any long-term action must have an upper limit and visible progress.

### Confident tone on any topic

The assistant responds equally cheerfully to “how much is 2 + 2” and “what is my balance in the account”. The second one he doesn't know, but it sounds the same. It’s not a design problem of a prompt — it’s an interface design problem that doesn’t distinguish between sources of confidence.

### A Memory No One Warned About

The assistant remembers the previous conversation, but the user does not know about it. A week later, he sees a strange answer and doesn't understand where the context came from. Memory must be visible and manageable, otherwise it works against trust.

### "Testing ourselves."

A team of five has been chasing the feature for two weeks and believes it has checked. Five people cover no language, no typos, no strange domains, no provocations. Without a living set of cases, any “works for us” is self-deception.

### "Standard from Best Answer"

Take a beautiful screen of a successful dialogue and use it as a sample. After a month, the model was updated, the answers became different, the screen is no longer played - and the team argues whether it became worse or just different. The standard should be a description of behavior, not a picture of a good day.

## Questions for AI-fichi design review

These questions should be asked not at the end, but at the stage when the layout can still be changed without pain.

• Where in the interface does the user see that the assistant does not know something or is not sure?
• What happens if the answer comes in 20 seconds instead of two?
• What actions can an assistant take without confirmation, and do we agree with this list?
• If the user wants to check the assistant, where will he get the source?
• What does the screen look like when the tool returned the error instead of the void?
• What will be the first thing to break in this interface if the model changes tomorrow?
• What behavior is normal and what is an incident, and who responds to it?
• In a month’s time, the user’s complaint will tell us exactly what happened, or will we guess?

## Short practical outcome

The AI feature is not checked like a regular button. It doesn't have one right answer, it has a range of what's acceptable, and the design challenge is to describe that range so that the command and the interface understand where it ends.

Three things that distinguish mature work with unpredictable behavior from naive work. First, behavior is described by words, not guessed by the layout. The second is a live set of cases where the feature is checked before and after each change, not “I’m working.” Third, the interface honestly shows boundaries: uncertainty, sources, failures, long waits. Everything else is nuanced on top of these three pillars.

The designer who knows how to assemble it ceases to be the author of screens and becomes the author of behavior. Behavior is what the user actually remembers.