aijelen.hu / Articles / Guide

AI Visibility Is a Distribution: How Many Runs You Need

Treating AI visibility as a statistical distribution requires minimum sample sizes and repeated testing to distinguish signal from platform noise.

By Zoltán Varga2026-11-054 min read

Why One Answer Is an Anecdote

A single query result tells you nothing about your brand's actual presence in an AI system. It captures only one moment in a dynamic environment where the output changes based on invisible factors like server load or geographic routing. Treating one 'yes' or 'no' as a definitive metric creates a false sense of security or panic.

The core issue is that Large Language Models operate on probability, not deterministic logic. Even with identical inputs and zero temperature settings, the system may route your request to different model checkpoints or apply varying safety filters. This means the output you see is a random draw from a larger set of possible responses, not a fixed truth.

If you base strategy on a single run, you risk optimizing for noise. You might invest resources to fix a problem that only appeared once due to a temporary glitch, or ignore a critical drop in visibility because one lucky run showed your brand. Reliable measurement requires observing the pattern across many attempts to separate random fluctuation from real trends.

Per-User Context Variability

Variability in AI answers is a designed feature, not a bug. Platforms actively test different versions of their models, adjust system prompts, and manage context windows to improve performance. These internal changes happen constantly and affect whether your brand appears in the final response without any external signal changing.

Your brand might appear when the system routes your query to a specific data center but disappear when routed elsewhere. This geographic and infrastructure-level variance means that a brand's visibility is not a single number but a range of probabilities. A high mention rate in one region does not guarantee the same rate in another.

This variability also stems from the user's implicit context. Even if you use the same prompt, the platform may interpret the intent differently based on subtle cues or previous interactions in the session. The model samples from a distribution of possible answers, and your brand's inclusion depends on where that sample lands relative to the threshold for mention.

Statistical Power and Minimum Samples

To detect meaningful changes in visibility, you need a sample size large enough to overcome random noise. Research indicates that a minimum of 30 independent runs per query is required to establish a stable baseline for the mention rate. Below this number, the standard error is too high to distinguish between a real shift and statistical fluctuation.

With 30 runs, you can calculate the mean mention rate and the standard deviation. This allows you to report visibility as a range, such as 'appears in 60% of queries with a margin of error of ±10%.' This approach provides a realistic view of performance and prevents overreaction to outliers. It transforms a binary 'seen or not seen' into a measurable probability.

If the variation coefficient remains high even after 30 runs, it suggests structural instability in how the platform handles your brand. This could indicate that the brand is on the edge of the model's recall threshold or that the query intent is ambiguous. In such cases, increasing the sample size to 50 or 100 runs helps narrow the confidence interval and clarify the true performance level.

Control Variables for Repeatable Tracking

Consistent measurement requires strict control over the variables you can influence. You must keep the prompt text identical, use the same model version if available, and avoid changing the session context between runs. Any deviation introduces new noise that makes it impossible to compare results over time.

Temperature settings are critical. While setting temperature to zero reduces randomness, it does not eliminate variability caused by backend infrastructure or system prompt updates. You must document these parameters explicitly. If the platform updates its underlying model without notice, your historical data may become incomparable unless you account for this drift.

You should also control for the query type. A direct brand query behaves differently than a category or problem-solving query. Segment your tests by intent to understand where your brand is strong and where it is fragile. A brand might have robust visibility in direct queries but fail completely in comparative scenarios, revealing specific gaps in authority or content representation.

Repeatability Protocol and Three-Layer Output

Measuring visibility involves checking three distinct layers of the output pipeline. Layer 1 is the raw model generation, which might include your brand. Layer 2 is the application processing that may filter, truncate, or reformat the text. Layer 3 is what the user actually sees on the screen. A brand mentioned in Layer 1 can vanish before reaching Layer 3.

Most active probing tools only measure Layer 1. They send a query and parse the raw API response. This misses the reality that safety filters or UI truncation might hide the mention from the user. To get a true picture of visibility, you must verify if the mention survives the full pipeline to the final user interface.

A repeatable protocol requires running the same query multiple times and recording the outcome at each layer. If you see a mention in the raw output but not in the rendered UI, you have identified a specific bottleneck. This distinction is vital for reporting because business outcomes depend on what the user sees, not what the model generated internally.

AI-assisted: drafted with a language model from the author's research notes, then edited and checked before publication. Author: Zoltán Varga, AI visibility researcher and AI ecosystem architect, Budapest.