AI Search Benchmark 01: Methodology for 1,000 Commercial SaaS Queries

If AI answers were just repackaged Google results, the brands they recommend would closely track classic organic rankings, and the sources cited would be the same pages that rank. This benchmark measures how true that actually is on commercial, money-intent queries, where the stakes are highest and the training-data shortcuts weakest.

The answer decides how much of the game is still classic SEO and how much is genuinely new work: the degree to which answer engine optimization and generative engine optimization are separate disciplines rather than rebranded ones.

The AI Search Benchmark methodology fixes the query set before collection starts and logs every answer with its citations under the rules below.

• Query set: 1,000 commercial SaaS queries across 20 categories, split across category, comparison, alternative and recommendation intent. The full list ships with the dataset.
• Surfaces: ChatGPT (with browsing), Gemini and Google AI Mode, sampled in the same window from clean, logged-out sessions with memory and personalization disabled.
• Sampling conditions: the exact collection dates, country/location, language, and the model or product version where observable are recorded with every run and published with the dataset.
• What we log: every brand named, its position in the answer, sentiment of the mention, every cited URL and domain, and the classic Google top-10 for the same query, collected in the same window, as a control.
• Runs: each query sampled multiple times per surface to account for answer variance; the repetition count per query per surface is fixed before collection starts and reported with the results. We report frequency, not single answers.
• Coding rules: what counts as a "recommendation" versus a passing mention, and how sentiment is classified, are defined in a written codebook that ships with the study, written before collection, not after.
• Citations: deduplicated to the canonical domain; per-URL detail preserved in the raw data.
• Failures: refusals, errors and non-answers are logged and reported, not silently dropped.
• Dataset: the raw query-level data ships with the published study so the findings can be checked, not just believed.

Once the AI Search Benchmark has run, the published study will report, at minimum:

• How often recommendation-intent answers name brands that do not rank in Google's top 10 for the same query.
• How concentrated citation influence is: how few domains account for the majority of citations across the three surfaces, and how much of that influence is third-party.
• How often the three surfaces agree on the top recommendation, and how often they contradict each other.
• Per-category splits, full result tables and the citation census, with a downloadable dataset.

We publish numbers when we have them and not a moment earlier. A benchmark announced with invented findings would be exactly the kind of marketing this research program exists to replace.

Pilot published: a 100-query, single-surface teardown of one category, coded under this rulebook with the dataset attached, is at Teardown 01: project management software. It is not the benchmark; it is the method run once, in public, so the coding rules could be tested before the full collection. A second run on CRM software, Teardown 02, produced codebook amendment v1.4 and the first side-by-side comparison of two categories under one rulebook.

The same measurement discipline runs inside every client engagement: it's how our GEO program and AI Search engagements get scored, and the citation census feeds the source maps behind Source Authority campaigns.

AI answers vary by session, region, account state and time, so the AI Search Benchmark's frequencies will tell you about the sampling window, not a permanent state. The query set is SaaS-weighted, and other categories may behave differently. Citation visibility differs by surface, so cross-surface citation comparisons are directional. We publish limitations so the findings can be trusted; a benchmark without them is marketing.