Metehan & Konitzny: ChatGPTs Retrieval System leaked - Web Search, Index etc.
Key Takeaways:
Metehan analyzed SSE (server-sent events) in the ChatGPT UI and found a detailed debug view of ChatGPTs retrieval system:
For a single "best AI visibility tools" prompt, ChatGPT ran 5 search rounds, wrote 18 different queries (hidden queries), made 50 engine calls, pulled 228 results, fetched 223 URLs with selected chunks, and cited 16.
The page body is not HTML. It is a markdown-like text render. That render is what gets cut into blocks of roughly 170 words and scored. The model does not see your full page. It sees one to three of those blocks per source
ChatGPT pulls pre-scored passage chunks from its search index cache. When a page is already indexed, the passage chunks and their neural relevance scores are carried straight into the fetch payload and normalized to 1.0000. Fresh or unindexed pages are split into hundreds of small fragments at runtime.
One prompt produces a 150,000-line JSON dump
A per-result score, plus a score object that breaks that score into its components:
A per-chunk score for every block of every fetched page, and which blocks were kept for the prompt
A should_fetch decision on each result, with crawl date and publication date
A second ranking pass done in the model's reasoning, where domains are re-ordered before the answer is written
The exact prompt the model receives, with the word budget it is given per source
Separate result types for shopping and local queries, with their own fields
Segonzac identified a plain-text field called "search_engine" in ChatGPT's turn-stream
he found that 70-80% of retrievals on a paid account turned to "serpapi"
while others showed lots of internal sources such as labrador-news-all, labrador-news-7d, labrador-wiki, labrador-web-fallback, labrador-images-nocache, labrador-arxiv, bing-image
Konitzny analyzed the search engines ChatGPTs retrieval system leaked in its data streams:
labrador: 42 engines, internal
The in-house index, and by far the deepest. It carries dedicated engines for
news windows (1 day, 7 days, all time),
wiki,
arxiv,
STEM,
legal,
finance,
four separate medical and
legal knowledge engines,
Reddit,
YouTube,
PDFs,
images,
places, and even
synthetic content.
labrador-web: 6 engines, internal General web retrieval, including the Google redirect fallbacks.
serpapi: 9 engines, external Split across three sources: serpapi_auto_news (5), serpapi_auto_web (2), and serpapi_auto_image (2).
mai_grounding: 5 engines, external
Two on mai_grounding for webpages, three on mai_grounding_news, all flagged mai-news-beta.
The rest: 7 engines, external
Bing (2), Fortis (1), Fortis Barebone (1), Getty (1), Yelp (1), Foursquare (1).
Konitzny created a complete list of ChatGPTs Search Engines used during retrieval
Rudzki further outlines the shopping layer or rather ChatGPTs shopping index:
Five live production experiments [A/B tests identified in Server-sent Events], all testing different variations of the index against real shopping traffic, point to a team actively building a product index and validating whether it's ready to rely on
BM25 lexical component: Narrows the field down to 10 sources for ranking. Ten turns out to be enough.
rerank400: Reranks a pool of 400 candidates.
prod400: 400 products used as the input pool.
ann4096: Approximate nearest neighbor, meaning vector search, run at two different dimensions, 12,288 and 4,096.
Further, Metehan found ChatGPT inlcuding notes on "inauthentic mentions" within list responses similar to Ray
What does it mean for AI SEO?
Sophisticated Monitoring is very important: keep track of the retrieval patterns in your specific industry and prompts
which sources are part of the retrieval already?
which ones are dropped during scoring and re-ranking?
which paragraphs are chosen for a certain prompt and page?
Optimize pages to contain high information density and information gain to match for the roughly 170 word blocks using clear and precise headlines
Technical SEO: check the rendering of your content or rather passages/paragraphs using html2text library - that is what is going into ChatGPTs retrieval process
Sources:
