Understanding the Citation Discrepancies: Large Language Models vs. Google Rankings

In the evolving landscape of online search and information retrieval, large language models (LLMs) are showing significant differences in how they cite sources compared to traditional search engines like Google. A recent study by Search Atlas, an SEO software company, sheds light on these disparities by systematically comparing the citation practices of OpenAI’s ChatGPT, Google’s Gemini, and another retrieval model known as Perplexity against Google’s search results.

The Study: An Overview

This comprehensive analysis incorporated a staggering 18,377 matched queries, aimed at uncovering the gaps between traditional search engine visibility and the citation practices of various AI platforms. Understanding these differences is crucial for marketers, content creators, and anyone interested in the dynamics of information retrieval today.

Perplexity: The Closest to Traditional Search

Perplexity stands apart from other models due to its architecture that supports live web retrieval. This feature implies that its citations would resemble search results more closely, a hypothesis supported by the study’s findings.

Across the evaluated dataset, Perplexity demonstrated a median domain overlap with Google results of around 25–30%. Furthermore, it showcased a median URL overlap of almost 20%. It’s worth noting that Perplexity shared a total of 18,549 domains with Google, accounting for approximately 43% of the domains it cited. This high overlap suggests that if your website has strong rankings in Google, you’re likely to see comparable visibility in Perplexity’s answers.

ChatGPT and Gemini: A More Selective Approach

In contrast, ChatGPT and Gemini adopt a distinctly different method. ChatGPT showed significantly lower overlap with Google; its median domain overlap lingered around 10–15%. It only shared 1,503 domains with Google, comprising about 21% of its cited domains. The URL matches often fell below the 10% mark, indicating a selective retrieval process.

Gemini’s behavior appeared less consistent, with some responses exhibiting minimal overlap with Google’s results, while others mirrored them more closely. Overall, Gemini’s shared domains with Google amounted to just 160, which represented a mere 4% of the domains within Google’s output—even though these domains constituted 28% of Gemini’s citations. This inconsistency reveals that Gemini may not be as reliable for retrieving similar information or citations as Perplexity.

Implications for Visibility

The findings of this study call into question the assumption that ranking well on Google guarantees citations by LLMs. Perplexity’s structure emphasizes real-time web searching, making its citation patterns more aligned with traditional search rankings. Therefore, if a site is already performing well in Google, it stands a better chance of being cited similarly in Perplexity’s answers.

On the other hand, ChatGPT and Gemini’s dependence on pre-trained knowledge leads them to draw from a narrower set of sources. This means that traditional SEO signals and rankings may exert less influence on the selection of cited sources. Consequently, those relying on these tools may face challenges in achieving visibility, especially if their domains don’t conform to the more limited, curated retrieval habits of these models.

Study Limitations

It’s essential to approach these findings with caution, given several limitations present in the study. For one, the dataset was largely skewed towards Perplexity, which made up a remarkable 89% of matched queries. In comparison, OpenAI accounted for 8%, and Gemini a mere 3%. This limitation raises questions about the broader applicability of the results.

Additionally, the researchers employed semantic similarity scoring to match queries. While this method aimed to reflect similar information requirements, it’s important to recognize that the paired queries were not identical user searches. A similarity threshold of 82% was utilized, based on OpenAI’s embedding model, potentially sidelining variations in user intent.

The analysis also captured data over a two-month window, offering only a snapshot in time. To understand whether these overlap patterns are consistent, a longer timeframe would be necessary for evaluation.

What Lies Ahead?

Looking forward, the implications of this study suggest that retrieval-based systems like Perplexity may place greater importance on traditional SEO signals and domain authority when it comes to visibility. Conversely, reasoning-focused models like ChatGPT and Gemini might rely less on these conventional indicators, fostering a divergence in information retrieval strategies.

As the landscape of online information continues to evolve with the integration of AI, understanding these varying citation practices becomes indispensable for navigating the world of digital content and search optimization. In this shifting scenario, keeping abreast of how different models retrieve and cite information will be crucial for anyone looking to maintain online visibility and relevance.