Wissenswertes

Key Takeaways: Metehan analyzed Common Crawl (CC) data and found a correlation to citation frequency in LLM Chats such as ChatGPT, Perplexity and others: Most major LLMs were trained on CC data (64% of models studied, 80%+ of GPT-3 tokens) CC prioritizes high-authority domains in its crawling via Harmonic Centrality These same domains tend to be cited most frequently by LLMs Feel free to use his tool to analyze Common Crawl authority scores for your domain or industry:...

Search Engines moved beyond keywords into understanding language in the vector space Maximum Inner Product Search (MIPS) aims at maximizing the inner product of a given query for a set of vectors