OpenAI launched SearchGPT on July 26, 2024, as its first foray into the search engine market. This launch included the introduction of OAI-SearchBot, a web crawler designed to index content and surface websites in SearchGPT’s search results.
A significant number of top news and media publishers have blocked OAI-SearchBot. According to Originality.ai, 14 of the top 1,000 website publishers, including The New York Times, Wired, The New Yorker, Vogue, Vanity Fair, and GQ, have implemented blocks on this bot.
The primary reasons for blocking OAI-SearchBot include concerns about privacy, intellectual property rights, and the potential for AI systems to replicate or closely mimic their content. Many publishers fear that allowing AI crawlers unrestricted access could lead to their content being repurposed without proper attribution or compensation.
This blocking is part of a larger trend where top websites are increasingly restricting AI web crawlers. The blocking of OAI-SearchBot and other AI crawlers could significantly impact the development and effectiveness of AI-driven search engines like SearchGPT.
