See Local News
Get Balanced News From Your U.S. State.
See Local News
See all locals

Now you can block OpenAI’s web crawler

Posted on AllSides August 9th, 2023
From The Left

OpenAI now lets you block its web crawler from scraping your site to help train GPT models. 

OpenAI said website operators can specifically disallow its GPTBot crawler on their site’s Robots.txt file or block its IP address. “Web pages crawled with the GPTBot user agent may potentially be used to improve future models and are filtered to remove sources that require paywall access, are known to gather personally identifiable information (PII), or have text that violates our policies,” OpenAI said in the blog post. For sources that don’t fit the excluded criteria,...

Read full story

More News about Technology from the Left, Center and Right

From the Left

From the Center

From the Right