Generally speaking, AI bots on the web are a bad thing. They’re consuming resources and stealing all of our content, but blocking them might make things worse.
As I’ve shared with many people over the last year, I don’t want AI bots to crawl my site, but I really don’t want them to not crawl my site. At the end of the day, having your content show up when people are using AI tools is a good thing, so letting the bots crawl your site is the logical move.
The other challenge is accessibility. Publishers are increasingly blocking tools like The Internet Archive (as explained here by Techdirt), which can be a slippery slope.
When it comes to blocking The Internet Archive, Techdirt says:
What makes this particularly frustrating is that the internet’s openness was never supposed to have asterisks. The fundamental promise wasn’t “publish something and it’s accessible to all, except for technologies we decide we don’t like.” It was just… open. You put something on the public web, people can access it. That simplicity is what made the web transformative.
Here is where the slippery slope comes in.
Now we’re carving out exceptions based on who might access content and what they might do with it. And once you start making those exceptions, where do they end? If the Internet Archive can be blocked because AI companies might use it, what about research databases? What about accessibility tools that help visually impaired users? What about the next technology we haven’t invented yet?
This is a real concern. People say “oh well, blocking machines is different from blocking humans,” but that’s exactly why I mention assistive tech for the visually impaired. Machines accessing content are frequently tools that help humans—including me. I use an AI tool to help fact check my articles, and part of that process involves feeding it the source links. But increasingly, the tool tells me it can’t access those articles to verify whether my coverage accurately reflects them.
As they said repeatedly in the article, there’s not an easy solution. These AI companies are making a lot of money off of our content, so something needs to change. For most of us, though, simply blocking them isn’t a good move for anyone.
Check out the full article from Techdirt for more.


