Dave

I guess the problem is that you’ve gotta be super careful. If you just block all data centres, you won’t be able to federate with most of the fediverse. It’s got to only be applied to the front end website, not to the API and especially not to the federation endpoints.

Tools like Anubis do the filtering pretty well anyway. Some 75% of traffic we get to the website is blocked without even getting a challenge, because of their list of IPs, user agents, etc.

Ah that would make more sense!

I dont believe it is that easy to tell if an IP is coming from a data center

The bad scraping bots are proxying their traffic via resiential IPs, so it doesn’t even matter. Presumably this is through the free TV apps that you see mentioned on Lemmy occasionally, or through compromised devices.

Ultimately this has been a recent-ish problem, and Lemmy simply doesn’t have this sort of function yet. It’s a goo thought, though.

As the other commenter said, bad bots are not easy to tell. OpenAI, Anthropic, etc identify themselves so are easier to block. They also tend to respect a robots.txt. The issue is the b=bots that don’t limit themselves, and route through hundreds of residential IPs. You literally see dozens of IPs all with approximately the same number of requests, like 80,000 per IP. If you block them they route through others. They seem to have an endless number of residential IPs to route through, presumably due to the things you see on Lemmy about the TV apps that proxy traffic even when the TV is off. I notice a lot via South American or South East Asian countries, which lead me to suspect at least some of these residential IPs are proxying through compromised devices that might not have up to date security patches available.

Lemmy is built as a separate web app and back end that communicate via APIs. It makes it easy to support apps and alternate web interfaces, unlike Piefed which only started adding an API reasonably recently and doesn’t have a fraction of the eco system that Lemmy has.

But I think an important part is that Lemmy is explicitly customised for users. While bots are getting the same logged out version, the intent of Lemmy is not to be scraped by bots but it’s to be custom for each user. You’re supposed to log in, subscribe to communities you like, block people you don’t like, and have posts you’ve already seen hidden (if you set this in your settings).

Serving a custom page to each user based on their settings, blocks, and subscriptions isn’t really suited to static pages.

Parts will be cached but it’s per instance. For a 100 user instance, what are the chances that a recent cache of All page 74 is available? Bots are crawling every page, and most pages are content from last month, last year, or 3 years ago.

No it’s not, it was more of a tangential comment.

I think this is the photography site he’s talking about.

He mentioned “so many scrapers hit my site that it saturated my upload with generated content feeding the them.” which seemed unusual to me. I’m not sure why such a site would have a public upload. It doesn’t feel like a site that would have a technical challenge with the scrapers, but I also get the impression the main issue is the thought of hosting all his photos just for AI scrapers to train their image generation models.

Lemmy is particularly susceptible. Hosting a static site with a limited number of pages is fine. But Lemmy needs to check all posts, find the votes, sort them, and compile the page. It does that for practically every page load, and so loading a page once is quite “expensive”.

Then add on to that the almost unlimited pages of content for bots to crawl through, bots just get caught endlessly scraping pages. Even powerful servers struggle, which is why most Lemmy instances have implemented some sort of protection against bots.