It’s impossible, i got this instance to just see lemmy from my own instance, but no, it was slow as hell the whole week, i got new pods, put postgres on a different pod, pictrs on another, etc.
But it was slow as hell. I didn’t know what it was until a few hours before now. 500 GETs in a MINUTE by ClaudeBot and GPTBot, wth is this? why? I blocked the user agents, etc, using a blocking extension on NGINX and now it works.
WHY? So google can say that you should eat glass?
Life is now hell, if before at least someone could upload a website, now even that is painfull.
Sorry for the rant.
I highly recommend using Anubis as a proxy for your entire instance. It’s a little complicated to get going, but it stops any and all AI scrapers with a denial of access. Having a robots.txt works, but only so much, because some of these bots do not respect it. And, honestly, with the way Sam Altman talks about the people he’s stolen and scraped from, I don’t think anyone should be surprised.
But, I have Anubis running on my personal website and I’ve tested to see if ChatGPT can see it, and it cannot. Good enough for me