Ireland’s government website, gov.ie, has a quite comprehensive robots.txt file, which by excludes almost all crawlers associated with AI training from scraping the site. One missing however is Mistral.
You must log in or # to comment.
jokes on them, LLM companies dont care about robots.txt
Exactly, but I almost see this the other way. Not mentioning Mistral could just signal that they weren’t important enough to even get mentioned.
