minus-squareShitpostCentral@lemmy.worldtoTechnology@lemmy.world•AI companies are violating a basic social contract of the web and and ignoring robots.txtlinkfedilinkEnglisharrow-up16·9 months agoYou’re second point is a good one, but you absolutely can log the IP which requested robots.txt. That’s just a standard part of any http server ever, no JavaScript needed. linkfedilink
You’re second point is a good one, but you absolutely can log the IP which requested robots.txt. That’s just a standard part of any http server ever, no JavaScript needed.