|
by Ice Cream Jonsey 09/09/2025, 10:48am PDT |
|
 |
|
 |
|
We were getting hammered by all manner of "bots," in so much as I think they are web crawlers and bots that Amazon is apparently using for an AI tool. We'll see if it took and if so, I will turn the search function back on. There were other web crawlers that I also blocked. I updated robots.txt, which I don't really think Amazon respected. I've heard that some crawlers will pretend to be AmazonBot, so maybe that was it.
If it was 100% the real Amazon, I think it's really irresponsible of them to unleash their site-crushing content harvesters on a small site hosted by Dreamhost of all places without contact or compensation. I didn't wake up as a newborn hating corporations, but I have probably spent 2 hours on this over the last week trying to fix things. Bezos is out this week bragging to everyone about his jewel encrusted yacht that he could still afford after a divorce where he lost billions. Ole Pissjugs didn't even bother offering me free year of Amazon Prime in exchange for taking my time and destroying our site. If you guys were seeing the 503s, then they were coming back to their crawler. No adjustments made, if anything they started scanning the site more. What a bunch of pigs.
But again, maybe it's not genuinely them, I didn't bother doing a reverse lookup on the addresses identifying themselves as Amazon. There were several other web crawlers I hadn't heard of that were here every hour, ripping away. |
|
 |
|
 |
|
|
|