Spiderbots Descend

-=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- (c) WidthPadding Industries 1987 0\|644\|0 -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=-
Socoder -> Site & Server -> Spiderbots Descend
Fri, 11 Oct 2024, 05:42
cyangames	Just a word of warning, bytespider and claudebot are mass crawling again, if you run a website, block them via .htaccess -=-=- Web / Game Dev, occasionally finishes off coding games also!
Fri, 11 Oct 2024, 05:42
Jayenkai	Logs for the past hour suggest... GPTBot : 7,193 Misc (Actual users, maybe?) : 2165 BingBog : 705 Meta : 219 Bytedance (TikTok) : 166 Google : 143 Claude : 40 -=-=- ''Load, Next List!''
Mon, 14 Oct 2024, 05:26
HoboBen	Dunno what your opinion is on ChatGPT is ethically. Figure, if we post here, it's public, and ChatGPT has as much right as any body to lurk the forum. But it's gonna package up what we say and regurgitate it without attribution and destroy the environment at the same time. And it's slurping up more data than anybody else. Do you block it? -=-=- blog \| work \| code \| more code
Mon, 14 Oct 2024, 06:09
Jayenkai	I don't block it. But discuss whether I should. To be fair, all the various AI's would have scraped the site half-a-dozen times each, by now, even long before it was a "thing" to avoid. Bing's ALWAYS scraping stuff, and they're chucking all that into CoPilot, and Google Google the Google Google, so the only "real" option is to pretty much block EVERYTHING. -=-=- ''Load, Next List!''
Mon, 14 Oct 2024, 09:50
Jayenkai	Stats for the past hour GPTBot : 8,741 Bing : 688 Probably actual users : 269 Google : 294 Bytedance : 167 CCBot : 58 Meta : 34 DotBot : 9 Claude : 5 -=-=- ''Load, Next List!''
Fri, 08 Nov 2024, 10:25
Jayenkai	Stats for the past hour Google : 3,124 dataforseo : 521 Bing : 453 Probably actual users : 184 Bytedance : 97 Meta : 59 AmazonBot : 26 Claude : 8 DotBot : 4 GPTBot : 1 .. Google.. And I double checked the IPs, and they're definitely Google. Half of them are listed as GoogleBot, and the other half "Google Other"...!?! So, I presume their AI crap. I mean.. What am I supposed to do about that!? Block Google!?!?! -=-=- ''Load, Next List!''
Fri, 08 Nov 2024, 15:08
cyangames	I think you can change the crawl rate within webmaster tools...well search console nowadays -=-=- Web / Game Dev, occasionally finishes off coding games also!
Fri, 08 Nov 2024, 16:25
Jayenkai	Oh yeah, I forgot about "you have to fart about online". Quite why it can't pay attention to the settings in robots.txt, I've no idea. -=-=- ''Load, Next List!''
Mon, 11 Nov 2024, 03:23
cyangames	It is a mystery.
Mon, 25 Nov 2024, 10:17
Jayenkai	Been writing a Log-File scanner tool, today. I can now, fairly easily, access 24-hour bot stats! Woot! Users : 44335 (2599 by IP address) GPTBot : 62140 Claude : 61504 Google : 36549 Bing : 19265 Facebook : 5296 Bytedance : 4151 DotBot : 538 AmazonBot : 22 Data4SEO : 4 -=-=- ''Load, Next List!''