A Vicious Spider Bot (AhrefsBot)


Subject: A Vicious Spider Bot (AhrefsBot)
I put this here because - "Who ever reads the chat forums"? Besides, I don't speak'a de Italiano to be able to add it there too. (Sorry MG) :mricy:

Reference:

http://ahrefs.com/robot/

You are advised to add this to your robots.txt in the root of the Public-Folder and the robots.txt in root folders of anything else you may have on your server.

user-agent: AhrefsBot
disallow: /

My log is absolutely full of "already has more than 'max_user_connections' active connections"

And this ugly bot from:

6:14:02pm to 7:47:26pm - When I stopped it with a quick "Deny All - Everything" until I could work out what was happening, had made 2,992 queries on the database.

But at the same time through its actions, had cached 2,992 files for me. So I suppose I should be grateful for that and not complain about the DDos attack.

:censored:

Profile PM  
Subject: Re: A Vicious Spider Bot (AhrefsBot)
http://www.icyphoenix.com/viewtopic.php?f=12&t=8773 :wink:

Subject: Re: A Vicious Spider Bot (AhrefsBot)
Um?

Sorry, but only users granted special access can read topics in this forum.

Is there something I should know?

:P

Profile PM  
Subject: Re: A Vicious Spider Bot (AhrefsBot)
Quote:
Thanks for reporting, it may happen from time to time that a new crawler opens many pages... I'll try to limit its access if it continues.
(from MG)

Profile PM  
Subject: Re: A Vicious Spider Bot (AhrefsBot)
Ooopz, I thought thought you were one of those special cases, I'll turn it arround and add the link to here in the other thread (it's about reporting ahrefs and 80legs behaving badly on icy sites (whois page).

Subject: Re: A Vicious Spider Bot (AhrefsBot)
Informpro wrote: [View Post]
Quote:
Thanks for reporting, it may happen from time to time that a new crawler opens many pages... I'll try to limit its access if it continues.
(from MG)


It can be controlled no doubt - limit Crawl-Delay: 360000, but as it has no respect for how it goes about its business, I'd class it as a web-reaper and ban it completely.

;)

Joshua203 wrote: [View Post]
Ooopz, I thought thought you were one of those special cases, I'll turn it arround and add the link to here in the other thread (it's about reporting ahrefs and 80legs behaving badly on icy sites (whois page).


Hehehe! Nope! :mryellow:

This thing appeared to have had 16 streams running when I killed it. Probably Out-DDos'd itself. :LOL:

Profile PM  

Page 1 of 1


  
You cannot post new topics
You cannot reply to topics
You cannot edit your posts
You cannot delete your posts
You cannot vote in polls
You cannot attach files
You can download files
You cannot post calendar events

   

This is a "Lo-Fi" version of our main content. To view the full version with more information, formatting and images, please click here.

Powered by Icy Phoenix based on phpBB
Generation Time: 0.2232s (PHP: 8% SQL: 92%)
SQL queries: 15 - Debug Off - GZIP Enabled