I put this here because - "Who ever reads the chat forums"? Besides, I don't speak'a de Italiano to be able to add it there too. (Sorry MG)

Reference:

http://ahrefs.com/robot/

You are advised to add this to your robots.txt in the root of the Public-Folder and the robots.txt in root folders of anything else you may have on your server.

user-agent: AhrefsBot
disallow: /

My log is absolutely full of "already has more than 'max_user_connections' active connections"

And this ugly bot from:

6:14:02pm to 7:47:26pm - When I stopped it with a quick "Deny All - Everything" until I could work out what was happening, had made 2,992 queries on the database.

But at the same time through its actions, had cached 2,992 files for me. So I suppose I should be grateful for that and not complain about the DDos attack.

 

http://www.icyphoenix.com/viewtopic.php?f=12&t=8773  

Um?

Sorry, but only users granted special access can read topics in this forum.

Is there something I should know?

 

Quote:

Thanks for reporting, it may happen from time to time that a new crawler opens many pages... I'll try to limit its access if it continues.

(from MG)

Ooopz, I thought thought you were one of those special cases, I'll turn it arround and add the link to here in the other thread (it's about reporting ahrefs and 80legs behaving badly on icy sites (whois page).

Informpro wrote: [View Post]

Quote:

Thanks for reporting, it may happen from time to time that a new crawler opens many pages... I'll try to limit its access if it continues.

(from MG)

It can be controlled no doubt - limit Crawl-Delay: 360000, but as it has no respect for how it goes about its business, I'd class it as a web-reaper and ban it completely.

Joshua203 wrote: [View Post]

Ooopz, I thought thought you were one of those special cases, I'll turn it arround and add the link to here in the other thread (it's about reporting ahrefs and 80legs behaving badly on icy sites (whois page).

Hehehe! Nope!  

This thing appeared to have had 16 streams running when I killed it.  Probably Out-DDos'd itself.