Icy Phoenix

     
 


Post new topic  Reply to topic 
Page 1 of 1
 
 
Reply with quote Download Post 
Post A Vicious Spider Bot (AhrefsBot) 
 
I put this here because - "Who ever reads the chat forums"? Besides, I don't speak'a de Italiano to be able to add it there too. (Sorry MG)

Reference:

http://ahrefs.com/robot/

You are advised to add this to your robots.txt in the root of the Public-Folder and the robots.txt in root folders of anything else you may have on your server.

user-agent: AhrefsBot
disallow: /

My log is absolutely full of "already has more than 'max_user_connections' active connections"

And this ugly bot from:

6:14:02pm to 7:47:26pm - When I stopped it with a quick "Deny All - Everything" until I could work out what was happening, had made 2,992 queries on the database.

But at the same time through its actions, had cached 2,992 files for me. So I suppose I should be grateful for that and not complain about the DDos attack.

 
 



 
mortSend private message  
Back to topPage bottom
Icy Phoenix is an open source project, you can show your appreciation and support future development by donating to the project.

Support us
 
Reply with quote Download Post 
Post Re: A Vicious Spider Bot (AhrefsBot) 
 
http://www.icyphoenix.com/viewtopic.php?f=12&t=8773  
 




____________
www.DutchaGoGo.com (development/under construction ...Forever?¿?)
 
Joshua203Send private messageVisit poster's website  
Back to topPage bottom
Reply with quote Download Post 
Post Re: A Vicious Spider Bot (AhrefsBot) 
 
Um?

Sorry, but only users granted special access can read topics in this forum.

Is there something I should know?

 
 



 
mortSend private message  
Back to topPage bottom
Reply with quote Download Post 
Post Re: A Vicious Spider Bot (AhrefsBot) 
 
Quote:
Thanks for reporting, it may happen from time to time that a new crawler opens many pages... I'll try to limit its access if it continues.
(from MG)
 



 
InformproSend private message  
Back to topPage bottom
Reply with quote Download Post 
Post Re: A Vicious Spider Bot (AhrefsBot) 
 
Ooopz, I thought thought you were one of those special cases, I'll turn it arround and add the link to here in the other thread (it's about reporting ahrefs and 80legs behaving badly on icy sites (whois page).
 




____________
www.DutchaGoGo.com (development/under construction ...Forever?¿?)
 
Joshua203Send private messageVisit poster's website  
Back to topPage bottom
Reply with quote Download Post 
Post Re: A Vicious Spider Bot (AhrefsBot) 
 
Informpro wrote: [View Post]
Quote:
Thanks for reporting, it may happen from time to time that a new crawler opens many pages... I'll try to limit its access if it continues.
(from MG)


It can be controlled no doubt - limit Crawl-Delay: 360000, but as it has no respect for how it goes about its business, I'd class it as a web-reaper and ban it completely.



Joshua203 wrote: [View Post]
Ooopz, I thought thought you were one of those special cases, I'll turn it arround and add the link to here in the other thread (it's about reporting ahrefs and 80legs behaving badly on icy sites (whois page).


Hehehe! Nope!  

This thing appeared to have had 16 streams running when I killed it.  Probably Out-DDos'd itself.  
 



 
mortSend private message  
Back to topPage bottom
Post new topic  Reply to topic  Page 1 of 1
 


Display posts from previous:    

HideWas this topic useful?

Link this topic
URL
BBCode
HTML




 
Permissions List
You cannot post new topics
You cannot reply to topics
You cannot edit your posts
You cannot delete your posts
You cannot vote in polls
You cannot attach files
You can download files
You cannot post calendar events


  

 

  cron