https://www.icyphoenix.com/viewtopic.php?f=35&t=7342
-----------------------------------
spydie
Sun 29 Aug, 2010 12:04

Massive Spider Attack
-----------------------------------
Hi

Got a small problem with one of my users pages.

Results that he´s having massive spider attacks from rumanien sites, but all using an american spider,

called 80.legs

turns out to use up to 80 conections or more at the time, driving the server cpu use, nearly into red area.

 [img]http://www.icyphoenix.com/files/images/6197/spider.png[/img] 

the funny thing is, it uses ip´s from rumania etc 

but we can´t off course block IP by IP, by the time you finnish with your list, he´s back with new IP´s. And look´s like theres no way to block them by spider id ( tried it, but it´s still there).


Any idea how to get rid of this ???

Edit:

Searching 80.legs website, i found [url=http://www.80legs.com/spider.html]This

Any idea how to include this spider into the robot block list??

By htaccess or by robot text


-----------------------------------
Mighty Gorgon
Tue 31 Aug, 2010 13:10

Re: Massive Spider Attack
-----------------------------------
.htaccess is the best way!


-----------------------------------
tormentor
Tue 31 Aug, 2010 22:35

Re: Massive Spider Attack
-----------------------------------
Deny the access from all the range of the network (if you don't care of Romanians)

for example: I found today on the error log someone that was trying to find files that doesn't exist my place (usually they search for the "links" mod or the "knowledge base" mod (I've uninstalled it)).

So in order to prevent the IP change I banned the whole network, you can find the range easily without calculation using whois.domaintools.com

for example if the ip is

58.22.0.0

the range is from 58.22.0.0 to 58.22.1.255

and the line in the .htaccess is

deny from 58.22.0.0/15

(you can find the range "58.22.0.0/15" in the site's page, but there are other site's that can make the calculation for you, I forgot the links, sorry  :oops: )


-----------------------------------
spydie
Wed 01 Sep, 2010 00:18

Re: Massive Spider Attack
-----------------------------------
well for first thing i did put the spider id in the tracker *008*.

took about 10 minutes , and all 80 legs conections were gone.

now i probably start to block ukrainiens "ru" range.

turns out that they´re trying to access referers list


