https://www.icyphoenix.com/viewtopic.php?f=4&t=2950
-----------------------------------
novice programmer
Tue 16 Oct, 2007 18:02

SEO - Robots list
-----------------------------------
This topic is for help ban of spam and un-wanted robots. use this with [url=http://www.icyphoenix.com/viewtopic.php?f=4&t=1529]this other topic: (Mail Spammers list and spam users)

As some people don´t want their sites indexed by any search robot/engine, I started this topic for reporting search robots.
There are some robots that ignore the disallow or go to that directories. Also there are ons that recopilate the "mailto:" HTML´s lines and send spam to that adresses.

Please, use this format for the robot reporting:

[cell class=mg_codetitle]Good/Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses:
The user-agent the robot gives:
Why it is good or bad:
Other data:
 
[/cell]

I don´t mind if cells are not used, but use the horizontal lines if you´re going to report various engines in same post.


-----------------------------------
novice programmer
Tue 16 Oct, 2007 18:12

Re: Robots List
-----------------------------------
Now, the list:

[spoiler]
[hide]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 207.96.148.8
The user-agent the robot gives: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)
Why it is bad: Bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by rdprm.gouv.qc.ca, PQ, CA.
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 209.237.238.230
The user-agent the robot gives:
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot of alexa.com, US. Weird operation, does not respect the robots standard, as an unknown bot (empty UA string) can not be dis-allowed. Sending an anonymous bot from non-resolving IP addresses with no associated PTR records does not add to profesionalism and trust.
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 209.237.238.235
The user-agent the robot gives:
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot of alexa.com, US. Weird operation, does not respect the robots standard, as an unknown bot (empty UA string) can not be dis-allowed. Sending an anonymous bot from non-resolving IP addresses with no associated PTR records does not add to profesionalism and trust.
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 209.237.238.224
The user-agent the robot gives:
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot of alexa.com, US. Weird operation, does not respect the robots standard, as an unknown bot (empty UA string) can not be dis-allowed. Sending an anonymous bot from non-resolving IP addresses with no associated PTR records does not add to profesionalism and trust.
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 64.15.129.5 - ip-64-15-129-5.reverse.privatedns.com
The user-agent the robot gives: http://www.picsearch.com/bot.html
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, 69.46.0.142 hosted by NOC4HOSTS, US. Content harvester with faked UA ?
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 69.46.0.142
The user-agent the robot gives: http://www.picsearch.com/bot.html
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, 69.46.0.142 hosted by NOC4HOSTS, US. Content harvester with faked UA ?
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 66.90.110.*
The user-agent the robot gives: T8Abot/v0.0.7-beta (3724461@gmail.com)
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by FDC Servers, fdcservers.net, US. Massive operation using many IP addresses (66.90.110.199 ... 66.90.110.254)
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 66.90.95.*
The user-agent the robot gives: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown massive bot using various IP addresses of a /24 subnet range (66.90.64.1 ... 66.90.64.255), hosted by FDC Servers, fdcservers.net, US. Looks like a parallel operation, but is probably just one host using a /24 address segment. Nasty and ugly.
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 208.53.158.58
The user-agent the robot gives: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by FDC Servers, fdcservers.net, US. Looks like the same operation as the 66.90.64.0/24. Reverse lookup translated to an empty domain name "."
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 67.103.244.51 - h-67-103-244-51.lsanca54.covad.net
The user-agent the robot gives: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by covad.net, CA, US
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 216.22.50.242 - srv.iffsu.com
The user-agent the robot gives: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot of iffsu.com, domain registered by DomainsByProxy.com, US, on GoDaddy. Hosted by Infini Corp, MA, US. Domain name 'srv.iffsu.com' does not resolve at all, iffsu.com does.
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 211.240.76.66
The user-agent the robot gives: Java/1.4.1_04
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by elim.net, KR
 
[/cell]
[cell class=mg_codetitle]Normal robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 202.96.51.*
The user-agent the robot gives: lanshanbot/1.0 - bot/1.0 - lanshanbot/1.0 - msrabot/1.0 (+http://search.msn.com/msnbot.htm)
Other data: unknown bot, hosted by Eastern-Network Co., CN. Using illegally faked UA "(+http://search.msn.com/msnbot.htm)". Although they seem to read robots.txt, this operation is apparently just another fishy content harvester.
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 64.15.139.* - ip-___.reverse.privatedns.com
The user-agent the robot gives: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by privatedns.com, CA. Crawling at a low creeping rate. Content harvester.
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 57.69.14.65 - nyhqoutbound.unicef.org
The user-agent the robot gives: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot of unicef.org (??!). Harvesting at a pace of 1 request every 12 seconds. Is this real, or are they 0wned?
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 209.200.225.* - d___.lunarpages.com
The user-agent the robot gives: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by lunarpages.com, US. Content harevster, crawling at a low creeping rate. Apparently parallel operation of a widely spread bot net involving many other nodes (many more than just those 2 listed above).
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 64.71.167.*
The user-agent the robot gives: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown slow creeping bot using various IP addresses of a /26 subnet range (64.71.167.73 ... 117), hosted by McColo Corp, NJ, US. Content harvester keeping a low profile but crawling and sucking the entire site's content over weeks. Looks like a parallel operation, but is probably just one host using a /26 address segment.
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 66.220.20.*
The user-agent the robot gives: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown slow creeping bot using various IP addresses of a /25 subnet range (66.220.20.1 ... 66.220.20.127), hosted by Hurricane Electric, he.net, US. Content harvester keeping a low profile but crawling and sucking the entire site's content over weeks. Looks like a parallel operation, but is probably just one host using a /25 address segment.
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 84.19.182.205 - ns.km20802.keymachine.de
The user-agent the robot gives: Java/1.4.1_04
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot of keymachine.de, hosted by keyweb.de DE
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 83.114.231.217 - aste-genev-bois-152-1-33-217.w83-114.abo.wanadoo.fr  	
The user-agent the robot gives: Mo College 1.9
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by wanadoo.fr, FR
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 83.114.105.251 - aste-genev-bois-152-1-39-251.w83-114.abo.wanadoo.fr
The user-agent the robot gives: Mo College 1.9
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by wanadoo.fr, FR
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 82.121.29.106 - aste-genev-bois-152-1-15-106.w82-121.abo.wanadoo.fr
The user-agent the robot gives: Mo College 1.9
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by wanadoo.fr, FR
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 210.176.59.13
The user-agent the robot gives: Java/1.4.1_04
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by pccw.com, HK
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 83.130.33.11 - igld-83-130-33-11.inter.net.il
The user-agent the robot gives: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by inter.net.il, IL
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 85.10.192.172 - cenitz.pingworks.net
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot of pingworks.net, hosted by hetzner.de, DE. Malicious forms exploiting bot trying to inject SPAM into PHP forms input.
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 217.76.144.121
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by arsys.es, ES
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 221.187.28.238 - p4238-ipbf401marunouchi.tokyo.ocn.ne.jp
The user-agent the robot gives: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: nknown bot, hosted by oce.ne.jp, JP
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 61.187.164.70
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by CHINANET-HN-HY hntelecom.net.cn, CN
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 218.39.69.26
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by HANANET-INFRA, hanaro.com, KR
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 216.7.179.20 - stxant393-01.syntryx.com
The user-agent the robot gives: lwp-trivial/1.41
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot of syntryx.com, hosted by Jay Kaplan, NJ, US; UA morphing ...
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 216.7.179.21 - stxant393-02.syntryx.com
The user-agent the robot gives: Syntryx ANT Scout Chassis Pheromone
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot of syntryx.com, hosted by Jay Kaplan, NJ, US
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 64.92.202.124 - stxant02.thenewpush.com
The user-agent the robot gives: Syntryx ANT Scout Chassis Pheromone
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by thenewpush.com, US
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 38.118.25.59
The user-agent the robot gives: Mozilla/4.0 (compatible; MSIE 6.0; Windows XP)
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by COGENT, US
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 193.219.242.141
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: guestbook spider bot, hosted by golis.net, SO.
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 61.95.26.170 - dsl-61-95-26-170.request.com.au
The user-agent the robot gives: Java/1.5.0_06
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by powertel.com.au, AU
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 67.164.135.213 - c-67-164-135-213.hsd1.tx.comcast.net
The user-agent the robot gives: Mozilla/4.08
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by comcast.net, US
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 88.136.38.152 - 88-136-38-152.adslgp.cegetel.net
The user-agent the robot gives: mozilla 4.0
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by cegetel.net, FR
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 145.99.244.242 - bnc.glined.info
The user-agent the robot gives: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by surfnet.nl, NL
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 70.84.128.244 - 244.70-84-128.reverse.theplanet.com
The user-agent the robot gives: "LWP::Simple/5.803" and "Poirot"
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by theplanet.com, US
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 67.76.130.25 - fl-67-76-130-25.sta.sprint-hsd.net
The user-agent the robot gives: Java/1.5.0_06
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by sprint-hsd.net, US
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 62.68.160.38 - 3e44a026.adsl.enternet.hu
The user-agent the robot gives: Java/1.5.0_06
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by enternet.hu, HU
 
[/cell]
[cell class=mg_codetitle]Bad robot:[/cell][cell class=mg_quotediv]
Ip/s that the robot uses: 81.18.67.*
The user-agent the robot gives: WEP Search 00
Why it is bad: bot starts spidering the site without even looking for a /robots.txt.
Other data: unknown bot, hosted by net263.com, CN
 
[/cell]



[/hide]
[/spoiler]

As this list is so high, I give the lists for download.
Last file in next of my posts in this topic.

Source:
http://www.kloth.net

Last file download:
[url=http://www.icyphoenix.com/download.php?id=667]Download from icyphoenix.com


-----------------------------------
jz
Tue 16 Oct, 2007 20:47

Re: Robots List
-----------------------------------
thanks


-----------------------------------
ThE KuKa
Tue 16 Oct, 2007 22:01

Re: Robots list
-----------------------------------
Nice [b]novice programmer[/b] :)


-----------------------------------
ChYvI
Wed 17 Oct, 2007 02:29

Re: Robots List
-----------------------------------
Thanks for the info.


-----------------------------------
Mighty Gorgon
Tue 23 Oct, 2007 23:01

Re: Robots list
-----------------------------------
Wow, thanks for this!  :mricy:


-----------------------------------
tonyf12
Fri 26 Oct, 2007 21:54

Re: Robots List
-----------------------------------
Thanks for the list.


-----------------------------------
novice programmer
Mon 05 Nov, 2007 23:49

Re: Robots List
-----------------------------------
This is the last part


-----------------------------------
amakusa
Thu 29 Nov, 2007 19:43

Re: Robots List
-----------------------------------
Thank you !! 

very usefull info.

 :mrblue:


-----------------------------------
wizzard
Sat 05 Jan, 2008 15:00

Re: Robots List
-----------------------------------
Cheers mate. Hope this will provide with some indepth info on the topic.


-----------------------------------
SilentDefender
Tue 06 Jan, 2009 18:20

Re: Robots List
-----------------------------------
Thanks, could do with something like this


-----------------------------------
novice programmer
Sun 11 Jan, 2009 00:25

Re: Robots List
-----------------------------------
You can add the robots there to your ctracker blocker by the http agent the requests use.

I am working around SQL queries.....


-----------------------------------
EFN
Wed 28 Jan, 2009 23:23

Re: Robots List
-----------------------------------
[quote user="novice programmer" post="35573"]You can add the robots there to your ctracker blocker by the http agent the requests use.

I am working around SQL queries.....[/quote]

Thanks man! Nice Work!
;)


-----------------------------------
stunnah83
Wed 28 Jan, 2009 23:32

Re: Robots List
-----------------------------------
This can be very useful, thanks for this  :mrviolet:


-----------------------------------
kalkulatorkredytowy
Sat 14 Jul, 2012 14:23

Re: SEO - Robots List
-----------------------------------
Thank you so much for this list. I have been struggling from keeping spammers away.


