BotReports.com - Crawlers en Tue, 07 Apr 2015 02:00:08 UT Big Medium 2.0.8 http://blogs.law.harvard.edu/tech/rss http://www.botreports.com/crawlers/index.shtml ArchitextSpider Fri, 14 Mar 2014 07:39:26 UT http://www.botreports.com/crawlers/architextspider.shtml 34312d27bfacc165896de33aa3284e55-548 User-Agent Crawlers Bots Starting With A AboutUsBot <div>AboutUsBot/VERSION (PURPOSE; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)</div> <div>&#160;</div> <div>AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)</div> Mon, 10 Mar 2014 06:45:13 UT http://www.botreports.com/crawlers/aboutusbot.shtml 34312d27bfacc165896de33aa3284e55-546 User-Agent Crawlers Bots Starting With A VoilaBot Sun, 09 Mar 2014 17:13:08 UT http://www.botreports.com/crawlers/voilabot.shtml 34312d27bfacc165896de33aa3284e55-545 User-Agent Crawlers Bots Starting With V 4seohuntBot Sun, 09 Mar 2014 17:11:04 UT http://www.botreports.com/crawlers/4seohuntbot.shtml 34312d27bfacc165896de33aa3284e55-544 User-Agent Crawlers Bots Starting With 4 200PleaseBot Sun, 09 Mar 2014 17:07:21 UT http://www.botreports.com/crawlers/200pleasebot.shtml 34312d27bfacc165896de33aa3284e55-543 User-Agent Crawlers Bots Starting With 2 ccbot CCBot/2.0 (http://commoncrawl.org/faq/) Fri, 07 Mar 2014 08:16:04 UT http://www.botreports.com/crawlers/ccbot.shtml 34312d27bfacc165896de33aa3284e55-542 User-Agent Crawlers Bots Starting With C gsa-crawler Thu, 06 Mar 2014 05:45:21 UT http://www.botreports.com/crawlers/gsa-crawler.shtml 34312d27bfacc165896de33aa3284e55-538 User-Agent Crawlers Bots Starting With G grapeshot <div>Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)</div> <div>&#160;</div> <div>Mozilla/5.0 (compatible; grapeFX/0.9; crawler@grapeshot.co.uk</div> Thu, 06 Mar 2014 05:17:10 UT http://www.botreports.com/crawlers/grapeshot.shtml 34312d27bfacc165896de33aa3284e55-537 User-Agent Crawlers Bots Starting With G envolk envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html) Thu, 06 Mar 2014 05:04:16 UT http://www.botreports.com/crawlers/envolk.shtml 34312d27bfacc165896de33aa3284e55-536 User-Agent Crawlers Bots Starting With E UnwindFetchor/1.0 UnwindFetchor/1.0 (+http://www.gnip.com/) Wed, 05 Mar 2014 06:41:19 UT http://www.botreports.com/crawlers/unwindfetchor10.shtml 34312d27bfacc165896de33aa3284e55-534 User-Agent Crawlers Bad Bots Bots Starting With U WebDataCentreBot <p>Mozilla/5.0 (compatible; WebDataCentreBot/1.0; http://WebDataCentre.com/)</p> <p>Mozilla/5.0 (compatible; WebDataCentreBot/1.0; http://WebDataCentre.com/)</p> Wed, 05 Mar 2014 06:25:40 UT http://www.botreports.com/crawlers/webdatacentrebot.shtml 34312d27bfacc165896de33aa3284e55-533 User-Agent Crawlers Bots Starting With W city review Cityreview Robot (+ http://www.cityreview.org/crawler/) Wed, 05 Mar 2014 06:22:45 UT http://www.botreports.com/crawlers/city-review.shtml 34312d27bfacc165896de33aa3284e55-532 User-Agent Crawlers Bots Starting With C Speedy Spider Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/) Wed, 05 Mar 2014 06:19:57 UT http://www.botreports.com/crawlers/speedy-spider.shtml 34312d27bfacc165896de33aa3284e55-531 User-Agent Crawlers Bots Starting With S NaverBot Wed, 05 Mar 2014 06:15:33 UT http://www.botreports.com/crawlers/naverbot.shtml 34312d27bfacc165896de33aa3284e55-529 User-Agent Crawlers Bots Starting With N Aboundexbot Wed, 05 Mar 2014 04:41:22 UT http://www.botreports.com/crawlers/aboundexbot.shtml 34312d27bfacc165896de33aa3284e55-528 User-Agent Crawlers Bots Starting With A