==================================================================================

==========

Sumber :

https://github.com/mthcht/awesome-lists/blob/main/Lists/suspicious_http_user_agents_list.csv

==================================================================================
==========

Lemon-Duck-*

Mozilla/5.0 (*-bit) dnstwist

unknown

LocusSoftware, NetInstaller

* (Linux; *

* (X11; *

* (X11; *Firefox*

* MASP)*

* PowerShell/*

* Toolbar *

* Wyzo/*

* masscan/*

*${jndi:ldap://*

*(Hackintosh*

*(Nikto/*

*-parrot-686*

*-parrot-amd64*

*/.insufficient/.*

*/.josephine/.*

*/.justice/.*

*/.snventor/.*

*/bin/bash*

*/etc/passwd*

*127.0.0.1*

*; Linux x86_64*
*; echo $*

*; googleweblight)*

*Arachni/*

*Arch Linux*

*ArchLinux*

*C:\Users\*

*CPython/*

*EICAR-STANDARD-ANTIVIRUS-TEST-FILE*

*HKCU\Software\Microsoft\Windows\*

*HeadlessChrome*

*HeadlessEdg*

*Kali linux*

*Mozilla/4.0 (Hydra)*

*NDES client *

*Nmap Scripting Engine*

*OpenVAS*

*QQDownload*

*SELECT*FROM*WHERE*

*SELECT*FROM*Win32_*

*Schtasks /create*

*SearchToolbar*

*UCCAPI/16.0.13328.20130 OC/16.0.13426.20234*

*Windows NT 123.9*

*WindowsPowerShell/*

*X11; TAILS;*

*YAYAYAY*

*Your Moms Smart Vibrator*

*\Microsoft Windows *\Windows Defender\*

*burpcollaborator.net*

*cmd.exe*

*kali1-686*
*kali1-amd64*

*nmap icap-client/*

*powershell.exe*

*wget http*

*yahoomailproxy*

.net backdor

01h4x.com

0xa10xa1HttpClient

1 space

123

1,23513E+12

1,25122E+11

2 spaces

20112211

23591

360Spider

404checker

404enemy

404search

4M5yC6u4stom5U8se3r

5.1 ...

80legs

9,01785E+11

91castInstallKernel

???

ADmantX

AE632AE3-FACB-4C1B-8906-FB65A13B01B4

AIBOT

ALIZER

ALittle\ Client
API-Guide test program

ASPSeek

AV1

AV2010

AVP2006IE

AYAYAYAY1337

Abonti

Aboundex

Aboundexbot

Access down

Accessing

Acunetix

AdVantage

AdiseExplorer

AdsTxtCrawlerTP

AfD-Verbotsverfahren

AgavaDwnl

Agent and 5 or 6 digits

AhrefsBot

AiHitBot

Aipbot

Alawar Toolbar

Aldi Bot

Alexibot

AllSubmitter

Alligator

AlphaBot

Anarchie

Anarchy

Anarchy99

Ankit
Anthill

AntiSpyware

AntiVermeans

AntiVerminser

AntiVirGear

AntivirXP

AnyDesk/*

AnyDesk

Apache (compatible...

Apexoo

App4

Apropos

AskBar

AskPBar

AskSearchAssistant

AsmUpdater

Aspiegel

Asteria md5

Asterias

Atomseobot

Attach

AutoDL\/1.0

AutoHotkey

AwarioRssBot

AwarioSmartBot

B Register

B1D3N_RIM_MY_ASS

BBBike

BDCbot

BDFetch
BLEXBot

BackDoorBot

BackStreet

BackWeb

Backlink-Ceck

BacklinkCrawler

Badass

Bandit

Barkrowler

BatchFTP

Battleztar\ Bazinga

BetaBot

BigFoot

BitTorrent/*

Bitacle

BlackWidow

Black\ Hole

Blackboard

Blah/2

Blow

*AAAAAAAAAA*

*=

Q2hyb21l*

QXBwbGVXZWJLaX*

RGFsdmlr*

TW96aWxsY*

rclone/v*

XMRig *

ccminer*

BlowFish

BndDriveLoader
BndVeano4GetDownldr

BoBrowser

Boardreader

Bolt

BotALot

BrandThunderHelper

Brandprotect

Brandwatch

Buck

Buddy

BuiltBotTough

BuiltWith

Bullseye

Bundle

BunnyLoader

BunnyLoader_Dropper

BunnyRequester

BunnyShell

BunnySlippers

BunnyStealer

BunnyTasks

BuzzSumo

Bytespider

C slash

CATExplorador

CCBot

CFS Agent

CFS_DOWNLOAD

CODE87

CPUSH_UPDATER

CS Fingerprint Module
CSHttp

CTTBasic

C\\WINDOWS\\system32\\NetLogom.exe

Calculon

Callstranger Vulnerability Checker

Cat

CazoodleBot

Cegbfeieh

CensysInspect

Charon/Inferno

CheTeam

CheeseBot

CherryPicker

ChilkatUpload

ChinaClaw

Chlooe

Chnome

Citoid

Claritybot

CleancopUpdate

Clever Internet Suite

ClickAdsByIE

Cliqzbot

Cloud\ mapping

Cocolyzebot

Cogentbot

Collection Info

Collector

CommonName Agent

Connector v1.2

Coolstreaming Tool-Bar
Copier

CopyRightCheck

Copyscape

Cosmos

Cr3dOv3r-Framework

Craftbot

Crawling\ at\ Home\ Project

CrazyBro

CrazyWebCrawler

Crescent

CrunchBot

Curious

Custo

CustomExchangeBrowser

CustomSpy

CyotekWebCopy

DARecover

DBLBot

DEBUT.TMP

DIALER

DIIbot

DInstaller2

DNS Extractor

DSInstall

DSearch

DTS\ Agent

DataCha0s

DatabaseDriverMysqli

Demon

Deusu

Devil
Digincore

DigitalPebble

DirBuster-*

Dirbuster

Disco

Discobot

Discoverybot

Dispatch

DittoSpyder

DnBCrawler-Analytics

DnyzBot

Docker-Desktop/*

DoctorVaccine

DomCopBot

DomainAppender

DomainCrawler

DomainSigmaCrawler

DomainStatsBot

Domains\ Project

Dotbot

Downing

Download Agent

Download App

Download Master

Download UBAgent

DownloadMR

Download\ Wonder

Downloader MLR 1.0.0

DrPCClean Transmit

Dragonfly

Drip
Dsreg/10.0 (Windows 10.0.19044.1826)

DuckTales

Dummy

ECCP/1.0

EELoader

EI

EMSCBVDFRT

EMail\ Siphon

EMail\ Wolf

ERRN2004 (Windows XP

ERRORNUKER

ESB

EVNUKER

EasyDL

Ebingbong

Ecxi

EirGrabber

ElectroSun

Envolo

EroCrawler

ErrCode

ErrorSafe

Errordigger.com related

Evil

Exabot

Example

Explorer

Express\ WebPictures

ExtLinksBot

Extractor

ExtractorPro
Extreme\ Picture\ Finder

EyeNetIE

Ezooms

Ezshop

FDM

FHscan

FUCKUSA

FULLSTUFF

FaceCooker

Fast Browser Search

FavUpdate

FemtosearchBot

FileDownloader

FileNolja

Fimap

Fire-Cloud

Firefox

Firefox/7.0

FlashGet

Flunky

Foobot

Forthgoer

Forthgoner

Freeuploader

FreezeInet

FrontPage

FunWebProducts

Fuzz

FyberSpider

Fyrebot

G-i-g-a-b-o-t
GAMEHOUSE.NET.URL

GETJOB

GLOBALx

GNU Wget*

GOOGLE

GPTBot

GT::WWW

GTBank

GUIDTracker

GalaxyBot

GameInfo

Gbot

GeekingToTheMoon

General Antivirus

GenericHttp/VER_STR_COMMA

Genieo

GermCrawler

GetRight

GetWeb

Getintent

Gigabot

GitHubCopilotChat/*

Go!Zilla

Go-Ahead-Got-It

GoZilla

Gootkit HTTP Client

Gotit

GrabNet

Grabber

Grafula

GrapeFX
GrapeshotCrawler

GridBot

GunnaWunna

GunnaWunnaBlueTips

HEADMasterSEO

HELLO

HMView

HTMLparser

HTTP Downloader

HTTP

HTTP::Lite

HTTPFILEDOWN

HTTPGETDATA

HTTPREAD

HTTPTEST

HTTP_CONNECT

HTTP_CONNECT_

HTTP_CONNECT_2

HTTP_FILEDOWN

HTTP_GET_COMM

HTTP_Query

HTTrack

Haansoft

HaosouSpider

HardCore Software For

Harvest

Havij

HaxerMen

HeartBeat_Sender

HelpSrvc

HelperH
Hloader

HonoluluBot

Hotbar

Http Down

HttpDownload

Huai_Huai

Humanlinks

HybridBot

I'm a mu mu mu ?

IBSBand

IDBTE4M

IDBot

IE

IE/1.0

IEMGR

IEP

IEToolbar

IE_6.0

IM Download

IM Downloader

INet

IOInstall

IOKernel

IRLbot

ISMYIE

ISUpd

ISecu

Iblog

Id-search

IlseBot

Image\ Fetch
Image\ Sucker

InHold

IndeedBot

Indy\ Library

InetURL

Inet_read

InfoBot

InfoBox

InfoNaviRobot

InfoTekies

Informer from RBC

InstallCapital

Installed OK

Installer

Intelliseek

InterGET

Internet 1.0

Internet Antivirus Pro

Internet Explorer (compatible

Internet Explorer 5.01

Internet Explorer 6.0

Internet Explorer

Internet HTTP Request

Internet

InternetSeer

Internet\ Ninja

InvokeAd

Iria

Iskanie

IstellaBot

JEDI-VCL
JOC\ Web\ Spider

JamesBOT

Jbrofuzz

JennyBot

JetCar

Jetty

JikeSpider

Joomla

Jorgee

JustView

Jyxobot

KKTone

KRMAK

KRSystem

Kenjin\ Spider

Keybot\ Translation-Search-Machine

Keyword\ Density

Kinza

Kozmosbot

Kvadrlson 1.0

LNSpiderguy

LWP::Simple

Lanshanbot

Larbin

Leap

LeechFTP

LeechGet

Lemon-Duck-*

LexiBot

Lftp

LibWeb
Libwhisker

LieBaoFast

Lightspeedsystems

Likse

Lilith-Bot*

LinkScan

LinkWalker

Linkbot

LinkextractorPro

LinkpadBot

LinksManager

LinqiaMetadataDownloaderBot

LinqiaRSSBot

LinqiaScrapeBot

Lipperhey

Lipperhey\ Spider

Litemage_walker

Live Enterprise Suite

LmaokaazLdr

Lmspider

Loands

Lobo Lunar

LockXLS

Locus NetInstaller

LogEvents

Loki/1.0

Ltx71

M0zilla

MBVDFRESCT

MFC_Tear_Sample

MIDown\ tool
MJ12bot

MS Internet Explorer

MSIE7 na

MSIECrawler

MTRobot

MYURL

Made by UltimateHackerzTeam

MadeByLc

Mag-Net

Magic NetInstaller

Magnet

Mail.RU_Bot

Majestic-SEO

Majestic12

Majestic\ SEO

MalwareWipe

MalwareWiped

MarkMonitor

MarkWatch

Mass\ Downloader

Masscan

Mata\ Hari

MauiBot

Mb2345Browser

Mbar

MeanPath\ Bot

MediaLabsSiteInstaller

Mediatoolkitbot

MegaIndex.ru

Megaupload

Metauri
MicroMessenger

Microgaming Install Program

Microsoft BITS/*

Microsoft Internet Browser

Microsoft Internet Explorer 6.0

Microsoft Internet Updater

Microsoft WinRM Client

Microsoft-CryptoAPI/*

Microsoft-WebDAV-MiniRedir/*

Microsoft\ Data\ Access

Microsoft\ URL\ Control

Minefield

Mirar_KeywordContent

Mirar_Toolbar

Mister\ PiX

Moblie Safari

Mojeek

Mojolicious

MolokaiBot

Momentum

MoonLight

Morfeus\ Fucking\ Scanner

Morpheus

Mozil1a

Mozila

Mozilla (libwhisker/*

Mozilla 1.02.45 biz

Mozilla FireFox

Mozilla-web

Mozilla/0.xx

Mozilla/3.0 (compatible
Mozilla/4.0 (SP3 WINLD

Mozilla/4.0 (SPGK

Mozilla/4.0 (compatible ICS

Mozilla/4.0 (compatible MSIE 7.0 na .NET CLR 2.0.50727 .NET CLR 3.0.4506.2152 .NET CLR 3.5.30729

Mozilla/4.0 (compatible

Mozilla/4.8 ru

Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko

Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.0

Mozilla/6.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/27.0 Iceweasel/25.3.0

Mozillar

Mozlila

Mr.4x3

MsgPlus3

Msrabot

Musobot

My Session

MyAgent

MyCustomUser

MyIE/1.0

MySearch

MyWay

MyWebSearch

Mz

MzApp

N1

NICErsPRO

NOPE

NPbot

NSIS_DOWNLOAD

NULL

Name\ Intelligence
Nameprotect

NateFinder

NavHelper

Navroad

NearSite

Needle

Nessus

NetAnts

NetInstaller

NetLyzer

NetMechanic

NetSpider

NetZIP

Net\ Vampire

Netcraft

Nettrack

Netvibes

NextGenSearchBot

Nibbler

Niki-bot

Nikto

NimbleCrawler

Nimbostratus

Nimo Software HTTP Retriever 1.0

Ninja

Nmap

Nuclei

Nutch

OK

ONANDON

OSSProxy
Octopus

Oemji

Offline\ Explorer

Offline\ Navigator

OnCrawl

OnionWClient / 1.0

OnionWClient

Open3

OpenLinkProfiler

OpenPage

OpenVAS

Openfind

Openvas

Opera/8.89

OrangeBot

OrangeSpider

Our_Agent

OutclicksBot

OutfoxBot

PATCHER

PCClearPlus

PCDoc11

PECL::HTTP

PHPCrawl

POE-Component-Client-HTTP

PTS

PWMI/1.0

PageAnalyzer

PageGrabber

PageScorer

PageThing.com
Page\ Analyzer

Pandalytics

Panscient

Papa\ Foto

Pavuk

PcPcUpdater

PcapXray

PeoplePal

Petalbot

Pi-Monster

Picscout

Picsearch

PictureFinder

Piepmatz

Pimonster

PinballCorp

Pivim Multibar

Pixray

PleaseCrawl

Pockey

Poker

PologiyKolokol

PopupBlockade/1.63.0.2/Reg

Powered By 64-Bit Alpha Processor

Presto

ProPowerBot

ProWebWalker

Probethenet

Proximic

ProxyDown

Psbot
Pu_iN

Pump

PxBroker

PyCurl

PycURL/*

Python/*

QQ

QQGame

QdrBi Starter

QueryN\ Metasearch

Quick-Crawler

Qvod

RBR

REKOM

RFRudokop v.1.1 account verification

ROGUE

RSSingBot

RX Bar

RangeCheck/0.1

RankActive

RankActiveLinkBot

RankFlex

RankingBot

RankingBot2

Rankivabot

RankurBot

Re-re

ReGet

ReadFileURL

RealDownload

Reaper
RebelMouse

Recorder

RedesScrapy

Releasexp

RepoMonkey

Report Runner

Rescue/9.11

Revolution Win32

RichCasino

Ripper

RocketCrawler

Rogerbot

SAH

SBIder

SBTCM

SEOkicks

SEOkicks-Robot

SEOlyticsCrawler

SEOprofiler

SEOstats

SERVER2_03

SF Installer

SHINI

SISTRIX

SK

SMTBot

SRInstaller

SRRecover

STBHOGet

STEROID Download

SUiCiDE/1.5
SalesIntelligent

Save

ScanAlert

Scanbot

ScoutJet

Scrapy

Scrapy/*

Screaming

ScreenerBot

ScrepyBot

SearchToolbar

Searchestate

SearchmetricsBot

Seekmo Toolbar

Seekport

SeekportBot

SemanticJuice

Semrush

SemrushBot

SentiBot

SenutoBot

SeoSiteCheckup

SeobilityBot

Seomoz

Session

SexTrackerWSI

Shodan

Sickloader

SideStep

Sidebar Client

Sidesearch
Siphon

SiteCheckerBotCrawler

SiteExplorer

SiteLockSpider

SiteSnagger

SiteSucker

Site\ Sucker

Sitebeam

Siteimprove

Sitevigil

Skype

Skypee

SlySearch

SmartDownload

SmartInstaller

SmartLoader

Sme32

Smileware Connection

Snake

Snapbot

Snatch-System

SnoopStick Updater

Snoopy

SocialRankIOBot

Sociscraper

SogouExplorerMiniSetup

SogouIMEMiniSetup

Sogou\ web\ spider

SomeTimes

Sosospider

Sottopop
SpaceBison

SpamBlockerUtility x.x.x

Spammen

SpankBot

Spanner

Spbot

SpeedRunner

Spinn3r

Sprout Game

SputnikBot

SpyDawn

SpyHealer

SpyLocked

Sqlmap

Sqlworm

Sqworm

Ssol NetInstaller

Steeler

Stripper

Sucker

Sucuri

Suggestion

SunShine

SunShineMoonLight

SuperBot

SuperHTTP

Surfbot

SurferPlugin

SurveyBot

Suzuran

Swiftbot
Swizz03r

Szukacz

T0PHackTeam

T8Abot

TALWinInetHTTPClient

TBONAS

TPSystem

TSA

TakeMyPainBack

Teleport

TeleportPro

Telesoft

Telesphoreo

Telesphorep

TestAgent

TheNomad

The\ Intraformant

Thumbor

TightTwatBot

Titan

Toata

Toolbar

Toweyabot

Tracemyfile

Travel Update

Trendiction

Trendictionbot

Trojan.Hijack.IrcBot.457 related

True_Robot

TryMedia_DM_2.0.0

Turingos
Turnitin

TurnitinBot

TwengaBot

Twice

Typhoeus - https://github.com/typhoeus/typhoeus

Typhoeus

U2Clean

UCmore

UDonkey

URLy.Warning

URLy\ Warning

USER_CHECK

UbrenQuatroRusDldr 096044

Ucheck

Ultimate Fixer

UnisterBot

Unknown

Update Internet Antivirus

Update1.0

Updater

Upflow

User Agent

User-Agent Mozilla/4.0 (compatible

User-Agent User-Agent

UtilMind HTTPGet

V-BOT

VB OpenUrl

VB\ Project

VCI

VHIbot/1.0

VMozilla
VP-EYE Downloader

VaccineKillerIU

Vacuum

Vagabondo

Valve/Steam HTTP Client

Varlok_11000

VelenPublicWebCrawler

VeriCiteCrawler

Version 1.23

VersionDwl

VidibleScraper

VirtualBox *

VirusHeat 4.3

VirusProtectPro

Virusdie

VisaoAPP

Visicom

VoidEYE

Voil

Voltron

VombaProductsInstaller

Vulners NMAP Plugin *

Vulture

WASALive-Bot

WBSearchBot

WEBDAV

WINDOWS_LOADS

WISENutbot

WPScan *

WPScan

WSHRAT*
WT Games App

WTA

WTInstaller

WTRecover

WT_GET_COMM

WWW-Collector-E

WWW-Mechanize

WWW::Mechanize

WWWOFFLE

Wallpapers

Wallpapers/3.0

WallpapersHD

WeSEE

WebAuto

WebBandit

WebCollage

WebCopier

WebDownloader

WebEnhancer

WebFetch

WebForm 1

WebFuck

WebGo\ IS

WebImageCollector

WebLeacher

WebPix

WebReaper

WebSauger

WebStripper

WebSucker

WebWhacker
WebZIP

Web\ Auto

Web\ Collage

Web\ Enhancer

Web\ Fetch

Web\ Fuck

Web\ Pix

Web\ Sauger

Web\ Sucker

Webalta

WebmasterWorldForumBot

Webshag

WebsiteExtractor

WebsiteQuester

Website\ Quester

Webster

Wfuzz/*

Whack

Whacker

Whatweb

Who.is\ Bot

WidgiToolbar-

Widow

Win95

WinButler

WinFixMaster

WinHTTP Example/1.0

WinHTTP*

WinHTTrack

WinHttpRequest

WinProxy
WinSoftware

WinTouch

WinXP Pro Service Pack 2

Windoss

Windows 10

Windows 7

Windows 8

Windows Explorer

Windows Updates Manager

Windows XP

Windows+NT+5.x

WindowsNT

WindowsShellClient/*

Windows_Installer

Wireshark/*

WiseGuys\ Robot

Wonderbot

Woobot

Wotbox

Wprecon

XXXwww

Xaldon\ WebSpider

Xaldon_WebSpider

Xenu

XieHongWei-HttpDown/2.0

Xmlst

XupiterToolbar

Yandesk

Yodao Desktop Dict

YoudaoBot

YourScreen123
ZAP

ZC XML-RPC C++ Client

ZC-Bridgev26

ZCOM Software

ZLoad-Software

Zadanie

Zade

Zauba

Zermelo

Zeus

Zilla

Zitebot

Zload

ZmEu

ZoomBot

ZoominfoBot

ZumBot

ZyBorg

_TEST_

aaaa

aaaabbb

abcd

ad-protect

adlib

adscanner

adsntD

aiohttp/*

alertup

altera forma

angel

antispyprogram
anycleaner

archive.org_bot

arquivo-web-crawler

arquivo.pt

atsu

auctionplusup

autoemailspider

aw v3

b5c3d0b28619de70bf5588505f4061f2

backlink-check

bdsclk

blahrx

boostsoftware-urlexists

browserbob.com

c \windows

c010101

cah.io.community

check1.exe

checkonline

chek

clark-crawler

coccocbot

cognitiveseo

com.plumanalytics

condi-bbos

contains loader

count

counters

crawl.sogou.com

crawler.feedback

crawler4j
cso

curl/*

dataforseo.com

dataforseobot

dbcount

default

demandbase-bot

descriptor but no string

dialno

dirhunt

doctorpro1

domainsproject.org

doshowmeanad loader v2.1

double dashes

downloader

dvadcat

dwplayer

eCatch

ekeoil

evc-batch

f**king

facebookscraper

fetcher

fian3manager

fs3update

get_site1

gettingAnswer

gh2008

giftz

globalupdate

go-diva
gobuster*

gomtour

google/dance

gopher

hacker

hbtools

hello crazyk

heritrix

hhh

hi

host

http-get-demo

i-scan

iDownloadAgent

iMeshBar

iMightJustPayMySelfForAFeature

iWin GameInfo Installer Helper

ic Windows NT 5.1 MSIE 6.0 Firefox/ Def

ieagent

iefeatsl

ieguideupdate

iexplore

indy

inetinst

instabid

install_s

internetVista\ monitor

ips-agent

isitwp.com

istsvc

iubenda-radar
kpangupdate

libsfml-network

libtorrent/*

lineguide

linkdexbot

lsosss

lwp-request

lwp-trivial

magpie-crawler

masscan/*

mc_v1.2.6

mediawords

medusa-stealer

mez

mfo4engo2m

microsoft

miip

mozzzzzzzzzzz

mrgud

ms

msIE 7.0

ms_ie

muhstik-scan

mypcdoctor

needit

neonabyupdate

netEstate\ NE\ Crawler

nethelper

nguideup

nimplant

NimPlant C2 Client
oBot

okcpmgr

openai

openai.com

opera

page\ scorer

pcBrowser

pint_agency

plumanalytics

polaris\ version

popup

probe-image-size

python-requests/*

python-requests/

qBittorrent/*

qwrqrwrqwrqwr

r00ts3c-owned-you

rc2.0/client

record

rhyno321

ripz

rqwrwqrqwrqw

ruler

runPatch.html

runUpdater.html

s1z.ru

satoristudio.net

scalaj-http

scan.lol

searchengine

searchtoolup
seobility

seocompany.store

seoscanners

seostar

serpstatbot

sexsearcher

sickness29a/0.1

single dash

sitechecker.pro

siteripz

skw00001

slayer

sogouspider

sp_auditbot

spyfu

spywareaxe

sqlmap/*

sureseeker

svchost

sysscan

szNotifyIdent

tAkeOut

thnall

trendiction.com

trendiction.de

uTorrent/*

ubermetrics-technologies.com

umbra

up2dash updater

update

updatesodui
uploader

vaccine

vb wininet

vikiller ctrl...

viruscheck

voyagerx.com

wbi_v0.90

webcount

webfile

webgains-bot

webmeup-crawler

webpros.com

webprosbot

wget 3.0

wget(*

wget

wget/*

windsoft

winlogon

x09

x09Mozilla

x22Mozilla

xehanort321

xmX*-AhKih-I-i5enQvO

xmlset_roodkcableoj28840ybtide

xpymep1.exe

xr - Worm.Win32.VB.cj related

xxx

zauba.io
zgrab

C2FunctionAgent

aws-cli/*

TeslaBrowser/5.5

SouthSide

MrBidenNeverKnow

Mozilla/4.0 (compatible; MSIE 8.0; Win32)

--------------------------------------------------------------------------------------------------------------------------------------
-------------------

===========================================================================

Sumber :

https://github.com/JayBizzle/Crawler-Detect/blob/master/tests/crawlers.txt

===========================================================================

TinEye-bot/0.51 (see http://www.tineye.com/crawler.html)

Whoismindbot/1.0 (+http://www.whoismind.com/bot.html)

Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.3; www.alertra.com)

R6_FeedFetcher(www.radian6.com/crawler)

CheckHost (http://check-host.net/)

YandeG 1.03

yacybot (/global; amd64 Linux 3.16-0.bpo.2-amd64; java 1.7.0_65; Europe/en)

http://yacy.net/bot.html

SEMrushBot

Protopage/3.0 (http://www.protopage.com)

yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html

StatoolsBot (+http://www.statools.com/bot.html)

adidxbot/2.0 (+http://search.msn.com/msnbot.htm)

Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)

Mozilla/5.0 (Linux; Android 5.0; Nexus 5 Build/LRX21O) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/46.0.2490.76 Mobile Safari/537.36 PTST/281

TurnitinBot (https://turnitin.com/robot/crawlerinfo.html)

Scrapy/1.0.5 (+http://scrapy.org)
yacybot (/global; amd64 Linux 4.4.0-31-generic; java 1.8.0_91; Europe/en) http://yacy.net/bot.html

XmlSitemapGenerator - http://xmlsitemapgenerator.org

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) Match by Siteimprove.com

Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)

WatchMouse/18990 (http://watchmouse.com/ ; gab)

WatchMouse/8.4.0.3 (http://watchmouse.com/ ; hkhkg02.watchmouse.net)

Mozilla/5.0 (compatible; LinkpadBot/1.06; +http://www.linkpad.ru)

Mozilla/5.0 (compatible; heritrix/1.14.2 +http://rjpower.org)

yacybot (webportal/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en)

http://yacy.net/bot.html

PercolateCrawler/4 ([email protected])

msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)

Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daum 4.1

Mozilla/5.0 (compatible; spbot/4.0.3; +http://www.seoprofiler.com/bot )

Mozilla/5.0 (compatible; LoadTimeBot/0.9; +http://www.loadtime.net/bot.html)

UnwindFetchor/1.0 (+http://www.gnip.com/)

nrsbot/5.0(loopip.com/robot.html)

SemrushBot/0.9

Mozilla/5.0 (compatible; UASlinkChecker/2.1; +https://udger.com/support/UASlinkChecker)

yacybot (/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; America/en)

http://yacy.net/bot.html

Netvibes (http://www.netvibes.com)

Acoon v4.1.0 (www.acoon.de)

msnbot/2.0b (+http://search.msn.com/msnbot.htm).

Mozilla/5.0 (compatible; ltbot/3.2.0.10 +http://www.kdsl.tu-darmstadt.de/de/kdsl/research-

program/crawling-and-semantic-structuring/)

smart.apnoti.com Robot/v1.34 (http://smart.apnoti.com/en/aboutApnotiWebCrawler.html)

HubSpot Links Crawler 2.0 http://www.hubspot.com/

yacybot (/global; x86 Windows XP 5.1; java 1.7.0_55; Europe/de) http://yacy.net/bot.html

Mozilla/5.0 (compatible; DCPbot/1.0; +http://domains.checkparams.com/)

Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)

WebDoc (abuse-webdoc at service.moquadv.com)

coccoc

Mozilla/5.0 (compatible; spbot/4.0b; +http://www.seoprofiler.com/bot )

Mozilla/5.0 (compatible; ExchangleBot/3.0; +https://www.exchangle.com/exchangling)

Mozilla/5.0 (compatible; Qwantify/2.0n; +https://www.qwant.com/)

OOZBOT/0.20 ( Setooz výrazný ako say-th-uuz, znamená mosty. ;

http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )

SpeedySpider - http://www.entireweb.com

Mozilla/5.0 (compatible; Heurekabot/3.1; +http://sluzby.heureka.cz/)

crawler for netopian (http://www.netopian.co.uk/)

L.webis/0.51 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)

Influencebot/0.9; (Automatic classification of websites; http://www.influencebox.com/;

[email protected])

Baiduspider+(+http://www.baidu.com/search/spider.htm)

Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1)

Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.28) Gecko/20120306 Firefox/99.0

YottaaMonitor

Scrapy/1.1.1 (+http://scrapy.org)

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/27.0.1453.116 Safari/537.36 HubSpot Marketing Grader

RyzeCrawler/1.1.1 ( http://www.domain2day.nl/crawler/)

eBot / v.1.0a (http://alfa.elchron.cz)

Sogou News Spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)

Crowsnest/0.5 (+http://www.crowsnest.tv/)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed
Insights) Chrome/22.0.1229 Safari/537.4

DoCoMo/2.0 N902iS(c100;TB;W24H12)(compatible; moba-crawler; http://crawler.dena.jp/)

Yeti/1.1 (NHN Corp.; http://help.naver.com/robots/)

Experibot_v1

Nekstbot - http://www.ipipan.waw.pl/nekst/nekstbot/

Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)

Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101

Safari/537.36; SSL-Crawler: http://crawler.dcsec.uni-hannover.de

CorpusCrawler 2.0.19 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus

Mozilla/5.0 (compatible; Semager/1.4; http://www.semager.de/blog/semager-bots/)

Mozilla/5.0 (compatible; dlcbot/0.1; +http://www.drlinkcheck.com/)

yacybot (freeworld/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; Europe/de)

http://yacy.net/bot.html

A6-Indexer/1.0 (http://www.a6corp.com/a6-web-scraping-policy/)

CopperEgg/RevealUptime/DallasTX(linode)

Mozilla/5.0 (compatible; Qwantify/2.1dw; +https://www.qwant.com/)/*

Curious George - www.analyticsseo.com

GozaikBot (www.gozaik.com;[email protected];www.gozaik.com/gozaikbot.html)

Pu_iN Crawler (+http://semanticjuice.com/)

Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev;

+http://www.openindex.io/en/webmasters/spider.html; systemsATopenindexDOTio)

yacybot (freeworld/global; i386 Linux 3.0.0-17-generic-pae; java 1.6.0_23; Europe/en)

http://yacy.net/bot.html

yacybot (-global; amd64 FreeBSD 9.2-RELEASE-p10; java 1.7.0_65; Europe/en)

http://yacy.net/bot.html

FAST Enteprise Crawler/6 (www dot fastsearch dot com)

Mozilla/5.0 (compatible; parsijoo-update-crawler; +http://www.parsijoo.ir/;

[email protected])

yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_25; Europe/en)
http://yacy.net/bot.html

Inspingbot/1.0 (+https://www.insping.com/)

OrgProbe/0.9.4 (+http://www.blocked.org.uk)

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider

Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/276

Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/42.0.2311.90 Safari/537.36 PTST/276

Mozilla/5.0 (compatible; SemrushBot/0.97~bl; +http://www.semrush.com/bot.html)

Feedspot http://www.feedspot.com

Zookabot/2.0;++http://zookabot.com

drupact/0.7; http://www.arocom.de/drupact

RobotsChecker/0.6 (+http://www.blocked.org.uk)

yacybot (/global; i386 Linux 3.13.0-37-generic; java 1.7.0_65; Europe/en) http://yacy.net/bot.html

Scrapy/1.0.3 (+http://scrapy.org)

Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot)

minicrawler/4.0.0~beta12

Slack-ImgProxy 0.66 (+https://api.slack.com/robots)

Mozilla/5.0 (compatible; WebCookies/1.0; +https://webcookies.org/faq/#agent)

NalezenCzBot/1.0 (http://www.nalezen.cz)

yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_03; Europe/en)
http://yacy.net/bot.html

Mozilla/5.0 (compatible; AhrefsBot/5.1; +http://ahrefs.com/robot/)

yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_51; Europe/de)

http://yacy.net/bot.html

yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_65; Europe/de) http://yacy.net/bot.html

Mozilla/4.0 compatible ZyBorg/1.0 ([email protected]; http://www.WISEnutbot.com)

Mozilla/5.0 (en-us) AppleWebKit/537.36(KHTML, like Gecko; Google-Adwords-DisplayAds-

WebRender;) Chrome/27.0.1453Safari/537.36

Crawler powered by contentDetection (www.mindup.de)

PagePeeker.com (info: http://pagepeeker.com/robots)

Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; +http://go.mail.ru/help/robots)

coccoc/1.0 (http://help.coccoc.com/)

Favicon downloader (+https://favico.be/bot.html)

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) ([email protected])

Mozilla/5.0 (compatible; DIY-SEOBot/0.1a; +http://www.upcity.com/bot.html)

Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/49.0.2623.112 Safari/537.36 PTST/276

Mozilla/5.0 (compatible; theoldreader.com)

Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)

hawkReader/1.8 (Link Parser; http://www.hawkreader.com/; Allow like Gecko) Build/f2b2566

Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.webarchiv.cz)

Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)

yacybot (/global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.7.0_75; Europe/en)

http://yacy.net/bot.html

Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)

yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_45; Europe/de) http://yacy.net/bot.html

larbin_2.6.3 [email protected]

yacybot (freeworld/global; amd64 Linux 3.10.17-gentoo; java 1.7.0_45; UTC/en)

http://yacy.net/bot.html

MeMoNewsBot/2.0 (http://www.memonews.com/en/crawler)

Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/281

Mozilla/5.0 (compatible; DCPbot/1.5; +http://domains.checkparams.com/)

Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html)

BlackBerry9000/4.6.0.167 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 ips-agent

Mozilla/5.0 (compatible; Baiduspider-cpro; +http://www.baidu.com/search/spider.html)

yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html

Mozilla/5.0 (compatible; electricmonk/3.1.1 +https://www.duedil.com/our-crawler/)

Hatena Antenna/0.5 (http://a.hatena.ne.jp/help)

Backlink-Ceck.de (+http://www.backlink-check.de/bot.html)

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)

commoncrawl.org/research//Nutch-1.7-SNAPSHOT

Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)

LapozzBot/1.4 (+http://robot.lapozz.com)

Mozilla/5.0 (WhatsMyIP.org HTTP_Compression_Test) http://whatsmyip.org/ua

Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0 (Following
Mediapartners-Google)

Mozilla/5.0 (compatible; ExaleadCloudview/6;)

ADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - [email protected]

msnbot/1.0 (+http://search.msn.com/msnbot.htm)

Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.79

Safari/537.4 LinkTiger 2.0

Pixray-Seeker/1.1 (Pixray-Seeker; [email protected])

mbot v.1.16

MXT/Nutch-1.12-SNAPSHOT (http://t.co/GSRLLKex24; informatique at mixdata dot com)

Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0.0

polybot 1.0 (http://cis.poly.edu/polybot/)

Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtweb01t.yandex.ru)

Mozilla/5.0 (compatible; memoryBot/1.21.46 +http://internetmemory.org/en/)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko)
Version/8.0.2 Safari/600.2.5 (Applebot/0.1)

yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_0; Europe/en) http://yacy.net/bot.html

yacybot (/global; x86 Windows 8.1 6.3; java 1.8.0_45; America/en) http://yacy.net/bot.html

Mozilla/5.0 (compatible; startmebot/1.0; +http://www.start.me/bot)

yacybot (amd64 Windows 7 6.1; java 1.6.0_21; Europe/fr) http://yacy.net/bot.html

yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_101; Asia/ru) http://yacy.net/bot.html

yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; Europe/de)

http://yacy.net/bot.html

alexa v0.1.4 (http://www.openwebspider.org/)

http://arachnode.net 1.4

Photon/1.0

NetpeakCheckerBot

GIDBot/3.0 (+http://www.gidnetwork.com/tools/gzip-test.php)

Yandex/1.01.001 (compatible; Win16; P)

w3dt.net httphr/2.0

yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_51; Europe/de) http://yacy.net/bot.html

iqdb/0.1 (+http://iqdb.org/)

Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/pages/crawler)

Motoricerca-Robots.txt-Checker/1.0 (http://tool.motoricerca.info/robots-checker.phtml)

Mozilla/5.0 (compatible; Tagoobot/3.0; +http://www.tagoo.ru)

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0 Evidon

([email protected])

Mozilla/5.0 (compatible; Nmap Scripting Engine; http://nmap.org/book/nse.html)

yacybot (i386 Linux 2.6.32-22-generic; java 1.6.0_20; Europe/de) http://yacy.net/bot.html

Mozilla/5.0 (Linux; Android 4.4.3; HTC One Build/KTU84L) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost

mindUpBot (datenbutler.de)

Mozilla/5.0 (compatible; monitis - premium monitoring service; http://www.monitis.com)

Mozilla/5.0 (compatible; RankSonicSiteAuditor/1.0; +https://ranksonic.com/ranksonic_sab.html)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111

Safari/537.36 DareBoost

Zookabot/2.1;++http://zookabot.com
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_24; Europe/de)
http://yacy.net/bot.html

Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool

+http://developers.google.com/structured-data/testing-tool/)

Mozilla/5.0 (compatible; Plukkie/1.2; http://www.botje.com/plukkie.htm)

focusbot/1.1

Mozilla/5.0 (compatible; idmarch Automatic.beta/1.2; +http://www.idmarch.org/bot.html)

CorpusCrawler 2.0.14 (http://corpora.fi.muni.cz/crawler/)

Mozilla/5.0 (compatible; XML Sitemaps Generator; https://www.xml-sitemaps.com) Gecko XML-

Sitemaps/1.0

ImplisenseBot 1.1

Promotion_Tools_www.searchenginepromotionhelp.com

Aboundex/0.2 (http://www.aboundex.com/crawler/)

CorpusCrawler 2.0.22 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus

yacybot (/global; amd64 no-os-name no-os-version; java no-java-version; Europe/en)

http://yacy.net/bot.html

LumpImageSearch/0.1 (+http://lump.co/about/bot)

Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/customer_webtxt_02.jsp)

Mozilla/5.0 (compatible; UASlinkChecker/2.0; +http://udger.com/support/UASlinkChecker)

GAChecker (+http://www.gachecker.com)

yacybot (freeworld/global; x86 Windows 7 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html

Influencebot/0.9; (Automatic classification of websites; http://www.influencebox.com/;

[email protected])User-Agent: Mozilla/5.0 (X11; Linux i686; rv:9.0) Gecko/20100101
Firefox/9.0

Experibot_v1 (https://dl.dropboxusercontent.com/u/8024465/site/Info.html)

DNSPod-reporting(http://www.dnspod.cn/reporting)

yacybot (freeworld/global; i386 Linux 3.2.0-23-generic; java 1.6.0_27; Europe/en)

http://yacy.net/bot.html

Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; +http://www.leiki.com)

PostPost/1.0 (+http://postpo.st/crawlers)

envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)

Snapbot/1.0 (Snap Shots, +http://www.snap.com)

Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/537.36 (KHTML, like
Gecko; Google-Publisher-Plugin) Chrome/27.0.1453 Mobile Safari/537.36
Mozilla/5.0 (compatible; Uptimebot/0.2.18; +http://www.uptime.com/uptimebot)

MIA DEV/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel
unhappy and do not want to be visited by our crawler send an email to [email protected];
http://spider.neofonie.de; [email protected])

Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0; PTST 2.386) like Gecko

Jyxobot/1

WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows

NT 5.1; ru-RU)

GC3pro+dir SEO Tools - Vers. 3.00b - For more informations: http://chkme.com/

Mozilla/5.0 (X11; Linux x86_64; rv:45.0; GTmetrix https://gtmetrix.com/) Gecko/20100101

Firefox/45.0

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0 AppEngine-Google;

(+http://code.google.com/appengine; appid: s~aeshortener)

Priceonomics Analysis Engine - Fetch/1.0

Ruky-Roboter (Version: 1.06, powered by www.ruky.de +http://www.ruky.de/bot.html)

Baiduspider+(+http://help.baidu.jp/system/05.html)

Openstat/0.1

Yandex/1.01.001 (compatible; Win16; m)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko)

Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)

Mozilla/5.0 (compatible; OpenindexSpider/Nutch-1.5-dev;

+http://www.openindex.io/en/webmasters/spider.html)

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Multiviewbot

CERT.at-Statistics-Survey/1.0 (http://www.cert.at/about/consec/content.html)

Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm)

yacybot (freeworld/global; x86_64 Mac OS X 10.6.8; java 1.6.0_29; Asia/ru) http://yacy.net/bot.html

gonzo/1[P] (+http://www.suchen.de/faq.html)

MixBot (+http://t.co/GSRLLKex24)

yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_60; Europe/en) http://yacy.net/bot.html

Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/

Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1
Safari/525.13

Mozilla/5.0 (compatible; Pi-Monster; https://pricepi.com/)

ThumbSniper (http://thumbsniper.com)
Shelob ([email protected])

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/537.36 (KHTML, like Gecko, Google-
Publisher-Plugin) Chrome/27.0.1453 Safari/537.36

Mozilla/5.0 (compatible; KaloogaBot; http://www.kalooga.com/info.html?page=crawler)

yacybot (/global; arm Linux 4.1.13+; java 1.8.0_40-internal; Etc/de) http://yacy.net/bot.html

CorporateNewsSearchEngine/Nutch-1.7 (http://pibs.co/news-search-engine)

Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/276

Yandex/1.01.001 (compatible; Win16; I)

FlightDeckReportsBot/2.0 (http://www.flightdeckreports.com/pages/bot)

Scrapy/0.24.4 (+http://scrapy.org)

ADmantX Platform Semantic Analyzer US - Turn - ADmantX Inc. - www.admantx.com -

[email protected]

Kyoto-Tohoku-Crawler/v1 (Mozilla-compatible; [email protected];

http://nlp.ist.i.kyoto-u.ac.jp/?crawling-kt)

Mozilla/5.0 (compatible; Scarlett/ 1.0; +http://www.ellerdale.com/crawler.html)

Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; [email protected])

www.deadlinkchecker.com Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML,

like Gecko) Chrome/46.0.2490.86 Safari/537.36

http://arachnode.net 1.2

Mozilla/5.0 (compatible; Plukkie/1.5; http://www.botje.com/plukkie.htm)

yacybot (freeworld/global; arm Linux 4.4.11-v7+; java 1.7.0_101; Etc/en) http://yacy.net/bot.html

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.16) Gecko/2009121601 Ubuntu/9.04 (jaunty)

Firefox/3.0.16 Specificfeeds- http://www.specificfeeds.com

Mozilla/5.0 (compatible; evc-batch/2.0)

Orbiter/1.2 (http://dailyorbit.com/)

crawler4j (https://github.com/yasserg/crawler4j/)

Mozilla/5.0 (compatible; SEOdiver/1.0; +http://www.seodiver.com/bot)

Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79

Safari/537.36 (https://shrinktheweb.com)

Scopia crawler 1.0 (+http://www.scopia.co)

yacybot (i386 Linux 2.6.23; java 1.6.0_06; Europe/en) http://yacy.net/bot.html

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET

CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
Mozilla/5.0 (compatible;
LXRbot/1.0;http://www.lxrmarketplace.com/,[email protected])

yacybot (freeworld/global; amd64 Linux 3.8.13-gentoo; java 1.7.0_21; Europe/de)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; Sitemap Generator/1.3; +http://www.check-

domains.com/sitemap/index.php)

DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)

Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)

Slack-ImgProxy 0.59 (+https://api.slack.com/robots)

Mozilla/5.0 (compatible; Ezooms/1.0; [email protected])

Mozilla/5.0 (compatible; YandexPagechecker/2.0; +http://yandex.com/bots)

CoinCornerBot/1.1 ( https://www.coincorner.com/BitcoinBot)

yacybot (freeworld/global; amd64 Linux 3.8.0-21-generic; java 1.6.0_27; Pacific/en)

http://yacy.net/bot.html

ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)

gonzo1[P] +http://www.suchen.de/faq.html

Mozilla/5.0 (compatible; MSIE or Firefox mutant;) Daum 4.1

Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620

Sogou web spider/4.02525A

Visbot/2.0 (+http://www.visvo.com/en/webmasters.jsp;[email protected])

Mozilla/5.0 (compatible; AcoonBot/4.10.8; +http://www.acoon.de/robot.asp)

WatchMouse/18990 (http://watchmouse.com/ ; d3.watchmouse.com)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Qt/4.8.1 Safari/534.34
ShoppimonAgent/1.0 ([email protected])

webinatorbot 1.1; +http://www.webinator.de

findlinks/2.0.5 (+http://wortschatz.uni-leipzig.de/findlinks/)

WikiDo/1.1 (http://wikido.com; [email protected])

yacybot (freeworld/global; amd64 Linux 3.5.0-27-generic; java 1.7.0_03; Europe/de)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; Uptimebot/0.1.73; +http://www.uptime.com/uptimebot)

Semantifire1/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )

Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)

Mozilla/5.0 compatible; yelpspider/yelpspider-1.0 (Crawlerbot run by Yelp Inc; yelpbot at yelp dot
com)

Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)

Mozilla/5.0 (compatible; emefgebot/beta; +http://emefge.de/bot.html)

Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev;

+http://www.openindex.io/en/webmasters/spider.html; systemsATopenindexDOTio)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75

Safari/537.36 Google (+https://developers.google.com/+/web/snippet/)

Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.de/crawler)

CCBot/1.0 (+http://www.commoncrawl.org/bot.html)

Lijit Crawler (+http://www.lijit.com/robot/crawler)

baypup/1.1 (Baypup; http://www.baypup.com/; [email protected])

Mozilla/5.0 (FauBot/0.1; +http://buzzvolume.com/fau/)

Mozilla/5.0 (compatible; NLNZ_IAHarvester2016/3.3.0 +https://natlib.govt.nz/publishers-and-

authors/web-harvesting/domain-harvest)

yacybot (/global; amd64 Linux 3.12.1; java 1.7.0_65; Europe/en) http://yacy.net/bot.html

yacybot (amd64 Windows 7 6.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html

SEOENGWorldBot/1.0 (+http://www.seoengine.com/seoengbot.htm)

Mozilla/5.0 (compatible; Finderbots finder bot; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-

[email protected])

Pompos/1.3 http://dir.com/pompos.html

Mozilla/5.0 (compatible; Gimme60bot/1.0; +http://gimme60.com) Firefox/16.0

Mozilla/5.0 (compatible; MJ12bot/v1.4.6; http://mj12bot.com/)

Mozilla/5.0 (compatible; Sysomos/1.0; +http://www.sysomos.com/; Sysomos)

urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html

al_viewer ([email protected])

LoadImpactRload/3.1.1 (Load Impact; http://loadimpact.com);

findlinks/2.0.2 (+http://wortschatz.uni-leipzig.de/findlinks/)

Mozilla/5.0 (compatible; spbot/3.1; +http://www.seoprofiler.com/bot )

Mozilla/5.0 (compatible; AcoonBot/4.11.0; +http://www.acoon.de/robot.asp)

WillyBot/1.1 (http://www.willyfogg.com/info/willybot)

Norton-Safeweb

WMCAI-robot (http://www.topicmaster.jp/wmcai/crawler.html)
rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,
[email protected])

Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; [email protected])

rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, [email protected])

ROR Sitemap Generator (http://www.rorweb.com)

http://domino.research.ibm.com/comm/research_projects.nsf/pages/sai-crawler.callingcard.html

MozillaTest/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )

Mozilla/5.0 (compatible; Peew/1.0; http://www.peew.de/crawler/)

Mozilla/5.0 (compatible; Website Analyzer/1.1; +http://www.check-domains.com/website-

analysis/website-analyzer.php)

Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)

PagePeeker.com

CorpusCrawler 2.0.10 (http://corpora.fi.muni.cz/crawler/)

yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_101; Europe/cs) http://yacy.net/bot.html

yacybot (/global; amd64 Linux 4.0.7-1-ck; java 1.8.0_45; Europe/de) http://yacy.net/bot.html

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/51.0.2704.103 Safari/537.36 PTST/284

yacybot (freeworld/global; i386 Linux 3.12-1-686-pae; java 1.7.0_21; Europe/fr)

http://yacy.net/bot.html

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; PTST 2.295)

CovarioIDS/1.1 (http://www.covario.com/ids; support at covario dot com)

Mozilla/5.0 (compatible; heritrix/1.14.4 +http://parsijoo.ir)

www.adressendeutschland.de

DialogSearch.com Bot 1.0;http://dialogsearch.com/webmasters

librabot/2.0 (+http://search.msn.com/msnbot.htm)

Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628

+http://www.archive.org/details/archive.org_bot)

wsAnalyzer/1.0; ++http://www.wsanalyzer.com/bot.html

Mozilla/5.0 (compatible; websays; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-

[email protected])

facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)

yacybot (/global; amd64 Linux 4.4.5-1-ARCH; java 1.8.0_77; America/en) http://yacy.net/bot.html

websitepulse checker/1.1 (compatible; MSIE 5.5; Netscape 4.75; Linux)

Mozilla/5.0 (compatible; memoryBot/1.21.14 +http://mignify.com/bot.html)

SalesIntelligent/v1.0

larbin_2.6.2 [email protected]

Mozilla/5.0 (compatible; SiteCondor; http://www.sitecondor.com)

yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_51; Europe/de)
http://yacy.net/bot.html

Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 43063)

yacybot (-global; amd64 Linux 2.6.32-042stab111.11; java 1.7.0_79; Europe/en)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; SemrushBot/0.96.2; +http://www.semrush.com/bot.html)

Linguee Bot (http://www.linguee.com/bot)

yacybot (/global; x86 Windows XP 5.1; java 1.7.0_51; Europe/de) http://yacy.net/bot.html

ICC-Crawler(Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)

Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.11) Gecko/20080109 (Charlotte/0.9t;

http://www.searchme.com/support/) (Charlotte/0.9t; http://www.searchme.com/support/)

Mozilla/5.0 (compatible; Goodzer/2.0; [email protected])

Acoon v4.10.5 (www.acoon.de)

CorpusCrawler 2.0.20 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus

Mozilla/5.0 (compatible; AMZNKAssocBot/4.0 +http://affiliate-program.amazon.com)

Mozilla/5.0 (compatible; MojeekBot/0.5; http://www.mojeek.com/bot.html)

yacybot (/global; amd64 Linux 4.2.0-22-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Qt/4.8.3 Safari/534.34
https://linkpeek.com

MetaTagRobot/0.2 (http://www.seocentro.com/tools/search-engines/metatag-analyzer.html)

Mozilla/5.0 (compatible; MJ12bot/v1.3.1; http://www.majestic12.co.uk/bot.php?+)

Mozilla/5.0 (compatible; Uptimebot/0.2.29; +http://www.uptime.com/uptimebot)

Mozilla/5.0 (compatible; Pandeo Bot; +http://pandeo.de/bot.php)

Pixray-Seeker/1.1 (Pixray-Seeker; http://www.pixray.com/pixraybot; [email protected])

EasyBib AutoCite (http://autocite-info.citation-api.com/)

Mozilla/5.0 (compatible; OptimizationCrawler/0.2; +http://www.domainoptima.com/robot)

AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; [email protected])

Gigabot/2.0
DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1;
+http://www.google.com/bot.html)

Morning Paper 1.0 (robots.txt compliant!)

Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html; Linux; Android 4; iPhone; CPU

iPhone OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0
Mobile/10A523 Safari/8536.25

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET
CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; GWX:RED; PTST 2.386)

SuperPagesUrlVerifyBot/1.0

CopperEgg/RevealUptime/LondonUK(linode)

Mozilla/5.0 (compatible; www.monitor.us - free monitoring service; http://www.monitor.us)

Mozilla/5.0 (compatible; LinkMarketbot/1.2; +http://www.linkmarket.com/)

Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider

wangling

Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)

RSSMicro.com RSS/Atom Feed Robot

GarlikCrawler/1.1 (http://garlik.com/, [email protected])

al_org_viewer ([email protected])

Mozilla/5.0 (compatible; JadynAveBot; +http://www.jadynave.com/robot)

dj-research/Nutch-1.11 (analytics@@demandjump.com)

Karneval-Bot (Version: 1.06, powered by www.karnevalsuchmaschine.de

+http://www.karnevalsuchmaschine.de/bot.html)

Baiduspider-image+(+http://www.baidu.com/search/spider.htm)\nReferer:
http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail

SauceNAO/1.0 (+http://saucenao.com/)

Mozilla/5.0 (compatible; JobdiggerSpider +http://www.jobdigger.nl/spider)

Yepi/1.0 (NHN Corp.; http://help.naver.com/robots/)

Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/)

SSL Labs (https://www.ssllabs.com/about/assessment.html)

Mozilla/5.0 (compatible; http://alyze.info)

GigablastOpenSource/1

Mozilla/5.0 (Windows; U; Windows NT 5.1;fr;rv:1.8.1) VoilaBotCollector BETA 0.1

(http://www.voila.com/)
Vorboss Web Crawler [[email protected]]/Nutch-2.3

Mozilla/5.0 (compatible; SecretSerachEngineLabs.com-SBSearch/0.9;

http://www.secretsearchenginelabs.com/secret-web-crawler.php)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Search Console)
Chrome/27.0.1453 Safari/537.36

YahooCacheSystem

search.KumKie.com

yacybot (webportal-global; x86 Windows Vista 6.0; java 1.7.0_25; Europe/en)

http://yacy.net/bot.html

Embedly [email protected]

stq_bot (+http://www.searchteq.de)

MSRBOT

Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)

CorpusCrawler 2.0.17 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus

Mozilla/5.0 (compatible; heritrix/3.1.0-RC1 +http://boston.lti.cs.cmu.edu/crawler_12/)

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) PTST/276

Mozilla/5.0 (compatible; WebCookies/1.0; +http://webcookies.org/faq/#agent)

L.webis/0.44 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)

yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; America/en)

http://yacy.net/bot.html

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/279

Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0

Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01i.yandex.ru)

Y!J-BSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/blog-search/)

Mozilla/5.0 (compatible; UptimeRobot/1.0; http://www.uptimerobot.com/)

NG-Search/0.86 (+http://www.ng-search.com)

ichiro/3.0 (http://help.goo.ne.jp/help/article/1142)

TwengaBot/1.1 (+http://www.twenga.com/bot.html)

WebImages 0.3 ( http://herbert.groot.jebbink.nl/?app=WebImages )

Mozilla/5.0 (compatible; aiHitBot/2.8; +http://endb-consolidated.aihit.com/)

RyzeCrawler/1.1.1 (+http://www.domain2day.nl/crawler/)

Mozilla/5.0 (Nekstbot; http://www.ipipan.waw.pl/nekst/nekstbot/)

Mozilla/5.0 (compatible; adidxbot/2.0; http://www.bing.com/bingbot.htm)

Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot-crawler.net)

WatchMouse/8.4.0.3 (http://watchmouse.com/ ; gblon01.watchmouse.net)

yacybot (/global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html

Page Analyzer v4.0 ( http://www.ranks.nl/ )

web_bh ([email protected])

findlinks/1.1.6-beta3 (+http://wortschatz.uni-leipzig.de/findlinks/)

Mozilla/5.0 (compatible; Findxbot/1.0; +http://www.findxbot.com)

findlinks/2.0 (+http://wortschatz.uni-leipzig.de/findlinks/)

Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/)

PagesInventory (robot http://www.pagesinventory.com)

Mozilla/5.0 (compatible; aiHitBot/1.0-DS; +http://www.aihit.com/)

tagSeoBot/1.0 (http://www.tagseoblog.de/tools)

Mozilla/5.0 (en-us) AppleWebKit/537.36 (KHTML, like Gecko; Google PP Default) Chrome/27.0.1453

Safari/537.36

404 Checker [http://www.404checker.com/user-agent]

CopperEgg/RevealUptime/

Mozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)

Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)

CopperEgg/RevealUptime/TokyoJapan

Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html)

GoSquared-Status-Checker/0.2

Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0 Safari/537.36

YottaaMonitor

WatchMouse/18990 (http://watchmouse.com/ ; bc.watchmouse.com)

yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_24; Europe/de)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; adbeat_bot; [email protected]; [email protected])

linkdexbot/Nutch-1.0-dev (http://www.linkdex.com/; crawl at linkdex dot com)

Heurekabot-Feed/1.0 (+http://sluzby.heureka.cz/napoveda/heurekabot/)

Mozilla/5.0 (compatible; Charlotte/1.1; http://www.searchme.com/support/)

yacybot (/global; amd64 Linux 3.10.0-229.7.2.el7.x86_64; java 1.8.0_45; Europe/en)
http://yacy.net/bot.html

LSSRocketCrawler/1.0 LightspeedSystems

Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; Jobboerse.com; http://www.xn--
jobbrse-d1a.com) Gecko/20100401 Firefox/24.0

findlinks/2.2 (+http://wortschatz.uni-leipzig.de/findlinks/)

Mozilla/5.0 (compatible; kulturarw3 +http://www.kb.se/om/projekt/Svenska-webbsidor---

Kulturarw3/)

Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online)

AppleWebKit/534.34

Mozilla/5.0 (compatible; SemrushBot/0.96.4; +http://www.semrush.com/bot.html)

woobot/2.0

GarlikCrawler/1.2 (http://garlik.com/, [email protected])

yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; AboutUsBot Johnny5/2.0; +http://www.AboutUs.org/)

yacybot (/global; amd64 Linux 3.10.0-327.22.2.el7.x86_64; java 1.7.0_101; Etc/en)

http://yacy.net/bot.html

Mozilla/5.0 (Linux; Android 4.1.2; Galaxy Nexus Build/JZO54K; GTmetrix http://gtmetrix.com/)

AppleWebKit/537.22 (KHTML, like Gecko) Chrome/26.0.1410.58 Mobile Safari/537.22

Nuhk/2.4 ( http://www.neti.ee/cgi-bin/abi/Otsing/Nuhk/)

Grahambot/0.1 (+http://www.sunaga-lab.com/graham-bot)

yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_91; Europe/de) http://yacy.net/bot.html

SEO Consulting; Redirect Checker Tool V.02; IP:

Mozilla/5.0 (compatible; Pro Sitemaps Generator; https://pro-sitemaps.com) Gecko Pro-

Sitemaps/1.0

yacybot (/global; amd64 Linux 4.4.10-antix.1-amd64-smp; java 1.8.0_101; Europe/en)

http://yacy.net/bot.html

Mozilla/5.0 (Windows NT 6.2; WOW64) Runet-Research-Crawler (itrack.ru/research/cmsrate;

[email protected])

Mozilla/5.0 (compatible; Crawler/0.9; http://linkfluence.net/)

ADmantX Platform Semantic Analyzer US Async - ADmantX Inc. - www.admantx.com -

[email protected]

GetProxi.es-bot/1.1 (http://getproxi.es/spiderinfo/)

Mozilla/5.0 (Windows NT 6.3; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/277

Pinterest/0.2 (+http://www.pinterest.com/)

CopperEgg/RevealUptime/AtlantaGA(linode)

OdklBot/1.0 ([email protected])

Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot)

yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_22; Europe/de)

http://yacy.net/bot.html

MojeekBot/0.2 (archi; http://www.mojeek.com/bot.html)

Sogou web spider/4.05252A

Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276

Mozilla/5.0 (compatible; MotoMinerBot/1.0; +https://motominer.com/Bot)

NG/2.0

Mozilla/5.0 (compatible; heritrix/1.14.2 +http://www.webarchiv.cz)

StackRambler/2.0 (MSIE incompatible)

Baiduspider+(+http://www.baidu.jp/spider/)

yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_25; Europe/de)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; houzzbot; +http://www.houzz.com/)

Woko robot 3.0

Mozilla/5.0 (compatible; Qwantify/2.0; +https://www.qwant.com/)

yacybot (/global; amd64 Linux 4.2.0-27-generic; java 1.8.0_66-internal; America/en)

http://yacy.net/bot.html

ADmantX Platform Semantic Analyzer - APAC - ADmantX Inc. - www.admantx.com -

[email protected]

Mozilla/5.0 (compatible; Uptimebot/0.2.40; +http://www.uptime.com/uptimebot)

Mozilla/5.0 (compatible; ExpertSearchSpider +http://www.expertsearch.nl/spider)

Mozilla/5.0 (compatible; coccocbot-web/1.0; +http://help.coccoc.com/searchengine)

Acoon v4.10.4 (www.acoon.de)

Mozilla/5.0 (compatible; memoryBot/1.20.210 +http://internetmemory.org/en/)

Readability/740ec9 - http://readability.com/about/

Mozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html)

yacybot (i386 Linux 2.6.28-gentoo-r5; java 1.5.0_18; Europe/en) http://yacy.net/bot.html

Mozilla/5.0 (compatible; suggybot v0.01a, http://blog.suggy.com/was-ist-suggy/suggy-webcrawler/)

yacybot (amd64 Windows 7 6.1; java 1.6.0_14; Europe/de) http://yacy.net/bot.html

yacybot (freeworld/global; amd64 Linux 3.3.4-1-ARCH; java 1.6.0_24; Europe/en)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; AportWorm/3.2; +http://www.aport.ru/help)

Mozilla/5.0 (compatible; memoryBot/1.20.235 +http://internetmemory.org/en/)

findlinks/2.6 (+http://wortschatz.uni-leipzig.de/findlinks/)

Mozilla/5.0 (compatible; Hailoobot/1.2; +http://www.hailoo.com/spider.html)

eCommerceBot (http://www.ehandel.se/botinfo.html)

Mozilla/5.0(compatible;Sosospider/2.0;+http://help.soso.com/webspider.htm)

yacybot (/global; amd64 Linux 4.3.0-gentoo-ARCH; java 1.7.0_85; Europe/en)

http://yacy.net/bot.html

Nymesis/2.0 (http://nymesis.com)

CopperEgg/RevealUptime/OregonUSA

uclassify.com/1.0

Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly.html) Gecko/2009032608

Firefox/3.0.8

Mozilla/5.0 (compatible; Prlog/1.0; +http://prlog.ru/)

Slack-ImgProxy 1.106 (+https://api.slack.com/robots)

AdnormCrawler www.adnorm.com/crawler

Mozilla/5.0 (compatible; YandexZakladki/3.0; +http://yandex.com/bots)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75

Safari/537.36 Google Favicon

Mozilla/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)

Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like
Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1;
+http://www.google.com/bot.html)

ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)

Mozilla/4.0 ([email protected])

bot-pge.chlooe.com/1.0.0 (+http://www.chlooe.com/)

Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/)

Mozilla/5.0 (compatible; Webmaster tools +http://sitexy.com/)

yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_40; Europe/de) http://yacy.net/bot.html

Mozilla/5.0 (compatible; MJ12bot/v1.4.1; http://www.majestic12.co.uk/bot.php?+)

Mozilla/5.0 (compatible; spbot/4.0.6; +http://www.seoprofiler.com/bot )

Mozilla/5.0 (compatible; EuripBot/2.0; +http://www.eurip.com)

findlinks/2.1 (+http://wortschatz.uni-leipzig.de/findlinks/)

Sogou web spider/4.025251

SETOOZBOT/5.0 ( http://www.setooz.com/bot.html )

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) ([email protected])

Mozilla/5.0 (compatible; spbot/2.0.4; +http://www.seoprofiler.com/bot )

Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0

mozilla/5.0 ([email protected])

Mozilla/5.0 (compatible; spbot/2.1; +http://www.seoprofiler.com/bot )

Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98

Safari/537.4 (StatusCake SSL Monitor)

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; PTST 2.385)

Mozilla/5.0 (compatible; evc-batch/2.0.20160608212921)

Mozilla/5.0 (compatible; Mail.RU_Bot/2.0)

seebot/2.0 (+http://www.seegnify.com/bot)

bl.uk_lddc_bot/3.3.0-LBS-2016-02
(+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)

CommaFeed/2.3.0-SNAPSHOT (https://www.commafeed.com)

OmniExplorer_Bot/5.91c (+http://www.omni-explorer.com) WorldIndexer

hledejLevne.cz/2.0

page_verifier (http://www.securecomputing.com/goto/pv)

url_test ([email protected])

Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20100101 Firefox/21.0 WordPress.com mShots

Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)

Is is up? (+http://isitup.org)

Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; [email protected]/)

TwengaBot-2.0 Champigny (+http://www.twenga.com/bot.html)

LivelapBot/0.2 (http://site.livelap.com/crawler)

HubSpot Crawler 1.0 http://www.hubspot.com/

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/45.0.2454.101 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
yacybot (/global; x86 Windows 10 10.0; java 1.8.0_73; Europe/de) http://yacy.net/bot.html

Scrapy/0.24.6 (+http://scrapy.org)

FAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)

Baiduspider+(+http://www.baidu.com/search/spider_jp.html)

Mozilla/5.0 (compatible; seplinkbot/1.0 )

Mozilla/5.0 (compatible; Falconsbot; +http://ws.nju.edu.cn/falcons/)

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/277

Mozilla/4.0 (Toread-Crawler/1.1; +http://news.toread.cc/crawler.php)

TinEye-bot/0.02 (see http://www.tineye.com/crawler.html)

yacybot (freeworld/global; i386 Linux 2.6.32-39-generic-pae; java 1.6.0_20; Europe/en)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; parsijoo; +http://www.parsijoo.ir/; [email protected])

Mozilla/5.0 (compatible; spbot/4.4.0; +http://OpenLinkProfiler.org/bot )

MaxPoint Bot (+http://www.maxpoint.com)

Mozilla/5.0 (compatible; Infohelfer/1.2.0; +http://www.infohelfer.de/crawler.php)

ExB Language Crawler 2.1.1 (+http://www.exb.de/crawler)

cg-eye interactive

ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry)

rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,
[email protected])

Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 47129)

Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:14.0) Gecko/20100101 Firefox/14.0.1 AppEngine-Google;

(+http://code.google.com/appengine; appid: s~metacdn-hr)

Mozilla/5.0 (compatible; WoTBoT; +https://www.wslta.com/WoTBoT.html)

HolmesBot (http://holmes.ge)

Baiduspider-image+(+http://www.baidu.com/search/spider.htm)

Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; no-

[email protected])

Mediapartners-Google

Mozilla/5.0 (compatible; MFGPagesBot/2.1; http://www.mfgpages.com)

larbin_2.6.2 [email protected]
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search;
http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com)
Gecko/20071127 Firefox/3.0.0.11

BlogPulseLive ([email protected])

WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)

yacybot (freeworld/global; i386 Linux 3.0.0-17-generic; java 1.6.0_23; America/en)

http://yacy.net/bot.html

ImplisenseBot 1.0

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; PTST 2.386)

gonzo2[P] +http://www.suchen.de/faq.html

Mozilla/5.0 (compatible; LXRbot/1.0; http://lxrseo.com/, [email protected])

Mozilla/5.0 (compatible; Arachnophilia/1.0; +http://arachnys.com/)

Mozilla/5.0 (compatible; CloudServerMarketSpider/1.0;

+http://cloudservermarket.com/spider.html)

kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler)

yacybot (webportal/global; x86_64 Mac OS X 10.9.2; java 1.6.0_65; Europe/de)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, [email protected])

yacybot (/global; amd64 Linux 4.1.19-gentoo; java 1.7.0_95; Europe/pl) http://yacy.net/bot.html

Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/276

Mozilla/5.0 (compatible; RankActiveLinkBot; +https://rankactive.com/resources/rankactive-linkbot)

audisto.com full crawler 3.26.431 (refer to in robots.txt as audisto, see https://audisto.com/bot)

Mozilla/5.0 (compatible; spbot/4.0.1; +http://www.seoprofiler.com/bot )

Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev;

+http://www.openindex.io/en/webmasters/spider.html)

mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)

gonzo2[p] (+http://www.suchen.de/faq.html)

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible;

HaosouSpider; http://www.haosou.com/help/help_3_2.html)

Mozilla/5.0 (compatible; spbot/2.0.1; +http://www.seoprofiler.com/bot/ )

WatchMouse/18990 (http://watchmouse.com/ ; uk)

yacybot (freeworld/global; amd64 Linux 3.1.10-hardened; java 1.7.0_03-icedtea; Europe/en)

http://yacy.net/bot.html

VeBot (+http://www.veinteractive.com/vebot)
Mozilla/5.0 (compatible; NLNZ_IAHarvester2013 +http://natlib.govt.nz/about-us/current-
initiatives/web-harvest-2012)

findlinks/1.1.3-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/)

Mozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.36 (KHTML, like Gecko; Google Web
Preview) Version/4.0 Mobile Safari/537.36

DialogSearch.com Bot 1.4;http://dialogsearch.com/webmasters

Mozilla/5.0 (compatible; GurujiBot/1.0; +http://www.guruji.com/en/WebmasterFAQ.html)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9) AppleWebKit/537.71 (KHTML, like Gecko) Version/7.0
Safari/537.71 (Rival IQ, rivaliq.com)

wscheck.com/1.0.0 (+http://wscheck.com/)

Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/30.0.1599.69 Safari/537.36 Webthumb/2.0

Mozilla/5.0 (compatible; FlipboardProxy/1.1; +http://flipboard.com/browserproxy)

WeSEE:Ads/PictureBot (http://www.wesee.com/bot/)

Mozilla/5.0 (compatible; Dataprovider/6.101; +https://www.dataprovider.com/)

Kyoto-Crawler/n1.0 (Mozilla-compatible; [email protected];

http://nlp.ist.i.kyoto-u.ac.jp/?crawling)

Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)

Mozilla/5.0 (compatible; spbot/4.0.9; +http://OpenLinkProfiler.org/bot )

Mozilla/5.0 (compatible; spbot/4.2.0; +http://OpenLinkProfiler.org/bot )

seo-nastroj.cz

LoadImpactPageAnalyzer/1.3.0 (Load Impact; http://loadimpact.com/)

CSS Certificate Spider (http://www.css-security.com/certificatespider/)

MetaGeneratorCrawler/1.3.2 (www.metagenerator.info)

Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot)

minicrawler/4.0.0~beta8

BLEXBot

Mozilla/2.0 (compatible; Ask Jeeves/Teoma)

Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot)

minicrawler/4.0.0~beta7

Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)

Mozilla/5.0 (compatible; spbot/4.4.1; +http://OpenLinkProfiler.org/bot )

ia_archiver (+http://www.alexa.com/site/help/webmasters; [email protected])

HeartRails Robot/0.1 (http://www.heartrails.com)

Mozilla/5.0 (compatible; Faveeo/1.0; +http://www.faveeo.com)

yacybot (/global; amd64 Linux 3.16.0-49-generic; java 1.7.0_79; Europe/en) http://yacy.net/bot.html

Castabot/0.1 (+http://topixtream.com/)

Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2010033101 Gentoo Firefox/3.0.5 (Dot
TK - spider 3.0)

Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;

+http://ws.daum.net/aboutWebSearch.html) Daumoa/2.0

istellabot-nutch/Nutch-1.10

Mail.RU/2.0

ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)

LexxeBot/1.0 ([email protected])

Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko)
Version/8.0 Mobile/12B410 Safari/600.1.4 (Applebot/0.1; +http://www.apple.com/go/applebot)

Mozilla/5.0 (compatible; FlipboardRSS/1.1; +http://flipboard.com/browserproxy)

Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)

yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.7.0_79; Europe/en)

http://yacy.net/bot.html

it2media-domain-crawler/1.0 on crawler-prod.it2media.de

yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_25; Europe/de) http://yacy.net/bot.html

yacybot (freeworld-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de)

http://yacy.net/bot.html

Mozilla/5.0 (WhatsMyIP.org HTTP_Headers) http://whatsmyip.org/ua

checkgzipcompression.com robot

Mozilla/5.0 eCairn-Grabber/1.0 (+http://ecairn.com/grabber)

RankurBot/3.3 (+http://rankur.com)

L.webis/0.50 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)

Speedy Spider (Submit your site at http://www.entireweb.com/free_submission/)

oBot

Snappy/2.0 ( http://www.urltrends.com/ )

Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess

(http://www.PowerMapper.com) /5.23.770.0

Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; )

flatlandbot/baypup (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot;
[email protected])

istellabot/Nutch-1.11

GetintentCrawler getintent.com

Covario-IDS/1.0 (Covario; http://www.covario.com/ids; support at covario dot com)

Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/crawler)

Mozilla/5.0 (compatible; MegaIndex.com/2.0; +http://megaindex.com/crawler)

Microsearch.ru/1.0; http://microsearch.ru/webmasters

TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)

BacklinkCrawler V (http://www.backlinktest.com/crawler.html)

FeedlyBot/1.0 (http://feedly.com)

Clickagy Intelligence Bot v2

GetURLInfo/1.0

DoCoMo/2.0 P900i(c100;TB;W24H11)(compatible; ichiro/mobile

goo;+http://help.goo.ne.jp/door/crawler.html)

Scrapy/1.1.0 (+http://scrapy.org)

yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; America/en) http://yacy.net/bot.html

Mozilla/5.0 (compatible; VSAgent/1.2)

Experibot_v1 [goo.gl/n6zrAf]

Mozilla/5.0 (compatible; DNS-Digger-Explorer/1.0; +http://www.dnsdigger.com)

boitho.com-robot/1.0

yacybot (/global; amd64 Windows Server 2012 6.2; java 1.7.0_51; Europe/de)
http://yacy.net/bot.html

RelateIQ Crawler www.relateiq.com

Mozilla/4.0 (compatible; Netcraft Web Server Survey)

Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot;

https://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0

yacybot (i386 Linux 2.6.9-023stab046.2-smp; java 1.6.0_05; Europe/en) http://yacy.net/bot.html

Scrubby/2.2 (http://www.scrubtheweb.com/)

Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:[email protected])

Fetch/2.0a (CMS Detection/Web/SEO analysis tool, see http://guess.scritch.org)

bitlybot/3.0 (+http://bit.ly/)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.exif-search.com)

yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_25; Europe/de)
http://yacy.net/bot.html

agentslug.com - website monitoring tool

SafeDNSBot (https://www.safedns.com/searchbot)

Feedbin

Mozilla/5.0 (compatible; proximic; +http://www.proximic.com)

Mozilla/5.0 (compatible; AcoonBot/4.11.1; +http://www.acoon.de/robot.asp)

Mozilla/5.0 (compatible; MagiBot/3.4.3; +http://magi.peak-labs.com/robots.txt)

Mozilla/5.0 (compatible; XML Sitemaps Generator; http://www.xml-sitemaps.com) Gecko XML-

Sitemaps/1.0

baypup/colbert (Baypup; http://sf.baypup.com/webmasters; [email protected])

MergeFlow-PageReader/0.91;+(+http://mergeflow.net/info/pagereader) Mozilla/5.0 (Windows)

compatible

Mozilla/5.0 (Mobile; rv:18.0) Gecko/18.0 Firefox/18.0 commoncrawl.org/research//Nutch-1.7-

SNAPSHOT

Mozilla/5.0 (compatible; spbot/4.1.0; +http://OpenLinkProfiler.org/bot )

Mozilla/5.0 (compatible; OsO; http://oso.octopodus.com/abot.html)

Mozilla/5.0 (compatible; gofind; +http://govid.mobi/bot.php)

CatchBot/2.0; +http://www.catchbot.com

Mozilla/5.0 (compatible; BusinessSeek.biz_Spider; http://www.businessseek.biz/)

Quora Link Preview/1.0 (http://www.quora.com)

radian6_default_(www.radian6.com/crawler)

DWDS-Crawler +http://odo.dwds.de/dwds-crawler.html

Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)

sogou spider

Mozilla/5.0 (compatible; GigablastOpenSource/1.0)

Mozilla/5.0 (compatible; HomeTags/1.0; +http://www.hometags.nl/bot)

CorpusCrawler 2.0.24 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus

Scooter/3.3

Zookabot/2.5;++http://zookabot.com
Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko)
Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0;
+http://yandex.com/bots)

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) Safari/534.34;

+http://sniptracker.com

Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record;

+http://archive.org/details/archive.org_bot)

es_com_viewer ([email protected])

Mozilla/5.0 (compatible; Uptimebot/0.2.14; +http://www.uptime.com/uptimebot)

yacybot (freeworld/global; amd64 Linux 3.0.0-14-generic; java 1.6.0_23; Europe/en)

http://yacy.net/bot.html

Mozilla/5.0 (X11; Linux x86_64; rv:35.0) Gecko/20100101 Firefox/35.0 DareBoost

yacybot (freeworld/global; amd64 Linux 2.6.32-41-server; java 1.6.0_26; Europe/de)

http://yacy.net/bot.html

yacybot (amd64 Linux 2.6.28-18-generic; java 1.6.0_16; GMT/en) http://yacy.net/bot.html

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET
CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; PTST 2.386)

KD Bot

Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.19) Gecko WebThumb/1.0

Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)

rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])

MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)

Superarama.com-Tarama-Botu-v.01

Mozilla/5.0 (compatible; Urlfilebot/2.2; +http://urlfile.com/bot.html)

Vegi bot (we follow your robots.txt settings before crawling, you can slow down the bot by change
the Crawl-Delay parameter in the settings.if you have an enquiry, please email to: abuse-
[email protected])

MnoGoSearch/3.3.9

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b

JyxobotRSS/0.06

Mozilla/5.0 (compatible; BigBozz/2.2.1; +http://www.bigbozz.com/)

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7

Safari/534.57.2 PTST/277

KiNShooboT (compatible; KiNShooboT/1.0.C; +http://www.kinshoo.com/bot.html)

SentiBot www.sentibot.eu (compatible with Googlebot)

Topicbot/1.0 (Mozilla;I;+http://92.42.190.57/)

IDG/IT (http://spaziodati.eu/)

LinkAider (http://linkaider.com/crawler/)

Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/searchengine)

yacybot (/global; amd64 Linux 2.6.32-573.3.1.el6.x86_64; java 1.7.0_85; Europe/en)

http://yacy.net/bot.html

bitlybot

Mozilla/5.0 (compatible; Mp3Bot/0.7; +http://mp3realm.org/mp3bot/)

WWW::LayeredExtractor::Handler::Feed/0.01

Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot.net)

Mozzila/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)

MetaTagRobot/2.1 (http://www.seocentro.com/tools/search-engines/metatag-analyzer.html)

Mozilla/5.0 (compatible; Goodzer/1.0)

yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_40; Europe/en) http://yacy.net/bot.html

Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3

(NetShelter ContentScan)

Facebot/1.0

SafeAds.xyz bot

Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/robots/)

yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/en) http://yacy.net/bot.html

scrapyproject (+http://www.profound.net)

Mozilla/5.0 (compatible; Online Domain Tools - Online Website Link Checker/1.2; +http://website-
link-checker.online-domain-tools.com)

Mozilla/5.0 (compatible; heritrix/2.0.2 +http://aihit.com)

Mozilla/5.0 (compatible; Scrubby/3.1; +http://www.scrubtheweb.com/help/technology.html)

Semantifire1/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )

Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0; ips-agent) Gecko/20100101 Firefox/14.0.1

UnisterBot (Mozilla/5.0 compatible; [email protected])

Mozilla/5.0 (compatible; Ezooms/1.0; [email protected])

Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; spbot/4.4.2; +http://OpenLinkProfiler.org/bot )

findlinks/2.1.3 (+http://wortschatz.uni-leipzig.de/findlinks/)

Mozilla/5.0 (compatible; MJ12bot/v1.2.5; http://www.majestic12.co.uk/bot.php?+)

Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/speedyspider/)

Mozilla/5.0 (compatible; kazbtbot/0.1; +http://kazbt.com/)

Orgbybot/OrgbyBot v1.3 (Spider; http://orgby.com/bot/ ; Orgby.com Search Engine)

Mozilla/5.0 (compatible; aiHitBot-DM/2.0.2 +http://www.aihit.com)

YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)

Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 40409)

Mozilla/5.0 (compatible; image.coccoc/1.0; +http://help.coccoc.com/)

yacybot (/global; amd64 Linux 3.19.2-1-ARCH; java 1.8.0_40; Europe/de) http://yacy.net/bot.html

Mozilla/5.0 (compatible; evc-batch/2.0.20161017175820)

datagnionbot (+http://www.datagnion.com/bot.html)

CopperEgg/RevealUptime/TokyoJP(linode)

MB-SiteCrawler

WatchMouse/18990 (http://watchmouse.com/ ; liz)

HybridBot (hybrid.ru/about. If our bot caused problems please contact us. Contact email:
[email protected])

Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 24522)

yacybot (/global; amd64 Linux 3.13.0-74-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html

findlinks/1.1.6-beta5 (+http://wortschatz.uni-leipzig.de/findlinks/)

yacybot (freeworld/global; amd64 Linux 3.13.0-24-generic; java 1.7.0_55; Europe/de)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; OpenfosBot/2.4; +http://www.openfos.com)

Nusearch Spider (www.nusearch.com)

WatchMouse/8.4.0.3 (http://watchmouse.com/ ; sesto02.watchmouse.net)

ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/univ-com/plan/crawl.html)

Mozilla/5.0 (compatible; spbot/1.2; +http://www.seoprofiler.com/bot/ )

Orbiter/1.3 (http://dailyorbit.com/)

CCBot/2.0

Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/about/bots/)

Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com)

AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-social)

yacybot (/global; x86 Windows 7 6.1; java 1.8.0_71; Europe/de) http://yacy.net/bot.html

Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia
530) like Gecko (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)

Dex Social Bot

yacybot (/global; amd64 Linux 4.7.6-200.fc24.x86_64; java 1.8.0_102; Etc/en)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; heritrix/1.14.3 +http://archive.org)

CRAZYWEBCRAWLER 0.9.10, http://www.crazywebcrawler.com

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/50.0.2661.102 Safari/537.36 PTST/277

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3; ips-agent) Gecko/20090824 Fedora/1.0.7-1.1.fc4

Firefox/3.5.3

SafeSearch microdata crawler (https://safesearch.avira.com, [email protected])

Zemanta Aggregator/0.9 +http://www.zemanta.com

Mozilla/5.0 (compatible; special_archiver/3.2.0

+http://www.loc.gov/webarchiving/notice_to_webmasters.html)

Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/crawler.html)

yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_45; Europe/de)

http://yacy.net/bot.html

Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89

Safari/537.1; 360Spider

FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php)

yacybot (/global; amd64 FreeBSD 10.3-RELEASE; java 1.8.0_77; GMT/en) http://yacy.net/bot.html

Searchie/1.0 (a Storm-based crawler; https://www.searchie.org; [email protected])

yacybot (freeworld/global; amd64 Linux 3.1.10-1-desktop; java 1.6.0_22; Europe/de)

http://yacy.net/bot.html

holmes/3.12.4 (http://morfeo.centrum.cz/bot)

WatchMouse/18990 (http://watchmouse.com/ ; se.watchmouse.com)

MXT/Nutch-1.10 (http://t.co/GSRLLKex24; informatique at mixdata dot com)

Mozilla/5.0 (compatible; Shareaholicbot/1.0; +http://www.shareaholic.com/bot)

ZoomInformation Bot

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; www.alertra.com)

Mozilla/5.0 (compatible; HomeTags/1.0; http://www.hometags.nl/bot)

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110515 HeartRails_Capture/1.0.4
(+http://capture.heartrails.com/) Namoroka/3.6.17

OmniExplorer_Bot/5.20 (+http://www.omni-explorer.com) WorldIndexer

WatchMouse/18990 (http://watchmouse.com/ ; it)

Mozilla/5.0 (compatible; imrbot/1.10.8 +http://www.mignify.com)

Mozilla/5.0 (compatible; spbot/4.0.2; +http://www.seoprofiler.com/bot )

Readability/1900e6 - http://readability.com/about/

Mozilla/5.0 (compatible; Dataprovider/6.92; +https://www.dataprovider.com/)

Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko;
Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25

VegeBot (we follow your robots.txt settings before crawling, you can slow down the bot by change
the Crawl-Delay parameter in the settings.if you have an enquiry, please email to: abuse-
[email protected])

yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en)

http://yacy.net/bot.html

FeedBucket/1.0 (+http://www.feedbucket.com)

topster.de HTTP-Header 1.0

MaxPointCrawler/Nutch-1.10 (maxpoint.crawler at maxpointinteractive dot com)

Mozilla/5.0 (Windows NT 6.0; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276

yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_05; Europe/es) http://yacy.net/bot.html

NETCRAFT

Mozilla/5.0 (compatible; WbSrch/1.0; +https://wbsrch.com)

WebCookies/1.0 (+http://webcookies.info/faq/#agent)

netEstate Impressumscrawler (+http://www.netestate.de/De/Loesungen/Impressumscrawler)

msnbot-media/2.0b (+http://search.msn.com/msnbot.htm)

CopperEgg/RevealUptime/DublinIEUSA

StormCrawler/1.0 (a Storm-based crawler; https://github.com/DigitalPebble/storm-crawler;

[email protected])

WatchMouse/8.4.0.3 (http://watchmouse.com/ ; uschi02.watchmouse.net)

Mozilla/5.0 (Linux; Android 5.0.2; SM-G920T Build/LRX22G) AppleWebKit/537.36 (KHTML, like

Gecko) Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost

Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.org)

Mozilla/5.0 (compatible; MJ12bot/v1.4.4 (domain ownership verifier);

http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; SemrushBot/0.99~bl; +http://www.semrush.com/bot.html)

WebWatch/Robot_txtChecker

Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/)

Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)

Toweyabot: toweya.com

Mozilla/5.0 (compatible; Infohelfer/1.4.3; +http://www.infohelfer.de/crawler.php)

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.1);

http://spinn3r.com/robot) Gecko/20021130

FeedCatBot/3.0 (+http://www.feedcat.net/)

LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com),

libot/Nutch-1.9 (http://www.linkedin.com; [email protected])

R6_CommentReader(www.radian6.com/crawler)

Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 70350)

Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+)

Domain Re-Animator Bot (http://domainreanimator.com) - [email protected]

Riddler (http://riddler.io/about.html)

Mozilla/5.0 (compatible; Esribot/1.0; http://www.esrihu.hu/)

Favicon downloader (http://favicon.netk6.com/)

findlinks/1.1.5-beta7 (+http://wortschatz.uni-leipzig.de/findlinks/)

ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl)

Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google favicon

GetFoundBot (+http://www.getfound.cz/getfoundbot/)

WatchMouse/8.4.0.3 (http://watchmouse.com/ ; usdal02.watchmouse.net)

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/48.0.2564.97 Safari/537.36 Yandex.Translate

Mozilla/5.0 (compatible; NLNZ_IAHarvester2013 +http://natlib.govt.nz/about-us/current-

initiatives/web-harvest-2013)

COMODOSpider/Nutch-1.2

Mozilla/5.0 (compatible; DomainMacroCrawler/0.1; +http://domainmacro.com)

Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.exif-search.com)

Mozilla/5.0 (compatible; MagiBot/3.6.2; +http://magi.peak-labs.com/robots.txt)

Mozilla/5.0 (compatible; Alexabot/1.0; +http://www.alexa.com/help/certifyscan;

[email protected])
psbot-image (+http://www.picsearch.com/bot.html)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0

(FlipboardProxy/1.6; +http://flipboard.com/browserproxy)

larbin_2.6.4 ([email protected])

yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/ru) http://yacy.net/bot.html

BacklinkCrawler (http://www.backlinktest.com/crawler.html)

Mozilla/5.0 (Windows NT 6.1; WOW64) adbeat.com/policy AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/48.0.2564.116 Safari/537.36

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7

Safari/534.57.2 PTST/276

Tools4noobs.com/1.0 Spider

yacybot (freeworld/global; amd64 Linux 2.6.32-40-generic; java 1.6.0_20; Europe/de)

http://yacy.net/bot.html

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider

(http://www.entireweb.com/about/search_tech/speedy_spider/)

istellabot/t.1

Mozilla/5.0 (compatible; Lipperhey-Kaus-Australis/5.0; +https://www.lipperhey.com/en/about/)

Mozilla/5.0 (compatible; Veoozbot/1.0; +http://www.veooz.com/veoozbot.html)

SemrushBot/Nutch-1.5-SNAPSHOT

Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

Mozilla/5.0 (X11; Linux x86_64; rv:47.0; GTmetrix https://gtmetrix.com/) Gecko/20100101

Firefox/47.0

CopperEgg/RevealUptime/Oregon(aws)

yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_102; Europe/en) http://yacy.net/bot.html

rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, [email protected])

yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_20; Europe/en) http://yacy.net/bot.html

Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0; BingPreview/1.0b) like Gecko

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR
2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDS; PTST 2.386)

Mozilla/5.0 (compatible; RavenCrawler/2.0; +https://raventools.com/seo-website-auditor/)

Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; Scrubby/3.2; +http://seotools.scrubtheweb.com/webpage-analyzer.html)

dubaiindex (adressendeutschland.de)
Mozilla/5.0 (compatible; spbot/5.0.2; +http://OpenLinkProfiler.org/bot )

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Netcraft SSL Server Survey - contact
[email protected])

Mozilla/5.0 (compatible; CompSpyBot/1.0; +http://www.compspy.com/spider.html)

Mozilla/5.0 (compatible; LA2; +http://www.zeerch.com/zeerch2/bot.php)

yacybot (/global; amd64 Linux 2.6.32-042stab108.8; java 1.7.0_91; America/en)

http://yacy.net/bot.html

updated/0.1-alpha (updated crawler; http://www.updated.com; [email protected])

CopperEgg/RevealUptime/FrankfurtGermany

Scrapy/1.1.2 (+http://scrapy.org)

yrspider (Mozilla/5.0 (compatible; YRSpider; +http://www.yunrang.com/yrspider.html))

Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots)

yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_31; Europe/de)

http://yacy.net/bot.html

Surphace Scout&v4.0 - scout at surphace dot com

yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr)

http://yacy.net/bot.html

MetaGeneratorCrawler/1.1 (www.metagenerator.info)

Porkbun/Mustache (Website Analysis; http://porkbun.com; [email protected])

Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)

Ruby, link_thumbnailer

yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/de)
http://yacy.net/bot.html

yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/de)

http://yacy.net/bot.html

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko)

Chrome/23.0.1271.64 Safari/537.11 GotSiteMonitor.com

Mozilla/5.0 (Linux; Android 4.4; Nexus 5 Build/KRT16M) AppleWebKit/537.36 (KHTML, like Gecko)
Version/4.0 Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost

Mozilla/5.0 (KeepRight OpenStreetMap Checker; http://keepright.at) Gecko/20100101 Firefox/22.0

Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0; topster.de Linkchecker 5.0) like Gecko
(194.228.205.74)

Mozilla/5.0 AppleWebkit/537.36 (KHTML, like Gecko) Trident/7.0; rv:11.0, Chrome/42.0.2311.135,

Edge/12.10136, http://www.shrinktheweb.com/, Webshot/0.9
topicbot/1.0 (Mozilla;I;+http://www.topic.bot/contact_page.html)

DataparkSearch/4.53 (+http://dataparksearch.org/bot)

Mozilla/5.0 (compatible; Linux; InfegyAtlas/1.0; en-US; [email protected])

Mozilla/5.0 (X11; Linux x86_64; rv:41.0; GTmetrix https://gtmetrix.com/) Gecko/20100101

Firefox/41.0

Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)

Mozilla/5.0 (compatible; parsijoo-update-crawler; +http://www.parsijoo.ir/;

[email protected])

Abrave Spider v4 Robot 1 (http://robot.abrave.co.uk)

Mozilla/5.0 (compatible; memoryBot/1.21.24 +http://internetmemory.org/en/)

Mozilla/5.0 (compatible; STINGbot/1.0; +http://136.186.231.16)

domainsbot (+http://www.domainsbot.com)

WEPA/3.1 (http://www.wepa.com/; [email protected])

Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/bots/)

Mozilla/5.0 (compatible; forensiq; +http://www.forensiq.com)

192.comAgent

JUST-CRAWLER(+http://www.justsystems.com/jp/tech/crawler/)

Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) SitemapProbe

HuaweiSymantecSpider/[email protected]+(compatible; MSIE 7.0; Windows

NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ;
http://www.huaweisymantec.com/en/IRL/spider)

Mozilla/5.0 ( Macintosh; Intel Mac OS X 10_10_1 ) AppleWebKit/600.2.5 ( KHTML, like Gecko )

Version/8.0.2 Safari/600.2.5 ( compatible; CloudServerMarketSpider/1.0;
+http://cloudservermarket.com/spider.html )

Mozilla/5.0 (compatible; MJ12bot/v1.3.0; http://www.majestic12.co.uk/bot.php?+)

Mozilla/5.0 (compatible; AcoonBot/4.10.6; +http://www.acoon.de/robot.asp)

Scrapy/1.0.1 (+http://scrapy.org)

seegnifybot/1.0.0 (http://www.seegnify.com/bot)

omgili/0.5 +http://omgili.com

Slack-ImgProxy 149 (+https://api.slack.com/robots)

yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de) http://yacy.net/bot.html

www.integromedb.org/Crawler

Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm

Mozilla/5.0 (compatible) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73
Safari/537.36 [email protected]

Mozilla/5.0 (compatible; IstellaBot/1.18.81 +http://www.tiscali.it/)

Gigabot/1.0

page_test ([email protected])

istellabot/Nutch-1.10

Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like
Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Mediapartners-Google/2.1;
+http://www.google.com/bot.html)

ldspider (http://code.google.com/p/ldspider/wiki/Robots)

yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_45; Europe/ru) http://yacy.net/bot.html

facebookplatform/1.0 (+http://developers.facebook.com)

Mozilla/5.0 (compatible; RTGI; http://linkfluence.net/)

yacybot (freeworld/global; amd64 Linux 3.12.43-52.6-default; java 1.8.0_40; Europe/en)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)

Mozilla/5.0 (compatible; DCPbot/1.2; +http://domains.checkparams.com/)

WatchMouse/18990 (http://watchmouse.com/ ; d2.watchmouse.com)

MnoGoSearch/3.3.15

Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot)

Web-sniffer/1.1.0 (+http://web-sniffer.net/)

Mozilla/4.0 (compatible; Fooooo_Web_Video_Crawl http://fooooo.com/bot.html)

50.nu/0.01 ( +http://50.nu/bot.html )

Mozilla/4.0 (compatible; MSIE 7.0; Windows; Windows NT 5.1) BrokenLinkCheck.com/1.1

Comodo-Certificates-Spider

Iframely/0.9.8 (+https://iframely.com/;)

MnoGoSearch/3.3.12

CopperEgg/RevealUptime/FrankfurtEU

Python-urllib/2.7 (+http://ella.juls.savba.sk/aranea_about)

Mozilla/5.0 (compatible; JobKereso; +http://www.kozvetlen-allasok.hu/robot.jsp info@kozvetlen-

allasok.hu)

Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112

Safari/537.36 PTST/201
Mozilla/5.0 (compatible; Google Keyword Tool;
+http://adwords.google.com/select/KeywordToolExternal)

WinWebBot/1.0; (Balaena Ltd, UK); http://www.balaena.com/winwebbot.html;

[email protected];)

AppleNewsBot

classbot (+http://allclasses.com)

yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_31; America/pt)
http://yacy.net/bot.html

Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko)
Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0;
+http://www.bing.com/bingbot.htm)

yacybot (/global; arm Linux 3.12.28+; java 1.7.0_71; Europe/en) http://yacy.net/bot.html

Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01e.yandex.ru)

CopperEgg/RevealUptime/TokyoJP

Mozilla/5.0 (compatible; AhrefsBot/3.1; +http://ahrefs.com/robot/)

CJNetworkQuality; http://www.cj.com/networkquality

psbot-page (+http://www.picsearch.com/bot.html)

DealGates Bot/1.1 by Luc Michalski (http://spider.dealgates.com/bot.html)

Pingdom GIGRIB (http://www.pingdom.com)

Mozilla/5.0 (compatible; SEOlyticsCrawler/3.0; +http://crawler.seolytics.net/)

Mozilla/5.0 (compatible; EventGuruBot/1.0; +http://www.eventguru.com/spider.html)

Python-urllib/2.7 (+http://aranea.juls.savba.sk/aranea_about.html)

yacybot (/global; amd64 Windows Vista 6.0; java 1.7.0_55; America/en) http://yacy.net/bot.html

The Lemur Web Crawler/Nutch-1.3 (Lemur Web Crawler; http://boston.lti.cs.cmu.edu/crawler_12/;

[email protected])

SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI)

MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

Feedfetcher-Nuesbyte; (+http://www.nuesbyte.com)

copyright sheriff (+http://www.copyrightsheriff.com/)

yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html

Touche (+http://www.touche.com.ve)

Panopta v1.1

Mozilla/5.0 (compatible; parsijoo-bot; +http://www.parsijoo.ir/; [email protected])

Zookabot/2.2;++http://zookabot.com

yacybot (/global; amd64 Linux 2.6.32-042stab093.4; java 1.7.0_65; Etc/en) http://yacy.net/bot.html

it2media-domain-crawler/2.0

yacybot (/global; amd64 Linux 4.2.0-27-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html

Mozilla/5.0 (Android; Mobile; rv:21.0) Gecko/21.0 Firefox/21.0 commoncrawl.org/research//Nutch-

1.7-SNAPSHOT

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; Data Center; .NET CLR 1.1.4322;
.NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR
3.5.30729; PTST 2.386)

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0

Safari/537.36 YottaaMonitor

CopperEgg/RevealUptime/AtlantaGAUSA

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/fr) http://yacy.net/bot.html

Mozilla/5.0 (compatible; MJ12bot/v1.2.4; http://www.majestic12.co.uk/bot.php?+)

Mozilla/5.0 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)

ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/ucri/plan/crawl.html)

Mozilla/5.0 (compatible; YYSpider; +http://www.yunyun.com/spider.html)

Mozilla/5.0 (iPhone; CPU iPhone OS 5_0_1 like Mac OS X) (compatible; Yeti-Mobile/0.1;

+http://help.naver.com/robots/)

Mozilla/5.0 (WhatsMyIP.org Text_to_Code_Ratio_Tool) http://whatsmyip.org/ua

Thumbnail.CZ robot 1.1 (http://thumbnail.cz/why-no-robots-txt.html)

Google-Calendar-Importer

Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) Websnapr/3.0

Safari/534.34

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/50.0.2661.75 Safari/537.36 PTST/276

WeSEE

librabot/2.0 (+http://academic.research.microsoft.com/)

Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0.1

Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) PTST/276

Cliqzbot/0.1 (+http://cliqz.com [email protected])

Scrubby/3.0 (+http://www.scrubtheweb.com/help/technology.html)
Pinterest/0.1 +http://pinterest.com/

Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 52327)

Comodo-Webinspector-Crawler 2.2.0, http://www.comodorobot.com

Iframely/0.6.0 (+http://iframely.com/;)

Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)

Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)

Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/blog-search/)

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/49.0.2623.112 Safari/537.36 PTST/276

Mozilla/5.0 (compatible; U; Hootsuite-WebFeed/1.0; mailto:[email protected])

RankFlex.com Webspider

TipTop http://feeltiptop.com

vebidoobot

Mozilla/5.0 (compatible; Dow Jones Searchbot)

Mozilla/5.0 (compatible; OrangeBot-Collector/2.0; [email protected])

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) MsnBot-Media

/1.0b

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6

Ara.com.tr AraBot 1.0

Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko)
Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile;
+http://www.google.com/mobile/adsbot.html)

Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01g.yandex.ru)

BUbiNG (+http://law.di.unimi.it/BUbiNG.html)

downnotifier.com monitoring

A6-Indexer

Mozilla/5.0 (compatible; htrix/1.14.4 ++https://webarchiv.onb.ac.at/robot.html)

MetaURI API/2.0 +metauri.com

Mozilla/5.0 (compatible; MixrankBot; [email protected])

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/44.0.2403.155 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)

Mozilla/5.0 Project 25499 (project25499.com)

COMODOspider/Nutch-1.0
Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html)

Uptime.com_(http://uptime.com/)

AcoonBot/4.10.5 (+http://www.acoon.de)

BDFetch

20090717_aa_page ([email protected])

Metaspinner/1.0 (Metaspinner Search Engine; http://www.meta-spinner.de/; support@meta-

spinner.de)

Mozilla/5.0 (compatible; DBLBot/1.0; +http://www.dontbuylists.com/)

Mozilla/5.0 (compatible; spbot/2.0; +http://www.seoprofiler.com/bot/ )

LYT.SR v1.5 +http://lyt.sr/ (running on http://lyt.sr)

Putin (+http://semanticjuice.com/)

Mozilla/5.0 (compatible; LA1; +http://www.zeerch.com/bot.php)

WatchMouse/18990 (http://watchmouse.com/ ; ny)

bitlybot/2.0

yacybot (/global; amd64 Linux 3.13.0-45-generic; java 1.7.0_75; Europe/en) http://yacy.net/bot.html

Webshot/0.9 ( http://www.shrinktheweb.com/ )

MetaURI API +metauri.com

Mozilla/5.0 (compatible; ProductoDownloadUrlBot/1.0; +http://www.producto.de/)

Google-Adwords-Instant (+http://www.google.com/adsbot.html)

yacybot (freeworld/global; amd64 Linux 3.8.0-19-generic; java 1.7.0_25; Europe/en)

http://yacy.net/bot.html

yacybot (freeworld/global; i386 Linux 2.6.32-5-686; java 1.6.0_18; Europe/en)

http://yacy.net/bot.html

Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/42.0.2311.90 Safari/537.36 PTST/277

Mozilla/5.0 (compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot)

HyperZbozi.cz Feeder/3.1

Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET

CLR 3.5.30729) SurveyBot/2.3 (DomainTools)

yacybot (/global; amd64 Windows Server 2012 R2 6.3; java 1.7.0_55; Europe/de)
http://yacy.net/bot.html

Google-AdWords-Express

Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)

SMTBot (similartech.com/smtbot)

Mozilla/5.0 (compatible; Lipperhey SEO Service; http://www.lipperhey.com/)

MetaInspector/4.7.2 (+https://github.com/jaimeiniesta/metainspector)

Mozilla/5.0 (Compatible; Vedma/0.91Beta; +http://www.vedma.ru/bot.htm)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) web-capture.net/0.9

Safari/537.21

Nokia6680/1.0 (4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1

(botmobi find.mobi/bot.html [email protected])

Mozilla/5.0 (compatible; Exploratodo/1.0; +http://www.exploratodo.com

iZSearch.com

Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)

bl.uk_lddc_bot/3.3.0-SNAPSHOT-2014-10-07T09:33:31Z
(+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)

yacybot (/global; amd64 Linux 3.14-0.bpo.1-amd64; java 1.7.0_55; Europe/de)

http://yacy.net/bot.html

HuaweiSymantecSpider/[email protected]+(compatible; MSIE 7.0; Windows

NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ;
http://www.huaweisymantec.com/cn/IRL/spider)

FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)

Mozilla/5.0 (compatible; DripfeedBot/2.0; +http://dripfeedbookmark.com/bot.html

Mozilla/5.0 (compatible; heritrix/2.0.2 +http://seekda.com)

yacybot (freeworld/global; i386 Linux 3.2.0-23-generic-pae; java 1.7.0_03; Europe/en)

http://yacy.net/bot.html

hawkReader/0.9b (Feed Parser; http://www.hawkreader.com; Allow like Gecko)

ruijie ([email protected])

CorpusCrawler 2.0.0 (http://corpora.fi.muni.cz/crawler/)

ExactSeek Crawler (http://www.exactseek.com/)/Nutch-1.4

WordPress.com; https://botd.wordpress.com

Mozilla/5.0 (compatible; Qwantify/2.0n; +https://www.qwant.com/)/*

Mozilla/5.0 (compatible; memoryBot/1.20.268 +http://internetmemory.org/en/)

ShowyouBot (http://showyou.com/crawler)

Yoleo Consumer v0.2

Mozilla/5.0 (compatible; IstellaBot/1.10.2 +http://www.tiscali.it/)

Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;
+http://ws.daum.net/aboutWebSearch.html) Daumoa/3.0

Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86

YaBrowser/15.12.0.6151 Safari/537.36 Yandex.Translate

Mozilla/5.0 (compatible; TeeRaidBot; +https://teeraid.com/bot/)

Mozilla/5.0 (compatible; Scopia Crawler 1.1; +http://www.scopia.co)

CopperEgg/RevealUptime/DublinIE(aws)

yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; Asia/ja) http://yacy.net/bot.html

RankurBot/Rankur2.1 (http://rankur.com; info at rankur dot com)

Gaisbot/3.0+([email protected];+http://gais.cs.ccu.edu.tw/robot.php)

Mozilla/5.0 (compatible; zitebot support [at] zite [dot] com +http://zite.com)

MetaTagRobot/2.0 (http://www.check-domains.com/website-analysis/website-analyzer.php)

Virusdie crawler/2.1

findlinks/2.0.4 (+http://wortschatz.uni-leipzig.de/findlinks/)

Mozilla/5.0 (compatible; AcoonBot/4.10.7; +http://www.acoon.de/robot.asp)

Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)

yacybot (-global; amd64 Linux 3.19.0-15-generic; java 1.8.0_45-internal; Europe/de)

http://yacy.net/bot.html

Google_Analytics_Snippet_Validator

SurcentroBot

Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com)

findlinks/1.1.3-beta9 (+http://wortschatz.uni-leipzig.de/findlinks/)

Mozilla/5.0 (compatible; memoryBot/1.20.199 +http://internetmemory.org/en/)

Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)

TwengaBot-Discover (http://www.twenga.fr/bot-discover.html)

Mozilla/5.0 (bl.uk_lddc_bot; Linux x86_64) PhantomJS/1.9.7

(+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)

panscient.com

Mozilla/5.0 (compatible; TestCrawler; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-

[email protected])

DataparkSearch/4.37-23012006 ( http://www.dataparksearch.org/)

GOFORITBOT ( http://www.goforit.com/about/ )

CopperEgg/RevealUptime/SydneyAustralia
msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)

Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/?tab=linkAnalyze)

MetaCommentBot; http://metacomment.io/about

Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html)

Twikle/1.0 , http://twikle.com , [email protected]

yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.8.0_60; Europe/de)
http://yacy.net/bot.html

Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl;

http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)

Comodo-Webinspector-Crawler 2.1

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.11 (KHTML, like Gecko)

Chrome/23.0.1271.64 Safari/537.11 KimonoLabs/0.2

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1 + FairShare-http://fairshare.cc)

Scrapy/0.24.5 (+http://scrapy.org)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Site-Shot/2.1

(http://www.site-shot.com/) Safari/534.34

Port Monitor check service 1.0 | http://www.port-monitor.com

Mozilla/5.0 (compatible; Uptimebot/0.1.43; +http://www.uptime.com/uptimebot)

yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en)

http://yacy.net/bot.html

SolomonoBot/1.04 (http://www.solomono.ru)

Easy-Thumb (https://www.easy-thumb.net/)

Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112

Safari/537.36 PTST/276

nic.it crawler v.1.221115 (7.35.0)

Mozilla/5.0 (compatible; memorybot/1.24.12 +https://internetmemory.net/en/bots)

Mozilla/5.0 (compatible; inoreader.com-like FeedFetcher-Google)

yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_72; Europe/de) http://yacy.net/bot.html

WatchMouse/8.4.0.3 (http://watchmouse.com/ ; demuc02.watchmouse.net)

Mozilla/5.0 (X11; Linux x86_64; rv:43.0; GTmetrix https://gtmetrix.com/) Gecko/20100101

Firefox/43.0

ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)

Mozilla/5.0 (compatible; UnisterBot; http://www.bluekiwi.de/misc/imprint)

Web-sniffer/1.0.31 (+http://web-sniffer.net/)

woobot

Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot;

https://www.jobboerse.com/bot.htm) Gecko/20100401 Firefox/24.0

AntBot/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; [email protected])

Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; Evrinid Iudex 1.0.0; +http://www.evri.com/evrinid)

adidxbot/1.1 (+http://search.msn.com/msnbot.htm)

Mozilla/5.0 (compatible; WebThumbnail/2.2; Website Thumbnail Generator;

+http://webthumbnail.org)

Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/)

EasyBib AutoCite (http://content.easybib.com/autocite/)

yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html

Mozilla/5.0 (compatible; woriobot +http://worio.com)

Mozilla/4.0 (compatible; Vagabondo/4.0; http://webagent.wise-guys.nl/)

SniffRSS/0.5beta (+http://www.blogator.com/)

woobot/1.1

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) WordPress.com mShots
Safari/537.36

ADmantX Platform Semantic Analyzer - Turn - ADmantX Inc. - www.admantx.com -

[email protected]

Mozilla/5.0 (compatible; UnisterBot; [email protected])

Mozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/)

UptimeDog Robot (www.uptimedog.com)

Googlebot/2.1 (+http://www.google.com/bot.html)

Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0 (NetShelter

ContentScan, contact [email protected] for information)

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; Media
Center PC 5.0; .NET CLR 3.5.30729; .NET4.0C; .NET CLR 3.0.30729; .NET4.0E) PTST/276

ServiceUptime.robot

Mozilla/4.0 (compatible;HostTracker/2.0;+http://www.host-tracker.com/)

SEO Browser

Leikibot/1.0 (+http://www.leiki.com)
yacybot (webportal-global; amd64 Windows 7 6.1; java 1.7.0_04; America/en)
http://yacy.net/bot.html

AppEngine-Google; (+http://code.google.com/appengine; appid: downforeveryoneorjustme)

Mozilla/5.0 (Windows NT 5.1) BrokenLinkCheck.com/1.1

Seobility (SEO-Check; http://bit.ly/1dJuuzs)

Mozilla/5.0 (seoanalyzer; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and

instructions.

Site24x7

websitepulse checker/3.0 (compatible; MSIE 5.5; Netscape 4.75; Linux)

Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)

Mozilla/5.0 (compatible; NerdByNature.Bot; http://www.nerdbynature.net/bot)

Abrave Spider v4 Robot 2 (http://robot.abrave.co.uk)

HubPages V0.2.2 (http://hubpages.com/help/crawlingpolicy)

lmspider ([email protected])

magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)

Mozilla/5.0 (compatible; WbSrch/1.1 +http://wbsrch.com

Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.7; Google-SearchByImage)

Gecko/2009021910 Firefox/3.0.7

Mozilla/5.0 (compatible; Najdi.si/3.1)

larbin_2.6.3 ([email protected])

Mozilla/5.0 (compatible; aiHitBot/1.1; +http://www.aihit.com/)

Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/50.0.2661.102 Safari/537.36 PTST/277

Mozilla/5.0 (YahooYSMcm/3.0.0; http://help.yahoo.com)

yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html

OOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )

findlinks/2.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/)

Mozilla/5.0 (compatible; Lipperhey Link Explorer; http://www.lipperhey.com/)

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6

(FlipboardProxy/1.1; +http://flipboard.com/browserproxy)

Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)

Mozilla/5.0 (compatible; memoryBot/1.24.11 +http://internetmemory.org/en/)

RED/1 (https://redbot.org/)

Twitterbot/1.0

Mozilla/5.0 (compatible; ptd-crawler; +http://bixolabs.com/crawler/ptd/; [email protected])

Open Web Analytics Bot 1.5.4

JikeSpider Mozilla/5.0 (compatible; JikeSpider; +http://shoulu.jike.com/spider.html)

Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)

Mozilla/5.0 (compatible; oBot/2.3.1; +http://www-

935.ibm.com/services/us/index.wss/detail/iss/a1029077?cntxt=a1027244)

elefent/Elefent 1.2 (A friendly web elefent.; http://elefent.eu/; [email protected])

Genderanalyzer/1.0

elefent/1.2 (Web Crawler; https://crawler.elefent.net; webmaster at elefent dot net)

Yandex/1.01.001 (compatible; Win16; H)

crawler4j for XQuery

Mozilla/5.0 (compatible; Plukkie/1.4; http://www.botje.com/plukkie.htm)

LinguaBot/v0.001-dev (MultiLinual Sarch Engine v0.001; LinguaSeek; admin at linguaseek dot com)

MrakyPraceCzBot/1.0 (http://www.mrakyprace.cz/about-crawler)

Y!J-BRJ/YATS crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)

WatchMouse/8.4.0.3 (http://watchmouse.com/ ; plgdn01.watchmouse.net)

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like

Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)

AskQuickly v2 (http://askquickly.org/)

Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)

Miniflux (http://miniflux.net)

Mozilla/5.0 (compatible; spbot/5.0.1; +http://OpenLinkProfiler.org/bot )

Mozilla/5.0 (compatible; wmtips.com/1.0; +http://www.wmtips.com/tools/)

Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)

Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)

Google favicon

Mozilla/5.0(compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)

MiaDev/0.0.1 (MIA Bot for research project MIA (www.MIA-marktplatz.de); http://www.mia-

marktplatz.de/spider; [email protected])
Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/33.0.1750.146 Safari/537.36 Specificfeeds- http://www.specificfeeds.com

Mozilla/5.0 (compatible; socialbm_bot/1.0; +http://spider.socialbm.net)

wp.com feedbot/1.0 (+https://wp.com)

Mozilla/5.0 (compatible; CligooRobot/2.0; +http://www.cligoo.de/wk/technik.php)

Slack-ImgProxy (+https://api.slack.com/robots)

Mozilla/5.0 (compatible; Online Domain Tools - HTTP Headers Online/1.0; +http://http-

headers.online-domain-tools.com)

Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)

WGETbot/1.0 (+http://wget.alanreed.org)

yacybot (/global; amd64 Linux 4.0.9-2-ck; java 1.8.0_51; Europe/de) http://yacy.net/bot.html

Y!J-BRI/0.0.1 crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html )

dlvr.it/1.0 (+http://dlvr.it/) Mozilla/5.0

yacybot (/global; arm Linux 4.1.13-v7+; java 1.8.0; Etc/de) http://yacy.net/bot.html

Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa.com/crawler.html)

yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/fr)

http://yacy.net/bot.html

Link Valet Online 1.2

Mozilla/5.0 (compatible; METASpider; +http://meta.ua/spider)

NeutrinoAPI/2.0.1

thumbshots-de-bot (+http://www.thumbshots.de/)

Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko)
Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Applebot/0.3;
+http://www.apple.com/go/applebot)

argenfybot

MergadoBot

Ant/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; [email protected])

CopperEgg/RevealUptime/SaoPauloBrazil

Mozilla/4.0 (compatible; Vagabondo/4.0/EU; http://webagent.wise-guys.nl/)

Mozilla/4.0 (compatible; DAWINCI ANTIPLAG SPIDER)

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276

HetrixTools.COM Uptime Monitoring Bot. https://hetrixtools.com/uptime-monitoring-bot.html

yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html

Mozilla/5.0 (Windows NT 6.1) (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)

Mozilla/5.0 (compatible; InbyBot/2.1; +http://inbyapp.com)

all_web1 ([email protected])

Mozilla/5.0 (compatible; BusinessSeek.biz_Directory; http://www.businessseek.biz/)

Wotbox/2.0 ([email protected]; http://www.wotbox.com)

Mediatoolkitbot ([email protected])

OpenWebSpider v0.1.2.B (http://www.openwebspider.org/)

Mozilla/5.0 (compatible; WebCorp/5.0; +http://www.webcorp.org.uk)

Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0

fastbot crawler beta 2.0 (+http://www.fastbot.de)

Sogou Web Spider

DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile

goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)

Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/searchengine)

XoviBot/1.0

Mozilla/5.0 (compatible; Online Domain Tools - Online Website Link Checker/1.1; +http://website-
link-checker.online-domain-tools.com)

WatchMouse/8.4.0.3 (http://watchmouse.com/ ; decgn02.watchmouse.net)

yacybot (i386 Linux 2.6.31-21-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html

ZnajdzFoto/ImageBot 2.0b

Mozilla/5.0 (compatible; heritrix/3.0.0-SNAPSHOT-20091120.021634 +http://crawler.archive.org)

Mozilla/5.0 (compatible; ProCogSEOBot/1.0; +http://www.procog.com/ )

Mozilla/5.0 (compatible; Plukkie/1.6; http://www.botje.com/plukkie.htm)

yacybot (freeworld/global; amd64 Linux 3.2.0-21-generic; java 1.7.0_03-icedtea; America/en)

http://yacy.net/bot.html

Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko)
Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)

Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +

http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0

Mozilla/5.0 (compatible; NLNZ_IAHarvester2014 +http://natlib.govt.nz/about-us/current-

initiatives/web-harvest-2014)
Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot)

Mozilla/5.0 (compatible; special_archiver/3.3.0

+http://www.loc.gov/webarchiving/notice_to_webmasters.html)

yacybot (freeworld/global; i386 Linux 3.4.2-linode44; java 1.6.0_27; America/en)

http://yacy.net/bot.html

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/47.0.2526.111 Safari/537.36 Yandex.Translate

yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en)

http://yacy.net/bot.html

FreeWebMonitoring SiteChecker/0.1 (+http://www.freewebmonitoring.com)

Mozilla/5.0 (compatible; Infohelfer/1.3.0; +http://www.infohelfer.de/crawler.php)

Kemvibot/1.0 (http://kemvi.com)

Mozilla/4.0 (compatible; HostTracker.com/1.0;+http://host-tracker.com/)

AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU

iPhone OS 3 0 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile Safari

Mozilla/5.0 (compatible; GeliyooBot/1.0; +http://www.geliyoo.com/)

yacybot (x86 Windows XP 5.1; java 1.6.0_21; Europe/de) http://yacy.net/bot.html

Mozilla/5.0 (compatible; archive.org_bot/3.1.2 +http://natlib.govt.nz/about-us/current-

initiatives/web-harvest-2012)

Mozilla/5.0 (compatible; StorygizeBot; http://www.storygize.com)

infohelfer/1.1.0 (http://www.infohelfer.de/)

yacybot (/global; amd64 Linux 3.16.0-41-generic; java 1.7.0_80; Europe/de) http://yacy.net/bot.html

SeoCheckBot (FischerNetzDesign Seo Checker, [email protected])

netEstate NE Crawler (+http://www.website-datenbank.de/)

Mail.Ru/1.0

binlar_2.6.3 ([email protected])

LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1

+http://www.linkedin.com)

bot.wsowner.com/1.0.0 (+http://wsowner.com/)

Ronzoobot/1.5 (http://www.ronzoo.com/about/)

Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.mixdata.com)

Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)

findlinks/1.1.6-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/)
Mozilla/5.0 (compatible; LoadTimeBot/0.7; +http://www.load-time.com/bot.html)

Mozilla/5.0 (compatible; MJ12bot/v1.4.2; http://www.majestic12.co.uk/bot.php?+)

Mozilla/4.0 (compatible; MSIE 5.01; GomezAgent 2.0; Windows NT)

Mozilla/5.0 (compatible; MJ12bot/v1.3.2; http://www.majestic12.co.uk/bot.php?+)

SEOENGBot/1.2 (+http://learn.seoeng.com/seoengbot.htm)

yacybot (i386 Linux 2.6.31-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html

Online Virus Scanner: http://tools.geek-tools.org

DwnldBot (+http://dwnld.me)

Mozilla/5.0 (compatible; OpenfosBot/2.1; http://www.openfos.com)

taptubot *** please read http://www.taptu.com/corp/taptubot ***

yacybot (freeworld/global; amd64 Linux 3.9.3-1-ARCH; java 1.7.0_21; America/en)

http://yacy.net/bot.html

Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot)

minicrawler/4.0.0~beta10

rogerbot/1.0 (http://www.moz.com/dp/rogerbot, [email protected])

Mozilla/5.0 (Windows NT 6.1; Win64; x64) KomodiaBot/1.0

Mozilla/5.0 (compatible; spbot/3.0; +http://www.seoprofiler.com/bot )

Mozilla/5.0 (compatible; DomainTunoCrawler/0.1; +https://www.domaintuno.com/robot)

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1;

+http://www.google.com/bot.html) Safari/537.36

yacybot (amd64 Linux 2.6.32-gentoo; java 1.6.0_17; Europe/en) http://yacy.net/bot.html

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like

Gecko; Google Page Speed Insights) Chrome/27.0.1453 Mobile Safari/537.36

Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa-update.com/crawler)

Cloud mapping experiment. Contact [email protected]

emefgebot/beta (+http://emefge.de/bot.html)

DoCoMo/2.0 D903i(c100;TB;W28H20) (compatible; Hatena-Mobile-Gateway/1.2;

+http://mgw.hatena.ne.jp/help)

ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html)

Mozilla/5.0 (compatible; SecurityResearch.bot; +http://besome1.info/securityresearchbot.html)

WPScan v2.9 (http://wpscan.org)

Ronzoobot/1.6 (http://www.ronzoo.com/about/)

livedoor ScreenShot/0.10
Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) KomodiaBot/1.0

Mozilla/5.0 (compatible; memoryBot/1.20.289 +http://internetmemory.org/en/)

Mozilla/5.0 (compatible; DomainTunoCrawler/0.1; +http://www.domaintuno.com/robot)

yacybot (freeworld/global; amd64 Linux 2.6.26-2-amd64; java 1.6.0_18; Europe/de)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa.com/crawler)

Mozilla/5.0 (compatible; spbot/5.0; +http://OpenLinkProfiler.org/bot )

Mozilla/5.0 (compatible; Webauskunft.org; http://www.webauskunft.org)

CCResearchBot/1.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT

Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)

linkdex.com/v2.0

Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 73393)

Mozilla/5.0 (compatible; SEODat/0.1 http://crawler.seodat.com)

yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en)

http://yacy.net/bot.html

CorpusCrawler 2.0.12 (http://corpora.fi.muni.cz/crawler/)

Mozilla/5.0 (compatible; spbot/4.3.0; +http://OpenLinkProfiler.org/bot )

Mozilla/5.0 (compatible; aiHitBot/2.7; +http://www.aihit.com/)

Mozilla/5.0 (compatible; SiteAnalyzerBot/4.0; +https://www.site-analyzer.com/)

DataparkSearch/4.40.1 (+http://www.dataparksearch.org/)

yacybot (freeworld-global; amd64 Linux 3.0.0-21-generic; java 1.6.0_26; America/en)

http://yacy.net/bot.html

findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/; YaCy 0.1; yacy.net)

CopperEgg/RevealUptime/SydneyAU(aws)

OpenAcoon v4.1.0 (www.openacoon.de)

semanticbot ([email protected])

Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 75425)

Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 11912)

Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Mozilla/5.0 (Windows NT 5.1; U; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-
ftgroup.com)

yacybot (-global; amd64 Linux 3.13.0-61-generic; java 1.7.0_79; Europe/en) http://yacy.net/bot.html

CorpusCrawler 2.0.25 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus

ichiro/3.0 (http://search.goo.ne.jp/option/use/sub4/sub4-1/)

R6_FeedFetcher_(www.radian6.com/crawler)

sosoimagespider+(+http://help.soso.com/soso-image-spider.htm)

Mozilla/5.0 (X11; Linux x86_64; rv:39.0; GTmetrix http://gtmetrix.com/) Gecko/20100101

Firefox/39.0

yacybot (freeworld/global; amd64 Linux 3.2.0-24-generic; java 1.6.0_24; Europe/en)

http://yacy.net/bot.html

Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-

guys.nl/)

CommaFeed/2.4.0-SNAPSHOT (https://github.com/Athou/commafeed)

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics)

Chrome/27.0.1453 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

yacybot (/global; x86 Windows 7 6.1; java 1.8.0_31; Europe/de) http://yacy.net/bot.html

Mozilla/5.0 (compatible; woriobot support [at] zite [dot] com +http://zite.com)

Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.at/crawler)

bl.uk_lddc_bot/3.3.0-SNAPSHOT-20140519-0844
(+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)

FFbot/1.0; +http://feedfury.com

Mozzila/5.0 (compatible; YamanaLab-Robot/1.0;

http://www.yama.info.waseda.ac.jp/~crawler/info.html)

LinqiaRSSBot/1.0 ([email protected])

FreeWebMonitoring SiteChecker/0.2 (+http://www.freewebmonitoring.com/bot.html)

Mozilla/5.0 (compatible; Web-Monitoring/1.0; +http://monoid.nic.ru/)

Mozilla/5.0 (compatible; heritrix/1.12.1b +http://netarkivet.dk/website/info.html)

Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research);

http://nlp.fi.muni.cz/projects/biwec/)

UrlTrends Ranking Updater/2.0 ( http://www.urltrends.com/ )

yacybot (/global; amd64 Linux 3.16.1-gentoo; java 1.7.0_55; Europe/en) http://yacy.net/bot.html

Mozilla/5.0 (compatible; gimme60bot/1.0 ; +http://gimme60.com)

CatchBot/5.0; +http://www.catchbot.com

elefent/Elefent 1.1 (Friendly web walker.; http://elefent.eu/; [email protected])

findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
Googlebot-Video/1.0

Mozilla/5.0 (compatible; XmarksFetch/1.0; +http://www.xmarks.com/about/crawler;

[email protected])

XRL/3.00 (Linux; i686; en-us) (+http://metamark.net/about)

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/50.0.2661.87 Safari/537.36 PTST/276

CorpusCrawler 2.0.8 (http://corpora.fi.muni.cz/crawler/)

Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)

Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://openindex.io/spider.html;

systemsATopenindexDOTio)

Ronzoobot/1.3 (http://www.ronzoo.com/about.php)

MetaInspector/2.2.1 (+https://github.com/jaimeiniesta/metainspector)

Mozilla/5.0 (compatible; WASALive-Bot ; http://blog.wasalive.com/wasalive-bots/)

Mozilla/5.0 (compatible; YioopBot; +http://www.yioop.com/bot.php)

Mozilla/5.0 (compatible; Mozilla/5.0; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-

[email protected])

Mozilla/5.0 (compatible; Kraken/0.1; http://linkfluence.net/; [email protected])

Mozilla/5.0 (compatible; Qseero; +http://www.q0.com)

yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.8.0_45; Europe/en)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; ProCogBot/1.0; +http://www.procog.com/spider.html)

Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)

Mozilla/5.0 (Anturis Agent)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Chromium/31.0.1650.63

Site-Shot/2.1 (http://www.site-shot.com/) Safari/534.34

webnumbrFetcher/1.0 (http://webnumbr.com/)

Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)

NIF/1.1 (http://www.newsisfree.com/robot.php users:)

Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103

Safari/537.36 PTST/284

DoCoMo/2.0 P901i(c100;TB;W24H11) (compatible; ichiro/mobile goo;

+http://help.goo.ne.jp/door/crawler.html)

Iframely/0.8.8 (+https://iframely.com/;)

yacybot (ppc Mac OS X 10.5.2; java 1.5.0_13; Europe/de) http://yacy.net/bot.html

Mozilla/5.0 (compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)

yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html

Mozilla/5.0 (compatible; FeedBooster; +http://feeds.qsensei.com)

yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; SputnikFaviconBot/1.2; +http://corp.sputnik.ru/webmaster)

Mozilla/5.0 (compatible; heritrix/3.1.1; UniLeipzigASV +http://corpora.informatik.uni-

leipzig.de/crawler_faq.html)

Mozilla/5.0 (compatible; OrangeBot/2.0; [email protected])

Yeti-FeedItemCrawler/1.0 (NHN Corp.; http://help.naver.com/robots/)

Mozilla/5.0 (compatible; SeznamBot/3.2-test1-1; +http://napoveda.seznam.cz/en/seznambot-intro/)

Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.crim.ca)

yacybot (/global; amd64 Linux 4.0.5-gentoo; java 1.8.0_51; Europe/en) http://yacy.net/bot.html

pr-cy.ru Screenshot Bot

GentleSource Short URL Checker ( http://www.gentlesource.com/ )

crazywebcrawler 0.9.0, http://www.crazywebcrawler.com

a.pr-cy.ru

Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko)
Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0;
http://www.bing.com/bingbot.htm)

findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)

ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)

yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_25; Europe/en) http://yacy.net/bot.html

Yeti/1.1 (Naver Corp.; http://help.naver.com/robots/)

Technoratibot/8.0

Setooz/Nutch-1.0 (http://www.setooz.com)

Mozilla/5.0 (Windows NT 5.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112

Safari/537.36 PTST/201

semanticbot

Mozilla/5.0 (compatible; JikeSpider; +http://shoulu.jike.com/spider.html)

Mozilla/5.0 (compatible; MetamojiCrawler/1.0; +http://www.metamoji.com/jp/crawler.html

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 -

James BOT - WebCrawler http://cognitiveseo.com/bot.html
Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0;
+http://ecommerce.bigdatacorp.com.br/faq.aspx) ppleWebKit/537.36 (KHTML, like Gecko)
Chrome/41.0.2272.118 Safari/537.36

Mozilla/5.0 (compatible; CloudServerMarketSpider/1.0;

+http://www.cloudservermarket.com/spider.html)

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0; Neustar WPM) Gecko/20100101 Firefox/38.0

Mozilla/5.0 (compatible;+ParchBot/1.0;++http://www.parchmenthill.com/search.htm)

Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)

MetaHeadersBot (+http://www.metaheaders.com/bot.html)

Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.4.2; trendiction media

ssppiiddeerr; http://www.trendiction.com/bot/; please let us know of any problems; ssppiiddeerr at
trendiction.com) Gecko/20071127 Firefox/2.0.0.11

voyager/2.0 (http://www.kosmix.com/crawler.html)

Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )

hivaBot/hivaBot-1.0 (Iranian Search Engine bot; http://yooz.ir; [email protected])

yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_55; Europe/de)

http://yacy.net/bot.html

AppEngine-Google; (+http://code.google.com/appengine; appid: s~closure-compiler-hrd)

larbin_2.5.0 ([email protected])

Scrapy/0.22.2 (+http://scrapy.org)

Ronzoobot/1.2 (http://www.ronzoo.com/about.php)

gooblog/2.0 (http://help.goo.ne.jp/contact/)

Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0

Safari/535.21 PagePeeker/2.1; +http://pagepeeker.com/robots/

WeSEE:Ads/PageBot (http://www.wesee.com/bot/)

yacybot (amd64 Linux 2.6.18-164.el5; java 1.6.0; Europe/de) http://yacy.net/bot.html

SemanticScholarBot/1.0 (+http://s2.allenai.org/bot.html)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535. Safari/535.22+;

(+http://immediatenet.com/thumbnail_api.html; free thumbnails)

Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 85747)

SBIder/Nutch-1.0-dev (http://www.sitesell.com/sbider.html)

Scrapy/1.0.4 (+http://scrapy.org)

SanszBot/1.7(http://www.sansz.org/sanszbot, [email protected]) ([email protected])

Mozilla/5.0 (compatible; KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36
[email protected]

Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3 like Mac OS X; en-GB) adbeat.com/policy

AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8F190 Safari/6533.18.5

Curious George - www.analyticsseo.com/crawler

Grammarly/1.0 (http://www.grammarly.com)

OpenCalaisSemanticProxy

yacybot (freeworld/global; amd64 Linux 2.6.32-49-server; java 1.6.0_27; Europe/en)

http://yacy.net/bot.html

netEstate RSS crawler (+http://www.rss-directory.info/)

ipv6-test.com validator

Mozilla/5.0 (compatible; Qualidator.com Bot 1.0;)

ichiro/2.0 ([email protected])

yacybot (freeworld/global; amd64 Linux 2.6.38-14-generic; java 1.6.0_22; Europe/en)

http://yacy.net/bot.html

cg-eye live

Mozilla/5.0 (compatible; OpenindexSpider; +http://www.openindex.io/en/webmasters/spider.html)

Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US) AppEngine-Google;

(+http://code.google.com/appengine; appid: s~virustotalcloud)

Mozilla/5.0 (compatible; heritrix/1.14.3.r6601 +http://www.buddybuzz.net/yptrino)

MetaGeneratorCrawler/1.3.4 (www.metagenerator.info)

Mozilla/5.0 (Linux; Android 5.0; LG-D855 Build/LRX21R.A1422018487) AppleWebKit/537.36 (KHTML,

like Gecko) Chrome/40.0.2214.109 Mobile Safari/537.36 DareBoost

Mozilla/5.0 (compatible; Nigma.ru/3.0; [email protected])

Mozilla/5.0 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer)

yacybot (/global; x86 Windows XP 5.1; java 1.7.0_71; Europe/de) http://yacy.net/bot.html

Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102

Safari/537.36 PTST/277

OpenWebSpider v0.1.4 (http://www.openwebspider.org/)

HubSpot Connect 1.0 (http://dev.hubspot.com/)

Page Valet/4.1pre5

msnbot/2.0b (+http://search.msn.com/msnbot.htm)

amibot - http://www.amidalla.de - [email protected] libwww-perl/5.831

SETOOZBOT/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html )

Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )

Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa 4.0

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) LinkCheck by Siteimprove.com

EdisterBot (http://www.edister.com/bot.html)

Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko)
Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0;
+http://www.similartech.com/smtbot)

Tarantula/BETA-0.5 (Tarantula de la Enlle, el mejor buscador en la red; http://www.enlle.com;

[email protected])

Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2

([email protected])

yacybot (freeworld/global; amd64 Linux 2.6.38-8-generic; java 1.6.0_22; Europe/en)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; SemrushBot/0.95; +http://www.semrush.com/bot.html)

CatchBot/3.0; +http://www.catchbot.com

Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: [email protected])

Mozilla/5.0 (compatible; RukiCrawler/1.0; +http://ruki.rezko.net)

adbeat_bot

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler

Mozilla/5.0 (compatible; YandexCatalog/3.0; +http://yandex.com/bots)

NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; [email protected])

Statastico/4.0

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko, Google Keyword
Suggestion) Chrome/10.0.648.127 Safari/534.16

Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko)
Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b

EuripBot/2.0 (+http://www.eurip.com) GetRobots

Mozilla/5.0 (compatible; MJ12bot/v1.2.1; http://www.majestic12.co.uk/bot.php?+)

Braintree-Webhooks

Nutch Checksem/Nutch-1.10

Mozilla/5.0 (compatible; HyperCrawl/0.2; +http://www.seograph.net/bot.html)

quickobot/quickobot-1 (Quicko Labs; http://quicko.co; robot at quicko dot co)

Whibse

Yahoo! Site Explorer Feed Validator

http://help.yahoo.com/l/us/yahoo/search/siteexplorer/manage/

Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)

Siteshooter B0t 2.1

GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm;

support at ginger software dot com)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed
Insights) Chrome/27.0.1453 Safari/537.36

eBot / v.0.1e (http://alfa.elchron.cz)

Mozilla/5.0 (compatible; memoryBot/1.20.247 +http://internetmemory.org/en/)

yacybot (x86 Windows 2003 5.2; java 1.6.0_20; America/de) http://yacy.net/bot.html

YandexSomething/1.0

Scoop

Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)

Mozilla/5.0 (compatible; SputnikBot/2.3; +http://corp.sputnik.ru/webmaster)

TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html)

Mozilla/5.0 (compatible; Ask Jeeves/Teoma;

+http://about.ask.com/en/docs/about/webmasters.shtml)

Mozilla/5.0 (compatible; CrawlForMe/3.0.2; [email protected])

Seologies/0.1 Bot

www.deadlinkchecker.com XMLHTTP/1.0 Mozilla/5.0 (Windows NT 6.1; WOW64)

AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36

dubaiindex (addressendeutschland.de)

yacybot (/global; x86 Windows 7 6.1; java 1.8.0_40; America/es) http://yacy.net/bot.html

Mozilla/5.0+(compatible;+PiplBot;++http://www.pipl.com/bot/)

LinkStats Bot

Mozilla/5.0 (compatible; adbeat-publisher-description-fetcher; [email protected];

[email protected])

CISPA Vulnerability Notification (https://notify.mmci.uni-saarland.de)

librabot/1.0 (+http://search.msn.com/msnbot.htm)

SeoCheckBot (Seomira Seo Check, http://www.kfsw.de/bot.html)

yacybot (x86 Windows 2003 5.2; java 1.6.0_16; Europe/de) http://yacy.net/bot.html

WatchMouse/8.4.0.3 (http://watchmouse.com/ ; itpda01.watchmouse.net)

yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html

BrainbruBot/1.0 (+http://www.brainbru.com/)

Docoloc

Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1

Safari/538.1 Prerender (+https://github.com/prerender/prerender)

dlvr.it/1.0 (+http://dlvr.it/)

Mozilla/5.0 (compatible; parsijoo-bot; +http://www.parsijoo.ir/; [email protected])

DataparkSearch/4.35-02122005 ( http://www.dataparksearch.org/)

Pixray-Seeker/2.0 (Pixray-Seeker; +http://www.pixray.com/pixraybot; [email protected])

Abrave v5.5 (http://robot.abrave.co.uk)

Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.20) Gecko/20090429

HeartRails_Capture/0.6 (+http://capture.heartrails.com/) BonEcho/2.0.0.20

yacybot (/global; amd64 Linux 3.19.0-26-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html

MLBot (www.metadatalabs.com/mlbot)

yacybot (freeworld/global; amd64 Linux 2.6.32-34-server; java 1.6.0_26; Europe/de)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; LinkpadBot/1.08; +http://www.linkpad.ru)

Mozilla/5.0 (compatible; SearchmetricsBot; http://www.searchmetrics.com/en/searchmetrics-bot/)

msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)

ZnajdzFoto/Image 2.0

Mozilla/5.0 (compatible; Magibot/4.0.0; +http://magi.peak-labs.com/robots.txt)

Mozilla/5.0 (compatible; spbot/4.0.8; +http://OpenLinkProfiler.org/bot )

Eurobot/1.2 (http://eurobot.ayell.eu)

Mozilla/5.0 (compatible; Embedly/0.2; snap; +http://support.embed.ly/)

Yaanb/1.5.001 (compatible; Win64;+http://www.yaanb.com/company/bot.hmtl)

yacybot (i386 Linux 2.6.31-16-generic; java 1.6.0_15; Europe/en) http://yacy.net/bot.html

Mozilla/5.0 (compatible; YioopBot; +http://173.13.143.74/bot.php)

CopperEgg/RevealUptime/Singapore(aws)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.24 (KHTML, like Gecko; Google Web Preview)
Chrome/11.0.696 Safari/534.24
yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_25; Europe/de)
http://yacy.net/bot.html

Server Density Service Monitoring v2

Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0~beta

TinEye/1.0; +http://www.tineye.com/

Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1);

http://spinn3r.com/robot) Gecko/2010040121 Firefox/3.0.19

GigablastOpenSource/1.0

Googlebot (gocrawl v0.4)

CSmartCrawler (email: abuse[at]adsadv[dot]com)

Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot)

minicrawler/4.0.0~beta9

KOCMOHABT (+http://kozmonavt.tk/) AppEngine-Google; (+http://code.google.com/appengine;

appid: s~web-explorer)

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110622 HeartRails_Capture/1.0.5

(+http://capture.heartrails.com/) Namoroka/3.6.18

PingSpot/1.0 (http://www.pingspot.org/)

AppEngine-Google; (+http://code.google.com/appengine; appid: s~buzzstream-tools-hrd)

AddThis.com (http://support.addthis.com/)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview)
Chrome/27.0.1453 Safari/537.36

ExactSeekCrawler/1.0

Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)

Mozilla/5.0 (compatible; jpg-newsbot/2.0; +http://vipnytt.no/bot.html)

g2reader-bot/1.0 (+http://www.g2reader.com/)

MetaGeneratorCrawler/1.3.12 (www.metagenerator.info)

QuerySeekerSpider ( http://queryseeker.com/bot.html )

KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo;

+http://help.goo.ne.jp/help/article/1142/)

Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)

Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0; WOW64; Trident/6.0;

BingPreview/1.0b)

Mozilla/5.0 (compatible; firmilybot/0.3; +http://www.firmily.com/bot.php

Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot;
http://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0

Mozilla/5.0 (compatible; metager2-verification-bot; +http://metager2.de/technology.php)

crawler4j (http://code.google.com/p/crawler4j/)

CirrusExplorer/1.1 (http://www.cireu.com/explorer.php)

Mozilla/5.0 (compatible; RankvalBot/1.0; +https://rankval.com)

Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.accelobot.com)

Mozilla/5.0 (compatible; BLEXBotTest/1.0; +http://webmeup.com/crawler.html)

nWormFeedFinder (http://www.nworm.com)

netEstate NE Crawler (+http://www.sengine.info/)

Mozila/5.0 (compatible; StudioFACA Search; +http://www.studiofaca.com/)

Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 92751)

wikiwix-bot-3.0

SEOCentro Page Keyword Analyzer v1.2

yacybot (amd64 Linux 2.6.18-164.15.1.el5xen; java 1.6.0_0; Europe/en) http://yacy.net/bot.html

Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)

Automattic Analytics Crawler/0.1; http://wordpress.com/crawler/

SemrushBot/0.92

http://www.uni-koblenz.de/~flocke/robot-info.txt

Mozilla/5.0 (compatible; idmarch Automatic.beta/1.3; +http://www.idmarch.org/bot.html)

Mozilla/5.0 (X11; Linux i686 on x86_64; rv:25.0; GTmetrix http://gtmetrix.com/) Gecko/20100101

Firefox/25.0

msnbot/2.0b (+http://search.msn.com/msnbot.htm)._

Stroke.cz (http://stroke.cz)

Mozilla (+http://web.nli.org.il/sites/NLI/English/collections/israel-
collection/internet_archive/Pages/default.aspx)

audisto.com full crawler 4.33.470 (refer to in robots.txt as audisto, see https://audisto.com/bot)

http://arachnode.net 2.5

SurveyBot/2.3 (Whois Source)

ant.com/Nutch-1.7 (http://ant.com)

CorpusCrawler 2.0.13 (http://corpora.fi.muni.cz/crawler/)

Seo Servis - Analyza zdrojoveho kodu

SEOENGBot/1.2 (+http://www.seoengine.com/seoengbot.htm)

Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)

Mozilla/5.0 (compatible; evc-batch/2.0.20160708184126)

HubSpot Marketing Grader

Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) LinkCheck by Siteimprove.com

nrsbot/6.0(loopip.com/robot.html)

Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup.com/crawler.html)

yacybot (freeworld-global; amd64 Windows 7 6.1; java 1.7.0_02-ea; America/en)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)

Mozilla/5.0 (compatible; Topicbot/12.0rc-2; +http://topicbot.awardspace.us/)

Mozilla/5.0 (compatible; Gimme60bot/1.0; +http://gimme60.com)

yacybot (i386 Linux 2.6.28-11-generic; java 1.6.0_13; Europe/de) http://yacy.net/bot.html

Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])

Kemvibot/1.0 (http://kemvi.com, [email protected])

CorpusCrawler 2.0.15 (http://corpora.fi.muni.cz/crawler/)

dotbot

Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko)
Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0;
+http://www.bing.com/bingbot.htm)

page_viewer ([email protected])

HootSuite Image proxy ([email protected])

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider for SpeedyAds

(http://www.entireweb.com/about/search_tech/speedy_spider/)

PINGOMETER_BOT_(HTTPS://PINGOMETER.COM)

Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-

guys.nl/)

VSE/1.0 ([email protected])

Scrapy/0.24.0 (+http://scrapy.org)

Speedy Spider (http://www.entireweb.com)

Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like

Gecko) Chrome/51.0.2704.106 Safari/537.36

WatchMouse/18990 (http://watchmouse.com/ ; hk)

Mozilla/5.0 (compatible; yoozBot-2.2; http://yooz.ir; [email protected])

Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 87214)

Mozilla/5.0 (compatible; discobot/2.0; +http://discoveryengine.com/discobot.html)

yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.8.0_11; Europe/de)

http://yacy.net/bot.html

Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot)

minicrawler/4.0.0~beta6

yacybot (freeworld/global; amd64 Linux 3.0.0-15-server; java 1.6.0_23; Europe/en)

http://yacy.net/bot.html

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0

(FlipboardProxy/1.1; +http://flipboard.com/browserproxy)

YisouSpider

Slack-ImgProxy 1.138 (+https://api.slack.com/robots)

mozilla/5.0 (compatible; webmastercoffee/0.7; +http://webmastercoffee.com/about)

Mozilla/5.0 (compatible; Uptimebot/0.2.22; +http://www.uptime.com/uptimebot)

Mozilla/5.0 (compatible; AhrefsBot/1.0; +http://ahrefs.com/robot/)

Mozilla/5.0 (compatible; Feedspotbot/1.0; +http://www.feedspot.com/fs/bot)

sg-Orbiter/1.0 (+http://searchgears.de/uber-uns/crawling-faq.html)

Site24x7 Tools

NextGenSearchBot 1 (for information visit

http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx)

Aboundex/0.3 (http://www.aboundex.com/crawler/)

yacybot (freeworld/global; i386 Linux 2.6.37.6-0.5-desktop; java 1.6.0_20; Europe/en)

http://yacy.net/bot.html

Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98

Safari/537.4 (StatusCake)

Mozilla/5.0 (compatible; DomainSONOCrawler/0.1; +http://domainsono.com)

Mozilla/5.0 (compatible; Online Domain Tools - Server Monitor/1.0; +http://server-

monitoring.online-domain-tools.com)

Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)

Mozilla/5.0 (compatible; spbot/4.0.4; +http://www.seoprofiler.com/bot )

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like

Gecko) Chrome/27.0.1453 Mobile Safari/537.36 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)
Factbot 1.09 (see http://www.factbites.com/webmasters.php)

CopperEgg/RevealUptime/SaoPauloBR(aws)

CRIM Crawler/Nutch-2.3 (Crawler du Centre de Recherche Informatique de Montr\xc3\xa9al (CRIM))

Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.webarchiv.cz)

Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com)

yacybot (i386 Linux 2.6.23; java 1.6.0_17; Europe/en) http://yacy.net/bot.html

Mozilla/5.0 (compatible; Cliqzbot/0.1 +http://cliqz.com/company/cliqzbot)

URLitor.com

RavenCrawler

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20130331 HeartRails_Capture/1.0.5

(+http://capture.heartrails.com/) Namoroka/3.6.18

Mozilla/5.0 (compatible; WebThumbnail/3.x; Website Thumbnail Generator;

+http://webthumbnail.org)

Kyoto-Crawler/2.0 (Mozilla-compatible; kyoto-crawler-contact(at)nlp(dot)kuee(dot)kyoto-

u(dot)ac(dot)jp; http://nlp.ist.i.kyoto-u.ac.jp/)

Mozilla/5.0 (compatible; SputnikImageBot/2.3; +http://corp.sputnik.ru/webmaster)

WatchMouse/8.4.0.3 (http://watchmouse.com/ ; lvps91-250-96-109.dedicated.hosteurope.de)

MetaGeneratorCrawler/1.3.3 (www.metagenerator.info)

LongURL API

Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html ; agentname at

setooz dot_com )

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6

(FlipboardProxy/2.0; +http://flipboard.com/browserproxy)

http://tools.geek-tools.org/link-counter/

SCFCrawler/Nutch-1.8 (Image Crawler for StolenCameraFinder.com;

http://www.stolencamerafinder.com/; [email protected])

Mozilla/5.0 (compatible; Uptimebot/0.2.41; +http://www.uptime.com/uptimebot)

Mozilla/5.0 (compatible; DomainAppender /1.0; +http://www.profound.net/domainappender)

Mozilla/5.0 (compatible; hypestat/1.0; +http://www.hypestat.com/bot)

yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/en)

http://yacy.net/bot.html

PagesInventory (robot +http://www.pagesinventory.com)

Sosospider+(+http://help.soso.com/webspider.htm)
Mozilla/5.0 (compatible; AntBot/1.0; +http://www.ant.com/)

PagesInventory (robot http://www.pagesinvenotry.com)

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;

http://www.changedetection.com/bot.html )

Wotbox/2.01 (+http://www.wotbox.com/bot/)

Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general;

[email protected])

Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research);

+http://nlp.fi.muni.cz/projects/biwec/)

CopperEgg/RevealUptime/DallasTXUSA

yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en)

http://yacy.net/bot.html

Bad-Neighborhood Link Analyzer (http://www.bad-neighborhood.com/)

Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)

Pixray-Seeker/2.0 (Pixray-Seeker; http://www.pixray.com/pixraybot; [email protected])

Cityreview Robot (+http://www.cityreview.org/crawler/)

yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_95; Europe/en) http://yacy.net/bot.html

Dlvr.it/1.0 (http://dlvr.it/)

Zookabot/2.4;++http://zookabot.com

AddThis.com robot [email protected]

Mozilla/5.0 (compatible; discoverybot/2.0; +http://discoveryengine.com/discoverybot.html)

TwengaBot

Mozilla/5.0 (compatible; 4SeoHuntBot; +http://4seohunt.biz/about.html)

Web-sniffer.me/1.0.0 (+http://web-sniffer.me/)

KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo;

+http://search.goo.ne.jp/option/use/sub4/sub4-1/)

gosquared-thumbnailer/1.0

yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Asia/zh) http://yacy.net/bot.html

BPImageWalker/2.0 (www.bdbrandprotect.com)

Acoon v4.10.3 (www.acoon.de)

Mozilla/4.0 (CMS Crawler: http://www.cmscrawler.com)

WP Engine Install Performance API

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.24; ips-agent) Gecko/20111107 Ubuntu/10.04 (lucid)
Firefox/3.6.24

Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/277

CamontSpider/1.0 +http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html

SEOENGBot/1.2 (+http://learn.seoengine.com/seoengbot.htm)

Y!J-BRJ/YATS crawler (http://listing.yahoo.co.jp/support/faq/int/other/other_001.html)

CopperEgg/RevealUptime/N.Virginia(aws)

Mozilla/5.0, Google-AdWords-Express

Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.youdao.com/help/webmaster/spider/;)

yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_91; Europe/en) http://yacy.net/bot.html

yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_09; Europe/de)

http://yacy.net/bot.html

Acoon v4.10.1 (www.acoon.de)

Mediatoolkitbot ([email protected])

Mozilla/5.0 (compatible; Speedy Spider;

http://www.entireweb.com/about/search_tech/speedy_spider/)

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) SPEng

2Bone_LinkChkr/1.0 libwww-perl/6.03

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6

(FlipboardBrowserProxy/0.0.5; +http://flipboard.com/browserproxy)

Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://napoveda.seznam.cz/en/seznambot-intro/)

Comodo SSL Checker

Comodo Spider 1.2

Who.is Bot

Mozilla/5.0 (compatible; online-webceo-bot/1.0; +http://online.webceo.com)

yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/en)
http://yacy.net/bot.html

psbot/0.1 (+http://www.picsearch.com/bot.html)

IlTrovatore-Setaccio/1.2 (http://www.iltrovatore.it/aiuto/faq.html)

Mozilla/5.0 (compatible; Twingly Recon; twingly.com)

Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com; [email protected])

Orgbybot/OrgbyBot v1.2 (Spidering the net for Orgby; http://www.orgby.com/ ; Orgby.com Search
Engine)
Browsershots

AdsBot-Google (+http://www.google.com/adsbot.html)

Qseero v1.0.0

Mozilla/5.0 (compatible; ltbot/0.3.6 +http://www.kdsl.tu-darmstadt.de/de/kdsl/research-

program/crawling-and-semantic-structuring/)

yacybot (x86 Windows XP 5.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html

visionutils/0.2

Mozilla/5.0 (compatible; Googlebot/2.1; https://www.deepcrawl.com/bot)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Web Preview)
Chrome/22.0.1229 Safari/537.4

Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 8942)

Mozilla/4.0 (compatible; LinkMarket-Bot v1.2)

Buzzbot/1.0 (Buzzbot; http://www.buzzstream.com; [email protected])

WordPress.com; https://public-api.wordpress.com

Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html )

CopperEgg/RevealUptime/FremontCAUSA

Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.4.5; trendiction search;

http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com)
Gecko/20071127 Firefox/3.0.0.11

yacybot (/global; amd64 Linux 2.6.32-042stab094.8; java 1.7.0_79; America/en)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; Windows NT 6.1?; ZumBot/1.0; http://help.zum.com/inquiry)

Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0; topster.de Linkchecker 5.0) like Gecko
(90.177.52.133)

Mozilla/5.0+(compatible; Monitority/1.0; http://www.monitority.com/)

yacybot (freeworld/global; x86 Windows 7 6.1; java 1.8.0_25; Europe/de) http://yacy.net/bot.html

Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html)

Mozilla/5.0 (compatible; SemrushBot/1.1~bl; +http://www.semrush.com/bot.html)

YowedoBot/Yowedo 1.0 (Search Engine crawler for yowedo.com;

http://yowedo.com/en/partners.html; [email protected])

Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)

Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1;

+http://www.google.com/bot.html)

Mozilla/5.0 (compatible; IstellaBot/1.23.15 +http://www.tiscali.it/)

GetIntent Crawler (http://getintent.com/bot.html)

Mozilla/5.0 (compatible; Uptimebot/0.1.89; +http://www.uptime.com/uptimebot)

Cliqzbot

AboutUsBot

yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en) http://yacy.net/bot.html

Sufog/Nutch-2.2.1 (www.sufog.com; www.sufog.com)

Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/)

KOCMOHABT (+http://kozmonavt.tk/) Mozilla/5.0 (compatible; Web Explorer 1) AppEngine-Google;

(+http://code.google.com/appengine; appid: s~web-explorer)

yacybot (/global; amd64 Linux 3.16.0-53-generic; java 1.7.0_85; Europe/en) http://yacy.net/bot.html

L.webis/0.53 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)

Mozilla/5.0 (compatible; memoryBot/1.21.23 +http://internetmemory.org/en/)

ExB Language Crawler 2.1.5 (+http://www.exb.de/crawler)

Peeplo Screenshot Bot/0.20 ( abuse at peeplo dot_com )

Mozilla/5.0 (compatible; Qwantify/2.1w; +https://www.qwant.com/)/*

Experibot_v1 [bit.ly/2bLMJdw]

WikioFeedBot 1.0 (http://www.wikio.com)

SalesIntelligent/v1.1 (https://www.salesintelligent.com)

Mozilla/5.0 (compatible; ExaleadCloudView/5;)

Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot)

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/44.0.2403.157 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)

Mozilla/5.0 (compatible; Search17Bot/1.1; http://www.search17.com/bot.php)

FollowSite Bot ( http://www.followsite.com/bot.html )

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler - [email protected]

Mozilla/5.0 (compatible; BlinkaCrawler/1.0; +http://www.blinka.jp/crawler/)

Mozilla/5.0 (compatible; Qwantify/2.2w; +https://www.qwant.com/)/*

findlinks/1.1.6-beta2 (+http://wortschatz.uni-leipzig.de/findlinks/)

yacybot (/global; amd64 Linux 4.4.0-28-generic; java 1.8.0_91; Europe/pl) http://yacy.net/bot.html

yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.8.0_40; Europe/de)
http://yacy.net/bot.html
Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)

WatchMouse/8.4.0.3 (http://watchmouse.com/ ; chzrh01.watchmouse.net)

Iframely/0.8.8 (+http://iframely.com/;)

Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)

LoadImpactRload/3.2.0 (Load Impact; http://loadimpact.com);

RSSingBot (http://www.rssing.com)

RADaR-Bot/Nutch-1.3 (http://radar-bot.com/)

Ocelli/1.4 (http://www.globalspec.com/Ocelli)

Mozilla/5.0 (compatible; BuzzSumo; +http://www.buzzsumo.com/bot.html)

Mozilla/4.0 (compatible; IDwhois/2.0alpha-10102619; Linux; en-US; +http://idwhois.info/about)

omgilibot/0.4 +http://omgili.com

yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_23; Europe/de)

http://yacy.net/bot.html

Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)

Mozilla/5.0 (compatible; AhrefsBot/2.0; +http://ahrefs.com/robot/)

Mozilla/5.0 (compatible; VideoSurf_bot +http://www.videosurf.com/bot.html)

Symfony Spider (http://symfony.com/spider)

Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)

Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev;

+http://www.openindex.io/en/webmasters/spider.html)

ADmantX Platform Semantic Analyzer - ADform - ADmantX Inc. - www.admantx.com -

[email protected]

Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+)

cuwhois/1.0 (+http://www.cuwhois.com/)

Mozilla/5.0 (compatible; memoryBot/1.20.311 +http://internetmemory.org/en/)

Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-

guys.nl/; http://www.wise-guys.nl/)

yacybot (-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html

Mozilla/5.0 (compatible; Dazoobot/1.0; http://dazoo.fr)

Mozilla/5.0 (compatible; Semager/1.4c; +http://www.semager.de/blog/semager-bots/)

EuripBot/1.1 (+http://www.eurip.com) GetRobots

yacybot (freeworld/global; amd64 Linux 3.8.0-23-generic; java 1.6.0_27; Pacific/en)

http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.24-23-generic; java 1.6.0_07; Europe/de) http://yacy.net/bot.html

yacybot (freeworld/global; x86 Windows 2003 5.2; java 1.6.0_32; Europe/de)

http://yacy.net/bot.html

yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/de)

http://yacy.net/bot.html

socialbm_bot http://spider.socialbm.net

Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)

webinatorbot 1.0; +http://www.webinator.de

WorldBrewBot/2.1 (+http://www.marketbrew.com/)

BotOnParade, http://www.bots-on-para.de/bot.html

ltx71 - (http://ltx71.com/)

VSB-TUO/1.0 (+http://www.vsb.cz)

yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_101; Europe/en)

http://yacy.net/bot.html

CopperEgg/RevealUptime/TokyoJP(aws)

Mozilla/5.0 (compatible; etoolsbot/1.0; +https://etools.io/bots/)

Comodo-Webinspector-Crawler 2.2.2, http://www.comodorobot.com

Mozilla/5.0 (compatible; TWMBot/0.1; +http://thewebminer.com)

Mozilla/5.0 (compatible; Plukkie/1.3; http://www.botje.com/plukkie.htm)

Autonomy

yacybot (i386 Linux 2.6.30-2-686; java 1.6.0_0; SystemV/en) http://yacy.net/bot.html

Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.1.0

CatchBot/1.0; +http://www.catchbot.com

Sitedomain-Bot(Sitedomain-Bot 1.0, http://www.sitedomain.de/sitedomain-bot/)

Mozilla/5.0 (compatible; Uptimebot/0.2.42; +http://www.uptime.com/uptimebot)

yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; Europe/en) http://yacy.net/bot.html

SiteTruth.com site rating system

SiteUptime.com

yacybot (i386 Linux 2.6.24-23-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html

UrlTrends Ranking Updater/2.0

yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/de) http://yacy.net/bot.html

yacybot (freeworld/global; amd64 Linux 3.5.0-27-generic; java 1.7.0_25; Europe/en)
http://yacy.net/bot.html

GarlikCrawler/1.1 (http://garlik.com/, [email protected])

Mozilla/5.0 (compatible; ParsijooBot; +http://www.parsijoo.ir/; [email protected])

Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89

Safari/537.1; 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)

Willow Internet Crawler by Twotrees V2.1

KrOWLer/0.1.0, matentzn at cs dot man dot ac dot uk

SimplyFast.info Headers

yacybot (freeworld/global; amd64 Linux 2.6.32-431.29.2.el6.x86_64; java 1.7.0_65; Europe/en)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; +http://yandex.com/bots)

msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/278

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7

Safari/534.57.2 PTST/278

gipo-crawler/Nutch-1.10 (Global Internet Policy Observatory crawler)

Mozilla/5.0 (compatible; sukibot_heritrix/3.1.1 +http://suki.ling.helsinki.fi/eng/webmasters.html)

PayPal IPN ( https://www.paypal.com/ipn )

Mozilla/5.0 (compatible; KaloogaBot; http://kalooga.com/crawler)

eCairn-Grabber/1.0 (+http://ecairn.com/grabber) curl/7.15

Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML like Gecko;

+http://www.diffbot.com) Chrome/24.0.1295.0 Safari/537.15

ImageEngine/1.0

DER-bot (+https://goo.gl/9ZibHG)

yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_60; Europe/de)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; SemrushBot/0.96.3; +http://www.semrush.com/bot.html)

Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)

Mozilla/5.0 (compatible; YandexDirect/3.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; aiHitBot/2.9; +https://www.aihitdata.com/about)

Linguee Bot (http://www.linguee.com/bot; [email protected])

MetaGeneratorCrawler/1.3.9 (www.metagenerator.info)
TelegramBot (like TwitterBot)

Mozilla/5.0 (compatible; Siteliner/1.0; +http://www.siteliner.com/bot)

calculon spider/0.1.0 mailto:[email protected]

yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/de) http://yacy.net/bot.html

PocketParser/2.0 (+https://getpocket.com/pocketparser_ua)

montastic-monitor http://www.montastic.com

Mozilla/5.0 (compatible; trovitBot 1.0; +http://www.trovit.com/bot.html)

Mozilla/5.0 (Windows NT 6.1)+(compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)

Yaanb/1.5.001 (compatible; Win64;)

Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 70054)

Mozilla/5.0 (compatible; spbot/2.0.2; +http://www.seoprofiler.com/bot/ )

yacybot (freeworld-global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_24; Europe/en)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; ImageFetcher/5.6; +http://images.weserv.nl/)

CopperEgg/RevealUptime/N.VirginiaUSA

Huaweisymantecspider (compatible; MSIE 8.0; [email protected])

OrangeCrawler/Nutch-1.0 ([email protected])

CopperEgg/RevealUptime/AmsterdamHolland

Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)

Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like

Gecko) Chrome/50.0.2661.75 Safari/537.36

Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html) AppEngine-Google;

(+http://code.google.com/appengine; appid: s~natmacdevice)

yacybot (/global; amd64 Linux 3.2.41-042stab094.7; java 1.7.0_75; Europe/en)

http://yacy.net/bot.html

yacybot (x86 Windows Vista 6.1; java 1.6.0_13; Europe/de) http://yacy.net/bot.html

SmabblerBot/1.0 (http://www.smabbler.com/)

Mozilla/5.0 (compatible; grapeFX/0.9; [email protected]

Mozilla/5.0 (compatible; SemrushBot/1~bl; +http://www.semrush.com/bot.html)

MetaGeneratorCrawler/1.3.14 (www.metagenerator.info)

oku-taka-lab-bot (Mozilla-compatible; [email protected]; http://lr-

www.pi.titech.ac.jp/wp/)
Mozilla/5.0 (Linux; Android 4.1.2; Galaxy Nexus Build/JZO54K; GTmetrix https://gtmetrix.com/)
AppleWebKit/537.22 (KHTML, like Gecko) Chrome/26.0.1410.58 Mobile Safari/537.22

Mozilla/5.0 (compatible; Feedage/2.0; +http://www.feedage.com/bot.php)

yacybot (amd64 Linux 2.6.26-2-openvz-amd64; java 1.6.0_12; UTC/en) http://yacy.net/bot.html

yacybot (-global; amd64 Linux 2.6.32-042stab116.1; java 1.7.0_79; Europe/en)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; DKIMRepBot/1.0; +http://www.dkim-reputation.org)

Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider

Mozilla/5.0 (compatible; pub-crawler; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-

[email protected])

mfibot/1.1 (http://www.mfisoft.ru/analyst/; <[email protected]>; en-RU)

Visited by http://tools.geek-tools.org

peerindex

Mozilla/5.0 (compatible; janforman.com/2.4; +http://janforman.com/)

http://www.amagit.com/

yacybot (freeworld/global; amd64 Linux 2.6.32-40-server; java 1.6.0_20; Europe/en)

http://yacy.net/bot.html

Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko)
Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)

Mozilla/5.0 (compatible; NLNZ_IAHarvester2014 +https://natlib.govt.nz/publishers-and-

authors/web-harvesting/domain-harvest)

Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)

mahonie, neofonie search:robot/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research
project. If you feel unhappy and do not want to be visited by our crawler send an email to
[email protected]; http://spider.neofonie.de; [email protected])

Mozilla/5.0 (Linux; Android 5.0; Nexus 5 Build/LRX21O) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/46.0.2490.76 Mobile Safari/537.36 PTST/276

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; WOW64; Trident/5.0;

BingPreview/1.0b)

Mozilla/5.0 (compatible; OptimizationCrawler/0.2; +http://www.domainoptima.com/bot.html)

Mozilla/5.0 (compatible; Cloudinary/1.0)

rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])

Iframely/0.9.8 (+http://iframely.com/;)
yacybot (webportal-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en)
http://yacy.net/bot.html

Interdose AntiSpamBot/2.0 (+http://www.piep.net)

CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)

Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko)
Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Google Search Console)

rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, [email protected])

findlinks/2.5 (+http://wortschatz.uni-leipzig.de/findlinks/)

Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/277

yacybot (/global; x86 Windows 2003 5.2; java 1.7.0_67; Europe/de) http://yacy.net/bot.html

NeutrinoAPI/2.0.3

Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )

Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)

Site24x7 Agent

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/6.0; .NET4.0E; .NET4.0C; .NET
CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729) CrawlerProcess
(http://www.PowerMapper.com) /5.7.704.0

Mozilla/5.0 (compatible; Falconsbot; +http://iws.seu.edu.cn/services/falcons/contact_us.jsp)

Super Monitoring

Seznam-Zbozi-robot/3.0

LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/4.3

+http://www.linkedin.com)

Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; [email protected]

http://everyonesocial.com/)

Mozilla/5.0 (compatible; IstellaBot/1.01.18 +http://www.tiscali.it/)

Mozilla/5.0 (compatible; ToutiaoSpider/1.0; http://web.toutiao.com/media_cooperation/;)

Mozilla/5.0 (compatible; BegunAdvertising/3.0; +http://begun.ru/begun/technology/indexer/)

Netcraft SSL Server Survey - contact [email protected]

Mozilla/5.0 (compatible; Qualidator.com SiteAnalyzer 1.0;)

Mozilla/5.0 (compatible; OrangeBot/2.0; [email protected])

Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online)

seebot/1.0.0 (http://www.seegnify.com/bot)

Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)

Eurobot/1.1 (http://eurobot.ayell.eu)

Mozilla/5.0 (compatible; IXEbot; +http://medialab.di.unipi.it/IXEbot.html)

Mozilla/5.0 (GetLinkInfo.com - http://www.getlinkinfo.com)

Mozilla/5.0 (compatible; spbot/4.0.5; +http://www.seoprofiler.com/bot )

yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_31; America/en)

http://yacy.net/bot.html

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0 PTST/314

Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)

UXCrawlerBot

Pixray-Seeker/2.0 (http://www.pixray.com/pixraybot; [email protected])

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/50.0.2661.94 Safari/537.36 PTST/276

FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)

yacybot (freeworld/global; amd64 Windows 10 10.0; java 1.8.0_60; Europe/de)

http://yacy.net/bot.html

Technoratibot/7.0

Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; DomainDB-1.1;

http://domaindb.com/crawler/)

Abrave v6.0 (http://robot.abrave.com)

Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot)

minicrawler/4.0.0~beta11

DowntimeDetector/1.0 (+http://downforeveryoneorjustme.com)

Comodo Spider 1.1

Link Valet Online 1.1

yacybot (freeworld/global; amd64 Windows XP 5.2; java 1.7.0_04; America/en)

http://yacy.net/bot.html

li_viewer ([email protected])

Mozilla/5.0 (compatible; Mail.RU/2.0c)

Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/)

Mozilla/5.0 (compatible; Uptimebot/0.2.19; +http://www.uptime.com/uptimebot)

Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko)
Version/6.0 Mobile/10A5376e Safari/8536.25 ( compatible; CloudServerMarketSpider/1.0;
+http://cloudservermarket.com/spider.html )
yacybot (i386 Linux 2.6.28-13-generic; java 1.6.0_13; Europe/en) http://yacy.net/bot.html

linguatools-bot/Nutch-1.6 (searching for translated pages;

http://www.linguatools.de/linguatoolsbot.html; peter dot kolb at linguatools dot org)

Mozilla/5.0 (compatible; Lippershey/5.2; +https://www.lipperhey.com/en/about/)

NextGenSearchBot 69 (for information visit

http://www.zoominfo.com/business/nextgensearchbot.html)

Mozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

yacybot (freeworld-global; amd64 Linux 3.2.0-35-generic; java 1.7.0_09; Europe/en)

http://yacy.net/bot.html

peerindex/0.1 (http://www.peerindex.com/; crawler AT peerindex DOT com)

iqdb/0.2 (+http://iqdb.harry.lu/)

Mozilla/5.0 (compatible; LuminateBot/1.0; +http://www.luminate.com/bot/)

Mozilla/5.0 (X11; Linux x86_64) adbeat.com/policy AppleWebKit/537.21 (KHTML, like Gecko)

Firefox/34.0 Safari/537.21

AntBot/1.0 (http://www.ant.com)

Mozilla/5.0 (compatible; Thumbshots.ru; +http://thumbshots.ru/bot) Firefox/3

Mozilla/5.0 (compatible; value/1.1; +http://www.aihit.com/)

Digg Deeper/v1 (http://digg.com/about)

Elmer, the Thinglink ImageBot (http://www.thinglink.com/help/ThinglinkImageBot)

rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])

Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89

Safari/537.1; http://spyonweb.com [email protected]

BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; [email protected])

Mozilla/5.0 (compatible; dotSemantic/1.0; +http://www.dotsemantic.org)

TinEye/1.1 (http://tineye.com/crawler.html)

Mozilla/5.0 (compatible; SemrushBot/0.98~bl; +http://www.semrush.com/bot.html)

CorpusCrawler 2.0.9 (http://corpora.fi.muni.cz/crawler/)

aboutthedomain

Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot)

Mozilla/5.0 (compatible; Uptimebot/0.2.43; +http://www.uptime.com/uptimebot)

LinqiaMetadataDownloaderBot/1.0 ([email protected])
yacybot (freeworld/global; amd64 Linux 3.2.0-32-generic; java 1.6.0_24; Europe/en)
http://yacy.net/bot.html

NeumobBot/0.2.16 (+http://www.neumob.com/bot)

Mozilla/5.0 (compatible; Mp3Bot/0.4; +http://mp3realm.org/mp3bot/)

Miniflux (https://miniflux.net)

Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko;
Google Page Speed Insights) Version/8.0 Mobile/12F70 Safari/600.1.4

Digincore crawler bot. See https://www.digincore.com/crawler.html for rules and instructions.

Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko)
Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0;
http://www.bing.com/bingbot.htm)

NerdyBot

Nutch/2.2.1 (page scorer; http://integralads.com/site-indexing-policy/)

Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html;) Gecko/2008032620

Mozilla/4.0 (compatible; LinkMarket-Bot)

SeoCheckBot (Seo-Check, http://www.kfsw.de/bot.html)

Mozilla/5.0 (compatible; Whynder Magnet; http://whynder.com/magnet)

IdeelaborPlagiaat/1

x28-job-bot; +http://x28.ch/bot.html

Mozilla/5.0 (compatible; spbot/4.0.7; +https://www.seoprofiler.com/bot )

Mozilla/5.0 (compatible; RelateIQ Crawler/0.4; +www.relateiq.com; [email protected])

WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/)

Mozilla/5.0 (iPhone; U; CPU iPhone OS 10_0 like Mac OS X; en-us) AppleWebKit/602.1.38 (KHTML,
like Gecko) Version/10.0 Mobile/14A5297c Safari/602.1 (compatible; Mediapartners-Google/2.1;
+http://www.google.com/bot.html)

LinkWalker/3.0 (http://www.brandprotect.com)

Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html)

ia_archiver-web.archive.org

Mozilla/5.0 (compatible; Online Domain Tools - SEO Checker/1.0; +http://online-domain-tools.com)

Mozilla/5.0 (compatible; AhrefsBot/3.0; +http://ahrefs.com/robot/)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.13+ (KHTML, like Gecko)

Version/5.1.7 Safari/534.57.2; Shoppimon Analyzer (http://www.shoppimon.com/)

Mozilla/5.0 (compatible; NewShareCounts.com/1.0; +http://newsharecounts.com/crawler)

Mozilla/5.0 (compatible; Mozilla; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-
[email protected])

Mozilla/5.0 (compatible; CukBot; Not a spammer; http://www.companiesintheuk.co.uk/bot.html)

yacybot (freeworld/global; amd64 Linux 3.1.10-1.9-default; java 1.6.0_24; Europe/de)

http://yacy.net/bot.html

Nuhk/2.4 (+http://www.neti.ee/cgi-bin/abi/otsing.html)

elefent/Elefent 1.1 (Friendly web walker.; http://www.elefent.eu/; [email protected])

HubSpot Links Crawler 1.0 http://www.hubspot.com/

Mozilla/5.0 (compatible; Siteluxbot/1.0; +https://sitelux.co.uk/)

CopperEgg/RevealUptime/AmsterdamNL(softlayer)

Mozilla/5.0 (compatible; spbot/4.0.7; +http://OpenLinkProfiler.org/bot )

Mediumbot-MetaTagFetcher/0.1 (+https://medium.com/)

Mozilla/2.0 (compatible; Ask Jeeves/Teoma;

+http://about.ask.com/en/docs/about/webmasters.shtml)

Mozilla/5.0 (compatible; 008/0.85; http://www.80legs.com/webcrawler.html) Gecko/2008032620

yacybot (/global; amd64 Linux 4.4.0ct; java 1.7.0_91; Europe/de) http://yacy.net/bot.html

Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/)

yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.8.0_05; Europe/en) http://yacy.net/bot.html

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110622 HeartRails_Capture/1.0.4

(+http://capture.heartrails.com/) Namoroka/3.6.18

Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.0.1; compatible; iCjobs Stellenangebote Jobs;
http://www.icjobs.de) Gecko/20100401 iCjobs/3.2.3

KOCMOHABT (+http://www.KOCMOHABT.tk/) AppEngine-Google;

(+http://code.google.com/appengine; appid: s~web-explorer)

yacybot (freeworld/global; x86 Windows 7 6.1; java 1.6.0_31; Europe/de) http://yacy.net/bot.html

Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess

(http://www.PowerMapper.com) /5.21.754.0

Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots; +http://go.mail.ru/help/robots)

Mozilla/5.0 (compatible; Googlebot/2.1; +http://import.io)

ownCloud Server Crawler

Mozilla/5.0 (compatible; Qwantify/2.1n; +https://www.qwant.com/)/*

LoadImpactRload/3.1.5 (Load Impact; http://loadimpact.com);

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com;
http://www.girafa.com)
OmniExplorer_Bot/6.65a (+http://www.omni-explorer.com) WorldIndexer

Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)

anonymous coward/1.0 (a Storm-based crawler; https://github.com/DigitalPebble/storm-crawler;

[email protected])

Mozilla/5.0 (compatible; Uptimebot/1.0; +http://www.uptime.com/uptimebot)

Mozilla/5.0 (compatible; FoundSeoTool/1.0; +http://www.found.co.uk/seo-tool/)

findlinks/2.1.5 (+http://wortschatz.uni-leipzig.de/findlinks/)

yacybot (freeworld/global; amd64 Windows NT (unknown) 6.2; java 1.7.0_05; Africa/en)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; memorybot/1.20.71

+http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB)

ownCloud News/5.3.6 (+https://owncloud.org/; 1 subscriber;)

YottaBot Spider/Nutch-2.3

Mozilla/5.0 (compatible; WeViKaBot/1.0; +http://www.wevika.de/)

ThumbShots-Bot (+http://thumbshots.in/bot.html)

Mnogosearch-3.1.21

Favicon downloader (+https://favico.be/)

Mozilla/5.0 (compatible; Page2RSS/0.7; +http://page2rss.com/)

ExB Language Crawler 2.1.2 (+http://www.exb.de/crawler)

PrivateSearch/0.1.0 (Polite Nutch Crawler; grierforensics.com)

nodestackbot/0.1 ([email protected] http://nodestack.com/bot.html)

iskanie (+http://www.iskanie.com)

yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_67; Europe/de) http://yacy.net/bot.html

Mozilla/5.0 (compatible; Dataprovider Site Explorer; http://www.dataprovider.com/)

Mozilla/5.0 (compatible; akula/12.0rc-2; +http://k311.fd.cvut.cz/)

Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)

Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko)
Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1;
+http://www.google.com/bot.html)

Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/284

Mozilla/5.0 (compatible; Infohelfer/1.3.3; +http://www.infohelfer.de/crawler.php)

www.deadlinkchecker.com XMLHTTP/1.0

Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/)

Chrome/49.0.2623.87 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer)

Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess

(http://www.PowerMapper.com) /5.11.740.0

XING-contenttabreceiver/2.0

Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko)
Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)

Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html#relaunch)

Pingoscope

Mozilla/5.0 (compatible; um-LN/1.0; mailto: [email protected])

Mozilla/5.0 (compatible; Pagespeed/1.1 Fetcher; +http://www.pagespeed.de)

HTTP-Header-Abfrage/1.0 (http://www.internalscripts.de/werkzeuge/http-header-abfrage.php)

Mozilla/5.0 (compatible; 80bot/0.71; http://www.80legs.com/spider.html;) Gecko/2008032620

LoadImpactRload/3.0.3 (Load Impact; http://loadimpact.com);

Mozilla/5.0 (compatible; Peepowbot/1.0; +http://www.peepow.com/bot.php)

yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_26; Atlantic/en)

http://yacy.net/bot.html

WatchMouse/8.4.0.3 (http://watchmouse.com/ ; grath01.watchmouse.net)

Mozilla/5.0 (compatible; +http://servernfo.com/)

Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)

CopperEgg/RevealUptime/NewarkNJUSA

Mozilla/5.0 (compatible; archive.org_bot/3.3.0 +http://www.archive.org/details/archive.org_bot)

Mozilla/5.0 (compatible; Plukkie/1.1; http://www.botje.com/plukkie.htm)

Mozilla/5.0 (compatible; musobot/1.0; [email protected]; +http://www.muso.com)

yacybot (freeworld/global; amd64 Linux 3.10.17-gentooxen; java 1.6.0_31; Europe/nb)

http://yacy.net/bot.html

ResponseCodeTest/1.1

Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl by

Siteimprove.com

Acoon v4.9.5 (www.acoon.de)

Mozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5

SiteGuardian/2.0 (Internet Monitoring)

Mozilla/5.0 (compatible; DCPbot/1.4; +http://domains.checkparams.com/)

BoardReader Favicon Fetcher /1.0 [email protected]

baypup/colbert (Baypup; http://www.baypup.com/webmasters; [email protected])

Mozilla/5.0 (compatible; MonTools.com)

yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; Etc/de)

http://yacy.net/bot.html

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.51 (KHTML, like Gecko; Google Web Preview)
Chrome/12.0.742 Safari/534.51

linkapediabot (+http://www.linkapedia.com)

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12; ips-agent) Gecko/20050922 Fedora/1.0.7-1.1.fc4

Firefox/1.0.7

BCKLINKS 1.0

CorpusCrawler 2.0.21 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus

Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/support/robots.html)

Mozilla/5.0 (compatible; pingbot/2.0; +http://www.pingdom.com/)

Mozilla/5.0 (compatible; memoryBot/1.20.275 +http://internetmemory.org/en/)

Mozilla/5.0 (compatible; spbot/1.1; +http://www.seoprofiler.com/bot/ )

yacybot (/global; amd64 Linux 3.13.0-88-generic; java 1.7.0_101; Europe/en)

http://yacy.net/bot.html

BlogSearch/2 +http://www.icerocket.com/

Mozilla/5.0 (compatible; LoadTimeBot/0.81; +http://www.load-time.com/bot.html)

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0

(BrowserSpyBot/1.0) (http://browserspy.dk/)

Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)

CopperEgg/RevealUptime/NewarkNJ(linode)

Automattic Analytics Crawler/0.2; http://wordpress.com/crawler/

larbin_2.6.3 [email protected]

HatenaScreenshot

CopperEgg/RevealUptime/LondonUK

Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)

Mozilla/5.0 (compatible; evc-batch/2.0.20160902205649)

yacybot (freeworld/global; amd64 Linux 3.2.0-23-generic; java 1.6.0_24; Europe/en)

http://yacy.net/bot.html

WordPress.com;
Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google
(+https://developers.google.com/+/web/snippet/)

Mozilla/5.0 (compatible; ScoutJet; http://www.scoutjet.com/)

Mozilla/5.0 (compatible; Uptimebot/0.1.97; +http://www.uptime.com/uptimebot)

WebTarantula.com Crawler

WatchMouse/8.4.0.3 (http://watchmouse.com/ ; itmil01.watchmouse.net)

LinqiaScrapeBot/1.0 ([email protected])

Orbiter (+http://www.dailyorbit.com/bot.htm)

Mozilla/5.0 (compatible; janforman.com/2.5; +http://janforman.com/)

InternetSeer.com

Mozilla/5.0 (compatible; aiHitBot/2.9; +http://www.aihitdata.com/about)

Mozilla/5.0 (compatible; WebNL; +http://www.web.nl/webmasters/spider.html; [email protected])

coccoc/1.0 (http://help.coccoc.vn/)

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/53.0.2785.116 Safari/537.36 PTST/314

Mozilla/5.0 (compatible; Googlebot/2.1; startmebot/1.0; +https://start.me/bot)

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; .NET CLR 1.1.4322; .NET CLR
2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E; PTST 2.386)

Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)

Mozilla/5.0 (compatible; idmarch Automatic.beta/1.4; +http://www.idmarch.org/bot.html)

Mozilla/5.0 (compatible; tldstat.com/Bot; +http://tldstat.com/bot.html)

yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_111; Europe/de) http://yacy.net/bot.html

Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 84846)

Mozilla/5.0 (compatible; spbot/4.0a; +http://www.seoprofiler.com/bot )

Mozilla/5.0 (Windows NT 6.1; Win64; x64;

+http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0

WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html)

OOZBOT/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )

Mozilla/5.0 (compatible; memoryBot/1.21.16 +http://internetmemory.org/en/)

FAST-WebCrawler/3.8

Googlebot-Image/1.0

enlle punto com/Nutch-1.9

OpenAcoon v4.10.5 (www.openacoon.de)

yacybot (freeworld/global; amd64 Linux 3.0.0-21-generic; java 1.6.0_26; America/en)

http://yacy.net/bot.html

Microsearch.ru Bot 1.3;http://microsearch.ru/webmasters

Mozilla/5.0 (compatible; Google-Site-Verification/1.0)

Mozilla/5.0 (compatible; Qwantify/2.3w; +https://www.qwant.com/)/2.3w

rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,
[email protected])

HubSpot Connect 2.0 (http://dev.hubspot.com/)

Robozilla/1.0

Mozilla/5.0 (compatible; spbot/2.0.3; +http://www.seoprofiler.com/bot/ )

Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116

Safari/537.36 PTST/314

facebookexternalhit/1.1

PercolateCrawler/3.1.30 ([email protected])

Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)

Mozilla/5.0 (compatible; pdffillerbot/1.0; +http://www.pdffiller.com/en/support.htm)

fastbot crawler beta 4.0 (+http://www.fastbot.de)

SafeDNS search bot/Nutch-1.9 (https://www.safedns.com/searchbot; support [at] safedns [dot]

com)

Mozilla/5.0 (Nekstbot; http://nekst.ipipan.waw.pl/nekstbot/)

yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; US/en)

http://yacy.net/bot.html

yacybot (freeworld/global; amd64 Linux 3.0.0-12-generic; java 1.6.0_26; Europe/en)

http://yacy.net/bot.html

coccoc/1.0 ()

larbin_2.6.3 ([email protected])

yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_0; Europe/de) http://yacy.net/bot.html

Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0;

+http://ecommerce.bigdatacorp.com.br/faq.aspx) AppleWebKit/535.1 (KHTML, like Gecko)
Chrome/13.0.782.107 Safari/535.1

yacybot (amd64 Linux 2.6.31-22-server; java 1.6.0_18; Asia/en) http://yacy.net/bot.html

yacybot (/global; amd64 Windows Server 2012 6.2; java 1.8.0_91; Europe/de)
http://yacy.net/bot.html
Mozilla/5.0 (compatible; Netseer crawler/2.0; +http://www.netseer.com/crawler.html;
[email protected])

Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.mixdata.com)

Scrapy/0.25.1 (+http://scrapy.org)

Mozilla/5.0 (compatible; Scopia Crawler 1.2; +http://www.scopia.co)

cn_ab_viewer ([email protected])

Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv 11.0) like Gecko (compatible; Zombiebot/2.1;

+http://www.zombiedomain.net/robot/)

Mozilla/5.0 (compatible; Uptimebot/0.1.38; +http://www.uptime.com/uptimebot)

yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; Europe/de)

http://yacy.net/bot.html

Python-urllib/2.7 (+http://sketch.juls.savba.sk/aranea_about)

Mozilla/5.0 (compatible; spbot/4.0; +http://www.seoprofiler.com/bot )

yacybot (-global; amd64 Windows 8 6.2; java 1.7.0_55; Europe/de) http://yacy.net/bot.html

yacybot (/global; amd64 Linux 3.2.41-042stab094.7; java 1.7.0_79; Europe/en)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; UASlinkChecker/1.0; +http://user-agent-string.info/UASlinkChecker)

Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5;

http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0)

yacybot (freeworld/global; amd64 Linux 3.10.15-1-MANJARO; java 1.7.0_40; Asia/en)

http://yacy.net/bot.html

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/6.0; .NET4.0E; .NET4.0C; .NET
CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729) CrawlerProcess
(http://www.PowerMapper.com) /5.9.726.0

Scrubby/2.1 (http://www.scrubtheweb.com/)

Mozilla/5.0 (compatible; JobKeresoBot; +http://www.kozvetlen-allasok.hu/help.jsp; info@kozvetlen-

allasok.hu; 5.0

yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_66; Europe/de) http://yacy.net/bot.html

Mozilla/5.0 (X11; Linux x86_64 UXCrawlerBot http://zium.co)

Slackbot-LinkExpanding (+https://api.slack.com/robots)

yacybot (freeworld/global; amd64 Linux 3.12.15-gentoo; java 1.7.0_55; Europe/en)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; LinkpadBot/1.07; +http://www.linkpad.ru)

Mozilla/5.0 (compatible; ZB-1; +http://www.zeerch.com/bot.php)

Mozilla/5.0 (compatible; WbSrch/1.1 +http://wbsrch.com)

WeSEE:Search

Mozilla/5.0 (compatible; DCPbot/1.1; +http://domains.checkparams.com/)

yacybot (webportal-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_45; Europe/en)

http://yacy.net/bot.html

WordPress.com mShots; http://support.wordpress.com/contact/

Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; FeedFinder-2.0; http://bloggz.se/crawler)

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6

(FlipboardProxy/0.0.5; +http://flipboard.com/browserproxy)

Mozilla/5.0 (compatible; coccocbot-image/1.0; +http://help.coccoc.com/searchengine)

Mozilla/5.0 (X11; Linux x86_64; GTmetrix http://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like

Gecko) Chrome/31.0.1650.63 Safari/537.36

https_page_test ([email protected])

Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11 360Spider;

yacybot (freeworld/global; amd64 Linux 3.2.0-2-amd64; java 1.6.0_24; Europe/en)

http://yacy.net/bot.html

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/600.1.4 (KHTML, like Gecko)

Safari/600.1.4 (compatible; Princetonbot/1.0; +http://http://tigress-
web.princeton.edu/~fy/bot.html)

Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11;

360Spider

KeywordDensityRobot/0.8 (http://www.seocentro.com/tools/search-engines/keyword-density.html)

Mozilla/5.0 (compatible; aiHitBot-BP/1.1; +http://www.aihit.com/)

findlinks/1.1.6-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/)

Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608

Firefox/3.0.8

CopperEgg/RevealUptime/FremontCA(linode)

Mozilla/5.0 (compatible; NTENTbot; +http://www.ntent.com/ntentbot)

link_thumbnailer

KeywordDensityRobot/0.9 (http://www.seocentro.com/tools/search-engines/keyword-density.html)

Mozilla/5.0 (compatible;WI Job Roboter Spider Version 3;+http://www.webintegration.at)

Mozilla/5.0 (compatible; Mail.RU/2.0)

CopperEgg/RevealUptime/N.California(aws)

Searchie Bot/1.0 (a Storm-based crawler; https://www.searchie.org; [email protected])

^Nail (http://CaretNail.com)

MarketBrewBot/2.1 (+http://www.marketbrew.com/)

Updownerbot (+http://www.updowner.com/bot)

yacybot (-global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html

Yanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/)

Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general;

[email protected])

Mozilla/5.0 (compatible; Scopia Crawler 2.0; +http://www.scopia.co)

asafaweb.com

KrOWLer/0.0.1, matentzn at cs dot man dot ac dot uk

Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters;

[email protected])

findlinks/1.0 (+http://wortschatz.uni-leipzig.de/findlinks/)

SygolBot http://www.sygol.com

voltron

Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: [email protected])

Comodo-Webinspector-Crawler 2.2.1, http://www.comodorobot.com

Superarama.com-Tarama-Botu-v.0.1

yacybot (freeworld/global; amd64 Linux 3.2.0-3-amd64; java 1.6.0_24; Europe/en)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; SSLBot/1.0; http://www.sslstats.com/sslbot)

PagesInventory.com (robot http://www.pagesinvenotry.com)

Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3 like Mac OS X; en-US) adbeat.com/policy

AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8F190 Safari/6533.18.5

WordPress.com Reader

Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)

Woko 3.0

Gigabot/3.0 (http://www.gigablast.com/spider.html)

ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html)

bl.uk_lddc_bot/3.1.1
(+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)

Urlstat (http://bit.ly/1dJuuzs)

yacybot (/global; amd64 Windows 8.1 6.3; java 1.7.0_55; America/ru) http://yacy.net/bot.html
Bad Neighborhood Header Detector (http://www.bad-neighborhood.com/header_detector.php)

Mozilla/5.0 (compatible; GeliyooBot/1.0beta; +http://www.geliyoo.com/)

NalezenCzBot/1.0 (http://www.nalezen.cz/about-crawler)

WeSEE_Bot:we_help_monitize_your_site (http://www.wesee.com/bot/)

yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_55; Asia/en) http://yacy.net/bot.html

studylib download bot, see http://studylib.es/public/bot for additional info

SemrushBot/0.91

Jabse.com Crawler v.1.0 www.jabse.com/crawler.php//imagecrawler

Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com;
http://www.girafa.com)

BizwikiBot/0.3 (http://www.bizwiki.com/bizwikibot.htm)

Mozilla/5.0 (compatible; BIXOCRAWLER; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-

[email protected])

PostPost/1.0 (+http://postpost.com/crawlers)

Mozilla/5.0 (nomore404.com robot/1.1; +https://nomore404.com/)

Riddler (http://riddler.io/about)

Mozilla/5.0 (compatible; socketcrawler; http://nlp.fi.muni.cz/projects/biwec/)

internetVista monitor (Mozilla compatible)

2Bone_LinkChecker/1.0 libwww-perl/6.03

MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com)

DomainStatsBot/1.0 (http://domainstats.io/our-bot)

KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile

goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)

Mergadobot/3.0.2 (+http://mergado.cz)

DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;

+http://search.goo.ne.jp/option/use/sub4/sub4-1/)

SalesIntelligent/v1.1

SiteAnalyzerBot

DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;

+http://help.goo.ne.jp/help/article/1142/)

yacybot (/global; i386 Linux 3.16.3-gentoo; java 1.7.0_55; Australia/en) http://yacy.net/bot.html

Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; +http://go.mail.ru/help/robots)

Cliqz Bot (+http://www.cliqz.com)

SalesIntelligent (+https://www.salesintelligent.com)

Mozilla/5.0 (compatible; memoryBot/1.20.243 +http://internetmemory.org/en/)

gimme60 (Gimme60 Store ID Bot; gimme60.com)

Orbiter/1.0 (http://dailyorbit.com/)

msnbot/1.1 (+http://search.msn.com/msnbot.htm)

Mozilla/5.0 (compatible; memoryBot/1.20.241 +http://internetmemory.org/en/)

Mozilla/5.0 (compatible; akula/k311; +http://k311.fd.cvut.cz/)

Irokez.cz monitoring v1.2 - (http://www.irokez.cz, Irokez.cz, crawl)

CCBot/2.0 (http://commoncrawl.org/faq/)

Mozilla/5.0 (compatible; openstat.ru/Bot)

Mozilla/5.0 (compatible; GigaBot/1.0; +http://www.gigablast.com/ )

Readability/6a54d4 - http://readability.com/about/

percbotspider <[email protected]>

Mozilla/5.0 (compatible; spbot/1.0; +http://www.seoprofiler.com/bot/ )

HatenaScreenshot/1.0 (checker)

Mozilla/5.0 (compatible; AcoonBot/4.12.1; +http://www.acoon.de/robot.asp)

CRAZYWEBCRAWLER 0.9.2, http://www.crazywebcrawler.com

Mozilla/5.0 (compatible; Online Domain Tools - Online Sitemap Generator/1.1; +http://sitemap-

generator.online-domain-tools.com)

Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool

+https://search.google.com/structured-data/testing-tool)

OnPageBot (compatible; Googlebot 2.1; +https://bot.onpage.org/)

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/48.0.2564.103 Safari/537.36 Yandex.Translate

Sogou web spider/4.0

Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots DEV)

LinkWalker/2.0

SeoCheck (FischerNetzDesign Seo Checker, [email protected])

yacybot (freeworld/global; amd64 Linux 3.2.13-1-ARCH; java 1.7.0_03-icedtea; Europe/fr)

http://yacy.net/bot.html

SiteCheck - http://downforeveryoneorjustme.com

Robots_Tester_http_www.searchenginepromotionhelp.com
CRAZYWEBCRAWLER 0.9.4, http://www.crazywebcrawler.com

Mozilla/5.0 (compatible; heritrix/3.1.1 +http://places.tomtom.com/crawlerinfo)

Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.10) Gecko/20050716 Thunderbird/1.0.6 -

WebCrawler http://cognitiveseo.com/bot.html

Mozilla/5.0 (compatible; Cliqzbot/1.0 +http://cliqz.com/company/cliqzbot)

yacybot (/global; amd64 Linux 4.2.0-gentoo-r1; java 1.8.0_45; Australia/en) http://yacy.net/bot.html

DomainsDB.net MetaCrawler v.0.9.7c (http://domainsdb.net/)

Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; mowser; http://www.mowser.com)

www.probethenet.com scanner

Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 6573)

LCC (+http://corpora.informatik.uni-leipzig.de/crawler_faq.html)

Imagga Downloader/v0.14.3

SeopultContentAnalyzer/1.0

Mozilla/5.0 (compatible; +http://www.hosterstats.com/newgtld-websurvey.php)

Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; +http://docs.google.com)

Mozilla/5.0 (compatible;acapbot/0.1)

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;Alibaba.Security.Heimdall)

HTTPClient/1.0 (2.8.0, ruby 2.0.0 (2014-05-08))

Jetty/9.3.z-SNAPSHOT

Jetty/9.3.z-SNAPSHOT, SMUrlExpander

Jetty/9.3.z-SNAPSHOT, SMViewer

Liferea/0.x.x (Linux; en_US.UTF-8; http://liferea.sf.net/)

SynHttpClient-Built/5825

asynchttp

SMUrlExpander

KickFire

HAAARTLAND http client

PEAR HTTPRequest class ( http://pear.php.net/ )

Mozilla/4.0 (vBSEO; http://www.vbseo.com)

HAARTLAND http client

JS-Kit URL Resolver http://js-kit.com/

http://www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsp

Kml-Google; (+http://code.google.com/apis/kml) gzip

gvfs/1.20.3

request.js

Microsoft URL Control - 6.01.9782

git/2.1.4

IPS/1.0

OxfordCloudService/1.0

Mozilla/5.0 (compatible; informatique at mixdata dot com/0.6; +http://t.co/GSRLLKex24;

informatique at mixdata dot com)

Mozilla/5.0 (compatible; Feedwind/2.0; +http://feed.mikle.com/support/description/)

Mozilla/5.0 (compatible; Feedwind/3.0; +http://feed.mikle.com/support/description/)

Mozilla/5.0 (Macintosh; Intel Mac OS X 1068) AppleWebKit/534.50 (KHTML like Gecko) Version/5.1
Instapaper/4.0 (+http://www.instapaper.com/)

Mozilla/5.0 (compatible; Keywords Research; http://www.check-domains.com/keyword-

research/keywords-adwords.php)

CAAM/0.1 (CAAM; caam dot crwlr at gmail dot com)

i2kconnect/"1.7.0"

resolver/0.1.12 (+http://github.com/andris9/resolver)

DangDang

Mozilla/5.0 (compatible; Tetrahedron/1.0; +https://compute.info/tet)

PritTorrent/0.1

FeedBunch/1.0 (+http://www.feedbunch.com)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko)

www.HomepageScreenshot.de - Free ScreenShotService/1.0 Safari/537.21

Mozilla/5.0 (compatible; Licorne Image Snapshot 1.0; +http://www.jednorozec.cz)

Pinterest/0.2 (+https://www.pinterest.com/bot.html)

Pinterest/0.1 (+https://www.pinterest.com/bot.html)

Mozilla/5.0 (iplabel; Windows NT 6.3; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0

Mozilla/5.0 zgrab/0.x

Zend_Http_Client

Zend\Http\Client
Zao/0.1 (http://www.kototoi.org/zao/)

YOURLS v1.7.1 +http://yourls.org/ (running on http://flnx.co)

yoogliFetchAgent/0.1

Yo-yo

Mozilla/5.0 (compatible; Yahoo Link Preview; https://help.yahoo.com/kb/mail/yahoo-link-preview-

SLN23615.html)

Mozilla/5.0 (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-

SLN24857.html)

Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile
(compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-
SLN24857.html)

xpymep.exe

xpymep2.exe

xpymep3.exe

Xenu Link Sleuth 1.2b

Xenu Link Sleuth 1.2c

Xenu Link Sleuth 1.2d

Xenu Link Sleuth 1.2e

Xenu Link Sleuth 1.2f

Xenu Link Sleuth 1.2g

Xenu Link Sleuth 1.2h

Xenu Link Sleuth 1.2i

Xenu Link Sleuth/1.3.7

Xenu Link Sleuth/1.3.8

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/40.0.2214.93 Safari/537.36 XaxisSemanticsClassifier/1.0 http://crystalsemantics.com

WWW-Mechanize/1.73

WordPress/4.0.1; http://bestcomp.hol.es

WordPress/4.3.3; http://homebusinessreviews.online

WordPress/4.3.3; http://legithomejobs.club

WordPress/4.4.2; http://cheapautoinsuranceinva.com

WordPress/4.4.2; http://citycollegeinmiami.com

WordPress/4.4.2; http://cookingideabook.com
WordPress/4.4.2; http://oldmanhealth.com

WordPress/4.4.2; http://sapcrmarchitecture.com

WordPress/4.4.2; http://www.spar3d.com

WordPress/4.5.2; http://1.we-careu.xyz

WordPress/x.x.x.x PHP/4.x.xxMozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML,

like Gecko) PhantomJS/1.6.1 Safari/534.34

Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.0.0

Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.25.0

Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.47.0

Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.48.0

Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.50.0

Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.56.0

Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.61.0

Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.62.0

Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.63.0

Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.64.0

WomlpeFactory/0.1 (+http://www.Womple.com/bot.html)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) wkhtmltoimage

Safari/534.34

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) wkhtmltopdf

Safari/534.34

Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)

Windows-RSS-Platform/2.0 (IE 11.0; Windows NT 10.0)

WhatsApp

WhatsApp/2.12.15/i

WhatsApp/2.12.16/i

WhatsApp/2.12.17/i

WhatsApp/2.12.449 A

WhatsApp/2.12.453 A

WhatsApp/2.12.510 A

WhatsApp/2.12.540 A

WhatsApp/2.12.548 A
WhatsApp/2.12.555 A

WhatsApp/2.12.556 A

WhatsApp/2.16.1/i

WhatsApp/2.16.13 A

WhatsApp/2.16.2/i

WhatsApp/2.16.42 A

WhatsApp/2.16.57 A

WhatWeb/0.4.8-dev

http://www.almaden.ibm.com/cs/crawler [wf84]

WeLikeLinks - WeAreNotEvil

Websquash.com (Add url robot)

WeCrawlForThePeace - We are not Evil

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) webscreenie Safari/537.21

Mozilla/3.0 (compatible; WebMon 1.0.11; Windows XP)

Mozilla/5.0 (X11; BSD Four) AppleWebKit/534.34 (KHTML, like Gecko) webkit2png Safari/534.34

WebFetch

WebIndex

webcollage/1.114

webcollage/1.117

webcollage/1.125

webcollage/1.129

webcollage/1.93

Mozilla/3.0 (compatible; WebCapture 2.0; Auto; Windows)

W3C_Unicorn/1.0

W3C_I18n-Checker/1.0

W3C-mobileOK/DDC-1.0

W3C-checklink

W3C-checklink/2.90 libwww-perl/5.64

W3C-checklink/3.6.2.3 libwww-perl/5.64

W3C-checklink/4.2 [4.20] libwww-perl/5.803

W3C-checklink/4.2.1 [4.21] libwww-perl/5.803

W3C-checklink/4.3 [4.42] libwww-perl/5.805

W3C-checklink/4.3 [4.42] libwww-perl/5.808

W3C-checklink/4.3 [4.42] libwww-perl/5.820

W3C-checklink/4.5 [4.154] libwww-perl/5.823

W3C-checklink/4.5 [4.160] libwww-perl/5.823

w3af.org

VYU2 (GNU; OpenRISC)

VORTEX/1.2 ( http://marty.anstey.ca/robots/vortex/)

Vortex/2.2 ( http://marty.anstey.ca/robots/vortex/)

Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/)

Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)

URLChecker

urlresolver

Upflow/1.0

Typhoeus - https://github.com/typhoeus/typhoeus

UdmSearch/3.1.x

Mozilla/5.0 Tweetminster

Mozilla/5.0 (compatible; +http://tweetedtimes.com)

truwoGPS/1.0 (GNU/Linux; U; i686; en-US; +http://www.lan4lano.net/browser.html )

Traackr.com

More-Testing (+http://www.theinternetrules.com)

The Expert HTML Source Viewer (http://www.expert-html.com)

htmlayout 3.3; above-Windows-7; www.terrainformatica.com )

WebFuck V2.1 T0PHackTeam www.t0p.xyz

Symfony2 BrowserKit

Mozilla/5.0 (compatible; SwiteScraper)

Summify (Summify/1.0.1; +http://summify.com)

Stratagems Kumo

Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu)

Sprinklr 2.0

spray-can/1.3.3
SortSite/4.4.500.0

Snoopy v2.0.0

SMRF URL Expander

silk/1.0 (+http://www.slider.com/silk.htm)/3.7

SiteBar/3.3.5 (Bookmark Server; http://sitebar.org/)

SiteBar/3.3.8 (Bookmark Server; http://sitebar.org/)

SimplePie/1.3.1 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20160226060242

ShortLinkTranslate

Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/)

SEOstats 2.0.9 https://github.com/eyecatchup/SEOstats

semanticdiscovery/0.1

SearchSight/2.0 (http://SearchSight.com/)

ScoutURLMonitor/5.9.8

Robosourcer/1.0

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; RetrevoPageAnalyzer;

+http://www.retrevo.com/content/about-us)

RestSharp/105.2.3.0

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.78.2 (KHTML, like Gecko)

Version/7.0.1(2.5.3) Safari(ReederForMac)

redback/v0-570-g26f8c96

RebelMouse/0.1 Mozilla/5.0 (compatible; http://rebelmouse.com) Gecko/20100101 Firefox/7.0.1

RealPlayer%20Downloader/1569 CFNetwork/720.3.13 Darwin/14.3.0 (x86_64)

RealPlayer%20Downloader/1701 CFNetwork/760.5.1 Darwin/15.5.0 (x86_64)

RealPlayer%20Downloader/1725 CFNetwork/720.3.13 Darwin/14.3.0 (x86_64)

RealPlayer%20Downloader/1750 CFNetwork/720.5.7 Darwin/14.5.0 (x86_64)

RealPlayer%20Downloader/1750 CFNetwork/760.0.5 Darwin/15.0.0 (x86_64)

RealPlayer%20Downloader/1750 CFNetwork/760.4.2 Darwin/15.4.0 (x86_64)

Qirina Hurdler v. 1.05 10.11.01 (+http://www.qirina.com/hurdler.html)

python-requests/1.1.0 CPython/2.7.6 Linux/3.13.0-58-generic

python-requests/1.2.0 CPython/2.7.3 Linux/3.2.0-29-virtual

python-requests/2.1.0 CPython/2.7.3 Linux/3.2.0-4-amd64

python-requests/2.10.0
python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-48-generic

python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-83-generic

python-requests/2.3.0 CPython/2.7.6 Linux/3.13.0-57-generic

python-requests/2.5.0 CPython/2.7.9 Linux/3.14.20-20.44.amzn1.x86_64

python-requests/2.6.0 CPython/2.7.3 Linux/3.2.0-4-amd64

python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-327.3.1.el7.x86_64

python-requests/2.7.0 CPython/2.7.6 Linux/3.13.0-85-generic

python-requests/2.8.0

python-requests/2.8.1

python-requests/2.9.1

Python-httplib2/0.7.4 (gzip)

Pulsepoint XT3 web scraper

Mozilla/5.0 (compatible; ProspectB2B/2.0; http://prospectb2b.com)

Mozilla/5.0 (Macintosh; Intel Mac OS X) PowerPoint/14.39.0

PostRank/2.0 (postrank.com; 1 subscribers)

PostRank/2.0 (postrank.com)

postano

Ploetz + Zeller (http://www.ploetz-zeller.de) Link Validator v1.0 ([email protected]) for ARIS

Business Architect

Pizilla++ ver 2.45

phpcrawl

Pearltrees

Pcore-HTTP/v0.23.20

Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.4) Gecko/2008102920 http://ow.ly web

crawler (.NET CLR 3.5.30729)

Mozilla/5.0 (compatible; Optimizer)

Mozilla/5.0 [en] (X11, U; OpenVAS 7.0.10)

JetBrains Omea Reader 1.0.x (http://www.jetbrains.com/omea_reader/)

JetBrains Omea Reader 2.0 Release Candidate 1 (http://www.jetbrains.com/omea_reader/)

okhttp/2.5.0

okhttp/4.9.0

oegp v. 1.3.0
Nuzzel

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134

Safari/537.36 http://notifyninja.com/monitoring

Notifixious/LinkChecker (http://notifixio.us)

http://www.nominet.org.uk/privacypolicy

node-superagent/1.8.3

node.io

Kimengi/nineconnections.com

newspaper/0.0.9.8

newspaper/0.1.7

newsme/1.0; [email protected]

NewsGator/2.0 Bot (http://www.newsgator.com)

NewsGator/2.5 (http://www.newsgator.com; Microsoft Windows NT 5.1.2600.0; .NET CLR

1.1.4322.2032)

NetTrack Anonymous Web Statistics https://nettrack.info/support.php

NetTrack.info Crawler

NetResearchServer(http://www.look.com)

NetResearchServer/2.5(loopimprovements.com/robot.html)

NetResearchServer/2.7(loopimprovements.com/robot.html)

NetResearchServer/2.8(loopimprovements.com/robot.html)

NetResearchServer/3.5(loopimprovements.com/robot.html)

NetResearchServer/4.0(loopimprovements.com/robot.html)

NetLyzer FastProbe (See http://netlyzer.com/report/www.domain.com for info)

check_http/v2.0.3 (nagios-plugins 2.0.3)

MVAClient

Link Validity Check From: http://www.w3dir.com/cgi-bin (Using: Hot Links SQL by Mrcgiguy.com)

Mojolicious (Perl)

mogimogi/1.0

Mindjet

Mindjet MindManager

Microsoft Windows Network Diagnostics

Microsoft Office Excel 2013

Microsoft Office Excel 2014

Microsoft Office Excel 2014 (16.0.6528) Windows NT 10.0

Microsoft Office Existence Discovery

Microsoft Office Mobile/15.0

Microsoft Office OneNote 2013

Microsoft Office PowerPoint 2013

Microsoft Office PowerPoint 2013 (15.0.4420) Windows NT 6.2

Microsoft Office PowerPoint 2013 (15.0.4641) Windows NT 6.2

Microsoft Office PowerPoint 2013 (15.0.4797) Windows NT 10.0

Microsoft Office PowerPoint 2013 (15.0.4797) Windows NT 6.2

Microsoft Office PowerPoint 2013 (15.0.4805) Windows NT 6.1

Microsoft Office PowerPoint 2013 (15.0.4805) Windows NT 6.2

Microsoft Office PowerPoint 2013 (15.0.4815) Windows NT 10.0

Microsoft Office PowerPoint 2014

Microsoft Office Word 2013

Microsoft Office Word 2013 (15.0.4623) Windows NT 6.2

Microsoft Office Word 2013 (15.0.4701) Windows NT 6.1

Microsoft Office Word 2013 (15.0.4797) Windows NT 10.0

Microsoft Office Word 2013 (15.0.4805) Windows NT 10.0

Microsoft Office Word 2013 (15.0.4805) Windows NT 6.1

Microsoft Office Word 2013 (15.0.4823) Windows NT 10.0

Microsoft Office Word 2014

Microsoft Office/16.0 (Windows NT 10.0; Microsoft Excel 16.0.13231; Pro)

Melvil Rawi/1.0

marketinggrader.com

Mandrill-Webhook/1.0

MailChimp.com

MagpieRSS/0.72 (+http://magpierss.sf.net)

http://www.mabontland.com

lwp-trivial/1.33

lwp-trivial/1.35
lwp-trivial/1.36

lwp-trivial/1.38

lwp-trivial/1.41

looksystems.net

LinkExaminer/1.01 (Windows)

VLC/2.2.4 LibVLC/2.2.4

[email protected]

Mozilla/5.0 (compatible; Jobrapido/1.1; +http://www.jobrapido.com)

JoBo/1.x (http://www.matuschek.net/jobo.html)

Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0

Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 001; +http://iodc.co.uk)

Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 005; +http://iodc.co.uk)

Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 006; +http://iodc.co.uk)

Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 007; +http://iodc.co.uk)

Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 011; +http://iodc.co.uk)

Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 013; +http://iodc.co.uk)

Mozilla/5.0 (compatible; IODC-Odysseus Survey 29691-100-070315021100-72; +http://iodc.co.uk)

Mozilla/5.0 (compatible; IODC-Odysseus Survey 29692-100-060415034231-192; +http://iodc.co.uk)

Mozilla/5.0 (compatible; IODC-Odysseus Survey 31193-100-080615161452-372; +http://iodc.co.uk)

Mozilla/5.0 (compatible; IODC-Odysseus Survey 37044-100-110516191742-025; +https://iodc.co.uk)

Mozilla/5.0 (compatible; IODC-Odysseus Survey 41437-100-090715214314-122; +http://iodc.co.uk)

Integrity/6

InfoWizards Reciprocal Link System PRO - (http://www.infowizards.com)

InDesign%20CC/11300 CFNetwork/720.4.4 Darwin/14.4.0 (x86_64)

inbound.li parser

InAGist URL Resolver (http://inagist.com)

igdeSpyder (compatible; igde.ru; +http://igde.ru/doc/tech.html)

IDG Twitter Links Resolver

Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)

Jersey/2.19 (HttpUrlConnection 1.7.0_85)

httpunit/1.x
httpssites_power/1 CFNetwork/758.3.15 Darwin/15.4.0

httpscheck (unknown version) CFNetwork/760.5.1 Darwin/15.5.0 (x86_64)

HttpComponents/1.1

http_requester/0.1

HTTP_Request2/@package_version@ (http://pear.php.net/package/http_request2) PHP/5.5.9-

1ubuntu4.14

HTTP_Request2/2.2.0 (http://pear.php.net/package/http_request2) PHP/5.3.29

HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.3.29

HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.5.33

HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.5.35

HTTP_Request2/2.3.0 (http://pear.php.net/package/http_request2) PHP/5.3.29

HTTP-Tiny/0.054

http-kit/2.0

HTMLParser/2.0

htdig/3.1.5

htdig/3.1.5 ([email protected])

htdig/3.1.5 (root@localhost)

htdig/3.1.5 ([email protected])

htdig/3.1.6 ([email protected])

htdig/3.1.6 ([email protected])

ht://check/2.0.0-rc1 (Linux 2.6.32-573.18.1.el6.x86_64 x86_64)

HEADMasterSEO

Mozilla/4.0 (compatible; grub-client-0.3.0; Crawl your own stuff with http://grub.org)

Mozilla/4.0 (compatible; grub-client-1.0.4; Crawl your own stuff with http://grub.org)

Mozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with http://grub.org)

Mozilla/4.0 (compatible; grub-client-1.0.6; Crawl your own stuff with http://grub.org)

Mozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with http://grub.org)

Mozilla/4.0 (compatible; grub-client-1.1.1; Crawl your own stuff with http://grub.org)

Mozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with http://grub.org)

Mozilla/4.0 (compatible; grub-client-1.3.1; Crawl your own stuff with http://grub.org)

Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org)

Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org)

Mozilla/4.0 (compatible; grub-client-1.5.3; Crawl your own stuff with http://grub.org)

Mozilla/4.0 (compatible; grub-client-2.3)

Mozilla/4.0 (compatible; grub-client-2.6.0)

Mozilla/4.0 (compatible; grub-client-2.6.1)

GoSpotCheck 1.0.6 (iPhone; iPhone OS 5.1.1; en_US)

GoScraper

GoogleProducer; (+http://goo.gl/7y4SX)

GoogleHC/

Google-HTTP-Java-Client/1.17.0-rc (gzip)

Google-HTTP-Java-Client/1.9.0-alpha

Mozilla/5.0 (compatible; Google-Apps-Script)

Go-http-client/1.1

Go 1.1 package http

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_6) AppleWebKit/534.24 (KHTML, like Gecko) (Contact:
[email protected])

Mozilla/5.0 (compatible; Funnelback) RPT-HTTPClient/0.3-3E

Mozilla/5.0 (compatible; Snacktory; +https://flynxapp.com)

Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; [email protected])

Flamingo_SearchEngine (+http://www.flamingosearch.com/bot)

Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Fetch API Request

FeedChecker-Zocle/1.0 (+https://zocle.com/zoclechecker)

FeedBurner/1.0 (http://www.FeedBurner.com)

FavOrg

Faraday v0.9.1

Mozilla/5.0 (Macintosh; Intel Mac OS X) Excel/14.10.0

Mozilla/5.0 (Macintosh; Intel Mac OS X) Excel/14.62.0

ExactSearch

EventMachine HttpClient

Mozilla/5.0 (compatible; MSIE 7.0 +http://www.europarchive.org)

Embed PHP Library

ECCP/1.2.1+([email protected])

ec2linkfinder

Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu)

Drupal (+http://drupal.org/)

Dragonfly File Reader

DNS-Tools Header-Analyzer

Dispatch/0.11.1-SNAPSHOT

[email protected]

CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp)

curb

Mozilla/4.0 (compatible; cron-job.org; http://cron-job.org/abuse/)

cosmos/0.9_([email protected])

Mozilla/5.0/Firefox/42.0 (contactbigdatafr at gmail.com)

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; MAXTHON 2.0);
Connect Us: [email protected]

Pattern/2.6 +http://www.clips.ua.ac.be/pattern

Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-0)

Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-1)

CC Metadata Scaper http://wiki.creativecommons.org/Metadata_Scraper

cb crawl (+http://www.companybooknetworking.com)

CapsuleChecker (http://www.capsulink.com/)

CakePHP

Branch-Passthrough

Branch Metrics API

boitho.com-dc/0.79 ( http://www.boitho.com/dcbot.html )

boitho.com-dc/0.81 ( http://www.boitho.com/dcbot.html )

boitho.com-dc/0.82 ( http://www.boitho.com/dcbot.html )

boitho.com-dc/0.83 ( http://www.boitho.com/dcbot.html )

boitho.com-dc/0.85 ( http://www.boitho.com/dcbot.html )

boitho.com-robot/1.1

Blackboard Safeassign
Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/42.0.2311.152 Safari/537.36 (BingLocalSearch)

BIGLOTRON (Beta 2;GNU/Linux)

Mozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/bnf.html)

Mozilla/4.0 (compatible; B-l-i-t-z-B-O-T)

Astute SRM

aria2/1.18.8

Mozilla/5.0 (compatible; archive-com.com/1.1; +http://archive-com.com/bot)

Mozilla/4.0 (compatible; Arachmo)

Apache-HttpClient/4.2.3 (java 1.5)

Apache-HttpClient/4.2.5 (java 1.5)

Apache-HttpClient/4.3.1 (java 1.5)

Apache-HttpClient/4.3.3 (java 1.5)

Apache-HttpClient/4.3.5 (java 1.5)

Apache-HttpClient/4.5 (Java/1.8.0_51)

Apache-HttpClient/4.5 (Java/1.8.0_72)

Apache-HttpClient/4.5 (Java/1.8.0_74)

Apache-HttpClient/4.5.1 (Java/1.7.0_79)

Apache-HttpClient/4.5.1 (Java/1.8.0_40)

Apache-HttpClient/4.5.2 (Java/1.8.0_72)

Apache-HttpClient/UNAVAILABLE (java 1.4)

Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.22; +http://software.schmorp.de/pkg/AnyEvent)

Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.23; +http://software.schmorp.de/pkg/AnyEvent)

Anemone/0.7.2

AndroidDownloadManager/6.0.1 (Linux; U; Android 6.0.1; SM-G900F Build/MMB29M)

AndroidDownloadManager/4.4.2 (Linux; U; Android 4.4.2; SM-T310 Build/KOT49H)

AndroidDownloadManager/4.4.2 (Linux; U; Android 4.4.2; SM-T210 Build/KOT49H)

Airmail 1.3.3 rv:237 (Macintosh; Mac OS X 10.9.3; sk_SK)

Python/3.5 aiohttp/0.21.6

AHC/1.0

Mozilla/4.0 (compatible; Advanced Email Extractor v2.61)

Pulp/1.5.2 (iPad; http://www.acrylicapps.com/pulp/)

Accoona-AI-Agent/1.1.1 (crawler at accoona dot com)

Accoona-AI-Agent/1.1.2

Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com)

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) [email protected]

scrutiny/6

RMA/1.0 (compatible; RealMedia)

PHP/5.5

NING/1.0

Mget/0.1.9

Java/1.4.1_04

Java/1.6.0

Java/1.6.0_04

Java/1.6.0_22

Java/1.6.0_24

Java/1.6.0_26

Java/1.6.0_31

Java/1.6.0_32

Java/1.6.0_34

Java/1.6.0_37

Java/1.7.0_101

Java/1.7.0_45

Java/1.7.0_51

Java/1.7.0_65

Java/1.7.0_80

Java/1.8.0_05

Java/1.8.0_20

Java/1.8.0_31

Java/1.8.0_40

Java/1.8.0_45

Java/1.8.0_51
Java/1.8.0_60

Java/1.8.0_65

Java/1.8.0_66

Java/1.8.0_73

Java/1.8.0_77

Java/1.8.0_91

Java/1.8.0_92

Goose/1.0.25

FDM 3.x

bluefish 0.6 HTML editor

2ip.ru CMS Detector (http://2ip.ru/cms/)

masscan/1.0 (https://github.com/robertdavidgraham/masscan)

VidibleScraper/1.0

WhereGoes? Redirect Tracer/1.0

unfurlist (https://github.com/Doist/unfurlist)

pshtt, https scanning

gofetch

COMODO DCV

CATExplorador/1.0beta (sistemes at domini dot cat; http://domini.cat/catexplorador.html)

ZmEu

RSSOwl/2.2.1.201312301314 (Windows; U; en)

WebClient/1.0

Microsoft-WebDAV-MiniRedir/10.0.10586

Microsoft-WebDAV-MiniRedir/10.0.14393

Microsoft-WebDAV-MiniRedir/5.1.2600

Microsoft-WebDAV-MiniRedir/6.0.6001

Microsoft-WebDAV-MiniRedir/6.1.7601

Microsoft-WebDAV-MiniRedir/6.3.9600

Microsoft Data Access Internet Publishing Provider Protocol Discovery

Test Certificate Info

Needle/1.4.2 (Node.js v6.9.2; linux x64)

DavClnt

Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko)
Version/6.0 Mobile/10A5376e Safari/8536.25 mixed-content-scan

Mozilla/5.0 (compatible; Semiocast HTTP client; http://semiocast.com/)

Mozilla/5.0 (compatible; The Drop Reaper; +http://tag1consulting.com/dropreaper.html)

/n software IPWorks HTTP/S Component - www.nsoftware.com

safe-agent-scanner

Mozilla/5.0 (compatible; ACHE/Unknown Version; +https://github.com/ViDA-NYU/ache; )

ANAM (github.com/dutchcoders/anam)

Dolphin http client/11.2.8(385) (Android)

Dolphin http client/11.3.4(405) (Android)

Calypso v/0.01

-Mozilla/4.0 (compatible; http://search.thunderstone.com/texis/websearch/about.html)

Mozilla/5.0 (compatible; Tweezler/2.0; http://tweezler.com)

Jeode/1.x.x

KeyCDN Perf Test

Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) YLT Chrome/27.0.1453.110

Safari/537.36

Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/51.0.2704.84 Safari/537.36 DMBrowser-UV

Taringa UGC www.taringa.net/shared-link.php

Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PingdomTMS/0.8.5

Safari/534.34

ganarvisitas/1.0 (+http://www.ganarvisitas.com/)

FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)

Mozilla/5.0 (compatible) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)

Mozilla/5.0 (compatible; Exabot PyExalead/3.0; +http://www.exabot.com/go/robot)

Mozilla/5.0 PhantomJS (compatible; Seznam screenshot-generator 2.1;

+http://fulltext.sblog.cz/screenshot/)

Mozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://napoveda.seznam.cz/en/seznambot-intro/)

Mozilla/5.0 (compatible; SeznamBot/3.2-test2; +http://napoveda.seznam.cz/en/seznambot-intro/)

Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://fulltext.sblog.cz/)

Mozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://fulltext.sblog.cz/)

Mozilla/5.0 (compatible; SeznamBot/3.2; +http://fulltext.sblog.cz/)

Seznam-Zbozi-robot/3.2.1

SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/)

Seznam-Zbozi-robot/3.2.2

Mozilla/5.0 (compatible; SeznamBot/3.2-test2; +http://fulltext.sblog.cz/)

Mozilla/5.0 (compatible; SeznamBot/3.2-test3; +http://fulltext.sblog.cz/)

Seznam-Zbozi-robot/3.3

SklikBot/2.0 ([email protected];+http://napoveda.sklik.cz/)

SeznamBot/3.0 (+http://fulltext.sblog.cz/)

Mozilla/5.0 (compatible; SeznamBot/3.1-test1; +http://fulltext.sblog.cz/)

Mozilla/5.0 (Linux; U; Android 4.1.2; cs-cz; Seznam screenshot-generator Build/Q3)

AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30

SeznamBot/3.0-test (+http://fulltext.sblog.cz/), I

SeznamBot/3.0 (HaF+http://fulltext.sblog.cz/)

SeznamBot/3.0-test (+http://fulltext.sblog.cz/)

SeznamBot/3.0-beta (+http://fulltext.sblog.cz/), I

SeznamBot/3.0-beta (+http://fulltext.sblog.cz/)

SeznamBot/3.0-alpha (+http://fulltext.sblog.cz/)

SeznamBot/2.0 (+http://fulltext.seznam.cz/)

SeznamBot/2.0-Test (+http://fulltext.sblog.cz/robot/)

Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; +http://fulltext.sblog.cz/screenshot/)

Mozilla/5.0 (compatible; FlipboardRSS/1.2; +http://flipboard.com/browserproxy)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0

(FlipboardProxy/1.2; +http://flipboard.com/browserproxy)

Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko)

Chrome/16.0.912.77 Safari/535.7 AppEngine-Google; (+http://code.google.com/appengine; appid:
s~feedly-social)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko)

Chrome/23.0.912.77 Safari/535.7 AppEngine-Google; (+http://code.google.com/appengine; appid:
s~feedly-nikon3)

Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)

Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01it.mtrs.yandex.ru)

Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko)
Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexBot/3.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; memoryBot/1.24.61 +http://internetmemory.org/en/)

Mozilla/5.0 (compatible; memoryBot/1.24.54 +http://internetmemory.org/en/)

BUbiNG - Research at Brno University of Technology - KNOT group -

http://knot.fit.vutbr.cz/crawling/ - Stop? http://law.di.unimi.it/BUbiNG.html

Mozilla/5.0 (compatible; MSIE 9.0; Firefox/50.1) Daum/4.1

Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.7

Daum Raam Edition b20140820 Safari/534.34

Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)

yacybot (/global; amd64 Linux 4.4.0-59-generic; java 1.8.0_121; Europe/de) http://yacy.net/bot.html

yacybot (-global; amd64 Linux 4.4.0-57-generic; java 9-internal; Europe/en) http://yacy.net/bot.html

yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_111; Europe/de) http://yacy.net/bot.html

yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_111; Europe/en) http://yacy.net/bot.html

yacybot (/global; amd64 FreeBSD 10.3-RELEASE-p7; java 1.7.0_95; GMT/en)

http://yacy.net/bot.html

yacybot (webportal-global; amd64 Linux 4.4.0-46-generic; java 1.8.0_91; Europe/en)

http://yacy.net/bot.html

yacybot (-global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.8.0_111; Europe/de)

http://yacy.net/bot.html

Mozilla/5.0 (compatible; special_archiver/3.3.0 [email protected]

+https://archive.org/details/archive.org_bot)

Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247

+http://archive.org/details/archive.org_bot)

Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.4 +http://www.archive.org)

SeobilityBot (SEO-Check; http://bit.ly/1dJuuzs)

Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.4

Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.3

Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.2

Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.1

Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)

Heurekabot-Feed/1.0 (+https://sluzby.heureka.cz/napoveda/heurekabot/)

Heurekabot-ImageFullText/1.0 (+https://sluzby.heureka.cz/napoveda/heurekabot/)

Heurekabot-ImageFullText/1.0 (+http://sluzby.heureka.cz/napoveda/heurekabot/)
Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63
Safari/537.36 QIHU 360SE; 360Spider

ZoomBot (Linkbot 1.0 http://suite.seozoom.it/bot.html)

Mozilla/5.0 (compatible; SOLOFIELD/1.0 +http://solofield.net/bot.html)

Qwantify/1.0

Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/bot)

Scrapy/1.3.0 (+http://scrapy.org)

Scrapy/1.0.5.post4+g4b324a8 (+http://scrapy.org)

Scrapy/1.2.1 (+http://scrapy.org)

Scrapy/0.16.5 (+http://scrapy.org)

Mozilla/5.0 (compatible; Dataprovider.com;)

Mozilla/5.0 (compatible; Dataprovider; https://www.dataprovider.com/spider/)

Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)

ICC-Crawler/2.0 (Mozilla-compatible; ; http://ucri.nict.go.jp/en/icccrawler.html)

DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html)

Mozilla/5.0 (compatible; SMTBot/1.0; http://www.similartech.com/smtbot)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) Chromium/31.0.1650.63

Site-Shot/2.1 (http://www.site-shot.com/) Safari/538.1

Mozilla/5.0 (Windows NT 6.1; WOW64; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/48.0.2564.82 Safari/537.36

Mozilla/5.0 (iPad; CPU OS 6_1_2 like Mac OS X; Neustar WPM) AppleWebKit/536.26 (KHTML, like
Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25

Mozilla/5.0 (Windows NT 6.0; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/48.0.2564.82 Safari/537.36

Mozilla/5.0 (Windows NT 6.1; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/48.0.2564.82 Safari/537.36

Mozilla/5.0 (Windows NT 6.0; rv:38.0; Neustar WPM) Gecko/20100101 Firefox/38.0

Mozilla/5.0 (Windows NT 6.1; rv:38.0; Neustar WPM) Gecko/20100101 Firefox/38.0

Mozilla/5.0 (Windows NT 6.1; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/52.0.2743.82 Safari/537.36

Mozilla/5.0 (Linux; Android 4.1.1; HTC6435LVW Build/JRO03C; Neustar WPM) AppleWebKit/537.22

(KHTML, like Gecko) Chrome/25.0.1364.169 Mobile Safari/537.22

Mozilla/5.0 (compatible; Whoiswebsitebot/0.1; +http://www.whoiswebsite.net)

Mozilla/5.0 (Linux; Android 5.0.2; SM-G920T Build/LRX22G) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840

Safari/537.36 DareBoost

Mozilla/5.0 (Linux; Android 5.0; LG-D855 Build/LRX21R.A1422018487) AppleWebKit/537.36 (KHTML,

like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost

Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_1 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like
Gecko) Version/10.0 Mobile/14A403 Safari/602.1 DareBoost

Mozilla/5.0 (Linux; Android 4.4.3; HTC One Build/KTU84L) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/54.0.2840 Mobile Safari/537.36 DareBoost

Mozilla/5.0 (Linux; Android 4.4; Nexus 5 Build/KRT16M) AppleWebKit/537.36 (KHTML, like Gecko)
Version/4.0 Chrome/54.0.2840 Mobile Safari/537.36 DareBoost

HubSpot Website Grader ([email protected])

HubSpot Marketing Grader, HubSpot Marketing Grader

com.hubspot.spooks.core.UserAgent@753535c0, HubSpot Marketing Grader

Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79

Safari/537.36 (http://www.shrinktheweb.com)

scrapy-redis (+https://github.com/rolando/scrapy-redis)

Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/55.0.2883.87 Safari/537.36 PTST/348

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/54.0.2840.99 Safari/537.36 PTST/345

TeamSpeak3-ImageFetcher-1.0

wf_crawler (http://www.websitefigures.com)

wf_crawler (http://websitefigures.com)

Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/55.0.2883.87 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)

TinEye-bot/0.61 (see http://www.tineye.com/crawler.html)

Mozilla/5.0 (compatible; Googlebot/2.1; https://deepcrawl.com/bot)

Googlebot deepcrawl

Mozilla/5.0 (compatible; evc-batch/2.0.20170110125844)

Mozilla/5.0 (compatible; evc-batch/2.0.20161220184139)

Mozilla/5.0 (compatible; evc-batch/2.0.20161122223033)

Mozilla/5.0 (compatible; evc-batch/2.0.20161026010401)

Mozilla/5.0 (X11; Linux x86_64; rv:49.0; GTmetrix https://gtmetrix.com/) Gecko/20100101
Firefox/49.0

Mozilla/5.0 (Linux; Android 4.3; Galaxy Nexus Build/JWR66Y; GTmetrix https://gtmetrix.com/)

AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.68 Mobile Safari/537.36

Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like

Gecko) Chrome/54.0.2840.71 Safari/537.36

Mediumbot-MetaTagFetcher/0.3 (+https://medium.com/)

Mediumbot-ProxyFetcher/0.1 (+https://medium.com/)

YOOBI.de - WebBot 1.0

YOOBI - WebBot 1.0

Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20160309-0050; UniLeipzigASV

+http://corpora.informatik.uni-leipzig.de/crawler_faq.html)

IAS crawler (page scorer; http://integralads.com/site-indexing-policy/)

Mozilla/5.0 (compatible; JobKeresoBot; +https://www.kozvetlen-allasok.hu/help.jsp;

[email protected]; 5.0

Mozilla/5.0 (compatible; JobKereso; +https://www.kozvetlen-allasok.hu/robot.jsp info@kozvetlen-

allasok.hu)

pyspider/0.3.9 (+http://pyspider.org/)

Mozilla/5.0 (X11; U; Linux x86_64; de-DE; rv:1.9.0.19) Gecko/2010120923 ThumbShotsBot (KFSW

3.0.6-3)

Mozilla/5.0 (compatible; Discordbot/1.0; +https://discordapp.com)

Mozilla/5.0 (compatible; Dead Link Checker; http://www.dead-link-checker.com/)

Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess

(http://www.PowerMapper.com) /5.24.776.0

Mozilla/4.0 (compatible; BOTW Spider; +http://botw.org)

TwengaBot-2.0

Iframely/1.0.0 (+https://iframely.com/;)

Iframely/0.8.5 (+http://iframely.com/;)

rogerbot/1.2 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+phaser-testing-

[email protected])

rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected]

Mozilla/5.0 (compatible; Sitemap Generator/1.3; http://www.check-

domains.com/sitemap/index.php) Gecko Check-domains)

mindUpBot
Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0;
+http://bigweb.bigdatacorp.com.br/faq.aspx) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/41.0.2272.118 Safari/537.36

Contacts Crawler (+http://www.scrapinghub.com)

TA SEO Crawler v0.1 URLData.pm ([email protected])

Mozilla/5.0 (compatible; scanbot/1.0; +http://dazzlepod.com/ip/)

ArielisBot/1.1 (+http://arielis.com/

RedesScrapy/0.24.1 (+http://g2pi.tsc.uc3m.es/es)

Blackboard Safeassign/0.1 (a Storm-based Blackboard Safeassign web-crawler;

https://github.com/DigitalPebble/storm-crawler; [email protected])

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:48.0) Gecko/20100101 Firefox/48.0 glindahl-

cocrawler/0.01 (+http://www.pbm.com/~lindahl/glindahl-cocrawler.html)

botrobin/Nutch-1.13-SNAPSHOT (http://smarter.codes/bot-robin/; [email protected])

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.63

Safari/537.36 (compatible; aylienbot/0.2; +http://www.aylien.com/bot.html)

GloomarBot/1.1 (https://www.gloomar.com/bot)

Hatena::Bookmark/2.00 (Hatena::Bookmark; master;)

Opendi Screenshot Bot [email protected]

Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; [email protected])

Mozilla/5.0 (compatible; KAZ.KZ_Bot/3.0)

BuckyOHare/1.3 (Googlebot/2.1; +https://hypefactors.com/webcrawler)

uipbot/1.0 ([email protected])

gMozilla/5.0 (compatible; Infohelfer/1.4.3; +http://www.infohelfer.de/crawler.php)

Mozilla/5.0 (compatible; lincobot/0.3; http://www.linksco.com/about/robot.html) AppEngine-

Google; (+http://code.google.com/appengine)

Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) MapperCmd/5.24.780.0

Seeker v.1 http://LookSeek.com/seeker/

AppManager RPT-HTTPClient/0.3-3E

LinkAlarm/3.1

Mozilla/5.0 (X11; U; Linux i686) Firefox/2.0.0.11 wprecon.com survey

Jeffrey's Exif Viewer (http://regex.info/exif)

Untiny

Mozilla/5.0 (Grabify.link - https://grabify.link/expander)

biNu image cacher ([email protected])

Mozilla/5.0 (compatible; Google-Youtube-Links)

ping.blo.gs/2.0

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; QQDownload 1.7; .NET CLR 2.0.50727)

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; QQDownload 663; SLCC2; .NET CLR
2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET CLR 1.1.4322)

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; QQDownload 760; .NET CLR
2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0E; .NET4.0C; .NET CLR
1.1.4322)

Mozilla/5.0 (X11; Linux x86_64; rv:52.0.2) Gecko/20100101 Firefox/52.0.2 Ghost Inspector

Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0 Ghost Inspector

Mozilla/5.0 (X11; Linux x86_64; rv:50.1.0) Gecko/20100101 Firefox/50.1.0 Ghost Inspector

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) QtWebEngine/5.3.0

Safari/538.1 Ghost Inspector

Mozilla/5.0 (X11; Linux x86_64; rv:45.0.1) Gecko/20100101 Firefox/45.0.1 Ghost Inspector

Mozilla/5.0 (X11; Linux x86_64; rv:46.0.1) Gecko/20100101 Firefox/46.0.1 Ghost Inspector

Mozilla/5.0 (X11; Linux x86_64; rv:47.0.2) Gecko/20100101 Firefox/47.0.2 Ghost Inspector

Mozilla/5.0 (X11; Linux x86_64; rv:48.0.2) Gecko/20100101 Firefox/48.0.2 Ghost Inspector

Mozilla/5.0 (X11; Linux x86_64; rv:49.0.2) Gecko/20100101 Firefox/49.0.2 Ghost Inspector

Netpursual/1.0

Kaspersky Lab CFR link resolver [email protected]

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453

Safari/537.36 (compatible; Google-HotelAdsVerifier/2.0)

b0t

"echocrawl 2.0"

masscan/1.0

Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko)
Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexBot/3.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexAccessibilityBot/3.0; +http://yandex.com/bots)

Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko)
Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0;
+http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexDirectDyn/1.0; +http://yandex.com/bots

Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexPagechecker/1.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YaDirectFetcher/1.0; Dyatel; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexCalendar/1.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexSitelinks; Dyatel; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexMetrika/3.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexVertis/3.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)

jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)

Mozilla/5.0 (compatible; Abonti/0.8 - http://www.abonti.com)

CloudEndure Scanner ([email protected])

AFB/3.0 (+http://allloadin.com)

TLSProbe/1.0 (+https://scan.trustnet.venafi.com/)

Sens.ai http://sovereign.ai/contact

eContext/1.0 (eContext Classification Engine)

Owler

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Slack/1.2.6

Chrome/45.0.2454.85 AtomShell/0.34.3 Safari/537.36 Slack_SSB/1.2.6

Mozilla/5.0 (compatible; ChangesMeter/1.9.1; http://intuiware.com/apps/changes-meter)

Mozilla/5.0 (compatible; Gookey.co/1.0; +http://gookey.co/)

TrapitAgent/0.1 (feed processor; +http://trapit.com/about)

sqlmap/1.0-dev-nongit-201612050a8c (http://sqlmap.org)

UniversalFeedParser/3.3 +http://feedparser.org/

TulipChain/5.xx (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://apple.com/)

Mac_OS_X/10.2.8

scalaj-http/1.0
HggH PhantomJS Screenshoter

fasthttp

fasthttp, Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/53.0.2785.101 Safari/537.36

lua-resty-http/0.10 (Lua) ngx_lua/10000

GIS-LABS:CertMon

GuzzleHttp/6.2.1 PHP/7.0.15-0ubuntu0.16.04.4

http-request/v0.7.0 (http://git.io/tl_S2w) node.js/v0.10.29

intraVnews/1.x

MailChimp

G-i-g-a-b-o-t

Symfony BrowserKit

VCPP71 http-get 1.0a

Jaunt/1.2

Mozilla/5.0 (compatible; Owler/0.4; +; )

MovableType/x.x

OMSC

websitepulse agent/0.1 (compatible; MSIE 5.5; Netscape 4.75; Linux)

Mozilla/5.0 (compatible; Uptimia; www.uptimia.com)

ssl-tools.net

Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0; .NET CLR 1.1.4322; .NET CLR
1.0.3705; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152;
.NET CLR 3.5.30729) DMBrowser/2.1

keycdn-tools/perf

keycdn-tools/br

keycdn-tools/h2

keycdn-tools/perf

keycdn-tools/br

keycdn-tools/h2

KOCMOHABT (+https://kozmonavt.tk/) Mozilla/5.0 (compatible; Web Explorer 2)

Donuts Content Explorer (www.donuts.domains)

Netsparker
Mozilla/5.0+(compatible; MxToolbox/Beta7; http://www.mxtoolbox.com/)

Apache-HttpAsyncClient/4.1.3 (Java/1.8.0_121)

Tenon.io Automated Web Accessibility Tester

ApacheBench/2.3

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Clarsentia)

Mozilla/5.0 (compatible; historious/2.0; +http://historio.us/)

Mozilla/5.0 (compatible; uCrawlr/1.0 ; +https://blog.ucoz.ru/upolicy)

Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)

Brodie/1.0 (+http://www.15miles.com/)

Jobg8 URL Monitor

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) wpif Safari/537.21

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) imgsizer Safari/537.21

Mozilla/5.0 (compatible; AppBeat/1.0; +http://www.appbeat.io/)

Mozilla/5.0 (compatible; Chirp/1.0; +http://www.binarycanary.com)

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SiteMonitor)

Mixnode/3.0.1 (+https://www.mixnode.com/; Speed analysis; no data is stored)

Pingability.com Monitoring Service

SiteSucker for macOS/2.10.6

Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)

Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)

Sideqik +http://www.sideqik.com

SiteSucker/2.4.6

Vulnbusters Meter (see http://vulnbusters.com for details)

Express WebPictures (www.express-soft.com)

gobyus

Search37/1.0 (http://www.search37.com; [email protected])

Search37/1.2 (http://www.search37.com; [email protected])

Mozilla/5.0 (compatible; SiteIndexed/1.2; +http://www.siteindexed.com)

Jersey/2.25.1 (Apache HttpClient 4.5)

URL Verifier by Daanav.com

trendspottr.com
Kerrigan/2.0 ([email protected])

Keyword Extractor 5000; [email protected]

Citoid (Wikimedia tool; learn more at https://www.mediawiki.org/wiki/Citoid)

grokkit ([email protected])

akka-http/10.0.10

POE-Component-Client-HTTP/0.943 (perl; N; POE; en; rv:0.943000)

RecurPost/1.0 https://recurpost.com (KHTML, like Gecko)

Mozilla/5.0 Firefox Windows WhoRunsCoinHive.com

Mozilla/5.0 (compatible; Kelny/2.0; +http://www.kelny.com/)

colly - https://github.com/gocolly/colly

WebCopier vx.x

gSOAP/2.8

http.rb/2.2.2

Grabber - https://sites.google.com/3ziele.de/grabber/

JobFeed discovery script (http://www.jobfeed.com)

JoinVision Generic Site Loader 1.0

SSI-LinkPreviewGenerator/1.0

SnowHaze Search/1.0 [email protected]

Saleslift

Perlu - HTML Analyzer and Relationship Builder 1.1

Offline Explorer 1.*

Mozilla/5.0 (compatible; Wappalyzer; https://www.wappalyzer.com)

AngleSharp/0.9.9

Email%20Extractor%20Lite/26 CFNetwork/720.3.13 Darwin/14.3.0 (x86_64)

HubSpot Page Screenshot Service

NewRelicPinger/1.0 (543245)

APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)

updown.io daemon 2.2

updown.io daemon 2.3

updown.io daemon 2.2

updown.io daemon 2.3

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81
Safari/537.36 PrintFriendly.com

Unshorten.It!/1.0 (https://unshorten.it/)

github-camo (65c9ebe0)

github-camo (abb93c82)

github-camo (bacd3d5b)

github-camo (df02412a)

Jobsearch1.5

HubSpot Content Strategy link check

Twisted PageGetter

node-urllib/2.25.3 Node.js/6.12.2 (Linux 4.4; x64)

sentry/8.18.0 (https://sentry.io)

Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/55.0.2883.87 Safari/537.36 (compatible; Sitebulb/1.1; +https://sitebulb.com)

Thinklab (thinklab.com)

HubSpot inbound link reporting check

SetCronJob/1.0 (+http://www.setcronjob.com/)

Burf Search Engine

Apache Droid

Mozilla/5.0 (compatible; adscanner/)

hackney/1.11.0

Mozilla/5.0 (compatible; MuckRack/1.0; +http://muckrack.com)

Mozilla/5.0 (compatible; dotMailer content retrieval service)

Nexgate Ruby Client

Miniature.io/2.0 (Linux; en-us)

[email protected]

Mozilla/5.0 (compatible; Your-Website-Sucks/2.0; +http://www.Your-Website-

Sucks.net/WWW.SOMEWEBSITE.COM)

axios/0.17.1

Wtrace

PostmanRuntime/7.1.1

hackney/1.11.0
Lavf/57.25.100

makecontact/1.0 (+https://makecontact.io)

QrafterPro/4600 CFNetwork/889.9 Darwin/17.2.0

Buck/2.2; (+https://app.hypefactors.com/media-monitoring/about.html)

Go http package

HonesoSearchEngine/1.0

orion-semantics.com 0.1

ExperianCrawlUK (andrew dot swanton at phgroup dot com)

Trendsmap Resolver/0.0.1

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts; .NET CLR 1.1.4322;
PeoplePal 6.2)

Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent

Mozilla/3.0 (compatible; Indy Library)

VB Project

webster-internet.de pad browser

LWP::Simple/5.805

Microsoft Data Access Internet Publishing Provider DAV

MSFrontPage/4.0

Microsoft Data Access Internet Publishing Provider Cache Manager

VB Project

eCatch/3.0

MSFrontPage/12.0

Mozilla/4.7 (compatible; OffByOne; Windows 2000) Webster Pro V3.4

Mozilla/4.7 (compatible; OffByOne; Windows 98) Webster Pro V3.2

webster-internet.de pad browser

Trendsmap Resolver/0.0.1

Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 4.0.30319.17929)

Arukereso/1.0

Trendsmap Resolver/0.0.1

Flock/14.15 (Android 2.9; fr_BE;)

Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.0.16) Gecko/2010021003 Firefox/3.0.16

Flock/2.5.6
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1.14) Gecko/20080414 Firefox/2.0.0.14
Flock/1.1.2

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070801 Firefox/2.0.0.6

Flock/0.9.0.2

AppStoreScraperZ/1 CFNetwork/889.9 Darwin/17.2.0 (x86_64)

Everwall Link Expander ( http://everwall.com )

Wfuzz/2.2.2

The Knowledge AI [email protected]

The Knowledge AI

Mozilla/5.0 (compatible; DataXu/1.0; +http://dataxu.com)

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/60.0.3112.113 Safari/537.36 (+https://whatis.contentkingapp.com)

AdAuth/1.0; +https://www.adauth.com

twibble.io

JolokiaPwn, https://matmannion.com/jolokiapwn/, [email protected]

CookieReports.com/0.1

GoogleCloudMonitoring

kube-probe/1.9+

Mozilla/5.0 Chrome/10.0.613.0 Safari/534.15 Zombie.js/5.0.7

Microsoft Office/16.0 (Microsoft Outlook Mail 16.0.7167; Pro)Mozilla/4.0 (compatible; MSIE 7.0;
Windows NT 6.1; Win64; x64; Trident/7.0; .NET CLR 2.0.50727; SLCC2; .NET CLR 3.5.30729; .NET CLR
3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; Tablet PC 2.0; InfoPath.3; Microsoft Outlook
14.0.6025; ms-office; MSOf

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Win64; x64; Trident/7.0; .NET CLR 2.0.50727;
SLCC2; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; InfoPath.3;
.NET4.0E; Microsoft Outlook 14.0.7167; ms-office; MSOffice 14)

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/7.0; SLCC2; .NET CLR
2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E;
InfoPath.3; Microsoft Outlook 15.0.4805; ms-office; MSOffice 15)

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 10.0; WOW64; Trident/7.0; .NET4.0C; .NET4.0E;
.NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; Tablet PC 2.0; MALNJS; Microsoft
Outlook 16.0.6001; ms-office; MSOffice 16)

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR
2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E;
InfoPath.3; Microsoft Outlook 14.0.6025)
Outlook-Express/7.0 (MSIE 7.0; Windows NT 6.1; WOW64; Trident/7.0; SLCC2; .NET CLR 2.0.50727;
.NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; McAfee; .NET4.0C; .NET4.0E;
GWX:QUALIFIED; TmstmpExt)

Mail/3445.4.7 CFNetwork/889.9 Darwin/17.2.0 (x86_64)

SPDYCheck SPDY Protocol Tester, see http://spdycheck.org/about.html

SpamExperts/128741 (Virus Scan)

Unshorten.It! /1.0 (unshorten.it)

Monit/5.25.2

Mozilla/5.0 (compatible; PingAdmin.Ru/1.1; +http://pingadmin.ru/)

Mozilla/5.0 (compatible; localsearch-web/2.0; +http://www.localsearch.ch/en/legal-notice)

Mozilla/5.0 (X11; Datanyze; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/65.0.3325.181 Safari/537.36

annotate_google; http://ponderer.org/download/annotate_google.user.js

Apache OpenOffice/4.1.2

Bidtellect/0.0.346.0

BlockNote.Net

Bloglines/3.1 (http://www.bloglines.com)

DemandbasePublisherAnalyzer/0.1 (http://www.demandbase.com; [email protected])

endo/1.0 (Mac OS X; ppc i386; http://kula.jp/endo)

Feedable/0.1 (compatible; MSIE 6.0; Windows NT 5.1)

Feedreader 3.xx (Powered by Newsbrain)

FeedshowOnline (http://www.feedshow.com)

FeedZcollector v1.x (Platinum) http://www.feeds4all.com/feedzcollector

ImageVisu/v4.x.x

InetURL:/1.0

Mixmax-LinkPreview 1.0 (+https://sdk.mixmax.com)

Mozilla/5.0 (compatible; DNS SSL/TLS HTTP and HTML scanner/0.1; +https://www.htmlyse.com/)

Mozilla/5.0 (compatible; DNS SSL/TLS HTTP HTML Website Security Scanner/0.2 beta;
+https://www.htmlyse.com/)

Mozilla/5.0 (compatible; Scrutiny/8; +https://peacockmedia.software/mac/scrutiny/

NetNewsWire/2.x (Mac OS X; http://ranchero.com/netnewswire/)

og-scraper/1.0.0
Plagger/0.x.xx (http://plagger.org/)

raynette_httprequest/1.0

RssReader/1.0.xx.x (http://www.rssreader.com) Microsoft Windows NT 5.1.2600.0

ScrapeBox Page Scanner

SearchExpress

Snarfer/0.x.x (http://www.snarfware.com/)

unirest-java/1.3.11

WinPodder (http://winpodder.com)

WordupinfoSearch/1.0

13TABS/13TABS-1.14 (https://www.13tabs.com/aboutUs.jsp)

777Siters/1.05 (+http://777s.ru/siters/)

7Siters/1.07 (+http://7ooo.ru/siters/)

Aranea Web Corpora Project (Sk Crawl) (+http://unesco.uniba.sk/guest)

censys.io internet census

Chromaxa/1.0 Sonar/1.0

compatible; Licorne Uri Checker 1.0; +http://www.jednorozec.cz)

CronDroid v0.80a http://www.crondroid.com

Cronjob.de

Cronless/2.0 (+http://cronless.com)

EasyCron/1.0 (https://www.easycron.com/)

EDS EXIF/1.0 (https://www.exif-search.com)

EDS EXIF/1.0 (https:/exif-search.com)

faviconkit/1.0 (https://faviconkit.com)

HomePage Rss Reader 1.0

Hong Kong Education City/1.0 (built with Hong Kong Education City 1.10; http://www.hkedcity.net/;
[email protected])

https://getroot.sh survey

isUp.li Website Monitoring 2

JAHHO check

MB-SitemapGenerator

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; http://basicstate.com/)

Mozilla/5.0 (compatible; Aprc/2.9.15-24; +https://aprc.it/) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/70.0.3528.4 Safari/537.36

Mozilla/5.0 (compatible; Google AppsViewer; http://drive.google.com)

Mozilla/5.0 (compatible; GoogleSites)

Mozilla/5.0 (compatible; iThemes Sync/1.0; +http://ithemes.com/sync)

Mozilla/5.0 (compatible; iThemes Sync/1.0; ITX WP-Admin API/1.0; +http://ithemes.com/sync)

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; CHKD 1.2; Akamai_Site_Analyzer)

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; WAC-OFU)

Mozilla/5.0 (compatible; WebDataStats/1.0 ; +https://webdatastats.com/policy.html)

Mozilla/5.0 (Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143

Safari/537.36 Nodemeter 0.7

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:41.0) Gecko/20100101 Firefox/55.0

BrandVerity/1.0 (http://www.brandverity.com/why-is-brandverity-visiting-me)

Mozilla/5.0 (Windows NT 6.1; rv:12.0; StumbleUpon; [email protected]) Gecko/20100101

Firefox/12.0

Mozilla/5.0 (Windows NT 6.1; rv:40.0; StumbleUpon; [email protected]) Gecko/20100101

Firefox/40.1

Mozilla/5.0 (Windows NT 6.1; stumbleupon.com) Chrome/58.0.3029.110 Safari/537.36

Mozilla/5.0 AppleWebKit (compatible; s4a-probe/1.0; +https://www.seo4ajax.com/webscraper)

Mozilla/5.0 AppleWebKit (compatible; s4a/1.0; +https://www.seo4ajax.com/webscraper)

Mozilla/5.0 Gecko/20100101 Firefox/51.0.1 SnapSearch

NodePing

PocketImageCache/1.0

queuedriver/1.0 ([email protected])

Re-re Studio (+http://re-re.ru/)

Tarmot Gezgin/1.0 (compatible; TarmotGezgin/1.1; +http://www.tarmot.com/gezgin)

WDT.io

webeval.researchready.com - [email protected] for info.

WWW::Document/0.02

Y!J-BRW/1.0 (https://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716)

YoYs.net

YP.PL
Mozilla/5.0 (Windows NT 6.1; rv:40.0; StumbleUpon; [email protected]) Gecko/20100101
Firefox/40.1

Mozilla/5.0 (Windows NT 6.1; StumbleUpon; [email protected]) Chrome/58.0.3029.110

Safari/537.36

Mozilla/5.0 (Windows NT 6.1; stumbleupon.com) Chrome/58.0.3029.110 Safari/537.36

Mozilla/5.0 (compatible; tracemyfile/1.0)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)

HeadlessChrome/66.0.3336.0 Safari/537.36

Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/61.0.3163.100 Safari/537.36 Pix/1.0 (Criteo; x64-criteo)

CDR/1.7.1 Simulator/0.7(+http://timewe.net) Profile/MIDP-1.0 Configuration/CLDC-1.0

Email%20Extractor/1 CFNetwork/760.6.3 Darwin/15.6.0 (x86_64)

Email%20Extractor/29 CFNetwork/897.15 Darwin/17.5.0 (x86_64)

HTTP Banner Detection (https://security.ipip.net)

HTTP Banner Detection(security.ipip.net)

Mozilla/5.0 (compatible; SimpleScraper)

http_get

MeltwaterNews www.meltwater.com

Disqus/1.0

Datafeedwatch/2.1.x

Zabbix

Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; http://desktop.google.com/)

httpRequest

Google-Transparency-Report

Mozilla/5.0 (compatible; TrendsmapResolver/0.1)

Embarcadero URI Client/1.0

GetCode

Gregarius/0.5.2 ( http://devlog.gregarius.net/docs/ua)

GRequests/0.10

DAP/NetHTTP

Hadi Agent

Canary%20Mail/397 CFNetwork/893.13.1 Darwin/17.4.0 (x86_64)

Sendsay.Ru/1.0; https://Sendsay.Ru/; [email protected]

Mozilla/5.0 (Zoom.Mac 10.8.5 x86)

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/70.0.3538.77 Safari/537.36 TryJsoup/1.0 (+http://try.jsoup.org/)

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0 ; BacklinkHttpStatus)

SilverReader/1.0; http://silverreader.com

ExtractorPro

WebsiteExtractor

Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202

Safari/535.1 google_partner_monitoring FWSzVTDDBz14547302713138T

Outlook-iOS/696.1188109.prod.iphone (2.102.0)

Polymail/1.47 (ctrlla.Polymail; build:64; OS X 10.12.6) Alamofire/4.5.1

nyawc/1.8.1 CPython/3.6.5 Linux/3.10.0-862.9.1.el7.x86_64

commonscan.org cralwer v1.01

AdminLabs

WebGazer/1.0 (+https://www.webgazer.io)

HappyApps-WebCheck/1.0

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6; +feeder.co) AppleWebKit/537.36 (KHTML, like

Gecko) Chrome/67.0.3396.99 Safari/537.36

Mozilla/5.0 (compatible; TestURI; +http://testuri.org/)

WebSniffer/1.0 (+http://websniffer.cc/)

Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko)
Version/9.0 Mobile/13B143 Safari/601.1 SeoSiteCheckup (https://seositecheckup.co

Adstxtaggregator.com/1.0

Discourse Forum Onebox v2.0.0.beta10

faviconarchive.org

GreatNews/1.0

MemGator:1.0-rc2 <@WebSciDL>

URLTester/1 CFNetwork/974.1 Darwin/18.0.0 (x86_64)

URLTester/1 CFNetwork/975.0.3 Darwin/18.2.0 (x86_64)

Urlcheckr/2.0

SimpleChecker
reqwest/0.9.9

ReactorNetty/0.7.10.RELEASE

Mozilla/5.0 (compatible; woorankreview/2.0; +https://www.woorank.com/)

https://www.nominet.uk/privacy-notice

<a href='http://www.unchaos.com/'> UnChaos </a> From Chaos To Order Hybrid Web Search
Engine.([email protected])

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36(KHTML, like Gecko) Chrome/69.0.3464.0

Safari/537.36 Chrome-Lighthouse

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36(KHTML, like Gecko)

Chrome/61.0.3116.0 Mobile Safari/537.36 Chrome-Lighthouse

Trackuity (+http://trackuity.com)

deeris/1.0 (+http://www.deeris.com)

Mediametric

TextRazor Downloader (https://www.textrazor.com)

ddline.cn rank history

Web spyder

AppleSyndication/52.8

AutomaticWPTester/1 CFNetwork/975.0.3 Darwin/17.7.0

CSSCheck/1.2.2

dBpoweramp

Thumbor/6.4.1

Reeder/1000.02.01 CFNetwork/454.11.5 Darwin/10.5.0 (i386) (Macmini2)

worldping-api

Geziyor 1.0

adstxt-worker (www.column6.com)

adstxt.com/1.2

Mozilla/5.0 (compatible; ZnHTTP/7.80)

khttp/1.0.0-SNAPSHOT

IZaBEE/IZaBEE-1.01 (Buzzing Abound The Web; https://izabee.com; info at izabee dot com)

HTTPie/1.0.2

Craw/1.0

internetwache.org v3.4
looid.com Search Engine/0.1

OgScrper/1.0.0

NotionEmbedder

facebookexternalua

Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; DuplexWeb-Google/1.0)

AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Mobile Safari/537.36

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118

Safari/537.36 (compatible; Google-Read-Aloud;
+https://support.google.com/webmasters/answer/1061943)

FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)

Amazon Simple Notification Service Agent

cortex/1.0

AWS Security Scanner

HaxerMen

kubectl/v1.12.0 (linux/amd64) kubernetes/0ed3388

NetSystemsResearch studies the availability of various services across the internet. Our website is
netsystemsresearch.com

Mozilla/5.0(Linux;Android 5.1.1;OPPO A33 Build/LMY47V;wv) AppleWebKit/537.36(KHTML,link

Gecko) Version/4.0 Chrome/43.0.2357.121 Mobile Safari/537.36 LieBaoFast/4.51.3

sc-downloader/2.0 (http://www.searchconcepts.io)

[object Promise] delve.ai website analyse

attohttpc/0.9.0

Bewica-security-scan

BlogBridge 2.13 (http://www.blogbridge.com/)

Boost.Beast/248

DomainLabz

ExoRank

Fuzz Faster U Fool v1.0-rc1

Hexometer

longurl-r-package

looid.com Search Engine/0.1

Mozilla/5.0 (compatible; RyowlEngine/1.0; +https://ryowl.org)

Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0; https://app.limber.io) Gecko/20100101
Firefox/25.0

OSPScraper (+https://www.opensyllabusproject.org)

python-httpx/0.11.1

shortURL lengthener/0.1 +http://kevindriscoll.info/

Scamadviser-Frontend

Wolfram HTTPClient 12.

SEOlizer/1.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13

Mozilla/5.0 (pc-x86_64-linux-gnu) Siege/4.0.4

Filestack

Google-Ads-Overview Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/41.0.2272.118 Safari/537.36

Google-Ads-Overview Mozilla/5.0 (Linux; U; Android 6.0.1; generic) AppleWebKit/537.36 (KHTML,

like Gecko) Chrome/41.0.2272.118 Version/4.0 Mobile Safari/537.36

Google-Ads-Overview Mozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.36 (KHTML,

like Gecko) Chrome/41.0.2272.118 Version/4.0 Mobile Safari/537.36

Google-Ads-Overview Mozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.36 (KHTML,

like Gecko) Version/4.0 Mobile Safari/537.36

Mozilla/5.0 AppleWebKit/537.36 Chrome/114.0.5735.179 Safari/537.36 Google-Ads-Conversions

adreview/1.0

Google-speakr

Google-speakr,gzip(gfe)

PageFreezer

Prebid.js Scraper

Mr.4x3 Powered

Goldfire Server

OhDear.app (+https://ohdear.app/docs/checks/uptime)

Mozilla/5.0 (compatible;Cloudflare-Healthchecks/1.0;+https://www.cloudflare.com/; healthcheck-

id: 755842180c7ae904)

WinHttp-Autoproxy-Service/5.1

DynatraceSynthetic/

acebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)

Mozilla/5.0 (compatible; Google-Podcast)

Project25499 Scanner ([email protected])

GitHub-Hookshot/a6f2714

Fiery%20Feeds/272 CFNetwork/1125.2 Darwin/19.4.0 (x86_64)

FreshRSS/1.13.0 (Linux; https://freshrss.org)

Gofeed/1.0

Mozilla/5.0 (Windows NT 6.1) AppleWebKit/538.1 (KHTML, like Gecko) QuiteRSS/0.18.12

Safari/538.1

php-requests/1.7

nghttp2/1.40.0

WinHTTP/1.1

Mozilla/5.0 (compatible; Javelin; +https://about.javelin.io/)

swcd (unknown version) CFNetwork/1125.2 Darwin/19.4.0

YahooMailProxy; https://help.yahoo.com/kb/yahoo-mail-proxy-SLN28749.html

Mozilla/5.0 (Java) outbrain

DHSH

DHSH Virus Check

DHSH Ads Check

DHSH Statistik Chec

DHSH Law Check

DHSH User Check

Mozilla/5.0+(compatible; Cula/2.0; https://cula.io/)

Winds: Open Source RSS & Podcast app: https://getstream.io/winds/

XenForo/2.x (https://www.recreationalflying.com)

TECHIAITH_CY (+https://techiaith.cymru)

ScopeContentAG-HTTP-Client www.thescope.com/0.1

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Lush Http Client

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/78.0.3904.97 Safari/537.36 The National Archives UK Government Web Archive:
http://www.nationalarchives.gov.uk/webarchive/; [email protected]

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/79.0.3945.88 Safari/537.36 (compatible; NameFo; https://namefo.com/)

Mozilla/5.0 (Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79

Safari/537.36 (compatible; Statically-Screenshot; +https://statically.io/screenshot)
Mozilla/5.0 (compatible; Domains Project/1.0.8; +https://domainsproject.org)

Mozilla/5.0 (compatible; AffiliateLabz/1.0; +https://affiliatelabz.com/)

Mozilla/5.0 (compatible; 2GDPR/1.2; https://2gdpr.com)

Mozilla/5.0 (+http://www.sli-systems.com/) Mammoth/0.1

Microsoft.Data.Mashup (https://go.microsoft.com/fwlink/?LinkID=304225)

LinkPreview/1.5 (http://linkpreview.net)

https://gdnplus.com:Gather Analyze Provide.

Feedshow/x.0 (http://www.feedshow.com; 1 subscriber)

apimon.de/http-apis#redirect

ZoteroTranslationServer/WMF (mailto:[email protected])

yomins.com/0.3

WakeletLinkExpander/0.1

ultimate_sitemap_parser/0.5

Swurl Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/77.0.3865.120 Safari/537.36

postplanner.com/site-scraping

Nuclei (@pdiscoveryio)

myseosnapshot/1.0

Contextual Code Sites Explorer

Corax - [email protected]

Florienzh/1.0

BKCTwitterUnshortener/0.1 (Contact: [email protected])

Syndirella/0.91pre

PodcastAddict/v2 - Dalvik/2.1.0 (Linux; U; Android 6.0.1; SM-G920F Build/MMB29K)

PodcastAddict/v5 (+https://podcastaddict.com/; Android podcast app)

PDF24 URL To PDF

MacOutlook/16.41.20091302 (Intelx64 Mac OS X 10.14.4 (Build 18E226))

HubSpot-Link-Resolver

Railgun/5.3.3

RSSMix/3.0

SearchWP
Stripe/1.0 (+https://stripe.com/docs/webhooks)

ZoominfoBot (zoominfobot at zoominfo dot com)

Asana/1.4.0 WebsiteMetadataRetriever

mattermost-5.19.0

WP Rocket/Homepage Preload

Anthill

cf-facebook/6.8.48 Node.js/v6.11.1

LinkAnalyser/2.0

PHP/{5|6|7}.{3|2}.{1|2|3|4|5|6|7|8|9|0}{1|2|3|4|5|6|7|8|9|0}

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0

Observatory/82.0

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) Project-Resonance (http://project-

resonance.com/) (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36

Mozilla/5.0 (compatible; DNSResearchBot/2.1; +http://195.37.190.77)

Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)

Mozilla/5.0 (compatible; tchelebi/1.0; +http://tchelebi.io)

OnalyticaBot

Pleroma 2.1.50-495-gf9ece1a7-develop+dev; https://search.fedi.app <[email protected]>;

Bot

SmerBot/0.1 ([email protected]) Language Model Dataset Fetcher

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)

HeadlessChrome/71.0.3578.98 Safari/537.36 Prerender (+https://github.com/prerender/prerender)

Bitrix link preview

Mikrotik/6.x Fetch

Mikrotik/7.x Fetch

HTTPing v2.5

Mozilla/5.0 (compatible; JungleKeyThumbnail/1.1; +http://www.junglekey.fr/)

postfix-mta-sts-resolver

go-mtasts/1.0

Mail::STS

Google-SMTP-STS

Netlab360
TLS tester from https://testssl.sh/dev/

TBI-HttpOpenPlugin/0.1.0 (+https://leakix.net/)

TBI-WebScanner/0.0.1 (+https://leakix.net/)

quic-go-HTTP/3

CISPA Web Analyser (https://notify.cispa.saarland)

webtech/1.2.11

Hardenize (https://www.hardenize.com)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:46.0) Gecko/20100101 Firefox/46.0 | Hardenize

(https://www.hardenize.com)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:46.0) Gecko/20100101 Firefox/46.0 |

Hardenize/v1.1196.1 (https://www.hardenize.com)

Expanse indexes the network perimeters of our customers. If you have any questions or concerns,
please reach out to: [email protected]

Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per
day to identify customers&#39; presences on the Internet. If you would like to be excluded from our
scans, please send IP addresses/domains to: [email protected]

PageThing http://pagething.com

PageThing.com

Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; PageThing http://pagething.com); rv:1.9;

Gecko/2008052906 Firefox/3.0

ureq

ureq/2.0.0-rc1

ureq/2.0.1

URL/Emacs Emacs/26.1 (X11; x86_64-pc-linux-gnu)

unshortenit 0.4.0

Statically-Images/2.0

Statically-Icon/1.4.1

RadioPublicImageResizer/2.0

nettle (+https://www.nettle.sk)

NihilScio - Educational search engine - +https://www.nihilscio.it/NihilScio.htm

Miro-HttpClient/1.0

mio_httpc 0.9.4
Amazon-Route53-Health-Check-Service (ref c96ad03c-acdf-48c6-8ec7-45806bf08123; report
http://amzn.to/1vsZADi)

pnpm/6.18.0 npm/? node/v16.11.1 linux x64

Gradle/7.1 (Linux;4.15.0-162-generic;amd64) (Oracle Corporation;11.0.10;11.0.10+9)

npm/6.14.12 node/v10.24.1 linux x64 ci/drone

pip/21.3.1
{"ci":null,"cpu":"x86_64","distro":{"name":"macOS","version":"11.6"},"implementation":{"name":"C
Python","version":"3.9.7"},"installer":{"name":"pip","version":"21.3.1"},"openssl_version":"OpenSSL
1.1.1l 24 Aug
2021","python":"3.9.7","setuptools_version":"59.2.0","system":{"name":"Darwin","release":"20.6.0
"}}

pip/21.0.1
{"ci":null,"cpu":"x86_64","distro":{"id":"stretch","libc":{"lib":"glibc","version":"2.24"},"name":"Debia
n
GNU/Linux","version":"9"},"implementation":{"name":"CPython","version":"3.7.9"},"installer":{"na
me":"pip","version":"21.0.1"},"openssl_version":"OpenSSL 1.1.0l 10 Sep
2019","python":"3.7.9","setuptools_version":"53.0.0","system":{"name":"Linux","release":"4.19.123
-coreos"}}

twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1

tqdm/4.62.3 CPython/3.8.12

Aether

PHP-AYMAPI/1.0

Adyen HttpClient 1.0

affilimate-puppeteer

docker/19.03.8 go/go1.12.17 git-commit/afacb8b7f0 kernel/4.19.123-coreos os/linux arch/amd64

UpstreamClient(Docker-Client/19.03.8 \(linux\))

Apache Ant/1.10.9

Synapse/1.36.0

GoogleProber

Lucee (CFML Engine)

RapidLoad/1.0

SAP NetWeaver Application Server (1.0;750)

Y!J-DLC/1.0 (https://support.yahoo-net.jp/PccSearch/s/article/H000007955)

ColdFusion

node-fetch

awin.com - site screen shotter

chatterino-api-cache/1.0 link-resolver

gobuster/3.1.0

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114

Safari/537.36 DatadogSynthetics

Frontman v1.9.3 linux amd64

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/87.0.4280.88 Safari/537.36 ForusP

Mozilla/5.0 (compatible; seocompany/0.6; +; )

Visual Rights Group (https://visualrightsgroup.com/) Scanner

Mozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io)

Mollie HTTP client/1.0

Inboxb0t-Image/2.0

fragFINN.de HTTP-Collector

SpeedKit

Amazon CloudFront

link-check/5.2.0

Mozilla/5.0/Firefox/42.0 - nbertaupete95(at)gmail.com