Well-behaved crawlers
Read the rules. Stayed off the lawn. Every client below fetched our robots.txt and has never requested anything under /lawn/, as far back as our records go. Thank you. If one of them ever does, it moves to the Wall of Shame.
Generated 2026-11-02 09:08 UTC. Rebuilt every few minutes. Also available as well-behaved.json. Listed: 93 groups, 474 clients (IP address and user agent pairs), 689 robots.txt fetches, 6269 requests.
Clients that never fetched robots.txt are not listed: we can't tell whether they followed it. "Saw the bait" means the client requested a page carrying hidden links into /lawn/ (the homepage or /sitemap.xml). Requests counts every request, any path. Rows are ordered by most recently seen. "Observed" lines are the strongest observed traits of a row's clients: what they sent, not a verdict on who or what they are. Following robots.txt is listed here whatever else a client did.
Verified crawlers
User agent matched a known crawler and the source address passed that vendor's published verification (reverse DNS or published IP ranges). Only these rows show individual IPs.
| Client | Networks | Clients | robots.txt fetches | Requests | Saw the bait | First seen Last seen | |||
|---|---|---|---|---|---|---|---|---|---|
| 24h | 7d | 30d | all | ||||||
| Anthropic verified AS16509 AMAZON-02 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +…Observed: Came from a network that X4BNet's public list classifies as datacenter or hosting. (3 of 3 clients) | 216.73.216.94/32+1 more | 4 | 6 | 2 | 2 | 2 | 13 | yes | 2026-09-27 04:36Z 2026-11-02 08:47Z |
| OpenAI verified AS8075 MICROSOFT-CORP-MSN-AS-BLOCK Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like…Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like…Observed: Sent no Accept-Language header. Mainstream browsers always send one. (39 of 40 clients)Observed: Came from a network that X4BNet's public list classifies as datacenter or hosting. (40 of 40 clients) | 51.8.102.11/32+37 more | 40 | 87 | 2 | 22 | 71 | 87 | no | 2026-09-27 05:14Z 2026-11-02 03:59Z |
| Google verified AS15169 GOOGLE Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)Observed: Sent no Accept-Language header. Mainstream browsers always send one. (20 of 20 clients)Observed: Came from a network that X4BNet's public list classifies as datacenter or hosting. (20 of 20 clients) | 66.249.66.160/32+17 more | 20 | 53 | 1 | 21 | 57 | 69 | yes | 2026-09-27 16:58Z 2026-11-02 00:04Z |
| Apple verified AS714 APPLE-ENGINEERING Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, li…Observed: Sent no Accept-Language header. Mainstream browsers always send one. (8 of 8 clients) | 17.166.24.217/32+5 more | 8 | 8 | 0 | 15 | 15 | 15 | no | 2026-10-27 23:21Z 2026-10-31 22:38Z |
| Ahrefs verified AS16276 OVH Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)Observed: Sent no Accept-Language header. Mainstream browsers always send one. (20 of 20 clients)Observed: Came from a network that X4BNet's public list classifies as datacenter or hosting. (20 of 20 clients) | 15.235.27.3/32+17 more | 20 | 20 | 0 | 6 | 17 | 20 | no | 2026-10-01 19:08Z 2026-10-31 19:39Z |
| Amazon verified AS14618 AMAZON-AES Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +…Observed: Sent no Accept-Language header. Mainstream browsers always send one. (1 of 1 client)Observed: Came from a network that X4BNet's public list classifies as datacenter or hosting. (1 of 1 client) | 3.217.82.254/32 | 1 | 1 | 0 | 0 | 1 | 1 | no | 2026-10-14 07:42Z 2026-10-14 07:42Z |
| Perplexity verified AS14618 AMAZON-AES Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.…Observed: Sent no Accept-Language header. Mainstream browsers always send one. (2 of 2 clients)Observed: Came from a network that X4BNet's public list classifies as datacenter or hosting. (2 of 2 clients) | 18.97.9.98/32 | 2 | 2 | 0 | 0 | 2 | 2 | no | 2026-10-12 00:44Z 2026-10-14 04:27Z |
| Microsoft verified AS8075 MICROSOFT-CORP-MSN-AS-BLOCK Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +ht…Observed: Sent no Accept-Language header. Mainstream browsers always send one. (2 of 2 clients)Observed: Came from a network that X4BNet's public list classifies as datacenter or hosting. (2 of 2 clients) | 40.77.167.108/32 | 2 | 3 | 0 | 0 | 3 | 3 | no | 2026-10-11 12:26Z 2026-10-13 07:48Z |
| Yandex verified AS208398 TELETECH Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) | 87.250.224.129/32 | 1 | 1 | 0 | 0 | 0 | 1 | no | 2026-09-27 06:53Z 2026-09-27 06:53Z |
Claimed, unverifiable
User agent names a known crawler whose vendor publishes no verification method we could find. The claim is not confirmed.
| Client | Networks | Clients | robots.txt fetches | Requests | Saw the bait | First seen Last seen | |||
|---|---|---|---|---|---|---|---|---|---|
| 24h | 7d | 30d | all | ||||||
| Meta claimed, unverifiable AS32934 FACEBOOK facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko…Observed: Sent no Accept-Language header. Mainstream browsers always send one. (94 of 94 clients)Other clients making this claim requested /lawn/: see the Wall of Shame. | 2a03:2880:2ff::/48+10 more | 94 | 107 | 15 | 60 | 99 | 107 | no | 2026-09-28 00:16Z 2026-11-01 19:36Z |
Spoofed crawler user agents
User agent claimed a known crawler, but the source address failed that crawler's published verification. Listed under the network the requests came from; nothing here is attributed to the crawler named.
| Client | Networks | Clients | robots.txt fetches | Requests | Saw the bait | First seen Last seen | |||
|---|---|---|---|---|---|---|---|---|---|
| 24h | 7d | 30d | all | ||||||
| AS396982 GOOGLE-CLOUD-PLATFORM spoofed UA User agent claimed OpenAI; failed OpenAI's published verification Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +ht…Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1…Observed: Sent no Accept-Language header. Mainstream browsers always send one. (2 of 2 clients)Observed: Came from a network that X4BNet's public list classifies as datacenter or hosting. (2 of 2 clients)+1 more observed traits in well-behaved.jsonOther clients from this network requested /lawn/: see the Wall of Shame. | 34.146.33.0/24 | 2 | 3 | 0 | 0 | 90 | 90 | yes | 2026-10-23 14:28Z 2026-10-25 11:09Z |
| AS396982 GOOGLE-CLOUD-PLATFORM spoofed UA User agent claimed Anthropic; failed Anthropic's published verification Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot…Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +…Observed: Sent no Accept-Language header. Mainstream browsers always send one. (2 of 2 clients)Observed: Came from a network that X4BNet's public list classifies as datacenter or hosting. (2 of 2 clients)+1 more observed traits in well-behaved.jsonOther clients from this network requested /lawn/: see the Wall of Shame. | 34.146.33.0/24 | 2 | 2 | 0 | 0 | 59 | 59 | no | 2026-10-21 15:47Z 2026-10-25 11:09Z |
| AS396982 GOOGLE-CLOUD-PLATFORM spoofed UA User agent claimed Common Crawl; failed Common Crawl's published verification CCBot/2.0 (https://commoncrawl.org/faq/)Observed: Sent no Accept-Language header. Mainstream browsers always send one. (1 of 1 client)Observed: Came from a network that X4BNet's public list classifies as datacenter or hosting. (1 of 1 client)+1 more observed traits in well-behaved.jsonOther clients from this network requested /lawn/: see the Wall of Shame. | 34.146.33.0/24 | 1 | 1 | 0 | 0 | 36 | 36 | no | 2026-10-23 14:29Z 2026-10-25 11:09Z |
Other clients, by network
No known crawler user agent. Attributed only to the network (ASN) the requests came from.
| Client | Networks | Clients | robots.txt fetches | Requests | Saw the bait | First seen Last seen | |||
|---|---|---|---|---|---|---|---|---|---|
| 24h | 7d | 30d | all | ||||||
AS396982 GOOGLE-CLOUD-PLATFORM anonymousMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko…Mozilla/5.0 (compatible; KimiBot/1.0; +https://kimi.ai/)+3 more user agentsObserved: Sent no Accept-Language header. Mainstream browsers always send one. (4 of 23 clients)Observed: Came from a network that X4BNet's public list classifies as datacenter or hosting. (23 of 23 clients)+2 more observed traits in well-behaved.jsonOther clients from this network requested /lawn/: see the Wall of Shame. | 16.216.88.0/24+4 more | 23 | 50 | 2 | 7 | 390 | 390 | yes | 2026-10-06 15:15Z 2026-11-02 05:41Z |
AS132203 TENCENT-NET-AP-CN Tencent Building, Kejizhongyi Avenue anonymousMozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (…Observed: Claimed a browser user agent, but its TLS handshake did not offer HTTP/2. Mainstream browsers always offer it. (3 of 3 clients)Observed: Claimed a browser user agent, but used HTTP/1.1 over HTTPS although HTTP/2 was offered. (3 of 3 clients)+1 more observed traits in well-behaved.jsonOther clients from this network requested /lawn/: see the Wall of Shame. | 43.131.32.0/24 | 3 | 3 | 2 | 4 | 7 | 7 | yes | 2026-10-10 17:06Z 2026-11-02 03:22Z |
AS398324 CENSYS-ARIN-01 anonymousMozilla/5.0 (compatible; CensysInspect/1.1; +https://about.censys.io/)Observed: Sent no Accept-Language header. Mainstream browsers always send one. (45 of 45 clients) | 66.132.159.0/24+5 more | 45 | 48 | 9 | 45 | 160 | 160 | yes | 2026-10-05 21:50Z 2026-11-02 02:46Z |
AS3215 France Telecom - Orange anonymousMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like…Other clients from this network requested /lawn/: see the Wall of Shame. | 92.150.119.0/24 | 1 | 1 | 1 | 1 | 1 | 1 | no | 2026-11-02 01:40Z 2026-11-02 01:40Z |
AS701 UUNET anonymousMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, li…Mozilla/5.0 (iPhone; CPU iPhone OS 18_7 like Mac OS X) AppleWebKit/605.1.15 (KH…+1 more user agentsObserved: Claimed a browser user agent and fetched 10 or more pages over HTTPS without ever requesting /favicon.ico. (2 of 3 clients)Other clients from this network requested /lawn/: see the Wall of Shame. | 100.12.184.0/24 | 3 | 6 | 25 | 29 | 29 | 70 | yes | 2026-09-27 04:33Z 2026-11-01 22:45Z |
AS14618 AMAZON-AES anonymousMozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, lik…Mozilla/5.0 (compatible; getdomaindata/1.0; +https://getdomaindata.com/bot)+2 more user agentsObserved: Claimed a browser user agent, but its TLS handshake did not offer HTTP/2. Mainstream browsers always offer it. (26 of 30 clients)Observed: Sent no Accept-Language header. Mainstream browsers always send one. (30 of 30 clients)+2 more observed traits in well-behaved.json | 3.215.197.0/24+27 more | 30 | 37 | 3 | 12 | 40 | 40 | yes | 2026-10-04 05:28Z 2026-11-01 19:19Z |
AS264617 GRUPO PANAGLOBAL 15 S.A anonymousApcosysBot/1.0 (+https://apcosys.net)Observed: Sent no Accept-Language header. Mainstream browsers always send one. (3 of 3 clients) | 2a0a:20c0:f001::/48 | 3 | 33 | 2 | 10 | 33 | 33 | no | 2026-10-07 15:46Z 2026-11-01 18:08Z |
AS16509 AMAZON-02 anonymousvisionheight.com/scan Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/12…Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like…+2 more user agentsObserved: Claimed a browser user agent, but its TLS handshake did not offer HTTP/2. Mainstream browsers always offer it. (1 of 8 clients)Observed: Sent no Accept-Language header. Mainstream browsers always send one. (7 of 8 clients)+2 more observed traits in well-behaved.jsonOther clients from this network requested /lawn/: see the Wall of Shame. | 3.129.187.0/24+5 more | 8 | 21 | 3 | 12 | 55 | 61 | yes | 2026-09-29 01:30Z 2026-11-01 17:52Z |
AS4837 CHINA169-BACKBONE CHINA UNICOM China169 Backbone anonymousMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chr…Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.h…Other clients from this network requested /lawn/: see the Wall of Shame. | 116.179.32.0/24 | 8 | 8 | 2 | 2 | 8 | 8 | no | 2026-10-08 18:40Z 2026-11-01 17:51Z |
AS218785 TC-DATACENTER - TC DATACENTER LIMITED anonymousMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/1…Observed: Claimed a browser version that sends Sec-Fetch-* headers to HTTPS sites, but never sent them. (1 of 1 client)Observed: Claimed a browser user agent, but its TLS handshake had the same JA4 fingerprint as clients identifying as go-http-client. (1 of 1 client)+5 more observed traits in well-behaved.jsonOther clients from this network requested /lawn/: see the Wall of Shame. | 45.138.12.0/24 | 1 | 4 | 0 | 666 | 739 | 743 | yes | 2026-09-28 10:41Z 2026-11-01 09:05Z |
AS24940 HETZNER-AS anonymousMozilla/5.0 (compatible; QlyzeBot/1.0; +https://app.qlyze.io/bot)serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/; abus…+8 more user agentsObserved: Claimed a browser user agent, but its TLS handshake had the same JA4 fingerprint as clients identifying as curl. (3 of 13 clients)Observed: Claimed a browser version that sends Sec-Fetch-* headers to HTTPS sites, but never sent them. (2 of 13 clients)+5 more observed traits in well-behaved.jsonOther clients from this network requested /lawn/: see the Wall of Shame. | 46.225.147.0/24+9 more | 13 | 21 | 0 | 12 | 244 | 263 | yes | 2026-09-28 20:11Z 2026-11-01 05:31Z |
AS137976 SIAM-AS-AP MD Firozur Rahman anonymousMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko…Other clients from this network requested /lawn/: see the Wall of Shame. | 103.119.23.0/24 | 1 | 1 | 0 | 1 | 1 | 1 | no | 2026-11-01 02:30Z 2026-11-01 02:30Z |
AS23033 WOW anonymousMozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help…Observed: Sent no Accept-Language header. Mainstream browsers always send one. (1 of 1 client)Observed: Came from a network that X4BNet's public list classifies as datacenter or hosting. (1 of 1 client) | 216.244.66.0/24 | 1 | 7 | 0 | 6 | 9 | 14 | yes | 2026-09-27 05:09Z 2026-10-31 23:39Z |
AS51167 CONTABO anonymousMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko…+1 more user agentsObserved: Claimed a browser version that sends Sec-Fetch-* headers to HTTPS sites, but never sent them. (2 of 3 clients)Observed: Claimed a browser user agent, but its TLS handshake had the same JA4 fingerprint as clients identifying as curl. (2 of 3 clients)+3 more observed traits in well-behaved.jsonOther clients from this network requested /lawn/: see the Wall of Shame. | 169.58.252.0/24 | 3 | 3 | 0 | 953 | 956 | 956 | yes | 2026-10-08 21:46Z 2026-10-31 21:19Z |
AS207990 HR-CUSTOMER anonymousMozilla/5.0 (iPhone; CPU iPhone OS 26_3_0 like Mac OS X) AppleWebKit/605.1.15 (…Mozilla/5.0 (compatible; ElevenlabsCrawler/1.0; +https://elevenlabs.io/docs/bot…Observed: Came from a network that X4BNet's public list classifies as datacenter or hosting. (2 of 2 clients) | 150.129.92.0/24 | 2 | 2 | 0 | 2 | 2 | 5 | yes | 2026-09-27 04:44Z 2026-10-31 13:45Z |
AS12816 MWN-AS anonymousMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko…Observed: Claimed a browser version that sends Sec-Fetch-* headers to HTTPS sites, but never sent them. (2 of 2 clients)Observed: Claimed a browser user agent, but its TLS handshake did not offer HTTP/2. Mainstream browsers always offer it. (2 of 2 clients)+2 more observed traits in well-behaved.json | 138.246.253.0/24 | 2 | 9 | 0 | 2 | 9 | 9 | no | 2026-10-03 14:53Z 2026-10-31 07:28Z |
Tor onion service anonymousMozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:140.0) Gecko/20100101 Firefox/…Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Firefox/128.0Other clients from this network requested /lawn/: see the Wall of Shame. | | 3 | 3 | 0 | 1 | 2 | 8 | yes | 2026-10-01 22:44Z 2026-10-31 07:08Z |
AS14061 DIGITALOCEAN-ASN anonymousMozilla/5.0Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko…+2 more user agentsObserved: Sent no Accept-Language header. Mainstream browsers always send one. (7 of 7 clients)Observed: Came from a network that X4BNet's public list classifies as datacenter or hosting. (7 of 7 clients) | 69.55.49.0/24+4 more | 7 | 9 | 0 | 146 | 287 | 404 | yes | 2026-10-01 13:27Z 2026-10-31 06:46Z |
AS8193 BRM-AS anonymousMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/1…Other clients from this network requested /lawn/: see the Wall of Shame. | 84.54.71.0/24 | 1 | 1 | 0 | 1 | 1 | 1 | no | 2026-10-31 01:42Z 2026-10-31 01:42Z |
AS203020 HOSTROYALE anonymousMozilla/5.0 (compatible; ElevenlabsCrawler/1.0; +https://elevenlabs.io/docs/bot…Observed: Came from a network that X4BNet's public list classifies as datacenter or hosting. (1 of 1 client)Other clients from this network requested /lawn/: see the Wall of Shame. | 31.42.115.0/24 | 1 | 1 | 0 | 21 | 21 | 21 | no | 2026-10-30 16:32Z 2026-10-30 16:33Z |
AS135377 UCLOUD-HK-AS-AP UCLOUD INFORMATION TECHNOLOGY HK LIMITED anonymousMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like…Observed: Claimed a browser version that sends Sec-Fetch-* headers to HTTPS sites, but never sent them. (14 of 14 clients)Observed: Claimed a browser user agent, but its TLS handshake did not offer HTTP/2. Mainstream browsers always offer it. (10 of 14 clients)+2 more observed traits in well-behaved.json | 101.36.105.0/24+10 more | 14 | 14 | 0 | 18 | 54 | 84 | yes | 2026-09-29 14:51Z 2026-10-30 14:29Z |
AS1267 ASN-WINDTRE IUNET anonymousMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like…Other clients from this network requested /lawn/: see the Wall of Shame. | 151.72.37.0/24 | 1 | 1 | 0 | 5 | 5 | 5 | no | 2026-10-30 13:53Z 2026-10-30 13:54Z |
AS20473 AS-VULTR anonymousMozilla/5.0 (compatible; ElevenlabsCrawler/1.0; +https://elevenlabs.io/docs/bot…Observed: Came from a network that X4BNet's public list classifies as datacenter or hosting. (2 of 2 clients)Other clients from this network requested /lawn/: see the Wall of Shame. | 152.39.175.0/24 | 2 | 2 | 0 | 8 | 8 | 8 | yes | 2026-10-30 08:03Z 2026-10-30 10:06Z |
AS7029 WINDSTREAM anonymousMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; SEOJuice-Search…Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko…Other clients from this network requested /lawn/: see the Wall of Shame. | 77.47.157.0/24 | 2 | 2 | 0 | 2 | 6 | 6 | yes | 2026-10-24 11:20Z 2026-10-30 04:58Z |
AS202425 INT-NETWORK anonymous(empty user agent)Observed: Sent no Accept-Language header. Mainstream browsers always send one. (3 of 3 clients)Observed: Came from a network that X4BNet's public list classifies as datacenter or hosting. (3 of 3 clients) | 80.82.77.0/24 | 3 | 3 | 0 | 6 | 6 | 9 | yes | 2026-09-28 22:48Z 2026-10-30 04:23Z |
AS35830 BTTGROUP-AS anonymousMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko…Observed: Claimed a browser version that sends Sec-Fetch-* headers to HTTPS sites, but never sent them. (1 of 1 client)Observed: Claimed a browser user agent, but its TLS handshake did not offer HTTP/2. Mainstream browsers always offer it. (1 of 1 client)+1 more observed traits in well-behaved.json | 130.49.79.0/24 | 1 | 1 | 0 | 2 | 2 | 2 | no | 2026-10-30 00:42Z 2026-10-30 00:42Z |
AS63199 CDSC-AS1 anonymousMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/8…Observed: Claimed a browser version that sends Sec-Fetch-* headers to HTTPS sites, but never sent them. (1 of 1 client)Observed: Claimed a browser user agent, but its TLS handshake did not offer HTTP/2. Mainstream browsers always offer it. (1 of 1 client)+3 more observed traits in well-behaved.json | 148.153.56.0/24 | 1 | 1 | 0 | 12 | 12 | 12 | yes | 2026-10-29 23:46Z 2026-10-29 23:46Z |
AS59651 ASBLG anonymousMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko…Observed: Claimed a browser version that sends Sec-Fetch-* headers to HTTPS sites, but never sent them. (4 of 6 clients)Observed: Claimed a browser user agent, but its TLS handshake did not offer HTTP/2. Mainstream browsers always offer it. (4 of 6 clients)+1 more observed traits in well-behaved.json | 130.49.10.0/24 | 6 | 6 | 0 | 3 | 9 | 9 | yes | 2026-10-25 23:45Z 2026-10-29 22:27Z |
AS12989 BLACKHOST AS12989.NET anonymous(empty user agent)Observed: Sent no Accept-Language header. Mainstream browsers always send one. (1 of 1 client)Observed: Came from a network that X4BNet's public list classifies as datacenter or hosting. (1 of 1 client) | 86.54.31.0/24 | 1 | 1 | 0 | 3 | 3 | 3 | yes | 2026-10-29 18:59Z 2026-10-29 18:59Z |
AS42708 GLESYS glesys.com anonymousMozilla/5.0 (Linux; Android 14; SM-S901B) AppleWebKit/537.36 (KHTML, like Gecko…Mozilla/5.0 (Linux; U; Android 13; sk-sk; Xiaomi 11T Pro Build/TKQ1.220829.002)…+3 more user agentsObserved: Came from a network that X4BNet's public list classifies as datacenter or hosting. (5 of 5 clients)Other clients from this network requested /lawn/: see the Wall of Shame. | 178.73.252.0/24+2 more | 5 | 5 | 0 | 1 | 4 | 5 | no | 2026-10-01 13:43Z 2026-10-28 06:38Z |
Showing 30 of 80, most recently seen first. The full list is in well-behaved.json.
What the labels mean
- verified
- User agent matched a known crawler and the source address passed that vendor's published verification. Only these rows show individual IPs.
- claimed, unverifiable
- User agent claims a known crawler whose vendor publishes no verification method we could find. Not confirmed.
- spoofed UA
- User agent claimed a known crawler, but the source address failed that vendor's published verification. Shown under the network it came from, not the name it claimed.
- anonymous
- No known crawler user agent. Attributed only to the network (ASN).
Networks other than verified crawlers are shown as a /24 (IPv4) or /48 (IPv6) at most. Windows (24h, 7d, 30d) count raw requests; "all" also includes daily summaries of older traffic. Listing here states what our logs show and nothing more.