← Wall of Shame · Well-behaved crawlers

Meta

claimed, unverifiable

User agent claims to be Meta. We found no published way to verify that claim, so it is not confirmed; the networks the requests came from are listed below.

Networks seen: AS32934 FACEBOOK.

Totals by window

WindowHours heldPagesMax depthBytesSessionsRead the rulesFirst seenLast seen
24h773.931044517328.5 MB678302026-11-01 10:03Z2026-11-02 10:03Z
7d6226.0588527472.3 GB4416902026-10-26 11:40Z2026-11-02 10:03Z
30d8249.93116441473.0 GB6217002026-10-04 11:05Z2026-11-02 10:03Z
all8249.93116441473.0 GB6217002026-10-04 11:05Z2026-11-02 10:03Z

"Read the rules" counts sessions that fetched robots.txt before their first disallowed page.

Observed traits

What these clients sent and how they walked the maze, from the raw logs we keep (the last 90 days). Each line is an observation, not a verdict on who or what a client is (how they are measured). A client is one address with one user agent; 1386 clients here.

ObservedClients
Sent no Accept-Language header. Mainstream browsers always send one.119

Fastest pace: 13 disallowed pages within one minute from one client.

TLS fingerprints (JA4), most used first: t13d3111h2_e8f1e7e78f70_5ac7197df9d2 (1386).

Daily

Pages per day, last 90 days (peak 160431).

Day (UTC)PagesHours heldBytesSessions
2026-11-0236017284.70141.4 MB2673
2026-11-01140775988.79296.9 MB7057
2026-10-3196128685.29287.7 MB5636
2026-10-301476971054.83479.5 MB7098
2026-10-291604311103.81463.6 MB7404
2026-10-2893990650.44242.8 MB4318
2026-10-27141047970.54281.2 MB6486
2026-10-2670302502.19130.9 MB3715
2026-10-259907.753.9 MB201
2026-10-24210122.248.2 MB377
2026-10-2337066270.0376.7 MB2430
2026-10-2278927546.08210.4 MB4141
2026-10-2174243515.92144.4 MB3478
2026-10-2067079465.07156.1 MB3059
2026-10-18522449.7723.7 MB997
2026-10-1711205117.0952.2 MB2344
2026-10-14108012.105.1 MB645
2026-10-13150.6260.1 kB14
2026-10-12170.1369.5 kB17
2026-10-10742.33355.3 kB74
2026-10-0460.2121.5 kB6

Top user agents

User agentPages
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webm…674778
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharin…347115
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/cra…75649
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/doc…47174
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36 Edg/144.0.0.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/doc…7779
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:147.0) Gecko/20100101 Firefox/147.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))2758
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:147.0) Gecko/20100101 Firefox/147.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))1636
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36 Edg/146.0.0.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/doc…1596
Mozilla/5.0 (X11; Linux x86_64; rv:147.0) Gecko/20100101 Firefox/147.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))1534
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.c…1492
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36 Edg/143.0.0.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/doc…962
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:147.0) Gecko/20100101 Firefox/147.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))604
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36 Edg/142.0.0.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/doc…354
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.7632.6 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/w…332
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))178
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36 Edg/141.0.0.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/doc…172
Mozilla/5.0 (X11; Linux x86_64; rv:146.0) Gecko/20100101 Firefox/146.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))152
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36 Edg/144.0.0.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.c…149

Networks

Shown as /24 (IPv4) or /48 (IPv6) networks. Individual addresses are only ever published for verified crawlers.

57.141.20.0/24 2a03:2880:f804::/48 2a03:2880:f814::/48

Sample paths

Recent disallowed URLs fetched, most recent first.

Generated 2026-11-02 10:03 UTC.