Get off my lawn.

What is this?

A tarpit. Everything under /lawn/ is a maze of generated pages. Each page is stitched together from a word-level Markov chain trained on a few public-domain novels, links to between ten and twenty deeper pages, and is sent at roughly sixteen bytes per second. There is no bottom. A crawler that follows the links can spend minutes per page and never run out.

Well-behaved crawlers never see any of it. Search engines and AI crawlers that honour robots.txt are never trapped, never logged as violators, and never shamed. If you are a person who clicked something odd, nothing bad happened: go have a sandwich.

Methodology

What counts as a violation

A request for any path under /lawn/. That prefix is disallowed for every user agent in the robots.txt shown at the bottom of this page, and has been since the site launched. Nothing else counts.

How visitors reach the lawn

Only through links a human would never follow: entries in /sitemap.xml and links on this page that are hidden from view and from screen readers and marked rel="nofollow". Every one of them points into the disallowed prefix.

What is logged

Every request to every route: time, client IP, origin ASN and country (from the public iptoasn.com dataset), user-agent string, how the request arrived (HTTPS or plain HTTP and the HTTP version, Tor, Gopher or Gemini), path (with query parameter names but never their values; for Gopher searches and Gemini queries, only that one was sent), maze depth, bytes sent, how long the connection was held and how it ended, the names of the request headers sent plus the Referer (likewise without query values), Accept and Accept-Language/-Encoding values, the HTTP and TLS versions used, the JA4 fingerprint of the TLS handshake, and the reverse-DNS name of the client address. Raw records are kept for 90 days, then reduced to per-day counts. Only what is described under "What is published" ever appears on this site.

How crawlers are identified

User-agent strings are matched against a list of known crawlers. Where the vendor publishes an official verification method, we use it: reverse DNS with a forward-confirming lookup, or the vendor's published IP ranges. The result is one of four labels:

• verified: the user agent claims a crawler and the vendor's own verification method confirms it.
• spoofed UA: the user agent claims a crawler but the vendor's verification method says the request did not come from them. We show who it really came from (by ASN), not who it claimed to be.
• claimed, unverifiable: the user agent claims a crawler whose vendor publishes no verification method. We show the claim, and we do not treat it as confirmed.
• anonymous: no known crawler claimed. Attributed only to the network (ASN) it came from.

Time held

For each request, the time between the first byte of the request and the end of our response (or the client hanging up). Sessions are runs of requests from the same IP and user agent with gaps of less than ten minutes.

"Read the rules, ignored them"

Within one session, the client fetched /robots.txt before its first request into /lawn/.

What is published

Only facts: organisation or network, user-agent string, number of disallowed fetches, time held, and the observed traits below; for well-behaved crawlers, how often they read robots.txt and visited. A crawler is listed as well-behaved only if it fetched robots.txt and has never requested anything under /lawn/. We make no claims about anyone's intent. Individual IP addresses are published only for verified crawlers, which run from the vendor's own published infrastructure. Everything else is shown at most as a /24 (IPv4) or /48 (IPv6) network plus its ASN, so no individual residential address is ever published.

Observed traits

Each wall page lists, from the raw records of the last 90 days, how many of its clients (one address with one user agent) showed each of these traits. They describe what was sent and how the maze was walked. They are not a verdict on who or what a client is, and none is published per address.

• A browser user agent whose TLS handshake has the same JA4 fingerprint as clients that name an HTTP library (python-requests, curl, Go, …) in their user agent. JA4 summarises the handshake: protocol and TLS versions, offered ciphers, extensions, signature algorithms and the first application protocol offered. Different TLS software produces different fingerprints, whatever user agent it sends.
• A browser user agent whose TLS handshake does not offer HTTP/2, or that uses HTTP/1.1 over HTTPS although HTTP/2 is offered. Mainstream browsers offer and use HTTP/2.
• A browser version that sends Sec-Fetch-* headers to HTTPS sites (Chrome 76, Edge 79, Firefox 90, Safari 16.4 and later) that never sent them; no Accept-Language header at all; HEAD requests from a browser user agent; ten or more pages over HTTPS from a browser user agent without ever requesting /favicon.ico (no page here names an icon, so browsers ask for that one).
• Thirty or more maze pages within one minute; maze pages at evenly spaced intervals (twenty or more gaps varying by less than a quarter of their average); most followed maze links fetched while the page they came from was still being served; maze links followed ten or more levels deep; mostly error responses.
• A network that X4BNet's public list classifies as datacenter or hosting. People use VPNs on such networks too, so this is context, not evidence on its own.

Each page also shows the fastest pace one client reached and the most common JA4 fingerprints. The well-behaved page shows each row's strongest traits too (all of them are in well-behaved.json): reading robots.txt and staying off the lawn is listed whatever else a client did.

Over Tor

Visitors of the onion mirror have no IP address. Tor gives each connection a circuit number, which is logged in its place and never published. Their requests appear on the wall as one network, "Tor onion service". Crawler claims made over Tor cannot be verified, so a known crawler's user agent is shown as claimed, unverifiable.

Machine-readable data

/shame/feed.json (all offenders), /shame/well-behaved.json (crawlers that followed the rules) and /shame/blocklist.txt (CIDRs of verified crawlers plus spoofers above a violation threshold). Formats and terms: Using this data.

Corrections

If you run a crawler listed here and believe the classification is wrong, publish an official verification method (reverse DNS or an IP range list) and it will be used.

Using this data

The data behind the wall is published for anyone to use: in a blocklist, in research, or to keep an eye on the crawler you run. It is licensed under Creative Commons Attribution 4.0 (CC BY 4.0): use, share and adapt it for any purpose, as long as you credit it. A line like this is enough, wherever you show or redistribute the data: Data: getoffmyfuckinglawn.com (CC BY 4.0), linked to https://getoffmyfuckinglawn.com/#data. Say if you changed it.

/shame/feed.json

A JSON array, one object per offender: per identity status, claimed organisation and network (ASN). Ordered by time held, longest first. Fields:

• org: the organisation the user agent claims; for anonymous entries, the network's label ("AS64500 EXAMPLE-NET").
• status: verified, user_triggered (verified, and the vendor says robots.txt may not apply), spoofed, unverifiable or anonymous, as defined under Methodology.
• asn (number, or null when unknown or over Tor) and asn_org.
• cidrs: addresses, under the rule in "What is published": individual addresses (/32, /128) only for verified entries, otherwise /24 or /48 networks.
• hours_held, pages (requests under /lawn/) and max_depth, all time.
• first_seen and last_seen: RFC 3339 times in UTC.

/shame/well-behaved.json

A JSON array of clients that fetched robots.txt and never requested /lawn/, most recently seen first. Same org, status, asn, asn_org, cidrs, first_seen and last_seen as above (spoofed entries are named after their network), plus claimed_org, robots_fetches, requests (any path), saw_bait (fetched a page carrying hidden links into the maze), trait_clients, and traits: [{"trait": …, "clients": …}], the observed traits and how many of the entry's clients showed each.

/shame/blocklist.txt

One CIDR per line, after # comment lines giving the generation time and the rules: addresses of verified crawlers that fetched disallowed paths, and the networks of spoofed crawler user agents above a violation threshold. Anonymous and unverifiable clients are never listed, so this is not a list of every scraper; it is a list of the ones we can name.

Freshness and fetching

The files are rebuilt every few minutes (observed traits at most hourly) and served with Last-Modified, so conditional requests (If-Modified-Since) are answered with 304. Fetching more often than every five minutes gains nothing. robots.txt allows everything outside /lawn/, these files included.

Stability

Fields are only ever added, never renamed, removed or given a new meaning; ignore fields you don't know. A change that breaks that would go to a new file name, with the old file kept for at least 90 days and the change announced here. Timestamps and strings that are unknown are empty (""), and lists are never null.

What it is not

A record of what our logs show for this one site, with nothing added. "Verified" means the vendor's own verification method passed; nothing here is a claim about anyone's intent.

For crawler operators

The fix is one line of code you already wrote: read robots.txt, then do what it says.