ClawHarborBot

What this crawler does, how often it does it, and how to stop it.

User agent

What it fetches

ClawHarbor reads a public website twice, for two different reasons.

• Once, at onboarding. A crawl records the site's shape: colour tokens, type scale, page structure, and the CSS selectors that identify a listing. The output is a manifest describing that shape. It contains no page content — no listings, no descriptions, no images, no font files.
• Live, per request. When someone asks an assistant a question, the page that answers it is fetched at that moment, cached briefly, and discarded. Nothing is accumulated into a database.

Politeness

• robots.txt is fetched and honoured before anything else, and re-checked on every crawl.
• Requests to one origin are serialised behind a per-origin delay — never parallel, never bursty.
• Responses are cached, so repeated questions cost the origin nothing.
• If an origin fails, we serve the cached answer labelled stale rather than retrying in a loop.

No third-party origins are configured on this host right now.

How to block it

Add this to your robots.txt and the crawler stops on its next check:

Blocking by user agent or IP at your edge also works and needs no notice to us. If you would rather an interface be removed outright, use the removal form — that is faster and it is honoured without argument.