GuthIndexBot

GuthIndexBot is the crawler of Guth, a personal assistant app. It reads a small, fixed list of public, official help pages (how to cancel, return, book, renew or report something, and who to contact) so Guth can point people to the right official page instead of guessing.

How it behaves

• User-Agent: GuthIndexBot/1.0 (+https://index.getguth.com/bot; support@getguth.com)
• It obeys robots.txt for the token GuthIndexBot (or *), re-read daily, following RFC 9309: if robots.txt can't be reached (5xx or a network error) it doesn't crawl that site; if a page answers 401 or 403 it stops and doesn't retry for a week.
• At most one request every 2 seconds per site (longer if your Crawl-delay says so), a handful of sites at a time, and it backs off on 429 and 5xx responses.
• It revisits a page about once a week, with If-None-Match / If-Modified-Since.
• GET requests only. No cookies, no logins, no forms, no pages behind a sign-in, and it honours noindex.
• It stores the page's readable text (as markdown) with its address and fetch time. It does not collect personal data.
• If your site offers llms.txt or markdown versions of pages, or serves text/markdown, it prefers those.

Opting out

Add this to your robots.txt and GuthIndexBot will stop within a day, and drop your pages from the index on its next visit:

User-agent: GuthIndexBot
Disallow: /

Or email support@getguth.com with your domain; we add it to a do-not-crawl list and remove stored pages.