Official crawler
CitewareBot
CitewareBot is operated by Citeware to fetch public product websites. A listing is drafted only after the live page opens. If this crawler is blocked, that listing cannot be created or refreshed.
Identification
- User-agent
- CitewareBot/1.0 (+https://citeware.io/bot)
- Operator
- Citeware (citeware.io)
- Purpose
- Public product pages for catalog listings
Scope
Requests
Public HTML, Open Graph images, linked pricing pages, and llms.txt when published.
Does not
Sign in, submit forms, complete challenges, or access gated or paid content.
Allow this crawler
robots.txt
Permit CitewareBot on public marketing pages.
User-agent: CitewareBot Allow: /
Cloudflare
Skip Bot Fight Mode, Super Bot Fight Mode, and WAF challenges for CitewareBot so the public HTML is returned instead of a verification page.