How we work
What our crawler does, and what it never does.
When someone requests a review, our crawler visits that site's public pages once. This page explains exactly what it does there, and how to block it.
What it reads
- The address it was given, plus up to nine more public pages on the same site, chosen by type: pricing, features or product, about, FAQ, contact, policies, one help or blog page, and the cart if it is public.
- One optional second address the requester provides, such as a Shopify App Store listing.
- Visible text, headings, links, meta tags, structured data, and which tracking scripts load and send a page-view request.
- Screenshots of each page at phone width (390 px) and desktop width (1440 px).
What it never does
- It never logs in, and never asks for or stores passwords for the site it reviews.
- It never submits forms, adds items to a cart, starts a checkout or places an order.
- It never visits private or internal network addresses.
- It never reads more than ten pages of a site per review, and stops at a fixed time limit.
- It never forwards the tracking requests your pages make: they are answered locally, so a review doesn't add visits to your analytics.
How it identifies itself
Every request carries this user agent:WebQABot/1.0 (+https://webqualityassurance.com/crawler)
It follows your robots.txt. To block it, add:
User-agent: WebQABot Disallow: /
How page content is treated
Text from crawled pages is treated as material to review, never as instructions. A page cannot change what the review checks or how it is scored.
Where the page content goes
The page text and screenshots are stored with the report so the requester can refer back to them, and are sent to Anthropic's AI model to write the review. They are deleted when the requester deletes the report or their account.
Questions or concerns
If our crawler visited your site and you have a question, write towebqualityassurance.mail@gmail.com.