How our crawler works.
Passiro Crawler scans websites for cookies, trackers, and consent compliance. It's respectful, transparent, and easy to control.
Identification
How to identify our crawler
The Passiro Crawler identifies itself in every request with the following User-Agent string:
You can identify our crawler by looking for PassiroCrawler in the User-Agent header.
What it does
Built to scan, not to scrape.
Cookie & tracker scanning
We load your pages in a headless browser and record every cookie and third-party request they trigger, so we can report exactly which trackers run before and after consent.
Screenshots
We take screenshots of scanned pages to create visual reports that show how your cookie banner and consent flow appear to visitors. Screenshots are stored securely and only shown to the site owner or in public reviews.
Link discovery
We follow internal links to discover pages on your site. We only follow links within the same domain and respect a maximum page limit. We also check your sitemap.xml if available.
Detection
What the consent check detects
The consent check loads each page in a real browser and records what happens before any consent is given. Everything reported on a review page is observed behaviour, not an assumption.
Consent platform and configuration
Which consent management platform runs on the site. For Cookiebot setups we also read the account ID and whether automatic script blocking is enabled.
Google Consent Mode v2
Whether consent defaults are declared before Google tags load, and whether all storage types default to denied.
IAB TCF v2
Whether the site exposes the Transparency and Consent Framework API used for programmatic advertising.
Cookies set before consent
Every cookie set before any interaction with the consent banner, classified against our database of known cookies.
Tracker requests
Third-party requests matched against more than 2,700 known tracker signatures, including whether each request set a cookie.
Cookie declaration drift
For Cookiebot sites, we compare the published cookie declaration with the cookies we actually observed on the site.
Consent platforms we recognise
Sites using a platform we do not recognise, or no platform at all, are reported as having no consent management platform detected.
Crawl behavior
Respectful by design.
Rate limited
Maximum 2 requests per second. We never flood your server.
Template-aware crawling
Our crawler intelligently detects page templates. On e-commerce sites with thousands of product pages sharing the same layout, we automatically identify the template and move on to other unique page types. This means full coverage with minimal requests.
No data extraction
We don't scrape or store your content, prices, or user data. We take screenshots for visual reports and analyze cookies, scripts, and consent behavior for compliance, nothing else.
Timeout protection
Each page has a 10-second timeout, and total crawl time is capped at 5 minutes. We never hang on your server.
Same-domain only
We only follow links within the domain being scanned. We never visit external websites during a scan.
Smart scanning
Sees your site the way users do.
To produce accurate scan results and clean screenshots, our crawler automatically handles the popups and overlays that appear on modern websites.
Cookie consent banners: automatically accepted across all major consent platforms (Cookiebot, OneTrust, Usercentrics, Didomi, and more) in all 24 EU languages.
Popups and modals: newsletter signups, promotional overlays, and other interrupting elements are detected and dismissed.
Backdrop overlays: dimming layers and full-screen backdrops are removed so screenshots show the actual page content.
Shadow DOM: we traverse shadow roots to find and dismiss consent elements that are hidden inside web components.
Customization
Fine-tune the crawler for your site.
Our automatic detection handles most websites, but some sites have custom popups, unique consent flows, or elements that need special treatment. Site owners can configure custom crawler rules in the site settings dashboard to handle these edge cases.
Click selectors
Specify CSS selectors of buttons or elements the crawler should click before scanning, for example, to dismiss a custom popup our automatic detection doesn't recognize.
Hide selectors
Force-hide specific elements before taking screenshots. Useful for sticky banners, chat widgets, or overlays that shouldn't appear in your compliance report.
Pre-set cookies
Set cookies before the crawler loads your pages, for example, a consent cookie to bypass the cookie banner entirely, or an A/B test cookie to scan a specific variant.
Ignored paths & page limits
Exclude specific URL paths from scanning and set a custom page limit. Useful if parts of your site are behind login or contain content you don't need scanned.
These settings are available in your site's Settings tab on the Passiro dashboard. Changes take effect on the next scan.
Your control
How to block our crawler
If you don't want the Passiro Crawler to visit your website, you can block it using your robots.txt file:
You can also block specific paths or sections of your site. We respect all standard robots.txt directives.
Alternatively, you can contact us and we'll add your domain to our exclusion list.
Questions about our crawler?
If you have questions, concerns, or need to report an issue with the Passiro Crawler, we're happy to help.
Contact us