For webmasters
Scrinly automation transparency
If this page appeared in your server logs, this explains the Scrinly-operated automation that linked here, the rules it follows, and how to stop it.
How to recognise it
Restricted, operator-run multi-page traversal carries this User-Agent. It is not a customer-facing API feature:
Mozilla/5.0 (compatible; Scrinly/3.0; +https://scrinly.com/bot)
We do not disguise this automation as a browser, and we do not rotate its identity. Scrinly has no proxy support of any kind — the API rejects proxy parameters outright and the platform it runs on provides no upstream proxy — so requests always arrive from Cloudflare's network as themselves. There is no mechanism here for evading a block.
Screenshot captures are different, and deliberately so
The public Scrinly product captures a specific publicly accessible or authorised page supplied by a customer. That request uses an ordinary browser user-agent because its purpose is to render the page as a visitor would see it, not to traverse the site.
Multi-page traversal—following links across pages without a separate
customer-supplied URL for each page—is restricted to Scrinly-operated internal
workflows. That activity carries the Scrinly identity and follows
robots.txt. If you see this user-agent, the block below stops it.
What it obeys
-
robots.txt, includingCrawl-delay. Fetched per origin before any page on it is requested, and honoured for theScrinlyuser-agent and for*. - One request per second per host, two at a time. Enforced centrally, so all Scrinly-operated traversal against the same host shares one budget.
- No media. Images, fonts, video and other heavy subresources are blocked before they leave the browser, reducing the bandwidth used by automated traversal.
robots.txt handling only for a domain
verified for the internal workflow, by requiring a token at
/.well-known/scrinly-verify.txt on that domain. The override
never applies to any other host — if traversal follows a link off the verified
domain, robots.txt applies again immediately. If the verification
file is absent, your robots.txt is obeyed.
How to block it
Add this to your robots.txt and we will stop:
User-agent: Scrinly Disallow: /
To slow it down rather than stop it:
User-agent: Scrinly Crawl-delay: 10
Changes take effect within 24 hours, which is how long a fetched
robots.txt is cached. Blocking at your edge or firewall works
too, and we will not attempt to work around it.
Public screenshot traffic
Scrinly's customer-facing API captures specific pages for screenshots, visual regions, comparisons, and monitoring. These are browser renders rather than crawler requests and therefore do not use the crawler identity above. Restricted automated traversal remains separately identifiable and governed by the controls on this page.
Contact
If Scrinly is causing you a problem — load, cost, anything — email hello@scrinly.com with a hostname and a rough timestamp and we will look into it. If you want a domain blocked at our end rather than yours, say so and we will add it.