Our research crawler
ThemeSmithResearchBot.
If you've seen it in your website's logs: here is what it is, what it reads, how fast it goes, and how to stop it.
1. What it is.
ThemeSmithResearchBot is the crawler ThemeSmith uses to read the public pages of UK business websites. We use what it reads to publish research about small business websites, such as how many show their prices or the details their regulator asks for, and to improve our own service. It identifies itself with this user agent:
ThemeSmithResearchBot/1.0 (+https://themesmith.ai/bot)
2. What it reads.
Only public pages: the ones anyone could open in a browser. It doesn't log in, fill in or submit forms, or place orders. It reads your robots.txt first and follows it. We also read pages from the Common Crawl archive, a free public copy of the web.
3. How fast it goes.
At most one request a second to any one website, and it gives up on a page after 20 seconds. We don't publish a fixed list of IP addresses; recognise it by the user agent above.
4. How to stop it.
Add these two lines to your website's robots.txt:
User-agent: ThemeSmithResearchBotDisallow: /
A rule for all crawlers (User-agent: *) works too. Or email privacy@themesmith.ai with your web address: we'll delete the pages we hold from your site and stop reading it.
5. What we do with what it reads.
We publish research as totals, such as “42.5% of law firms”. We never publish a business's name or details, never sell or share them, and don't use what it reads to send marketing to a sole trader or partnership. We keep copies of pages for up to 36 months.
Where a website belongs to a sole trader or partnership, its business details are personal data, and we read them on the basis of our legitimate interest in market research. The full wording is in § 2.10 of our privacy notice.