Safe by default
Blocks all bots unless you say otherwise, so non-production environments are not indexed by accident.
Open source Umbraco package
A config-driven robots.txt for Umbraco
Crumpled.RobotsTxt serves a robots.txt built from configuration. It blocks every bot by default, so staging, development and preview environments are not indexed by accident, and it lets you set different rules for each site and environment.
Safe by default
With no sites configured, every bot is blocked, so a forgotten staging site or an overlooked *.umbraco.io environment domain never ends up in a search engine. On an Umbraco Cloud live environment the default is to allow all bots.
Once you configure sites, any domain you have not listed is still blocked, unless you name a default ruleset to use instead.
What it does
Everything is configured in appsettings.json, with no code to write.
Multi-site and environment-aware
Different robots.txt rules for different domains and environments, such as production, staging and development.
Reusable rulesets
Define Allow and Disallow patterns for different user agents once, then reuse them across sites.
Content Signals
Control AI training, search and AI input use of your content with Content-Signal directives (version 3.1.0 and later).
Sitemap URLs
Include a sitemap URL for each site, generated for you.
Hot reload
Configuration changes are picked up without restarting the application.
Get started in two steps
-
Install
Add the package with dotnet add package Crumpled.RobotsTxt. It registers itself through an Umbraco composer, so there is no code to write.
-
Configure
Add your rulesets and sites under Crumpled:RobotsTxt in appsettings.json. Changes are picked up without a restart.
Need help?
Need help installing Crumpled RobotsTxt to your Umbraco installation?
Tell us about your Umbraco site and we will reply within one working day.
- Supported Umbraco versions
- 13 to 17
- License
- MIT