The question is not what crawl rules each tenant needs. The question is how to implement per-tenant crawl control when robots.txt is fundamentally a single file per hostname that applies to all URLs uniformly. Multi-tenant SaaS platforms face a structural constraint: if tenants share a domain (app.example.com/tenant-a/, app.example.com/tenant-b/), one robots.txt governs all of them. If tenants have subdomains (tenant-a.example.com), each gets its own robots.txt but the platform must generate and maintain thousands of them. Both approaches have failure modes that affect tenant SEO performance.
Subdomain-Based Tenant Isolation for Independent Crawl Control
Assigning each tenant a subdomain creates the cleanest crawl control separation. Each subdomain operates as an independent host from robots.txt perspective, per RFC 9309. A request to tenant-a.example.com/robots.txt returns tenant A’s directives. A request to tenant-b.example.com/robots.txt returns tenant B’s. No cross-contamination.
The implementation requires dynamic robots.txt generation at the application layer. When Googlebot requests /robots.txt for a given subdomain, the server identifies the tenant from the hostname and returns the appropriate directives. The generation should be template-based: a default set of rules that apply to all tenants (blocking admin paths, checkout flows, internal search) combined with tenant-specific overrides.
Shopify’s architecture demonstrates this model at scale. Each Shopify store operates on its own domain or subdomain, with a default robots.txt that blocks common low-value paths (/admin, /cart, /checkout, /search, /collections/*+*). Shopify introduced customization through the robots.txt.liquid template, allowing store owners to modify rules for their specific domain. For multi-market configurations using Shopify Markets, the request.host object enables different rules per domain or subdomain.
The caching strategy for dynamically generated robots.txt must account for two factors. First, the generated response should include appropriate Cache-Control headers to help Google’s robots.txt cache management. Second, the generation must be fast (under 100ms response time) because every Googlebot crawl session begins with a robots.txt fetch, and a slow robots.txt response delays the entire session.
Path-Based Multi-Tenancy and Shared robots.txt Constraints
The operational burden is the tradeoff. At 1,000+ tenants, the platform maintains 1,000+ robots.txt configurations. Each must be tested, each can break independently, and each introduces a potential failure point. A deployment error that affects the robots.txt generation function blocks crawling for every tenant simultaneously.
When tenants share a hostname and are differentiated by URL path (app.example.com/tenant-a/products/, app.example.com/tenant-b/products/), a single robots.txt must serve all tenants. This creates a fundamental constraint: every rule applies to every tenant.
The pattern architecture must be designed so that universal rules apply correctly across all tenants without creating conflicts. This requires standardized URL structures where tenant-specific segments appear in predictable positions.
Effective pattern design uses a tiered approach:
# Universal rules for all tenants
User-agent: *
Disallow: /*/admin/
Disallow: /*/checkout/
Disallow: /*/internal-search/
Disallow: /*/api/
# Allow important content paths for all tenants
Allow: /*/products/
Allow: /*/categories/
Allow: /*/blog/The wildcard patterns in this approach work because the URL structure enforces consistency: every tenant’s admin path follows the same pattern, every tenant’s product path follows the same pattern. When one tenant needs a custom rule that conflicts with another tenant’s needs, the path-based architecture cannot accommodate it in robots.txt.
The critical limitation: tenant-specific crawl preferences cannot be expressed in a shared robots.txt. If Tenant A wants to block /tenant-a/old-products/ while Tenant B wants to allow /tenant-b/old-products/, the path-based approach fails. The rule either applies to both or neither. This constraint must be communicated during tenant onboarding and documented as a platform limitation.
The meta robots and X-Robots-Tag fallback for per-page tenant control
Where robots.txt cannot provide tenant-specific granularity, per-page directives fill the gap. The <meta name="robots"> tag and X-Robots-Tag HTTP header operate at the individual URL level, independent of robots.txt. These directives can be customized per tenant without affecting other tenants.
The precedence relationship between robots.txt and meta robots/X-Robots-Tag is important. Robots.txt blocking prevents Googlebot from fetching the page, which means any per-page directives are invisible. For the per-page approach to work, the URLs must be crawlable (not blocked in robots.txt) so that Googlebot can read the meta robots or X-Robots-Tag directive.
The recommended architecture combines both levels: robots.txt for broad path-level control that applies universally (blocking admin, checkout, and internal tool paths), and per-page meta robots for tenant-specific crawl and indexing decisions. This gives each tenant granular control over their content’s indexation without requiring robots.txt modifications.
The implementation uses the platform’s page-rendering pipeline. When generating HTML for a tenant’s page, the application inserts the appropriate meta robots directives based on the tenant’s configuration. For non-HTML resources (PDFs, images, API responses), the X-Robots-Tag header provides the same control at the HTTP level.
This approach scales better than robots.txt customization because it operates within the application layer, where per-tenant logic already exists. Adding a tenant-specific meta tag to a page template is a standard application concern; managing thousands of robots.txt files is an infrastructure concern.
Tenant onboarding workflow must include crawl configuration as a required step
Many multi-tenant platforms treat crawl configuration as an afterthought, discovered only when a tenant reports indexing problems. By that point, Google may have already indexed content the tenant wanted blocked, or blocked content the tenant wanted indexed.
The onboarding checklist should include:
Default crawl rules review. Present the platform’s default robots.txt rules and explain what they block and allow. Tenants must understand the baseline before customizing.
Content classification. Tenants should identify which content sections should be indexable (product pages, blog, public pages) and which should not (user dashboards, admin interfaces, draft content, internal tools). This classification maps directly to crawl and indexing directives.
Meta robots defaults. Set default meta robots directives for each content type. New pages in tenant admin sections default to noindex. New product pages default to index,follow. These defaults can be overridden per page but provide a safe starting position.
Sitemap configuration. Each tenant needs a sitemap containing only their indexable URLs. For subdomain architectures, this is straightforward. For path-based architectures, the platform must generate per-tenant sitemaps that include only that tenant’s URLs within the shared domain.
Verification. After onboarding, verify that Googlebot can access intended pages and is blocked from unintended pages. The URL Inspection tool in Search Console provides per-URL verification. For subdomain tenants, each subdomain should be added as a Search Console property.
Templated Generation and Version Control for Tenant Overrides
At scale, configuration drift is the primary risk. Individual tenant files diverge from intended patterns through manual edits, migration errors, platform updates that do not propagate correctly, or edge cases in the generation logic.
Templated generation. All robots.txt files should be generated from templates, not edited directly. Tenant customizations are stored as configuration parameters that feed into the template engine. Direct file editing is not exposed to tenants or platform operators.
Version control for tenant overrides. Every tenant customization is tracked in version control with the tenant identifier, the date of change, and the reason. This creates an audit trail for diagnosing when and why a tenant’s crawl behavior changed.
Automated Compliance Monitoring and Update Propagation at Scale
Automated compliance checking. A scheduled process fetches each tenant’s robots.txt (via HTTP, exactly as Googlebot would), parses the response, and compares it against expected rules. Deviations trigger alerts. This catches generation failures, CDN caching issues, and configuration corruption before they affect crawl behavior.
Platform update propagation. When the platform updates default robots.txt rules (adding a new blocked path, changing a wildcard pattern), the update must propagate to all tenant configurations. The propagation should be staged: update a subset of tenants, verify no crawl anomalies, then roll out to the full tenant base.
Monitoring per-tenant crawl health. For subdomain architectures, each tenant’s Search Console property provides crawl stats. For path-based architectures, server log analysis filtered by tenant URL path provides equivalent data. Alert on sudden crawl volume changes per tenant, which may indicate a robots.txt generation failure.
Does a tenant’s robots.txt misconfiguration on a shared subdomain affect crawling of other tenants?
On subdomain-based architectures, each tenant’s subdomain has an independent robots.txt file, so a misconfiguration affects only that tenant’s crawl behavior. On path-based architectures sharing a single domain, one robots.txt file governs all tenants. A broad disallow rule intended for one tenant’s path could accidentally match another tenant’s URL patterns if the rules use wildcards carelessly. Path-based platforms must validate rule specificity during generation to prevent cross-tenant crawl blocking.
Does migrating a tenant from a path-based structure to a dedicated subdomain require updating robots.txt on both the old and new locations?
The migration requires a robots.txt file on the new subdomain to establish crawl rules for the tenant’s content at its new location. The shared robots.txt on the original domain should have its tenant-specific rules removed to prevent unnecessary parsing overhead. During the migration period, 301 redirects from old paths to new subdomain URLs handle the transition, and Googlebot fetches the new subdomain’s robots.txt independently on its first crawl attempt.
Does Google respect robots.txt rules that use non-standard directives like Crawl-delay or Request-rate?
Google does not support the Crawl-delay or Request-rate directives in robots.txt. These non-standard directives are recognized by some other crawlers (Bing supports Crawl-delay), but Googlebot ignores them entirely. Google manages its own crawl rate through the dynamic throttling mechanism based on server response time. Including these directives in a multi-tenant robots.txt file does no harm but provides no Google-specific benefit.
Sources
- How Google Interprets the robots.txt Specification — Google’s documentation on robots.txt path matching, host-level scope, and rule parsing
- Shopify: Customize robots.txt — Shopify’s developer documentation on dynamic robots.txt generation and multi-domain customization
- Shopify: Customize robots.txt Rules by Domain — Shopify’s per-domain robots.txt customization for multi-market stores
- Robots.txt Introduction and Guide — Google’s foundational robots.txt documentation covering syntax, scope, and best practices
