How to Dynamically Generate robots.txt Files Using PHP

To dynamically generate robots.txt files using PHP, you create a PHP script (e.g., robots.php) that outputs the robots.txt content as plain text and configure your web server to serve this script whenever robots.txt is requested. This allows you to customize the robots.txt content based on conditions like domain, environment, or user-agent.

Here is a step-by-step approach:

1. Create a PHP script (robots.php):

<?php
header('Content-Type: text/plain');

// Example: Customize based on domain
if ($_SERVER['HTTP_HOST'] == 'test.example.com') {
    echo "User-agent: *\nDisallow: /"; // Block all on test site
} else {
    echo "User-agent: *\nDisallow: /private/\nAllow: /public/\nSitemap: https://example.com/sitemap.xml";
}
?>

2. Configure your web server to rewrite requests for robots.txt to robots.php:

• For Apache, add this rule to your .htaccess file:

RewriteEngine On
RewriteRule ^robots\.txt$ /robots.php [L]

This makes requests to robots.txt serve the dynamic PHP script instead of a static file.

3. Benefits:

• You can generate different robots.txt content dynamically based on domain, environment (test/live), or other logic.
• Avoids manual updates and errors from uploading wrong robots.txt files.
• Allows conditional blocking or allowing of user-agents or paths.

4. Additional tips:

• Always set the content type header to text/plain to ensure proper interpretation by crawlers.
• You can read from a template file and modify it dynamically if preferred.
• Keep the output size reasonable (under 500 KB) for crawler compatibility.

This method is widely used to manage robots.txt dynamically for multi-domain or multi-environment websites using PHP and Apache rewrite rules.