7fbf9ec9e6
Crawl-Delay does have a big impact on the loadavg of the server, meaning: - (1) most requests are from crawlers and, - (2) most crawlers listen to Crawl-Delay. The prod server can handle the current loadavg just fine, let's let them up their game. Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
3 lines
38 B
Text
3 lines
38 B
Text
User-Agent: *
|
|
Allow: /
|
|
Crawl-Delay: 1
|