elixir/static/robots.txt
Théo Lebrun ad558b04c8 static: robots.txt: change Crawl-Delay from 1s to 2s
Recently crawlers have risen back up to high levels.
Let's ask them nicely to calm down.

Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
2025-01-14 10:49:51 +01:00

3 lines
38 B
Text

User-Agent: *
Allow: /
Crawl-Delay: 2