3 commits

Author	SHA1	Message	Date
Théo Lebrun	ad558b04c8	static: robots.txt: change Crawl-Delay from 1s to 2s ... Recently crawlers have risen back up to high levels. Let's ask them nicely to calm down. Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>	2025-01-14 10:49:51 +01:00
Théo Lebrun	7fbf9ec9e6	static: robots.txt: change Crawl-Delay from 5s to 1s ... Crawl-Delay does have a big impact on the loadavg of the server, meaning: - (1) most requests are from crawlers and, - (2) most crawlers listen to Crawl-Delay. The prod server can handle the current loadavg just fine, let's let them up their game. Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>	2025-01-09 10:48:14 +01:00
Théo Lebrun	e25b3d7c4b	static: add /robots.txt ... This is NOT being served by Elixir backend. The HTTP server in front must serve it. Commit robots.txt to store the file somewhere. Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>	2024-09-27 18:19:11 +02:00