How can I ensure robots.txt
is served as text/plain
[1] rather than text/html
?
Interesting question. It’s true that zola serve
should probably get this right (it does send text/xml
correctly for feeds), but ultimately web crawlers aren’t going to be looking at a zola serve
d dev site. For the real site, the webserver will determine the mimetype, and it’ll go off file extension unless specifically configured otherwise.
@mrec I don’t think Zola does get this right. If I make an empty site:
$ zola init
Welcome to Zola!
Please answer a few questions to get started quickly.
Any choices made can be changed by modifying the `config.toml` file later.
> What is the URL of your site? (https://example.com):
> Do you want to enable Sass compilation? [Y/n]:
> Do you want to enable syntax highlighting? [y/N]:
> Do you want to build a search index of the content? [y/N]:
Done! Your site was created in /private/tmp/z
Get started by moving into the directory and using the built-in server: `zola serve`
Visit https://www.getzola.org for the full documentation.
$ echo "hello from humans" > static/humans.txt
$ echo "hello from robots" > templates/robots.txt
$ zola --version
zola 0.20.0
$ zola serve
Building site...
Checking all internal links with anchors.
> Successfully checked 0 internal link(s) with anchors.
-> Creating 0 pages (0 orphan) and 0 sections
Done in 5ms.
Web server is available at http://127.0.0.1:1025 (bound to 127.0.0.1:1025)
Listening for changes in /private/tmp/z/{config.toml,content,sass,static,templates}
Press Ctrl+C to stop
And check the response type:
$ curl -i http://127.0.0.1:1025/robots.txt
HTTP/1.1 200 OK
content-type: text/html
content-length: 18
date: Wed, 02 Jul 2025 19:58:23 GMT
hello from robots
$ curl -i http://127.0.0.1:1025/humans.txt
HTTP/1.1 200 OK
content-type: text/plain
access-control-allow-origin: *
content-length: 18
date: Wed, 02 Jul 2025 19:58:26 GMT
hello from humans
You can see that robots.txt
is incorrectly served as text/html
. Are you saying that when it’s really deployed, it won’t be served as text/html
? I would have thought that the dev site is replicating the prod one.
No, I’m not saying zola serve
gets this right, I’m saying it probably should i.e. it ought to be fixed! Sorry for the misunderstanding.
But to your broader question: no, don’t think of zola serve
as replicating prod. It’s there to help you check content and styling and the like, but it’s not making any claim to be a full-featured web server. It doesn’t set any caching headers, for example, which is something you’d really want in a static site. Also, if you poke around in the /public
directory, you’ll notice that zola serve
doesn’t actually generate many of the “files” it serves as real files on disk - there’s no robots.txt
in there, for example.
I’d be confident that the real webserver won’t have this issue you’re seeing when you put things in production.
Ah, I see. You’re right: in production, it does serve it as plain/text; charset=utf-8
, as it should. Thanks @mrec!