Serve robots.txt as text/plain

How can I ensure robots.txt is served as text/plain [1] rather than text/html?

Interesting question. It’s true that zola serve should probably get this right (it does send text/xml correctly for feeds), but ultimately web crawlers aren’t going to be looking at a zola served dev site. For the real site, the webserver will determine the mimetype, and it’ll go off file extension unless specifically configured otherwise.

@mrec I don’t think Zola does get this right. If I make an empty site:

$ zola init
Welcome to Zola!
Please answer a few questions to get started quickly.
Any choices made can be changed by modifying the `config.toml` file later.
> What is the URL of your site? (https://example.com):
> Do you want to enable Sass compilation? [Y/n]:
> Do you want to enable syntax highlighting? [y/N]:
> Do you want to build a search index of the content? [y/N]:

Done! Your site was created in /private/tmp/z

Get started by moving into the directory and using the built-in server: `zola serve`
Visit https://www.getzola.org for the full documentation.

$ echo "hello from humans" > static/humans.txt

$ echo "hello from robots" > templates/robots.txt

$ zola --version
zola 0.20.0

$ zola serve
Building site...
Checking all internal links with anchors.
> Successfully checked 0 internal link(s) with anchors.
-> Creating 0 pages (0 orphan) and 0 sections
Done in 5ms.

Web server is available at http://127.0.0.1:1025 (bound to 127.0.0.1:1025)

Listening for changes in /private/tmp/z/{config.toml,content,sass,static,templates}
Press Ctrl+C to stop

And check the response type:

$ curl -i  http://127.0.0.1:1025/robots.txt
HTTP/1.1 200 OK
content-type: text/html
content-length: 18
date: Wed, 02 Jul 2025 19:58:23 GMT

hello from robots

$ curl -i  http://127.0.0.1:1025/humans.txt
HTTP/1.1 200 OK
content-type: text/plain
access-control-allow-origin: *
content-length: 18
date: Wed, 02 Jul 2025 19:58:26 GMT

hello from humans

You can see that robots.txt is incorrectly served as text/html. Are you saying that when it’s really deployed, it won’t be served as text/html? I would have thought that the dev site is replicating the prod one.

No, I’m not saying zola serve gets this right, I’m saying it probably should i.e. it ought to be fixed! Sorry for the misunderstanding.

But to your broader question: no, don’t think of zola serve as replicating prod. It’s there to help you check content and styling and the like, but it’s not making any claim to be a full-featured web server. It doesn’t set any caching headers, for example, which is something you’d really want in a static site. Also, if you poke around in the /public directory, you’ll notice that zola serve doesn’t actually generate many of the “files” it serves as real files on disk - there’s no robots.txt in there, for example.

I’d be confident that the real webserver won’t have this issue you’re seeing when you put things in production.

1 Like

Ah, I see. You’re right: in production, it does serve it as plain/text; charset=utf-8, as it should. Thanks @mrec!

1 Like