Serve robots.txt as text/plain

Support
1

How can I ensure robots.txt is served as text/plain [1] rather than text/html?

2

Interesting question. It’s true that zola serve should probably get this right (it does send text/xml correctly for feeds), but ultimately web crawlers aren’t going to be looking at a zola served dev site. For the real site, the webserver will determine the mimetype, and it’ll go off file extension unless specifically configured otherwise.

3

@mrec I don’t think Zola does get this right. If I make an empty site:

$ zola init
Welcome to Zola!
Please answer a few questions to get started quickly.
Any choices made can be changed by modifying the `config.toml` file later.
> What is the URL of your site? (https://example.com):
> Do you want to enable Sass compilation? [Y/n]:
> Do you want to enable syntax highlighting? [y/N]:
> Do you want to build a search index of the content? [y/N]:

Done! Your site was created in /private/tmp/z

Get started by moving into the directory and using the built-in server: `zola serve`
Visit https://www.getzola.org for the full documentation.

$ echo "hello from humans" > static/humans.txt

$ echo "hello from robots" > templates/robots.txt

$ zola --version
zola 0.20.0

$ zola serve
Building site...
Checking all internal links with anchors.
> Successfully checked 0 internal link(s) with anchors.
-> Creating 0 pages (0 orphan) and 0 sections
Done in 5ms.

Web server is available at http://127.0.0.1:1025 (bound to 127.0.0.1:1025)

Listening for changes in /private/tmp/z/{config.toml,content,sass,static,templates}
Press Ctrl+C to stop

And check the response type:

$ curl -i  http://127.0.0.1:1025/robots.txt
HTTP/1.1 200 OK
content-type: text/html
content-length: 18
date: Wed, 02 Jul 2025 19:58:23 GMT

hello from robots

$ curl -i  http://127.0.0.1:1025/humans.txt
HTTP/1.1 200 OK
content-type: text/plain
access-control-allow-origin: *
content-length: 18
date: Wed, 02 Jul 2025 19:58:26 GMT

hello from humans

You can see that robots.txt is incorrectly served as text/html. Are you saying that when it’s really deployed, it won’t be served as text/html? I would have thought that the dev site is replicating the prod one.

4

No, I’m not saying zola serve gets this right, I’m saying it probably should i.e. it ought to be fixed! Sorry for the misunderstanding.

But to your broader question: no, don’t think of zola serve as replicating prod. It’s there to help you check content and styling and the like, but it’s not making any claim to be a full-featured web server. It doesn’t set any caching headers, for example, which is something you’d really want in a static site. Also, if you poke around in the /public directory, you’ll notice that zola serve doesn’t actually generate many of the “files” it serves as real files on disk - there’s no robots.txt in there, for example.

I’d be confident that the real webserver won’t have this issue you’re seeing when you put things in production.

1 Like
5

Ah, I see. You’re right: in production, it does serve it as plain/text; charset=utf-8, as it should. Thanks @mrec!

1 Like