# Search Improvement

**URL:** <https://zola.discourse.group/t/search-improvement/344>\
**Category:** Feature requests\
**Created:** [March 4, 2020, 10:58am UTC](https://zola.discourse.group/t/search-improvement/344 "2020-03-04T10:58:26Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![SgtIria](https://yyz2.discourse-cdn.com/free1/user_avatar/zola.discourse.group/sgtiria/32/85_2.png) [@SgtIria](https://zola.discourse.group/u/SgtIria)\
**Post date:** [March 4, 2020, 10:58am UTC](https://zola.discourse.group/t/search-improvement/344/1 "2020-03-04T10:58:26Z")

</div>

I know you’ve previously discussed the inability to output custom fields in the RUST generated [elasticlunr.js](https://github.com/weixsong/elasticlunr.js) JSON index. I accept **custom search fields** would be mission creep and _Zola_ needs to stick to what it does best. `But...` 🙂 Would it be possible to specify a trim size for the page data?

For example; At it’s simplest, a `config.toml` variable that said `search_truncate = 200` i.e. first 200 hundred characters would be used to build the stem words and be held in the index JSON data, you would simply skip the rest. If the page is `< 200 characters` long you just roll with what you’ve got.

You change none of the current implementation other than to lookup a configuration variable which determines if you trim the page length data.

I think one of **zola’s USP’s** is the search built in. Implementing search for a static website can be a major pain or worse still cost money.

Implementing this basic limitation does a few things:

- It makes search possible for websites with a small number of pages, even when the pages contain a lot of character data/text.
- It gives people, who have 100’s of pages, the flexibility to reduce the file size of the `search_index.en.js` file to keep it usable before having to look for alternative search engine options.
- Possibility to set `search_truncate = 0` which would effectively make the search `title only`
- In theory it’s not breaking any of the current implementation, it all works exactly the same way, except the data you generate the search index is trimmed first.

At this point your thinking, omg these guys just don’t get it. It’s impossible 🙂 or hopefully, this one is doable 🙂

I look forward to your views.

---

<div class="post-metadata">

**Author:** ![keats](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@keats](https://zola.discourse.group/u/keats)\
**Post date:** [March 6, 2020, 7:04am UTC](https://zola.discourse.group/t/search-improvement/344/2 "2020-03-06T07:04:17Z")

</div>

I think there are many ways to improve search, from limiting the amount of text to selecting only title/description or some random field in [extra]  
I need to create a GH issue for that this weekend

---

<div class="post-metadata">

**Author:** ![SgtIria](https://yyz2.discourse-cdn.com/free1/user_avatar/zola.discourse.group/sgtiria/32/85_2.png) [@SgtIria](https://zola.discourse.group/u/SgtIria)\
**Post date:** [March 7, 2020, 11:34am UTC](https://zola.discourse.group/t/search-improvement/344/3 "2020-03-07T11:34:59Z")

</div>

Great, thanks. We’ve currently got a 5mb json file (around 250,000 words) which when gzip’d is around 500k over the network, this is fine on a desktop, but interestingly on mobile the JVM on the mobile browser takes ages 5+ seconds on older phones to decompress and scan/load the database it’s quite slow. So having options to tweak the size and how much is indexed would be a really valuable.

---

<div class="post-metadata">

**Author:** ![keats](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@keats](https://zola.discourse.group/u/keats)\
**Post date:** [March 7, 2020, 11:41am UTC](https://zola.discourse.group/t/search-improvement/344/4 "2020-03-07T11:41:30Z")

</div>

I’ve created [https://github.com/getzola/zola/issues/961](https://github.com/getzola/zola/issues/961) for that. We can continue discussing in this discourse thread though, the issue is there so I don’t forget it

---

<div class="post-metadata">

**Author:** ![Roman](https://avatars.discourse-cdn.com/v4/letter/r/ed8c4c/32.png) [@Roman](https://zola.discourse.group/u/Roman)\
**Post date:** [March 9, 2020, 6:19pm UTC](https://zola.discourse.group/t/search-improvement/344/5 "2020-03-09T18:19:48Z")

</div>

I use Zola in local knowledgebase and I’ve found only one solution we discussed [there](https://zola.discourse.group/t/building-search-index-with-title-and-description-only/198/13): to build my own version of zola binary and search index with just title&description. It works despite of some bounds, an example, I should use separate debian virtual server with ownbuilded zola and nginx to serve html files. It would be super useful to have settings string for search fields in config.toml

---

<div class="post-metadata">

**Author:** ![SgtIria](https://yyz2.discourse-cdn.com/free1/user_avatar/zola.discourse.group/sgtiria/32/85_2.png) [@SgtIria](https://zola.discourse.group/u/SgtIria)\
**Post date:** [March 23, 2020, 9:46am UTC](https://zola.discourse.group/t/search-improvement/344/6 "2020-03-23T09:46:14Z")

</div>

I’ve noticed the discussion on Zola git has started to talk about a completely new search engine, which when I took a look at it, does not even provide a stand alone static website javascript library. I’m not sure if I’m reading this the wrong way, but I’d like to make sure that Zola search evolves, rather than being replaced.

I (and I am sure lots of other people) have spent quite a bit of time and effort getting the current search functionality to work. It works fine, in fact it’s an elegant solution, the only downside is the lack of any ability to optimise the json index file size to suit different use cases (like mobile).

To be clear, I am happy with the way search is implemented, improvements would be welcome. Please don’t move away from a static website compatible implementation. Please consider carefully the impact of breaking changes, I’m sure you will 🙂

---

<div class="post-metadata">

**Author:** ![keats](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@keats](https://zola.discourse.group/u/keats)\
**Post date:** [March 23, 2020, 11:47am UTC](https://zola.discourse.group/t/search-improvement/344/7 "2020-03-23T11:47:56Z")

</div>

No the goal for any search engine in Zola would be to have something that works without a server so Tantivy is out. Then it’s just a matter of how good the results are, if we can have the same UX for search as right now but with a better search engine, why not.

---

<div class="post-metadata">

**Author:** ![keats](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@keats](https://zola.discourse.group/u/keats)\
**Post date:** [April 24, 2020, 2:35pm UTC](https://zola.discourse.group/t/search-improvement/344/8 "2020-04-24T14:35:43Z")

</div>

Anyone interested in making some changes to Zola regarding that?

IE select which fields to index, a max size for the context etc?

---

<div class="post-metadata">

**Author:** ![Roman](https://avatars.discourse-cdn.com/v4/letter/r/ed8c4c/32.png) [@Roman](https://zola.discourse.group/u/Roman)\
**Post date:** [April 25, 2020, 9:16pm UTC](https://zola.discourse.group/t/search-improvement/344/9 "2020-04-25T21:16:48Z")

</div>

I’ll be happy if it would be possible to map fileds in config

---

<div class="post-metadata">

**Author:** ![SgtIria](https://yyz2.discourse-cdn.com/free1/user_avatar/zola.discourse.group/sgtiria/32/85_2.png) [@SgtIria](https://zola.discourse.group/u/SgtIria)\
**Post date:** [May 15, 2020, 12:49pm UTC](https://zola.discourse.group/t/search-improvement/344/10 "2020-05-15T12:49:10Z")

</div>

I know this is probably not a huge priority for everyone, but over on [https://adeptenglish.com/](https://adeptenglish.com/) we now have an index file that’s 6mb (600kb over the wire).

On desktop this is no big deal, but on mobile the 600kb is not great but OK, its the decompression time to unzip the JavaScript to its 6mb is killing older mobile phones. Google lighthouse bench-marking is hideous.

It’s getting to the point where we are going to switch it off.

All it needs is the ability to truncate the amount of text taken from `page.content` in the short term. So we can index on first 300-500 words, and ignore the other 2000+ words in the article.

---

<div class="post-metadata">

**Author:** ![keats](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@keats](https://zola.discourse.group/u/keats)\
**Post date:** [May 15, 2020, 4:04pm UTC](https://zola.discourse.group/t/search-improvement/344/11 "2020-05-15T16:04:20Z")

</div>

I will take a PR for those kind of things, which fields to use + trimming.

---

<div class="post-metadata">

**Author:** ![SgtIria](https://yyz2.discourse-cdn.com/free1/user_avatar/zola.discourse.group/sgtiria/32/85_2.png) [@SgtIria](https://zola.discourse.group/u/SgtIria)\
**Post date:** [June 3, 2020, 11:39am UTC](https://zola.discourse.group/t/search-improvement/344/12 "2020-06-03T11:39:56Z")

</div>

Hi Keats, what does

> I will take a PR for those kind of things

Mean? Are you asking me to raise a PR? I don’t know how to do that, but I’ll have a go 🙂 if you show me how.

I’ve been looking at the `search_index.en.js` json output and interestingly enough I was able to reduce the file size just removing the full URL and replacing with a `\`.

**For example**

Using notepad++ I just global replaced `"https://adeptenglish.com/"` with `"/"`

So on a local search index the original was **6,713,339 bytes** (Before Gzip over the wire)

For example:

```auto
{"tf":1.0},"http://192.168.1.96:2015/language-courses/podcast-bundle-201-250/":{"tf":1.0},

```

Becomes:

```auto
{"tf":1.0},"/language-courses/podcast-bundle-201-250/":{"tf":1.0},

```

Now it’s **5,315,371 bytes**

It appears no functional change happened, i.e. it seems to work fine. Around a **20%** saving for free. Your mileage will vary based on domain length. This is a huge saving and should be trivial? What do you think?

I’m still very keen to see an improvement in the search but I’ll take a quick win

---

<div class="post-metadata">

**Author:** ![keats](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@keats](https://zola.discourse.group/u/keats)\
**Post date:** [June 4, 2020, 8:42am UTC](https://zola.discourse.group/t/search-improvement/344/13 "2020-06-04T08:42:14Z")

</div>

> [@SgtIria](#):
>
> I will take a PR for those kind of things

It means if anyone wants to implement it, I’ll merge it 🙂 It’s on the TODO list for the next release but it will be faster if someone else implements it since the list is long ([0.12 · GitHub](https://github.com/getzola/zola/projects/2)) and I don’t have a lot of time right now.

For this change, we need to add some options in the config and then just change what we are passing to the search index in [https://github.com/getzola/zola/blob/master/components/search/src/lib.rs](https://github.com/getzola/zola/blob/master/components/search/src/lib.rs) depending on those config options.  
Using the path instead of the permalink is a really good idea too.

---

<div class="post-metadata">

**Author:** ![Roman](https://avatars.discourse-cdn.com/v4/letter/r/ed8c4c/32.png) [@Roman](https://zola.discourse.group/u/Roman)\
**Post date:** [June 4, 2020, 9:04pm UTC](https://zola.discourse.group/t/search-improvement/344/14 "2020-06-04T21:04:31Z")

</div>

of the three of us looks like you’re the only one who will be able to make it 🙃

---

<div class="post-metadata">

**Author:** ![keats](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@keats](https://zola.discourse.group/u/keats)\
**Post date:** [June 29, 2020, 6:04pm UTC](https://zola.discourse.group/t/search-improvement/344/15 "2020-06-29T18:04:01Z")

</div>

I’ve pushed a commit to address that: [https://github.com/getzola/zola/pull/1038/commits/fb994c71d7d4ae5aa7875889077046db5bebde80](https://github.com/getzola/zola/pull/1038/commits/fb994c71d7d4ae5aa7875889077046db5bebde80)

Can you build the `next` branch and see if it works for you?

---

<div class="post-metadata">

**Author:** ![Roman](https://avatars.discourse-cdn.com/v4/letter/r/ed8c4c/32.png) [@Roman](https://zola.discourse.group/u/Roman)\
**Post date:** [June 29, 2020, 9:26pm UTC](https://zola.discourse.group/t/search-improvement/344/16 "2020-06-29T21:26:29Z")

</div>

I’ve tried with zola next build and config below (sorry for screens) and it couldn’t build search-index at all.

![изображение](https://global.discourse-cdn.com/free1/uploads/zola1/original/1X/d283c8db55a9e5bd51e1a8830df62aa4638ac56f.png)

 ![изображение](https://global.discourse-cdn.com/free1/uploads/zola1/original/1X/e80b3621533f67392e72fb340b2e13d9b80a2ccb.png)

---

<div class="post-metadata">

**Author:** ![keats](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@keats](https://zola.discourse.group/u/keats)\
**Post date:** [June 30, 2020, 12:12pm UTC](https://zola.discourse.group/t/search-improvement/344/18 "2020-06-30T12:12:36Z")

</div>

you still need `build_search_index = true` in your config

---

<div class="post-metadata">

**Author:** ![Roman](https://avatars.discourse-cdn.com/v4/letter/r/ed8c4c/32.png) [@Roman](https://zola.discourse.group/u/Roman)\
**Post date:** [June 30, 2020, 5:31pm UTC](https://zola.discourse.group/t/search-improvement/344/19 "2020-06-30T17:31:21Z")

</div>

doh! I didn’t know why I had read this [string](https://github.com/getzola/zola/pull/1038/commits/fb994c71d7d4ae5aa7875889077046db5bebde80#diff-936c4f6b6d85b600e91b6c4c1f5c1bd6R122) like “you should toggle build\_search\_index = false” to implement new features 🤯

I tried successfully `include_title = true` and `include_description = true` . It works! many thanks, hope to see it in the master branch

---

<div class="post-metadata">

**Author:** ![redFrik](https://avatars.discourse-cdn.com/v4/letter/r/bcef8e/32.png) [@redFrik](https://zola.discourse.group/u/redFrik)\
**Post date:** [July 21, 2020, 10:41am UTC](https://zola.discourse.group/t/search-improvement/344/20 "2020-07-21T10:41:49Z")

</div>

looking forward the new release with the new search features. thanks.

meanwhile, and sorry if this is stating the obvious, what worked for me was to defer loading of /search\_index.en.js and only initialise the search when/if a user start typing in the search field.

so assuming [https://github.com/getzola/zola/blob/master/docs/static/search.js](https://github.com/getzola/zola/blob/master/docs/static/search.js) - change the last seven lines to look something like this…

```
if (document.readyState === "complete" ||
    (document.readyState !== "loading" && !document.documentElement.doScroll)
) {
  delayedInitSearch();
} else {
  document.addEventListener("DOMContentLoaded", delayedInitSearch);
}

function delayedInitSearch() {
  function loadScript(url, callback) {
    var body = document.body
    var script = document.createElement('script')
    script.type = 'text/javascript'
    script.src = url
    script.onreadystatechange = callback
    script.onload = callback
    body.appendChild(script)
  }
  document.getElementById("search").addEventListener("input", function() {
    loadScript("/search_index.en.js", initSearch)
  }, {once: true})
}

```

and the large file will load when you type the first character. a small hickup will be noticed but that’s ok i find.

just a suggestion.

---

<div class="post-metadata">

**Author:** ![keats](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@keats](https://zola.discourse.group/u/keats)\
**Post date:** [July 24, 2020, 6:59pm UTC](https://zola.discourse.group/t/search-improvement/344/21 "2020-07-24T18:59:36Z")

</div>

Can you also defer the script load itself? So it doesn’t block anything but still happens in the background.

[Next page](https://zola.discourse.group/t/search-improvement/344.md?page=2)