Elasticsearch has a lot of small unknown game changing features. 

Search templates are one of those. When it suits your use case, it changes a lot your integration quality 

TL:DR 

Elasticsearch’s Search Template are interesting for : 

  • Simplify integration
  • Give a standard access to your indices without having to create any microservice
  • Separation of concerns
  • Avoid code duplication

What is a Search Template? 

Search templates are reusable scripts that handle the query complexity and let integrator use very complicated queries with just few parameters.

Elasticsearch’s JSON syntax is very verbose, and a good query for your innovative uniq use case can be very long and ask for a lot of iterations to give the best results

Then instead of creating your JSON query in your application code, it can be a good idea to just consume a template in your application and to keep the complexity of the query in your elasticsearch. 

Your project can then align itself with the team development best practices of separation of concerns. 

Your integrator plays with showing results on the screen, and your elasticsearch specialist works in kibana to improve your results quality. 

Another really interesting use case is when you have the same behaviors in different applications. For this article, we will prepare a query to find clients in a client database.
You’ll probably need to integrate it in your CRM (Salesforce or another one), your ERP coded in Java, and maybe your intranet coded in NodeJs. 
The behavior is the same and we want to have the same results for the same query in each application. 
To avoid code duplication, coding a dedicated micro-service, or recode everything 3 times, just call the search template and let Elasticsearch handle the logic. 

Search template exemple

A search template is basically a script with a mustache template for the query itself. 

It can handle

  • Complexe queries
  • Parameters
  • Conditional parts in query

When you create a template, your goal will be to deliver a simple first iteration, usable for the integrator:

POST _scripts/search_clients
{
  "script": {
    "lang": "mustache",
    "description": "Search integration for finding client",
    "source": {
      "query": {
        "match": {
          "fullname": "{{name.first}} {{name.last}}"
        }
      },
      "_source": ["id_client", "fistname", "lastname", "fullname", "company", "tel", "email", "website", "age", "gdpr_optin"]
    }
  }
}

To consume this template just use:

GET clients/_search/template
 {
  "id": "search_clients", 
  "params": {
    “name”: {
       "first": "jeremy", 
       "last": "gachet"
     }
  }
}

The result will be something like this: 

"hits" : [
      {
        "_index" : "clients",
        "_type" : "_doc",
        "_id" : "CL-00456",
        "_score" : 0.17402273,
        "_source" : {
          "website" : [
            "https://spoon-elastic.com/",
            "https://www.spoonconsulting.com/"
          ],
          "id_client" : "CL-00456",
          "gdpr_optin" : true,
          "company" : "Spoon Consulting",
          "tel" : "0000000000",
          "fullname" : "Jeremy GACHET",
          "email" : "[email protected]",
          "age" : 32,
          "lastname" : "GACHET"
        }
      }
    ]

Good! Our integrator can work on his side and we are ready to work on query improvement. 

In next iterations, we will add more flexibility to the integrator. 

He will be able to : 

  • Find all clients by
    • company name
    • Website
  • Find a client by client_id
  • Filter clients within a range of age
  • Filter by default clients who check the GDPR acceptance clause

First, we add other filters: 

POST _scripts/search_clients
{
  "script": {
    "lang": "mustache",
    "description": "Search integration for finding client",
    "source": {
      "query": {
        "bool": {
          "must": [
            {
              "match": {
              "fullname": "{{name.first}} {{name.last}}"
              }
            },
            {
              "match": {
                "company": "{{company}}"
              }
            }
          ],
          "filter": [
            {
              "term": {
                "website.keyword": "{{website}}"
              }
            },
            {
              "term": {
                "id_client.keyword": "{{id_client}}"
              }
            },
            {
              "range": {
                "age": {
                  "gte": "{{age.min}}",
                  "lte": "{{age.max}}"
                }
              }
            },
           { "term": {
                "gdpr_optin": true
            }}
          ]
        }
      },
      "_source": [
        "id_client",
        "fistname",
        "lastname",
        "fullname",
        "company",
        "tel",
        "email",
        "website",
        "age",
        "gdpr_optin"
      ]
    }
  }
}

It will work…. If you want all parameters to be mandatory. 

Step 2 Let’s add more flexibility with conditions in our template

{
      "query": {
        "bool": {
          "must": [
	    {"match_all": {}}
            {{#name}}
	    ,
            {
              "match": {
                "fullname": "{{#name.first}}{{name.first}}{{/name.first}} {{#name.last}}{{name.last}}{{/name.last}}"
              }
            }
            {{/name}}
            
            {{#company}}
	    ,
            {
              "match": {
                "company": "{{company}}"
              }
            }{}
            {{/company}}
          ],
          "filter": [
            { "term": {
                "gdpr_optin": true
            }},
            {{#website}}
            {
              "term": {
                "website.keyword": "{{website}}"
              }
            }
            {{/website}}
            {{#id_client}}
	    ,
            {
              "term": {
                "id_client.keyword": "{{id_client}}"
              }
            }
            {{/id_client}}

            {{#age}}
            ,
            {
              "range": {
                "age": {
                  "gte": "{{age.min}}{{^age.min}}0{{/age.min}}"
		{{#age.max}},{{/age.max}}
		{{#age.max}}
                  "lte": "{{age.max}}"
		{{/age.max}}
                }
              }
            }
            {{/age}}
          ]
        }
      },
      "_source": [
        "id_client",
        "fistname",
        "lastname",
        "fullname",
        "company",
        "tel",
        "email",
        "website",
        "age",
        "gdpr_optin"
      ]
    }

Third step, pass the template as a string to create it. This step can be really annoying. Be patient.

PRO TIPS: 
If we use the _render api on our template with the query

Use the _render api to generate you compiled template and understand more quickly the issues.

GET _render/template
{
  "source": "{\"query\":{\"bool\":{\"must\":[{\"match_all\":{}}{{#name}},{\"match\":{\"fullname\":\"{{#name.first}}{{name.first}}{{/name.first}}{{#name.last}}{{name.last}}{{/name.last}}\"}}{{/name}}{{#company}}, { \"match\": { \"company\": {  \"query\":\"{{company}}\",  \"boost\": 5  } } }{{/company}}],\"should\": [{{#name}} {  \"match_phrase\": {  \"fullname\":  \"{{#name.first}}{{name.first}}{{/name.first}} {{#name.last}}{{name.last}}{{/name.last}}\"} } {{/name}} ],\"filter\":[{\"term\":{\"gdpr_optin\": true }} {{#website}},{\"term\":{\"website.keyword\":\"{{website}}\"}}{{/website}}{{#id_client}},{\"term\":{\"id_client.keyword\":\"{{id_client}}\"}}{{/id_client}}{{#age}},{\"range\":{\"age\":{ {{#age.min}}\"gte\":\"{{age.min}}\"{{/age.min}}{{#age.max}},{{/age.max}}{{#age.max}}\"lte\":\"{{age.max}}\"{{/age.max}}}}}{{/age}}]}},\"_source\":[\"id_client\",\"fistname\",\"lastname\",\"fullname\",\"company\",\"tel\",\"email\",\"website\",\"age\",\"gdpr_optin\"]}",
  "params": {
    "name":{
      "first":"jeremy",
      "last": "gachet"
    },
    "company": "Spoon Consulting",
    "id_client": "CL-00456",
    "website": "https://spoon-elastic.com/",
    "age": {
      "min": 20,
      "max": 40
    }
  }
}

The result will show me an issue. I need to add a space between my fistname and my lastname

{
  "template_output" : {
    "query" : {
      "bool" : {
        "must" : [
          {
            "match_all" : { }
          },
          {
            "match" : {
              "fullname" : "jeremygachet"
            }
          },
          {
            "match" : {
              "company" : {
                "query" : "Spoon Consulting",
                "boost" : 5
              }
            }
          }
        ],
        "should" : [
          {
            "match_phrase" : {
              "fullname" : "jeremy gachet"
            }
          }
        ],
        "filter" : [
          {
            "term" : {
              "gdpr_optin" : true
            }
          },
          {
            "term" : {
              "website.keyword" : "https://spoon-elastic.com/"
            }
          },
          {
            "term" : {
              "id_client.keyword" : "CL-00456"
            }
          },
          {
            "range" : {
              "age" : {
                "gte" : "20",
                "lte" : "40"
              }
            }
          }
        ]
      }
    },
    "_source" : [
      "id_client",
      "fistname",
      "lastname",
      "fullname",
      "company",
      "tel",
      "email",
      "website",
      "age",
      "gdpr_optin"
    ]
  }
}

With this template our first query still works as if nothing had changed. And you can query with any combination of any parameters:

{
  "id": "search_clients",
  "params": {
    "name":{
      "first":"jeremy",
      "last": "gachet"
    },
    "company": "Spoon Consulting",
    "id_client": "CL-00456",
    "website": "https://spoon-elastic.com/",
    "age": {
      "min": 20,
      "max": 40
    }
  }
}

Improve scoring : 

What about improving scoring without changing anything to your integration? 

You can easily do your test cases, define the best query and then deploy it without touching anything in your apps. 

Let’s fine tune our query with a match phrase and some boost. 

POST _scripts/search_clients
{
  "script": {
    "lang": "mustache",
    "description": "Search integration for finding client",
    "source":  "{\"query\":{\"bool\":{\"must\":[{\"match_all\":{}}{{#name}},{\"match\":{\"fullname\":\"{{#name.first}}{{name.first}}{{/name.first}}{{#name.last}}{{name.last}}{{/name.last}}\"}}{{/name}}{{#company}}, { \"match\": { \"company\": {  \"query\":\"{{company}}\",  \"boost\": 5  } } }{{/company}}],\"should\": [{{#name}} {  \"match_phrase\": {  \"fullname\":  \"{{#name.first}}{{name.first}}{{/name.first}} {{#name.last}}{{name.last}}{{/name.last}} \"} } {{/name}} ],\"filter\":[{\"term\":{\"gdpr_optin\": true }}{{#website}},{\"term\":{\"website.keyword\":\"{{website}}\"}}{{/website}}{{#id_client}},{\"term\":{\"id_client.keyword\":\"{{id_client}}\"}}{{/id_client}}{{#age}},{\"range\":{\"age\":{ {{#age.min}}\"gte\":\"{{age.min}}\"{{/age.min}}{{#age.max}},{{/age.max}}{{#age.max}}\"lte\":\"{{age.max}}\"{{/age.max}}}}}{{/age}}]}},\"_source\":[\"id_client\",\"fistname\",\"lastname\",\"fullname\",\"company\",\"tel\",\"email\",\"website\",\"age\",\"gdpr_optin\"]}"
  }
}

And last test, the query must still works with only the first params: 

{
  "id": "search_clients",
  "params": {
    "name":{
      "first":"jeremy",
      "last": "gachet"
    }
  }
}

Last step (but not least), create an API doc to communicate a standard access to your index.

Conclusion

On large projects, Search templates are a very good method to industrialize processes.

It’s a little heavy for now but we can be sure that it will become easier on futur elastisearch releases. 

Spoon consulting is a certified partner of Elastic

As a certified partner of the Elastic company, Spoon Consulting offers a high level consulting for all kinds of companies.

Read more information on your personal use Elasticsearch use case on Spoon consulting’s posts

Or contact Spoon consulting now