In the latest releases of Elasticsearch and Kibana, a lot has been done to integrate runtime fields smoothly and unleash all it’s power naturally.
In a few clicks, you can create virtual values for formatting or calculating values without reindexing the whole index.
The runtime field can now be called in total transparency like any other elasticsearch _source field.
But it’s still a virtual field, calculated on the fly at query time, just like scripted fields.
So, it’s a very good solution to show valuable read only values, but even if it’s quite well optimised, when it comes to search or aggregation the performance can be a real issue.
Official documentation is pretty clear, the goal of this post is to clarify some blurry areas.
Note, We will not talk of painless script in details here. If you need more information I advise you to read this excellent post carefully : Elastic Runtime Field example repository
Creation in Kibana
In the discover clic on the three dot, and on “add field to index pattern”
And from lens’ formula:
Note, both options will create query time fields, and save it in the case of the discover visualisation) on the Kibana index pattern, not on the elasticsearch index itself. We will talk more of the difference soon.
How Elasticsearch’s runtime field works
At first the easier way to create and use a runtime field is to declare it in the request query.
It’s how kibana manage them.
# --------------------------------------
# query runtime field
#----------------------------------------
GET test_run/_search
{
"runtime_mappings": {
"test_run": {
"type": "keyword",
"script": {
"source": """
emit('query runtime test');
"""
}
}
},
"fields": ["*"] # you’ll need this to see the runtime field in response as it will not be in _source, note the best practice is to specify explicitly each needed field
}
Response :
"hits" : [
{
"_index" : "test_run",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.0,
"_source" : { },
"fields" : {
"test_run" : [
"query runtime test"
]
}
}
]
It works!
Note : to see how kibana builds a request query for a visualisation, click on the wheel > inspect > view: Request and click on the Tab named request. You can also see the raw response in the dedicated tab.
If you want to make the field available for all queries for an index and avoid a declaration on every single query, you can map the runtime field.
It will have the exact same behaviour as the runtime at query time, but you don’t need to add it on each request anymore.
Update your index mapping in elasticsearch
PUT test_run/_mapping
{
"runtime": {
"test_run": {
"type": "keyword",
"script": {
"source": """
emit('runtime test');
"""
}
}
}
}
Verify the mapping
GET test_run/_mapping
# response
{
"test_run" : {
"mappings" : {
"runtime" : {
"test_run" : {
"type" : "keyword",
"script" : {
"source" : """
emit('runtime test');
""",
"lang" : "painless"
}
}
}
}
}
}
And you can now query it:
GET test_run/_search?filter_path=hits.hits.fields
{
"fields": [
"status", "test_run"
],
"_source": false
}
how does index runfield react on already existing docs and how does it react on new docs ?
As fields are populated at query time (the famous “schema on read”) you can create fields whenever you want, it will be applied in real time.
POST test_run/_doc
{
"status": "ingested after script"
}
GET test_run/_search?filter_path=hits.hits.fields
{
"fields": [
"status", "test_run"
],
"_source": false
}
#response
{
"hits" : {
"hits" : [
{
"fields" : {
"test_run" : [
"runtime test"
]
}
},
{
"fields" : {
"test_run" : [
"runtime test"
],
"status" : [
"ingested after script"
]
}
}
]
}
}
Indexing field
Documentation said : To gain greater performance, you decide to index the very important new field
if you create a runtime field with the same name as a field that already exists in the mapping, the runtime field shadows the mapped field
PUT test_run/_mapping
{
"properties": {
"test_run": {
"type": "keyword",
"on_script_error": "fail",
"script": {
"source": """
emit('scipted test');
"""
}
},
"test_scripted": {
"type": "keyword",
"on_script_error": "fail",
"script": {
"source": """
emit('scipted also test');
"""
}
}
}
}
# verifying that everithing is good :
GET test_run/_mapping
#response
{
"test_run" : {
"mappings" : {
"runtime" : {
"test_run" : {
"type" : "keyword",
"script" : {
"source" : """
emit('runtime test');
""",
"lang" : "painless"
}
}
},
"properties" : {
"status" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"test_run" : {
"type" : "keyword",
"script" : {
"source" : """
emit('scipted test');
""",
"lang" : "painless"
}
},
"test_scripted" : {
"type" : "keyword",
"script" : {
"source" : """
emit('scipted also test');
""",
"lang" : "painless"
}
}
}
}
}
}
So If I request my first document why will i see ?
-> runtime test just like before
And if I try to add a new doc ?
POST test_run/_doc
{
"status": "ingested after script"
}
GET test_run/_search
{
"fields": [
"*"
],
"_source": false
}
# response
[
{
"fields" : {
"test_run" : [
"runtime test" # but my script is : emit('scipted test');
],
"test_scripted" : [
"scipted also test"
],
"status.keyword" : [
"ingested after script"
],
"status" : [
"ingested after script"
]
}
}
]
We got the runtime value just like before
It’s because the runtime field overrides the source field.
It’s a good feature to correct data on the fly but can be confusing
So no matter what you try you will still have values in this order :
- query runtime
- index runtime (if query runtime field does not exists)
- source / scripted field (if query and index runtime field does not exists)
Let’s illustrate it with our first search query :
GET test_run/_search?filter_path=hits.hits.fields
{
"runtime_mappings": {
"test_run": {
"type": "keyword",
"script": {
"source": """
emit('query runtime test');
"""
}
}
},
"_source": false
"fields": ["test_run"]
}
Elasticsearch will always return
query runtime test
Therefore if you want to replace the runtime field in index by a more performant scripted field, you’ll have to add to nullify the runtime first
PUT test_run/_mapping
{
"runtime": {
"test_run": null
}
}
Now we have the good scripted value in our test_run field
GET test_run/_search?filter_path=hits.hits.fields
{
"fields": ["test_run"],
"_source": false
}
Runtime field : dynamic mapping clarification
Let’s clean up our test :
DELETE runtime_index
And let’s recreate it with a dynamic mapping parameter set to runtime.
PUT runtime_index
{
"mappings": {
"dynamic": "runtime",
"runtime": {
"runtime_field": {
"type": "keyword"
}
},
"properties": {
"not_runtime_field": {
"type": "text"
}
}
}
}
Then we add a doc:
POST runtime_index/_doc/sxT3dn4B1c5z0ej01DAo
{
"my_new_text": "my value"
}
In the index mapping you’ll see a new
"runtime" : {
"my_new_text" : {
"type" : "keyword"
},
GET runtime_index/_mapping
#response
{
"runtime_index" : {
"mappings" : {
"dynamic" : "runtime",
"runtime" : {
"my_new_text" : {
"type" : "keyword"
},
"runtime_field" : {
"type" : "keyword"
}
},
"properties" : {
"not_runtime_field" : {
"type" : "text"
}
}
}
}
}
Now If we add a new doc with another new non mapped field :
POST runtime_index/_doc/
{
"my_new_text2": "my value"
}
GET runtime_index/_search
{
"fields": [
"*"
]
}
# response : {
"hits" : {
"hits" : [
{
"fields" : {
"my_new_text" : [
"my value"
]
}
},
{
"fields" : {
"my_new_text2" : [
"my value"
]
}
}
]
}
}
If we add a new fields on this doc it will just add more runtime fields :
POST runtime_index/_doc/KhSddn4B1c5z0ej0ZzBY
{
"my_new_text3": "my value",
"my_new_text4": "my value",
"my_new_text5": "my value",
"my_new_text6": "my value",
"my_new_text7": "my value",
"my_new_text8": "my value",
"my_new_text9": "my value",
"my_new_text10": "my value",
"my_new_text11": "my value",
"my_new_text12": "my value"
}
And the mapping :
{
"runtime_index" : {
"mappings" : {
"dynamic" : "runtime",
"runtime" : {
"my_new_text" : {
"type" : "keyword"
},
"my_new_text10" : {
"type" : "keyword"
},
"my_new_text11" : {
"type" : "keyword"
},
"my_new_text12" : {
"type" : "keyword"
},
"my_new_text2" : {
"type" : "keyword"
},
"my_new_text3" : {
"type" : "keyword"
},
"my_new_text4" : {
"type" : "keyword"
},
"my_new_text5" : {
"type" : "keyword"
},
"my_new_text6" : {
"type" : "keyword"
},
"my_new_text7" : {
"type" : "keyword"
},
"my_new_text8" : {
"type" : "keyword"
},
"my_new_text9" : {
"type" : "keyword"
},
"runtime_field" : {
"type" : "keyword"
}
},
"properties" : {
"not_runtime_field" : {
"type" : "text"
}
}
}
}
}
What is very interesting here is that we can then choose to modify the mapping on the runtime fields without reindexing, which really ease the early stage of your index definition:
A classical use case: When the first ingested doc is a number but the real type needs to be a float, elasticsearch will create a type long and throw an exception when a float is ingested.
Exemple :
POST runtime_index/_doc/sxT3dn4B1c5z0ej01DAo
{
"my_new_number": 1
}
GET runtime_index/_mapping
# my new number is a long
"my_new_number" : {
"type" : "long"
}
Let’s modify the type on the fly :
PUT runtime_index/_mapping
{
"runtime": {
"my_new_number" : {
"type" : "double"
}
}
}
GET runtime_index/_mapping
# my new number is now a double
"my_new_number" : {
"type" : "double"
},
You can now save a lot of time by testing your fields and preparing a clean index with a strict mapping for production
Note : runtime fields are not counted in the limit of 1000 fields max par index.
Then you can try to ingest blindly in a test index, analyse and clean up your index with kibana and then prépare a clean production ingestion easily.
Conclusion
Runtime fields are game changing in Elasticsearch. It will save us days of work and be more satisfying for non technical users.
I’m pretty sure that a lot of innovation will be added on this feature on next released.
Spoon consulting is a certified partner of Elastic
As a certified partner of the Elastic company, Spoon Consulting offers a high level consulting for all kinds of companies.
Read more information on your personal use Elasticsearch use case on Spoon consulting’s posts
