Elasticsearch search

last updated 2013-01

update 2026 - written for Elasticsearch 0.20 (2013). the ideas hold, but a lot of the API doesn't: facets were replaced by aggregations in 2.0, the filtered query and the top-level filter are gone (use a bool query's filter clause, and post_filter for filtering hits only), and so are mapping types

official documentation

back to Elasticsearch

search can be executed across indices and types, with a query string as a parameter, or using a request body

search is broadcast to all the index/indices shards; to limit the scope, you can use the routing parameter - e.g. using the user ID when searching through tweets by a given user. the routing parameter may be multivalued, CSV

see also highlighting, and other topics not covered here…

request body

the request body uses query DSL

$ curl -XGET 'http://localhost:9200/twitter/tweet/_search' -d '{
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}'

{
    "_shards":{
        "total" : 5,
        "successful" : 5,
        "failed" : 0
    },
    "hits":{
        "total" : 1,
        "hits" : [
            {
                "_index" : "twitter",
                "_type" : "tweet",
                "_id" : "1", 
                "_source" : {
                    "user" : "kimchy",
                    "postDate" : "2009-11-15T14:12:12",
                    "message" : "trying out Elastic Search"
                }
            }
        ]
    }
}

parameters

timeout - search timeout, bounding the search request to be executed within the specified time; default no timeout
from - the starting from index of the hits to return; default 0
size - the number of hits to return; default 10
search_type - the type of the search operation to perform - dfs_query_then_fetch, dfs_query_and_fetch, query_then_fetch, query_and_fetch; defaults to query_then_fetch; see search type for more details on the different types of search that can be performed

URI request

a search request can be executed purely using a URI by providing request parameters

$ curl -XGET 'http://localhost:9200/twitter/tweet/_search?q=user:kimchy'

parameters

q - query string (maps to the query_string query, see query string query for more details)
df - default field to use when no field prefix is defined
default_operator - default operator to be used - AND or OR, default OR
explain - include an explanation of how scoring of each hit was computed
fields - selective fields of the document to return, CSV; default is the internal _source field; an empty value will cause no fields to return
sort - sorting to perform - fieldName, fieldName:asc, or fieldName:desc; fieldName can either be an actual field, or _score; there can be several sort parameters (CSV, order is important)
track_scores - when sorting, set to true in order to return the score as part of each hit
timeout - search timeout, limiting the execution time; all results accumulated up to the timeout are returned; defaults to no timeout
from - starting from index of the hits to return; defaults to 0
size - number of hits to return; defaults to 10
lowercase_expanded_terms - should terms be automatically lowercased or not; default true
analyze_wildcard - should wildcard and prefix queries be analyzed or not; default false

query element

see query DSL for details

{
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

when doing things like facet navigation, sometimes only the hits need to be filtered by the chosen facet, and all the facets should continue to be calculated based on the original query. the filter element within the search request can be used to accomplish it

note, this is different compared to creating a filtered query with the filter, since this will cause the facets to only process the filtered results

in other words, using

{
    "query" : { "term" : { "message" : "something" } },
    "filter" : { "term" : { "tag" : "green" } },
    "facets" : { "tag" : { "terms" : { "field" : "tag" } } }
}

the filter will not change the facets (the results of facets will be the same as without the filter element), while the results set will be different

but using the filtered query:

{
    "query" : {
        "filtered" : {
            "query" : { "term" : { "message" : "something" } },
            "filter" : { "term" : { "tag" : "green" } }
        }
    },
    "facets" : { "tag" : { "terms" : { "field" : "tag" } } }
}

the filter field within the filtered query element will change the facets, influencing both the results set and the facets

to filter the facets, you can use the facet_filter element

from and size, pagination

{
    "from" : 0, "size" : 10,
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

indices and types

specific index and type

$ curl -XGET 'http://localhost:9200/twitter/user/_search?q=user:kimchy'

specific index, multiple types

$ curl -XGET 'http://localhost:9200/twitter/user,tweet/_search?q=user:kimchy'

multiple indices, all types

$ curl -XGET 'http://localhost:9200/twitter,facebook/_search?q=user:kimchy'

all indices, specific type

$ curl -XGET 'http://localhost:9200/_all/tweet/_search?q=user:kimchy'

all indices and types

$ curl -XGET 'http://localhost:9200/_search?q=user:kimchy'

sorting

the sort is defined on a per field level, with the special field name _score to sort by score:

{
    "sort" : [
        { "post_date" : {"order" : "asc"} },
        "user",
        { "name" : "desc" },
        { "age" : "desc" },
        "_score"
    ],
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

the sort values for each document returned are also returned as part of the response

missing numeric fields

numeric fields support specific handling for missing fields in a doc. the missing value can be _last, _first, or a custom value (that will be used for missing docs as the sort value). for example:

{
    "sort" : [
        { "price" : {"missing" : "_last"} }
    ],
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

missing mapping for field

by default, the search request will fail if there is no mapping associated with a field. the ignore_unmapped option allows to ignore fields that have no mapping and not sort by them. here is an example of how it can be used:

{
    "sort" : [
        { "price" : {"ignore_unmapped" : true} }
    ],
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

GeoDistance

you can also sort by _geo_distance:

{
    "sort" : [
        {
            "_geo_distance" : {
                "pin.location" : [-70, 40],
                "order" : "asc",
                "unit" : "km"
            }
        }
    ],
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

the _geo_distance pin may be provided as:

properties:         "pin.location" : { "lat" : 40, "lon" : -70 }
string:             "pin.location" : "40,-70"
geohash:            "pin.location" : "drm3btev3e86"
array:              "pin.location" : [-70, 40]

script based sorting

{
    "query" : {
        ....
    },
    "sort" : {
        "_script" : { 
            "script" : "doc['field_name'].value * factor",
            "type" : "number",
            "params" : {
                "factor" : 1.1
            },
            "order" : "asc"
        }
    }
}

note: for single field based sorting, use the custom_score query - it's faster

scores

when sorting on a field, scores are not computed. by setting track_scores to true, scores will still be computed and tracked

{
    "track_scores": true,
    "sort" : [
        { "post_date" : {"reverse" : true} },
        { "name" : "desc" },
        { "age" : "desc" }
    ],
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

note

beware that all relevant fields used for sorting have to be loaded to memory

when sorting on string fields, the field sorted on should not be analyzed/tokenized. for numeric types, it is recommended to explicitly set the type - short, integer, float, …

fields

by default, ES loads the internal _source field

the fields will automatically load stored fields (store mapping set to yes), or, if not stored, will load the _source and extract it from it (allowing to return a nested document object)

the * can be used to load all stored fields from the document

an empty array will cause only the _id and _type for each hit to be returned, for example:

{
    "fields" : [],
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

partial

you can also specify only parts of the document to be included or excluded from loading, using partial_fields (each of them supports multiple patterns):

{
    "query" : {
        "match_all" : {}
    },
    "partial_fields" : {
        "partial1" : {
            "include" : ["obj1.obj2.*", "obj1.obj4.*"],
            "exclude" : "obj1.obj3.*"
        }
    }
}

script fields

ES allows to return fields created by script evaluation:

{
    "query" : {
        ...
    },
    "script_fields" : {
        "test1" : {
            "script" : "doc['my_field_name'].value * 2"
        },
        "test2" : {
            "script" : "doc['my_field_name'].value * factor",
            "params" : {
                "factor"  : 2.0
            }
        },
        "test3" : {
            "script" : "_source.obj1.obj2" 
        }
    }
}

it's important to understand the difference between doc['my_field'].value and _source.my_field. the first, using the doc keyword, will cause the terms for that field to be loaded to memory (cached), which will result in faster execution, but more memory consumption. also, the doc[...] notation only allows for simple valued fields (can't return a JSON object from it) and makes sense only on non-analyzed or single term based fields. the _source on the other hand causes the source to be loaded, parsed, and then only the relevant part of the JSON is returned

the field used for facet calculations must be of type numeric, date/time or be analyzed as a single token - see the mapping guide for details on the analysis process

you can give the facet a custom name and return multiple facets in one request

request body

parameters

URI request

parameters

query element

filter element

from and size, pagination

indices and types

sorting

missing numeric fields

missing mapping for field

GeoDistance

script based sorting

scores

note

fields

partial

script fields

facets