paskvilwiki

up: databases & search

Elasticsearch

last updated 2013-12

update 2026 - written for Elasticsearch 0.20 (2013). the ideas hold, but a lot of the API doesn't: mapping types (_type, /index/type/id URLs) are gone since 8.0, facets were replaced by aggregations in 2.0, and the curl calls now need -H 'Content-Type: application/json' for a body (6.0+). check the current docs before copying anything

i was really surprised by Elasticsearch (ES further on) - the simplicity of setup and configuration, and its powers and options

installation

download, unpack, and run es/bin/elasticsearch. yes, that's it. amazing, isn't it?

what you get

after the above 30-sec setup, you have a search engine running on http://localhost:9200/, with automatic sharding (unlike with other systems, ES is sharded always - even on a single machine), replication, and much much more

a few highlights:

  • ES sports a neat RESTful API that communicates (almost) entirely in JSON
  • ES is schemaless, unless you want it to be
  • you can hint ES on many tasks - e.g. hint what shards to search in, etc.
  • indices are created on the fly, no need to precreate (yes, might be tougher to find a bug, but installation of a new system is a breeze)
  • you can specify what indices to search, or what document types, you can search over a group or all or just one
  • documents are versioned; also, adding a document with the same ID doesn't replace the old document - this might or might not be what you want

indexing

official documentation

example

let's start with an add-get example:

# lets add (type) _user_ to _twitter_ index, with ID _kimchy_
$ curl -XPUT 'http://localhost:9200/twitter/user/kimchy' -d '{ "name" : "Shay Banon" }'
> {"ok":true,"_index":"twitter","_type":"user","_id":"kimchy","_version":2}

$ curl -XGET 'http://localhost:9200/twitter/user/kimchy?pretty=true'
> {
>   "_index" : "twitter",
>   "_type" : "user",
>   "_id" : "kimchy",
>   "_version" : 1,
>   "exists" : true, "_source" : { "name" : "Shay Banon" }
> }

# lets add one more _user_ to _twitter_ with the same ID
$ curl -XPUT 'http://localhost:9200/twitter/user/kimchy' -d '{ "name" : "Shay Baror" }'
> {"ok":true,"_index":"twitter","_type":"user","_id":"kimchy","_version":2}

# note the increase in version number
$ curl -XGET 'http://localhost:9200/twitter/user/kimchy?pretty=true'
> {
>   "_index" : "twitter",
>   "_type" : "user",
>   "_id" : "kimchy",
>   "_version" : 2,
>   "exists" : true, "_source" : { "name" : "Shay Baror" }
> }

# now, lets search for "shay" users
$ curl -XGET 'http://localhost:9200/twitter/user/_search?q=name:shay&pretty=true'
> {
>   "took" : 491,
>   "timed_out" : false,
>   "_shards" : {
>     "total" : 5,
>     "successful" : 5,
>     "failed" : 0
>   },
>   "hits" : {
>     "total" : 1,
>     "max_score" : 0.625,
>     "hits" : [ {
>       "_index" : "twitter",
>       "_type" : "user",
>       "_id" : "kimchy",
>       "_score" : 0.625, "_source" : { "name" : "Shay Baror" }
>     } ]
>   }
> }

# to search only among all types in _twitter_ index
$ curl -XGET 'http://localhost:9200/twitter/_search?q=name:shay'

# finally, you may search all indices
$ curl -XGET 'http://localhost:9200/_search?q=name:shay'

# or just selected indices - _twitter_ and _facebook_
$ curl -XGET 'http://localhost:9200/twitter,facebook/_search?q=name:shay'

# or on all indices starting with _t_, excluding _twitter_
$ curl -XGET 'http://localhost:9200/t*,-twitter/_search?q=name:shay'

note that you get most of the useful information, and very little superfluous. of course, without the pretty=true parameter, you get the "normal" more compressed version of JSON

creating documents

you create/index documents by PUTting them to index as type with docid ID:

$ curl -XPUT 'http://localhost:9200/index/type/docid' -d '{"content":"trying out Elastic Search"}'

versioning

note that documents are versioned rather than replaced if PUT more than once

you can use the OCC in ES to make sure you're updating the document you started the update on:

$ curl -XPUT 'http://localhost:9200/twitter/tweet/1?version=1' -d '{"name":"Shay Boron"}'

this will result in a document with the same data as version 1, with the name field updated/added

note that if you're updating an old version (a new version appeared in the meantime), you'll get an error:

{"error":"VersionConflictEngineException[[twitter][3] [user][kimchy]: version conflict, current [2], provided [1]]","status":409}

it's up to you then to get the new version and update the data accordingly and PUT again

PUT-if-absent

to create a document only if it's not present in the index yet, use the create parameter (the following 2 calls are equivalent):

$ curl -XPUT 'http://localhost:9200/twitter/tweet/1?op_type=create' -d '{...}'
$ curl -XPUT 'http://localhost:9200/twitter/tweet/1/_create' -d '{...}'

automatic ID generation

you can create a document without providing an ID. note that the call is POST not PUT! (this, of course, automatically sets op_type to create)

$ curl -XPOST 'http://localhost:9200/twitter/tweet/' -d '{...}'

you'll receive the generated (UUIDv4) ID in the response:

{
    "ok" : true,
    "_index" : "twitter",
    "_type" : "tweet",
    "_id" : "6a8ca01c-7896-48e9-81cc-9f70661fcb32",
    "_version" : 1
}

indices and types

an index is automatically created if it doesn't exist. data type mapping is also automatically created/updated

indices can also be created "manually", as well as mappings

by setting action.auto_create_index to false in configuration, indices need to be created manually before use. same goes for type mapping - index.mapper.dynamic

you can also white/black-list indices by name, which are to be created automatically and manually, by setting action.auto_create_index to +aaa*,-bbb*,+ccc*,-*

routing

by default, the shard used to store the document is selected using a hash of the document's ID

you can "control" this by providing the ID used for hashing manually, using the routing parameter:

$ curl -XPOST 'http://localhost:9200/twitter/tweet?routing=kimchy' -d '{...}'

the main advantage of routing is that you can use this information later on when searching for documents, by providing the same ID:

$ curl -XGET 'http://localhost:9200/twitter/tweet/_search?routing=kimchy' -d '{"query":{...}}'

this will restrict the search only to shards that might store the documents for the given routing

deleting

official documentation

you can simply delete a document using the DELETE HTTP method:

$ curl -XDELETE 'http://localhost:9200/twitter/tweet/1'

to make sure you're deleting the document you really want to delete, you can provide the version parameter, and the delete will fail if there is a newer version of the document

note: if you used routing for creation of the document, you need to provide it for deletion also!

note: set the replication parameter to async if you want it executed asynchronously; in that case, the operation will return as soon as the document is removed on the primary shard, instead of waiting for update to all replicas

getting

official documentation

you can get a document from the index by type and ID:

$ curl -XGET 'http://localhost:9200/twitter/tweet/1'

{
    "_index" : "twitter",
    "_type" : "tweet",
    "_id" : "1", 
    "_source" : {
        "user" : "kimchy",
        "postDate" : "2009-11-15T14:12:12",
        "message" : "trying out Elastic Search"
    }
}

to simply check if the document exists, use HEAD:

$ curl -XHEAD 'http://localhost:9200/twitter/tweet/1'

get options

the GET API is realtime, that is, not affected by the refresh rate of the index (when data will become visible for search). you can disable this by setting the realtime parameter to false, or globally by setting action.get.realtime to false

you can also specify the fields parameter to get only selected fields from the document. you may also fetch sub-objects, using obj1.obj2 notation

$ curl -XGET 'http://localhost:9200/twitter/tweet/1?fields=title,content'

the type of the document is optional, and passing _all as the type for the GET call will return all documents from the index based on ID

the routing parameter is used as usual to specify the shard explicitly

multi get

official documentation

the _mget interface allows you to get multiple documents based on an index, (optionally) type and id (and possibly routing)

from various indices

$ curl 'http://localhost:9200/_mget' -d '{
    "docs" : [
        {
            "_index" : "test",
            "_type" : "type",
            "_id" : "1"
        },
        {
            "_index" : "test",
            "_type" : "type",
            "_id" : "2"
        }
    ]
}'

from same index, various types

$ curl 'http://localhost:9200/test/_mget' -d '{
    "docs" : [
        {
            "_type" : "type",
            "_id" : "1"
        },
        {
            "_type" : "type",
            "_id" : "2"
        }
    ]
}'

same index, same type

$ curl 'http://localhost:9200/test/type/_mget' -d '{ "ids" : ["1", "2"] }'

you can also specify fields to fetch:

$ curl 'http://localhost:9200/_mget' -d '{
    "docs" : [
        {
            "_index" : "test",
            "_type" : "type",
            "_id" : "1",
            "fields" : ["field1", "field2"]
        },
        {
            "_index" : "test",
            "_type" : "type",
            "_id" : "2",
            "fields" : ["field3", "field4"]
        }
    ]
}'

updating

official documentation

the operation gets the document, runs the script, and indexes back the result (also allows to delete, or ignore the operation). it uses versioning to make sure no updates have happened during the "get" and "reindex"

note: a full document reindex is still needed, just cutting down the network round-trip and using versioning to avoid conflicts

example

# create a document
$ curl -XPUT 'http://localhost:9200/test/type1/1' -d '{
    "counter" : 1,
    "tags" : ["red"]
}'

# update the counter - increment by 4
$ curl -XPOST 'http://localhost:9200/test/type1/1/_update' -d '{
    "script" : "ctx._source.counter += count",
    "params" : {
        "count" : 4
    }
}'

# add a tag (might add a duplicate, since it's just a list!)
$ curl -XPOST 'http://localhost:9200/test/type1/1/_update' -d '{
    "script" : "ctx._source.tags += tag",
    "params" : {
        "tag" : "blue"
    }
}'

# add a new field
$ curl -XPOST 'http://localhost:9200/test/type1/1/_update' -d '{
    "script" : "ctx._source.text = \"some text\""
}'

# remove a field
$ curl -XPOST 'http://localhost:9200/test/type1/1/_update' -d '{
    "script" : "ctx._source.remove(\"text\")"
}'

# delete documents with tag 'blue', or ignore/noop
$ curl -XPOST 'http://localhost:9200/test/type1/1/_update' -d '{
    "script" : "ctx._source.tags.contains(tag) ? ctx.op = \"delete\" : ctx.op = \"none\"",
    "params" : {
        "tag" : "blue"
    }
}'

# "full" document may also be POST'ed, causing the final document to be a merge of original and update
$ curl -XPOST 'http://localhost:9200/test/type1/1/_update' -d '{
    "doc" : {
        "name" : "new_name"
    }
}'

# if field does not exist, its "default" may be specified as 'upsert'
$ curl -XPOST 'http://localhost:9200/test/type1/1/_update' -d '{
    "script" : "ctx._source.counter += count",
    "params" : {
        "count" : 4
    },
    "upsert" : {
        "counter" : 1
    }
}'

update parameters

  • routing - select routing/shard
  • timeout - timeout waiting for a shard to become available
  • replication - the replication type for the delete/index operation (sync or async)
  • consistency - the write consistency of the index/delete operation
  • percolate - enables percolation and filters out which percolator queries will be executed
  • refresh - refresh the index immediately after the operation occurs, so that the updated document appears in search results immediately (might increase load and network load)
  • fields - return the relevant fields from the document updated; use _source to return the full updated source
  • retry_on_conflict - how many times to retry if there is a version conflict; default 0

official documentation

query DSL

indices management

create index

official documentation

ES provides support for multiple indices, including executing operations across several indices. each index created can have specific settings associated with it

$ curl -XPUT 'http://localhost:9200/twitter/' -d '{
    "settings" : {
        "number_of_shards" : 3,
        "number_of_replicas" : 2
    }
}'

you may also provide mappings settings (see below) when creating an index:

$ curl -XPOST 'http://localhost:9200/test' -d '{
    "settings" : {
        "number_of_shards" : 1
    },
    "mappings" : {
        "type1" : {
            "_source" : { "enabled" : false },
            "properties" : {
                "field1" : { "type" : "string", "index" : "not_analyzed" }
            }
        }
    }
}'

delete index

you can delete an index by simply

$ curl -XDELETE 'http://localhost:9200/twitter/'

note that you can specify multiple indices, or _all. if no index name is provided, all indices will be deleted!

open and close index

indices may be closed - they have no overhead (exc. keeping metadata up to date), and all read/write operations on the index are then disabled. when you open an index, it goes through the normal recovery process:

curl -XPOST 'localhost:9200/my_index/_close'
curl -XPOST 'localhost:9200/my_index/_open'

get settings

$ curl -XGET 'http://localhost:9200/twitter/_settings'

mappings

official documentation

mapping is the process of defining how a document should be mapped to the search engine, including its searchable characteristics such as which fields are searchable and if/how they are tokenized. in Elasticsearch, an index may store documents of different "mapping types". Elasticsearch allows one to associate multiple mapping definitions for each mapping type

mapping settings may be set while creating the index, or separately

the following "core types" are supported:

  • string
  • numbers - float, double, byte, short, integer, and long
  • date - all dates are UTC; format may be specified, and a long number of milliseconds since epoch can be used any time

put mapping

$ curl -XPUT 'http://localhost:9200/twitter/tweet/_mapping' -d '
{
    "tweet" : {
        "properties" : {
            "message" : {"type" : "string", "store" : "yes"}
        }
    }
}'

fields are not stored by default, just indexed (see below - common attributes)

mapping can be applied to multiple indices (passing CSV of index names), or to _all indices

when a mapping already exists under the given type, the two mapping definitions, the one already defined, and the new ones are merged. in general, if a different core type is defined, it is considered a conflict. new mapping definitions can be added to object types, and core type mapping can be upgraded to multi_field type

get mapping

$ curl -XGET 'http://localhost:9200/twitter/tweet/_mapping'

again, you can pass CSV of index names and of type names, or _all

delete mapping

$ curl -XDELETE 'http://localhost:9200/twitter/tweet/'

note, most times it makes more sense to reindex the data into a fresh index compared to deleting large chunks of it

common attributes

see the full list of field types' attributes

the following attributes are supported for all field types (exc. binary):

  • index_name - name of the field that will be stored in the index, defaults to the property/field name
  • store - yes or no to store the actual field in the index; default no (the JSON doc is stored, and the field can be extracted)
  • index - analyzed = indexed and searchable, tokenized; not_analyzed = indexed and searchable, but not tokenized; no = not searchable at all (still may be included in _all)
  • boost - boost value, default 1.0
  • null_value - null value to use when the field is not present in the document; default "do not add the field"
  • include_in_all - included in the _all field?; default yes
  • ignore_malformed - ignore malformed record? (e.g. string instead of number); default false [exc. for string, and bool]

specific attributes

date

binary

  • base64 representation of binary data, always stored and not indexed at all

example

{
    "tweet" : {
        "properties" : {
            "user" : { "type" : "string", "index" : "not_analyzed" },
            "message" : { "type" : "string", "null_value" : "na" },
            "postDate" : { "type" : "date" },
            "priority" : { "type" : "integer", "boost" : 3.0 },
            "rank" : { "type" : "float", "null_value" : 1.0 }
        }
    }
}

diagnostics

status

$ curl -XGET 'http://localhost:9200/twitter/_status'
  • recovery=true to see the status of recovery
  • snapshot=true for snapshot status

again, CSV indices or _all (the same as omitted) can be used

stats

$ curl -XGET 'http://localhost:9200/twitter/_stats'

again, CSV indices or _all (the same as omitted) can be used