Elasticsearch
0 likes1,814 views
Elasticsearch 1.1.0 includes several new features and improvements such as new aggregation types like cardinality and percentiles, significant terms aggregation, and improvements to terms and multi-field search. It also includes breaking changes to configuration, multi-fields, stopwords, and return values. New features for aggregations include bucketing and metrics aggregations as well as the ability to add sub-aggregations. Backup and restore capabilities were added through repositories and snapshots. The tribe feature allows federation across multiple clusters.
![{
"aggs" : {
"price_ranges" : {
"range" : {
"field" : "price",
"ranges" : [
{ "to" : 50 },
{ "from" : 100 }
]
},
"aggs" : {
"price_stats" : {
"stats" : { "field" : "price" }
}
}
}
}
}
{
"aggregations": {
"price_ranges" : {
"buckets": [
{
"to": 50,
"doc_count": 2,
"price_stats": {
"count": 2,
"min": 20,
"max": 47,
"avg": 33.5,
"sum": 67
}
}, …
]
}
}
}](https://image.slidesharecdn.com/elasticsearch-140403100432-phpapp01/85/Elasticsearch-11-320.jpg)
![SIGNIFICANT_TERMS
{
"query" : {
"terms" : {
"force" : [ "BritishTransport Police" ]
}
},
"aggregations" : {
"significantCrimeTypes" : {
"significant_terms" : { "field" : "crime_type" }
}
}
}
An aggregation that identifies terms that are significant rather than merely popular in a result set.
Significance is related to the changes in document frequency observed between everyday use in the
corpus and frequency observed in the result set.
1.1.0
{
"aggregations" : {
"significantCrimeTypes" : {
"doc_count": 47347,
"buckets" : [
{
"key": "Bicycle theft",
"doc_count": 3640,
"score": 0.371235374214817,
"bg_count": 66799
}, …
]
}
}
}](https://image.slidesharecdn.com/elasticsearch-140403100432-phpapp01/85/Elasticsearch-14-320.jpg)
More Related Content
Elasticsearch
- 1. ELASTICSEARCH What’s new since 0.90? techtalk @ ferret
- 2. • Latest stable release: Elasticsearch 1.1.0 • Released: 25.03.2014 • Based on Lucene 4.6.1
- 3. BREAKING CHANGES in versions 1.x
- 4. CONFIGURATION • The cluster.routing.allocation settings (disable_allocation, disable_new_allocation and disable_replica_location) have been replaced by the single setting: cluster.routing.allocation.enable: all|primaries|new_primaries| none • Elasticsearch on 64 bit Linux now uses mmapfs by default. Make sure that you set MAX_MAP_COUNT to a sufficiently high number. The RPM and Debian packages default this value to 262144.
- 5. MULTI-FIELDS Existing multi-fields will be upgraded to the new format automatically. "title": { "type": "multi_field", "fields": { "title": { "type": "string" }, "raw": { "type":“string", "index": "not_analyzed" } } } "title": { "type": "string", "fields": { "raw": { "type":“string", "index": "not_analyzed" } } }
- 6. STOPWORDS • Previously, the standard and pattern analyzers used the list of English stopwords by default, which caused some hard to debug indexing issues. • Now they are set to use the empty stopwords list (ie _none_) instead.
- 7. RETURNVALUES • The ok return value has been removed from all response bodies as it added no useful information. • The found, not_found and exists return values have been unified as found on all relevant APIs. • Field values, in response to the fields parameter, are now always returned as arrays. Metadata fields are always returned as scalars. • The analyze API no longer supports the text response format, but does support JSON andYAML.
- 8. DEPRECATIONS • Per-document boosting with the _boost field has been removed.You can use the function_score instead. • The custom_score and custom_boost_score is no longer supported. You can use function_score instead. • The field query has been removed. Use the query_string query instead. • The path parameter in mappings has been deprecated. Use the copy_to parameter instead.
- 10. AGGREGATIONTYPES • Bucketing aggregations Aggregations that build buckets, where each bucket is associated with a key and a document criterion. ! Examples: range, terms, histogram ! Bucketing aggregations can have sub-aggregations (bucketing or metric). The sub-aggregations will be computed for the buckets which their parent aggregation generates. • Metrics aggregations Aggregations that keep track and compute metrics over a set of documents. ! Examples: min, max, stats
- 11. { "aggs" : { "price_ranges" : { "range" : { "field" : "price", "ranges" : [ { "to" : 50 }, { "from" : 100 } ] }, "aggs" : { "price_stats" : { "stats" : { "field" : "price" } } } } } } { "aggregations": { "price_ranges" : { "buckets": [ { "to": 50, "doc_count": 2, "price_stats": { "count": 2, "min": 20, "max": 47, "avg": 33.5, "sum": 67 } }, … ] } } }
- 12. CARDINALITY The cardinality aggregation is a metric aggregation that allows to compute approximate unique counts based on the HyperLogLog++ algorithm which has the nice properties of both being close to accurate on low cardinalities and having fixed memory usage so that estimating high cardinalities doesn't blow up memory. { "aggs" : { "author_count" : { "cardinality" : { "field" : "author" } } } }
- 13. PERCENTILES A percentiles aggregation would allow to compute (approximate) values of arbitrary percentiles based on the t-digest algorithm. Computing exact percentiles is not reasonably feasible as it would require shards to stream all values to the node that coordinates search execution, which could be gigabytes on a high-cardinality field. 1.1.0 { "aggs" : { "load_time_outlier" : { "percentiles" : { "field" : "load_time" } } } } { ... "aggregations": { "load_time_outlier": { "1.0": 15, "5.0": 20, "25.0": 23, "50.0": 25, "75.0": 29, "95.0": 60, "99.0": 150 } } }
- 14. SIGNIFICANT_TERMS { "query" : { "terms" : { "force" : [ "BritishTransport Police" ] } }, "aggregations" : { "significantCrimeTypes" : { "significant_terms" : { "field" : "crime_type" } } } } An aggregation that identifies terms that are significant rather than merely popular in a result set. Significance is related to the changes in document frequency observed between everyday use in the corpus and frequency observed in the result set. 1.1.0 { "aggregations" : { "significantCrimeTypes" : { "doc_count": 47347, "buckets" : [ { "key": "Bicycle theft", "doc_count": 3640, "score": 0.371235374214817, "bg_count": 66799 }, … ] } } }
- 16. TERMS AGGREGATION • Before 1.1.0 terms aggregations return up to size terms, so the way to get all matching terms back was to set size to an arbitrary high number that would be larger than the number of unique terms. ! • Since version 1.1.0 to get ALL terms just set size=0
- 17. MULTI-FIELD SEARCH • The multi_match query now supports three types of execution: • best_fields (field-centric, default) Find the field that best matches the query string. Useful for finding a single concept like “full text search” in either the title or the body field. ! • most_fields (field-centric) Find all matching fields and add up their scores. Useful for matching against multi-fields, where the same text has been analyzed in different ways to improve the relevance score: with/without stemming, shingles, edge-ngrams etc. ! • cross_fields (term-centric) New execution mode which looks for each term in any of the listed fields. Useful for documents whose identifying features are spread across multiple fields, such as first_name and last_name, and supports the minimum_should_match operator in a more natural way than the other two modes.
- 18. CAT API since version 1.0.0
- 19. JSON is great… for computers. Human eyes, especially when looking at an ssh terminal, need compact and aligned text.The cat API aims to meet this need. $ curl 'localhost:9200/_cat/nodes?h=ip,port,heapPercent,name' 192.168.56.40 9300 40.3 Captain Universe 192.168.56.20 9300 15.3 Kaluu 192.168.56.50 9300 17.0Yellowjacket 192.168.56.10 9300 12.3 Remy LeBeau 192.168.56.30 9300 43.9 Ramsey, Doug
- 20. TRIBE NODES since version 1.0.0
- 21. The tribes feature allows a tribe node to act as a federated client across multiple clusters. tribe: t1: cluster.name: cluster_one t2: cluster.name: cluster_two elasticsearch.yml The merged global cluster state means that almost all operations work in the sam way as a single cluster: distributed search, suggest, percolation, indexing, etc. ! However, there are a few exceptions: • The merged view cannot handle indices with the same name in multiple cluster • Master level read operations (eg Cluster State, Cluster Health) will automati execute with a local flag set to true since there is no master. • Master level write operations (eg Create Index) are not allowed.These should performed on a single cluster.
- 22. BACKUP & RESTORE since version 1.0.0
- 23. REPOSITORIES $ curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{ "type": "fs", "settings": { "location": "/mount/backups/my_backup", "compress": true }}' Before any snapshot or restore operation can be performed a snapshot repository should be registered in Elasticsearch. Supported repository types: • fs (filesystem) • S3 • HDFS (Hadoop) • Azure
- 24. SNAPSHOTS $ curl -XPUT "localhost:9200/_snapshot/my_backup/snapshot_1" -d '{ "indices": "index_1,index_2" }' A repository can contain multiple snapshots of the same cluster. Snapshot are identified by unique names within the cluster. • The index snapshot process is incremental. • Only one snapshot process can be executed in the cluster at any time. • Snapshotting process is executed in non-blocking fashion
- 25. RESTORE $ curl -XPOST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -d '{ "indices": "index_1,index_2", "rename_pattern": "index_(.+)", "rename_replacement": "restored_index_$1" }' A snapshot can be restored using the following command: • The restore operation can be performed on a functioning cluster. • An existing index can be only restored if it’s closed. • The restored persistent settings are added to the existing persistent settings.
- 26. ELASTICSEARCH-PY Official low-level client for Elasticsearch
- 27. Features: • translating basic Python data types to and from json (datetimes are not decoded for performance reasons) • configurable automatic discovery of cluster nodes • persistent connections • load balancing (with pluggable selection strategy) across all available nodes • failed connection penalization (time based - failed connections won’t be retried until a timeout is reached) • thread safety • pluggable architecture Versioning: • There are two branches - master and 0.4. Master branch is used to track all the changes for Elasticsearch 1.0 and beyond whereas 0.4 tracks Elasticsearch 0.90.