Making your elastic cluster perform - Jettro Coenradie - Codemotion Amsterdam 2016
0 likes714 views
The document provides a comprehensive guide on optimizing performance for an Elasticsearch cluster, covering installation, configuration, indexing, and querying strategies. It highlights key considerations such as hardware selection, index design, and mapping while offering practical command examples for configuration and data management. Additionally, it emphasizes monitoring tools and logging settings to enhance cluster visibility and performance troubleshooting.
![/ETC/ELASTICSEARCH/ELASTICSEARCH.YML
cluster.name: playground
node.name: node-1
discovery.zen.ping.unicast.hosts: ["node-1", "node-2", "node-3"]
discovery.zen.minimum_master_nodes: 2
path.repo: /opt/es_snapshots/
script.inline: true](https://image.slidesharecdn.com/tqstjv3jqqolpl6bkzyo-signature-97ac0cba55d83fa6f2a2f253a387270677bf11da9177d0b7d555fb911fc34ba2-poli-160526084215/85/Making-your-elastic-cluster-perform-Jettro-Coenradie-Codemotion-Amsterdam-2016-18-320.jpg)
![curl -XGET "http://localhost:9200/meetups/_search" -d'
{
"query": {
"bool": {
"must": [
{
"match": {
"venue.city": "amsterdam"
}
},
{
"match": {
"description": "elasticsearch"
}
}
]
}}}'](https://image.slidesharecdn.com/tqstjv3jqqolpl6bkzyo-signature-97ac0cba55d83fa6f2a2f253a387270677bf11da9177d0b7d555fb911fc34ba2-poli-160526084215/85/Making-your-elastic-cluster-perform-Jettro-Coenradie-Codemotion-Amsterdam-2016-58-320.jpg)
![curl -XGET "http://localhost:9200/meetups/_search" -d'
{
"query": {
"bool": {
"must": [
{
"match": {
"description": "elasticsearch"
}
}
],
"filter": {
"term": {
"venue.city.raw": "Amsterdam"
}
}
}}}'](https://image.slidesharecdn.com/tqstjv3jqqolpl6bkzyo-signature-97ac0cba55d83fa6f2a2f253a387270677bf11da9177d0b7d555fb911fc34ba2-poli-160526084215/85/Making-your-elastic-cluster-perform-Jettro-Coenradie-Codemotion-Amsterdam-2016-60-320.jpg)
![curl -XGET "http://localhost:9200/meetups/_search" -d'
{
"_source": {
"include": ["venue.*", "group.name", "name"]
},
"query": {
"simple_query_string": {
"query": "elastic OR elasticsearch"
}
}
}'](https://image.slidesharecdn.com/tqstjv3jqqolpl6bkzyo-signature-97ac0cba55d83fa6f2a2f253a387270677bf11da9177d0b7d555fb911fc34ba2-poli-160526084215/85/Making-your-elastic-cluster-perform-Jettro-Coenradie-Codemotion-Amsterdam-2016-63-320.jpg)
!["buckets": [
{
"key": "Amsterdam",
"doc_count": 8
},
{
"key": "Ede",
"doc_count": 1
},
{
"key": "Leidschendam",
"doc_count": 1
},
{
"key": "Rotterdam",
"doc_count": 1
}
]](https://image.slidesharecdn.com/tqstjv3jqqolpl6bkzyo-signature-97ac0cba55d83fa6f2a2f253a387270677bf11da9177d0b7d555fb911fc34ba2-poli-160526084215/85/Making-your-elastic-cluster-perform-Jettro-Coenradie-Codemotion-Amsterdam-2016-68-320.jpg)
![[2016-05-11 16:25:02,105][DEBUG][index.search.slowlog.query]
[meetups-20160509113909]took[518.5micros],took_millis[0],
types[], stats[], search_type[QUERY_AND_FETCH], total_shards[1]
, source[{"size":0,"aggs":{"byCity":{"terms":{"field":
"venue.city.raw","size":10}}}}], extra_source[],](https://image.slidesharecdn.com/tqstjv3jqqolpl6bkzyo-signature-97ac0cba55d83fa6f2a2f253a387270677bf11da9177d0b7d555fb911fc34ba2-poli-160526084215/85/Making-your-elastic-cluster-perform-Jettro-Coenradie-Codemotion-Amsterdam-2016-82-320.jpg)
More Related Content
Similar to Making your elastic cluster perform - Jettro Coenradie - Codemotion Amsterdam 2016 (20)
Making your elastic cluster perform - Jettro Coenradie - Codemotion Amsterdam 2016
- 1. MAKING YOUR ELASTIC CLUSTER PERFORM Created by @jettroCoenradie
- 10. curl 'localhost:9200?pretty' { "name" : "Tatterdemalion", "cluster_name" : "elasticsearch", "version" : { "number" : "2.3.1", "build_hash" : "bd980929010aef404e7cb0843e61d0665269fc39", "build_timestamp" : "2016-04-04T12:25:05Z", "build_snapshot" : false, "lucene_version" : "5.5.0" }, "tagline" : "You Know, for Search" }
- 11. curl -XPOST 'localhost:9200/conferences/conference/1?pretty' -d ' { "name": "Codemotion Amsterdam", "location": "Kromhouthal" }'
- 12. THAT WAS EASY!
- 14. DESIGN YOUR CLUSTER How to install and configure? How many nodes? What hardware?
- 15. INSTALLATION Just download, unzip and run Use package manager: yum, apt Use ansible, chef or puppet
- 17. /ETC/DEFAULTS/ELASTICSEARCH # Heap size defaults to 256m min, 1g max # Set ES_HEAP_SIZE to 50% of available RAM, but no more than 31g ES_HEAP_SIZE=2g
- 18. /ETC/ELASTICSEARCH/ELASTICSEARCH.YML cluster.name: playground node.name: node-1 discovery.zen.ping.unicast.hosts: ["node-1", "node-2", "node-3"] discovery.zen.minimum_master_nodes: 2 path.repo: /opt/es_snapshots/ script.inline: true
- 19. How many nodes do I need?
- 21. Small production
- 22. Large production
- 23. What hardware do I need?
- 24. HARDWARE Prefer cores over clock speed Choose between 8-64Gb Prefer SSD
- 25. DESIGN YOUR INDICES How many shards? How many replicas? Time based indices?
- 26. What does an index look like?
- 28. How many shards do I need? Amount of docs or terms Indexing speeds Not bigger than than 50Gb Why not a lot of shards? Start small and test
- 29. How many replicas do I need?
- 30. Should I use Types?
- 31. Should I use Aliases?
- 32. Working with time based indices? Option to change shards per time period Use index templates
- 33. DESIGN YOUR MAPPING Do I need a mapping? What do analyzers do? Do I need an analyzer?
- 34. The default uses dynamic type mapping Make your mapping explicit: Date, Geo_point, long disable dynamic type mapping PUT /_settings { "index.mapper.dynamic":false }
- 35. A mapping is persistent, can only add new things.
- 36. Use multi field mapping: name PUT /conferences { "mappings": { "conference": { "properties": { "name": { "type": "string", "analyzer": "standard", "fields": { "raw": { "type": "string", "index": "not_analyzed" } } } }}}}
- 37. An analyzer creates terms out of data Has three components: Character filter - replace & with and Tokenizer - on whitespace, regexp, ngrams Filters - ascii folding, language specific, lowercase, stop words
- 38. CHOOSE THE RIGHT ANALYZER FOR THE JOB Custom using tokenizer and filters combinations Use the multi field approach for special analyzers. Do not analyze if you don't need it.
- 39. INDEXING DOCUMENTS How to improve indexing performance?
- 40. What happens when we index a document?
- 47. TIPS DISABLE OR DECREASE REFRESH RATE curl -XGET 'http://localhost:9200/meetups/_settings' -d '{ "index" : { "refresh_interval" : "-1" } }'
- 48. INDEX WITHOUT REPLICAS curl -XGET 'http://localhost:9200/meetups/_settings' -d '{ "index" : { "number_of_replicas" : 0 } }'
- 49. USE BULK Bulk request should be between 5-15Mb max Round robin requests over nodes
- 50. QUERYING DOCUMENTS How to make queries faster?
- 58. curl -XGET "http://localhost:9200/meetups/_search" -d' { "query": { "bool": { "must": [ { "match": { "venue.city": "amsterdam" } }, { "match": { "description": "elasticsearch" } } ] }}}'
- 59. How to make a query Faster?
- 60. curl -XGET "http://localhost:9200/meetups/_search" -d' { "query": { "bool": { "must": [ { "match": { "description": "elasticsearch" } } ], "filter": { "term": { "venue.city.raw": "Amsterdam" } } }}}'
- 61. Query context Filter context How well does it match? Does it match? Calculates score true/false Not-cacheable Cacheable
- 62. Use filter context if you do not need a score Don't ask for hits if you do not use them Request only the fields that you need
- 63. curl -XGET "http://localhost:9200/meetups/_search" -d' { "_source": { "include": ["venue.*", "group.name", "name"] }, "query": { "simple_query_string": { "query": "elastic OR elasticsearch" } } }'
- 64. Profile api to learn about the performance "profile": true
- 65. ANALYTICS FROM DOCUMENTS Why use not_analyzed fields?
- 66. Aggregations, maybe the reason why elasticsearch became so popular
- 67. curl -XGET "http://localhost:9200/meetups/_search" -d' { "size": 0, "aggs": { "byCity": { "terms": { "field": "venue.city.raw", "size": 10 } } } }'
- 68. "buckets": [ { "key": "Amsterdam", "doc_count": 8 }, { "key": "Ede", "doc_count": 1 }, { "key": "Leidschendam", "doc_count": 1 }, { "key": "Rotterdam", "doc_count": 1 } ]
- 70. Inverted index not suitable for aggregations
- 71. DOC_VALUES Stored on disk during indexing All fields except analyzed strings
- 72. FIELDDATA For analyzed strings Stored in the heap
- 73. MONITORING THE CLUSTER How can I see what elastic is doing? What numbers are important?
- 74. GET /_cluster/health { "cluster_name": "playground", "status": "yellow", "timed_out": false, "number_of_nodes": 1, "number_of_data_nodes": 1, "active_primary_shards": 55, "active_shards": 55, "relocating_shards": 0, "initializing_shards": 0, "unassigned_shards": 16, "delayed_unassigned_shards": 0, "number_of_pending_tasks": 0, "number_of_in_flight_fetch": 0, "task_max_waiting_in_queue_millis": 0, "active_shards_percent_as_number": 77.46478873239437 }
- 75. GET /_cluster/health?level=indices "indices": { "conferences": { "status": "yellow", "number_of_shards": 5, "number_of_replicas": 1, "active_primary_shards": 5, "active_shards": 5, "relocating_shards": 0, "initializing_shards": 0, "unassigned_shards": 5 } }
- 76. indices - field_data, filter_cache os - cpu, memory, load process - file descriptors, cpu, memory jvm - memory, garbage collection thread_pool - threads, rejected fs - disk space GET /_nodes/stats?human
- 77. _CAT API GET /_cat/health?v epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks ma 1462882362 14:12:42 playground yellow 1 1 56 56 0 0 GET /_cat/indices?v health status index pri rep docs.count docs.deleted store.size pri.store.size green open gridshore-logs-2016.01.19 5 0 1007 0 1.2mb green open .kibana 1 0 100 1 100.2kb yellow open topbeat-2016.05.04 5 1 170264 0 44.4mb green open meetups-20160509113909 1 0 11 0 67.3kb GET /_cat/fielddata?v id host ip node total aqj9L-DPR86J8CgYitcHsA 127.0.0.1 127.0.0.1 node-JC 0b
- 79. CLUSTER LOGS How to configure what is logged?
- 80. LOGGING Can be changed dynamically PUT /_cluster/settings { "transient" : { "logger.discovery" : "DEBUG" } }
- 81. SLOWLOG PUT /meetups/_settings { "index.search.slowlog.threshold.query.warn" : "10s", "index.search.slowlog.threshold.fetch.debug": "500ms", "index.indexing.slowlog.threshold.index.info": "5s" } PUT /_cluster/settings { "transient" : { "logger.index.search.slowlog" : "DEBUG", "logger.index.indexing.slowlog" : "WARN" } }
- 82. [2016-05-11 16:25:02,105][DEBUG][index.search.slowlog.query] [meetups-20160509113909]took[518.5micros],took_millis[0], types[], stats[], search_type[QUERY_AND_FETCH], total_shards[1] , source[{"size":0,"aggs":{"byCity":{"terms":{"field": "venue.city.raw","size":10}}}}], extra_source[],
- 83. QUESTIONS? Twitter: @jettroCoenradie Github: https://github.com/jettro Blog: https://amsterdam.luminis.eu/news/ Licence: http://creativecommons.org/licenses/by-nc-sa/3.0/