Keeping A Single Elasticsearch Instance Green
For testing and low-value data I run a single Elasticsearch instance.
This indices quickly go “yellow” because of unallocated shards.
Set replication to zero to prevent this.
Use curl on the command line to disable replicas for all indices:
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/*/_settings -d '{ "index" : { "number_of_replicas" : 0 } }'
The above will apply the setting to all existing indices, but not the future ones. Create a template to apply automatically to all newly created indices:
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_template/disablereplication -d '{"template":"*", "order":1, "settings":{"number_of_replicas":0}}'
Background
When I add data to my Elasticsearch via logstash, the system goes yellow:
# curl -X GET http://localhost:9200/_cat/indices?v health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open logstash-2018.04.05 U4iGNmGMR6uDz10vlr1Q3w 5 1 9 0 41.7kb 41.7kb green open .kibana Bv3f9WPET6e5Z4YXrAryrg 1 0 4 0 24.1kb 24.1kb yellow open logstash-2018.09.05 Imy0m6s3TkGb7V6xjz-1Dw 5 1 1 0 9kb 9kb yellow open logstash-2017.09.18 6CMD0EkDQ82j0-Hl24wTWQ 5 1 8 0 33.6kb 33.6kb yellow open logstash-2018.09.03 P2_Uqy8HQJuUqlvHWilucA 5 1 602 0 123.8kb 123.8kb yellow open logstash-2017.02.12 jL9KEVDBQkq_tUn3X5TpRw 5 1 207 0 68.2kb 68.2kb
Notice how the indices are yellow? We can check why:
# curl -X GET http://localhost:9200/_cluster/health?pretty
{
"cluster_name" : "soc-demo",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 26,
"active_shards" : 26,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 25,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.98039215686274
}
Notice the 25 unassigned shards? Let’s see why this is:
# curl -X GET http://localhost:9200/_cluster/allocation/explain?pretty
{
"index" : "logstash-2018.04.05",
"shard" : 2,
"primary" : false,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "INDEX_CREATED",
"at" : "2018-09-09T12:47:36.111Z",
"last_allocation_status" : "no_attempt"
},
"can_allocate" : "no",
"allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
"node_allocation_decisions" : [
{
"node_id" : "LwXDpjSoRl2EmRUHOHAl6w",
"node_name" : "elk-node-one",
"transport_address" : "1.2.3.4:9300",
"node_attributes" : {
"ml.machine_memory" : "8654823424",
"xpack.installed" : "true",
"ml.max_open_jobs" : "20",
"ml.enabled" : "true"
},
"node_decision" : "no",
"weight_ranking" : 1,
"deciders" : [
{
"decider" : "same_shard",
"decision" : "NO",
"explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[logstash-2018.04.05][2], node[LwXDpjSoRl2EmRUHOHAl6w], [P], s[STARTED], a[id=HtX67XZgR96REHSQZRcd1w]]"
}
]
}
]
}
That is the explanation: “the shard cannot be allocated to the same node on which a copy of the shard already exists”. Elasticsearch by default expects to replicate data across multiple nodes. In this test configuration, there is only one node.
So, I disable replication, either as a one-off or permanently (see above for the commands).
We can now check that the indices are all green:
# curl -X GET http://localhost:9200/_cat/indices?v health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open logstash-2018.04.05 U4iGNmGMR6uDz10vlr1Q3w 5 0 9 0 42.2kb 42.2kb green open .kibana Bv3f9WPET6e5Z4YXrAryrg 1 0 4 0 24.1kb 24.1kb green open logstash-2018.09.05 Imy0m6s3TkGb7V6xjz-1Dw 5 0 1 0 9.2kb 9.2kb green open logstash-2017.09.18 6CMD0EkDQ82j0-Hl24wTWQ 5 0 8 0 34kb 34kb green open logstash-2018.09.03 P2_Uqy8HQJuUqlvHWilucA 5 0 602 0 124.3kb 124.3kb green open logstash-2017.02.12 jL9KEVDBQkq_tUn3X5TpRw 5 0 207 0 68.7kb 68.7kb