Ggtcf

I have elasticsearch setup, so that it stores the data at two locations meaning in elasticsearch.yml I have

path.data: /path_one/es_data,/path_two/elasticsearch

I was hoping elasticsearch would automatically figure out where more space is available and store new incoming data wherever possible, but instead, I found it starts to crash when short of memory at any one location. So I would like to move one node from path_one to path_two.

Currently, it looks like this

ls -lha /path_one/es_data/nodes/0/indices/
drwxr-xr-x 3 elasticsearch elasticsearch 4.0K Mar 7 03:13 A4XXnhNdTwKILyeE39UosA
drwxr-xr-x 3 elasticsearch elasticsearch 4.0K Mar 7 03:13 C2BPWKL4T3-jHIfZXNKG6g
drwxr-xr-x 3 elasticsearch elasticsearch 4.0K Mar 7 03:13 c8mFFi56RAyRYNpHOUvG4g
drwxr-xr-x 6 elasticsearch elasticsearch 4.0K Mar 7 03:13 DEk-qwdnSLOHbP_-nAhSdw
drwxr-xr-x 3 elasticsearch elasticsearch 4.0K Mar 7 03:13 kV32aUcET1WrlKXWOunGhg
drwxr-xr-x 3 elasticsearch elasticsearch 4.0K Mar 7 03:13 pGmjsSJHRAiMUC5paYfjag
drwxr-xr-x 3 elasticsearch elasticsearch 4.0K Mar 7 03:13 T1k45bs2SUGHJ6dJniPjZg

ls -lha /path_two/elasticsearch/nodes/0/indices/
drwxr-xr-x 4 elasticsearch elasticsearch 4.0K Mar 7 03:13 A4XXnhNdTwKILyeE39UosA
drwxr-xr-x 4 elasticsearch elasticsearch 4.0K Mar 7 03:13 C2BPWKL4T3-jHIfZXNKG6g
drwxr-xr-x 4 elasticsearch elasticsearch 4.0K Mar 7 03:13 c8mFFi56RAyRYNpHOUvG4g
drwxr-xr-x 5 elasticsearch elasticsearch 4.0K Mar 7 03:13 DEk-qwdnSLOHbP_-nAhSdw
drwxr-xr-x 4 elasticsearch elasticsearch 4.0K Mar 7 03:13 pGmjsSJHRAiMUC5paYfjag
drwxr-xr-x 3 elasticsearch elasticsearch 4.0K Mar 7 03:13 

T1k45bs2SUGHJ6dJniPjZg
drwxr-xr-x 4 elasticsearch elasticsearch 4.0K Mar 7 03:13 XpHUz15oTbGG0Bvnf2xZsw

So my first question is, why are some of the nodes present at both locations? And my second question is whether I can just

1. stop elasticsearch


2. copy nodes over


3. restart elasticsearch

or whether I have to do more?

EDIT: I found some messages in the logfiles which look related

[2019-03-07T17:08:21,910][WARN ][o.e.c.r.a.DiskThresholdMonitor] [WU6cQ-o] high disk watermark [90%] exceeded on [WU6cQ-oTR2Ssg3LzoI4_yg][WU6cQ-o][/var/lib/elasticsearch/elasticsearch/nodes/0] free: 984.7mb[1.6%], shards will be relocated away from this node
[2019-03-07T17:08:51,944][WARN ][o.e.g.DanglingIndicesState] [WU6cQ-o] [[paper-index/XpHUz15oTbGG0Bvnf2xZsw]] can not be imported as a dangling index, as index with same name already exists in cluster metadata

so it seems like elasticsearch is trying to move indices, but can't because there are already copies of these indices at the other location? Can I just delete the copies at the location where there is more space?

An Elasticsearch instance corresponds to one node. Setting two locations in path.data doesn't mean that you have two nodes running on the same host, but that you're storing the data of your node on two locations (see documentation). Therefore, to answer your first question, it is to be expected that data of the same node is spread across locations.

As for your second question, I don't understand your process, mostly because I'm not sure that you're running multiple nodes.

From the Elastic documentation, all we know about data distribution across locations is that Elasticsearch stores the files related to the same shard in the same location.

Hope it helps