Re: a problem about elasticsearch

Vincent Praveen <vincent2...@...>

Hi Jason, 

We are also facing the same issue and we want to find out the root cause for this these are the steps we do 

We ingest 1000 edges in a single commit, I understand the error comes on the ES side.  We assume that the ES cannot complete the indexing of the 1000 inserted records in 30 seconds or 1000 record batch size is too large for it to process ( Note : we always had 1000 edge batches and there was no problem earlier we have about 200m+ edges inside already ) 

We want to know what could have triggered this message, and what steps happen in which order so we can dig deeper. 

1. After we ingest the 1000 edges and run commit, ES takes over and runs the index if this step fails will the indexing not happen at all for these 1000 records ? 
2. We noticed a recent slowness in our ingestion speed, could this be due to this ES index issue , if so anything we can do to overcome it ? 
3. For each batch commit, does the JG wait for the ES to complete the indexing before moving on to the next batch? 
4. We have multiple error messages in our logs and want to know whats the impact on the index or any possible data loss?


On Wednesday, August 8, 2018 at 10:07:03 PM UTC+8 Jason Plurad wrote:
The data is still stored in the storage backend (Cassandra, HBase). You could run a reindex operation to get ES populated with the data after addressing some of the possible data overload situations that Ted mentioned.

See also this thread from the Elasticsearch forum:

On Wednesday, July 25, 2018 at 11:08:26 AM UTC-4, Ted Wilmes wrote:
Hi jcbms,
This sort of thing usually means you're overloading your Elasticsearch server(s). Perhaps you're committing too much in a single transaction or you do not have enough resources to support the load you're placing on the ES server? I'd suggest watching the ES metrics and turning down the concurrency of your load and/or batch sizes, or perhaps expanding your ES resources.


On Wednesday, July 25, 2018 at 1:37:00 AM UTC-5, jcbms wrote:
I want to know if this problem will lost data? and how to prevent it  

在 2018年7月25日星期三 UTC+8下午2:30:27,jcbms写道:
janusgraph : 0.2.0
elasticsearch: 6.1.1

can you tell me why this happen?
06:21:01,169 ERROR ElasticSearchIndex:604 - Failed to execute bulk Elasticsearch mutation
java.lang.RuntimeException: error while performing request
at org.elasticsearch.client.RestClient$SyncResponseListener.get(
at org.elasticsearch.client.RestClient.performRequest(
at org.elasticsearch.client.RestClient.performRequest(
at org.janusgraph.diskstorage.indexing.IndexTransaction$
at org.janusgraph.diskstorage.indexing.IndexTransaction$
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(
at org.janusgraph.diskstorage.util.BackendOperation.execute(
at org.janusgraph.diskstorage.indexing.IndexTransaction.flushInternal(
at org.janusgraph.diskstorage.indexing.IndexTransaction.commit(
at org.janusgraph.diskstorage.BackendTransaction.commitIndexes(
at org.janusgraph.graphdb.database.StandardJanusGraph.commit(
at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(
at DNSSubmit.submit(
at LoadData$
at java.util.concurrent.ThreadPoolExecutor.runWorker(
at java.util.concurrent.ThreadPoolExecutor$

Join to automatically receive all group messages.