The data is still stored in the storage backend (Cassandra, HBase). You could run a reindex operation to get ES populated with the data after addressing some of the possible data overload situations that Ted mentioned.

See also this thread from the Elasticsearch forum:
https://discuss.elastic.co/t/restclient-java-lang-runtimeexception-error-while-performing-request/81078

On Wednesday, July 25, 2018 at 11:08:26 AM UTC-4, Ted Wilmes wrote:

Hi jcbms,

This sort of thing usually means you're overloading your Elasticsearch server(s). Perhaps you're committing too much in a single transaction or you do not have enough resources to support the load you're placing on the ES server? I'd suggest watching the ES metrics and turning down the concurrency of your load and/or batch sizes, or perhaps expanding your ES resources.

--Ted

On Wednesday, July 25, 2018 at 1:37:00 AM UTC-5, jcbms wrote:

I want to know if this problem will lost data? and how to prevent it  

在 2018年7月25日星期三 UTC+8下午2:30:27，jcbms写道：

janusgraph : 0.2.0

elasticsearch: 6.1.1

can you tell me why this happen?

06:21:01,169 ERROR ElasticSearchIndex:604 - Failed to execute bulk Elasticsearch mutation

java.lang.RuntimeException: error while performing request

at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:681)

at org.elasticsearch.client.RestClient.performRequest(RestClient.java:219)

at org.elasticsearch.client.RestClient.performRequest(RestClient.java:191)

at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:320)

at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:249)

at org.janusgraph.diskstorage.es.ElasticSearchIndex.mutate(ElasticSearchIndex.java:601)

at org.janusgraph.diskstorage.indexing.IndexTransaction$1.call(IndexTransaction.java:160)

at org.janusgraph.diskstorage.indexing.IndexTransaction$1.call(IndexTransaction.java:157)

at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)

at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)

at org.janusgraph.diskstorage.indexing.IndexTransaction.flushInternal(IndexTransaction.java:157)

at org.janusgraph.diskstorage.indexing.IndexTransaction.commit(IndexTransaction.java:138)

at org.janusgraph.diskstorage.BackendTransaction.commitIndexes(BackendTransaction.java:141)

at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:751)

at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1374)

at DNSSubmit.submit(DNSSubmit.java:104)

at LoadData$4.run(LoadData.java:160)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Hi Jason, 

We are also facing the same issue and we want to find out the root cause for this these are the steps we do 

We ingest 1000 edges in a single commit, I understand the error comes on the ES side.  We assume that the ES cannot complete the indexing of the 1000 inserted records in 30 seconds or 1000 record batch size is too large for it to process ( Note : we always had 1000 edge batches and there was no problem earlier we have about 200m+ edges inside already ) 

We want to know what could have triggered this message, and what steps happen in which order so we can dig deeper. 

Questions

1. After we ingest the 1000 edges and run commit, ES takes over and runs the index if this step fails will the indexing not happen at all for these 1000 records ? 

2. We noticed a recent slowness in our ingestion speed, could this be due to this ES index issue , if so anything we can do to overcome it ? 

3. For each batch commit, does the JG wait for the ES to complete the indexing before moving on to the next batch? 

4. We have multiple error messages in our logs and want to know whats the impact on the index or any possible data loss?

Regards,

Vincent 

On Wednesday, August 8, 2018 at 10:07:03 PM UTC+8 Jason Plurad wrote:

The data is still stored in the storage backend (Cassandra, HBase). You could run a reindex operation to get ES populated with the data after addressing some of the possible data overload situations that Ted mentioned.

See also this thread from the Elasticsearch forum:
https://discuss.elastic.co/t/restclient-java-lang-runtimeexception-error-while-performing-request/81078

On Wednesday, July 25, 2018 at 11:08:26 AM UTC-4, Ted Wilmes wrote:

Hi jcbms,

This sort of thing usually means you're overloading your Elasticsearch server(s). Perhaps you're committing too much in a single transaction or you do not have enough resources to support the load you're placing on the ES server? I'd suggest watching the ES metrics and turning down the concurrency of your load and/or batch sizes, or perhaps expanding your ES resources.

--Ted

On Wednesday, July 25, 2018 at 1:37:00 AM UTC-5, jcbms wrote:

I want to know if this problem will lost data? and how to prevent it  

在 2018年7月25日星期三 UTC+8下午2:30:27，jcbms写道：

janusgraph : 0.2.0

elasticsearch: 6.1.1

can you tell me why this happen?

06:21:01,169 ERROR ElasticSearchIndex:604 - Failed to execute bulk Elasticsearch mutation

java.lang.RuntimeException: error while performing request

at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:681)

at org.elasticsearch.client.RestClient.performRequest(RestClient.java:219)

at org.elasticsearch.client.RestClient.performRequest(RestClient.java:191)

at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:320)

at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:249)

at org.janusgraph.diskstorage.es.ElasticSearchIndex.mutate(ElasticSearchIndex.java:601)

at org.janusgraph.diskstorage.indexing.IndexTransaction$1.call(IndexTransaction.java:160)

at org.janusgraph.diskstorage.indexing.IndexTransaction$1.call(IndexTransaction.java:157)

at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)

at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)

at org.janusgraph.diskstorage.indexing.IndexTransaction.flushInternal(IndexTransaction.java:157)

at org.janusgraph.diskstorage.indexing.IndexTransaction.commit(IndexTransaction.java:138)

at org.janusgraph.diskstorage.BackendTransaction.commitIndexes(BackendTransaction.java:141)

at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:751)

at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1374)

at DNSSubmit.submit(DNSSubmit.java:104)

at LoadData$4.run(LoadData.java:160)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgra...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/d72cab3e-1d52-493c-87c6-6569fb1db54bn%40googlegroups.com.

Hi Vincent,

Recently I faced a lot of mutation errors. The simple solution is just upgrade your Janusgraph to v0.5.2.

Hope this helps you.

Thanks,
Abhay

On Tue, 18 Aug 2020 at 14:04, Vincent Praveen <vinc...@...> wrote:

Hi Jason, 

We are also facing the same issue and we want to find out the root cause for this these are the steps we do 

We ingest 1000 edges in a single commit, I understand the error comes on the ES side.  We assume that the ES cannot complete the indexing of the 1000 inserted records in 30 seconds or 1000 record batch size is too large for it to process ( Note : we always had 1000 edge batches and there was no problem earlier we have about 200m+ edges inside already ) 

We want to know what could have triggered this message, and what steps happen in which order so we can dig deeper. 

Questions

1. After we ingest the 1000 edges and run commit, ES takes over and runs the index if this step fails will the indexing not happen at all for these 1000 records ? 

2. We noticed a recent slowness in our ingestion speed, could this be due to this ES index issue , if so anything we can do to overcome it ? 

3. For each batch commit, does the JG wait for the ES to complete the indexing before moving on to the next batch? 

4. We have multiple error messages in our logs and want to know whats the impact on the index or any possible data loss?

Regards,

Vincent 

On Wednesday, August 8, 2018 at 10:07:03 PM UTC+8 Jason Plurad wrote:

The data is still stored in the storage backend (Cassandra, HBase). You could run a reindex operation to get ES populated with the data after addressing some of the possible data overload situations that Ted mentioned.

See also this thread from the Elasticsearch forum:
https://discuss.elastic.co/t/restclient-java-lang-runtimeexception-error-while-performing-request/81078

On Wednesday, July 25, 2018 at 11:08:26 AM UTC-4, Ted Wilmes wrote:

Hi jcbms,

This sort of thing usually means you're overloading your Elasticsearch server(s). Perhaps you're committing too much in a single transaction or you do not have enough resources to support the load you're placing on the ES server? I'd suggest watching the ES metrics and turning down the concurrency of your load and/or batch sizes, or perhaps expanding your ES resources.

--Ted

On Wednesday, July 25, 2018 at 1:37:00 AM UTC-5, jcbms wrote:

I want to know if this problem will lost data? and how to prevent it  

在 2018年7月25日星期三 UTC+8下午2:30:27，jcbms写道：

janusgraph : 0.2.0

elasticsearch: 6.1.1

can you tell me why this happen?

06:21:01,169 ERROR ElasticSearchIndex:604 - Failed to execute bulk Elasticsearch mutation

java.lang.RuntimeException: error while performing request

at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:681)

at org.elasticsearch.client.RestClient.performRequest(RestClient.java:219)

at org.elasticsearch.client.RestClient.performRequest(RestClient.java:191)

at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:320)

at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:249)

at org.janusgraph.diskstorage.es.ElasticSearchIndex.mutate(ElasticSearchIndex.java:601)

at org.janusgraph.diskstorage.indexing.IndexTransaction$1.call(IndexTransaction.java:160)

at org.janusgraph.diskstorage.indexing.IndexTransaction$1.call(IndexTransaction.java:157)

at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)

at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)

at org.janusgraph.diskstorage.indexing.IndexTransaction.flushInternal(IndexTransaction.java:157)

at org.janusgraph.diskstorage.indexing.IndexTransaction.commit(IndexTransaction.java:138)

at org.janusgraph.diskstorage.BackendTransaction.commitIndexes(BackendTransaction.java:141)

at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:751)

at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1374)

at DNSSubmit.submit(DNSSubmit.java:104)

at LoadData$4.run(LoadData.java:160)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/d72cab3e-1d52-493c-87c6-6569fb1db54bn%40googlegroups.com.

Hi Abhay, 

Thanks for your suggestion, But just curious is not this issue related to ES rather than JG, how come upgrading the JG will help with ES handling of indexes and mutation. 

Regards,

Vincent

On Wednesday, August 19, 2020 at 2:21:29 AM UTC+8 ab...@... wrote:

Hi Vincent,

Recently I faced a lot of mutation errors. The simple solution is just upgrade your Janusgraph to v0.5.2.

Hope this helps you.

Thanks,
Abhay

On Tue, 18 Aug 2020 at 14:04, Vincent Praveen <vinc...@...> wrote:

Hi Jason, 

We are also facing the same issue and we want to find out the root cause for this these are the steps we do 

We ingest 1000 edges in a single commit, I understand the error comes on the ES side.  We assume that the ES cannot complete the indexing of the 1000 inserted records in 30 seconds or 1000 record batch size is too large for it to process ( Note : we always had 1000 edge batches and there was no problem earlier we have about 200m+ edges inside already ) 

We want to know what could have triggered this message, and what steps happen in which order so we can dig deeper. 

Questions

1. After we ingest the 1000 edges and run commit, ES takes over and runs the index if this step fails will the indexing not happen at all for these 1000 records ? 

2. We noticed a recent slowness in our ingestion speed, could this be due to this ES index issue , if so anything we can do to overcome it ? 

3. For each batch commit, does the JG wait for the ES to complete the indexing before moving on to the next batch? 

4. We have multiple error messages in our logs and want to know whats the impact on the index or any possible data loss?

Regards,

Vincent 

On Wednesday, August 8, 2018 at 10:07:03 PM UTC+8 Jason Plurad wrote:

The data is still stored in the storage backend (Cassandra, HBase). You could run a reindex operation to get ES populated with the data after addressing some of the possible data overload situations that Ted mentioned.

See also this thread from the Elasticsearch forum:
https://discuss.elastic.co/t/restclient-java-lang-runtimeexception-error-while-performing-request/81078

On Wednesday, July 25, 2018 at 11:08:26 AM UTC-4, Ted Wilmes wrote:

Hi jcbms,

This sort of thing usually means you're overloading your Elasticsearch server(s). Perhaps you're committing too much in a single transaction or you do not have enough resources to support the load you're placing on the ES server? I'd suggest watching the ES metrics and turning down the concurrency of your load and/or batch sizes, or perhaps expanding your ES resources.

--Ted

On Wednesday, July 25, 2018 at 1:37:00 AM UTC-5, jcbms wrote:

I want to know if this problem will lost data? and how to prevent it  

在 2018年7月25日星期三 UTC+8下午2:30:27，jcbms写道：

janusgraph : 0.2.0

elasticsearch: 6.1.1

can you tell me why this happen?

06:21:01,169 ERROR ElasticSearchIndex:604 - Failed to execute bulk Elasticsearch mutation

java.lang.RuntimeException: error while performing request

at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:681)

at org.elasticsearch.client.RestClient.performRequest(RestClient.java:219)

at org.elasticsearch.client.RestClient.performRequest(RestClient.java:191)

at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:320)

at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:249)

at org.janusgraph.diskstorage.es.ElasticSearchIndex.mutate(ElasticSearchIndex.java:601)

at org.janusgraph.diskstorage.indexing.IndexTransaction$1.call(IndexTransaction.java:160)

at org.janusgraph.diskstorage.indexing.IndexTransaction$1.call(IndexTransaction.java:157)

at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)

at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)

at org.janusgraph.diskstorage.indexing.IndexTransaction.flushInternal(IndexTransaction.java:157)

at org.janusgraph.diskstorage.indexing.IndexTransaction.commit(IndexTransaction.java:138)

at org.janusgraph.diskstorage.BackendTransaction.commitIndexes(BackendTransaction.java:141)

at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:751)

at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1374)

at DNSSubmit.submit(DNSSubmit.java:104)

at LoadData$4.run(LoadData.java:160)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/d72cab3e-1d52-493c-87c6-6569fb1db54bn%40googlegroups.com.

Hi Anshul,

I am facing same issue? Did you got any solution for the issue?

Thanks,
Anjani