Topics

reindex job is very slow on ElasticSearch and BigTable


vamsi.lingala@...
 

we have imported around 4 billion vertices in janus graph.
We are using big table and elastic search

reindexing speed is very slow..around 2000 records per second
is there any way to speed it up?

MapReduceIndexManagement is not supported for BigTable


hadoopmarc@...
 

Thanks for reposting your issue on the janusgraph-users list!

Can you please show the entire stack trace leading to your error message?

Note that your issue might be related to:
https://github.com/JanusGraph/janusgraph/issues/2201

Marc


vamsi.lingala@...
 

Thanks a lot for your reply.
I don't get any error message.

The REINDEX step is very slow (2000/s) for the mixed index and fails after running for a few days.
Everytime indexing fails or the cluster is restarted the shards settings and replica declared for the Indices is reset to 1 (default values for es-7) and thus I have to recreate new indices after disabling old ones.

is there any better way to reindex fast?


hadoopmarc@...
 

I mean, what happens if you try to run MapReduceIndexManagement on BigTable. Apparently, you get this error message "MapReduceIndexManagement is not supported for BigTable" but I would like to see the full stack trace leading to this error message, to see where this incompatibility stems from. E.g. the code in:

https://github.com/JanusGraph/janusgraph/blob/d954ea02035d8d54b4e1bd5863d1f903e6d57844/janusgraph-hadoop/src/main/java/org/janusgraph/hadoop/MapReduceIndexManagement.java

reads:
HadoopStoreManager storeManager = (HadoopStoreManager) graph.getBackend().getStoreManager().getHadoopManager();
if (storeManager == null) {
    throw new IllegalArgumentException("Store manager class " + graph.getBackend().getStoreManagerClass() + "is not supported");
}

But this is not what you see.

Best wishes,    Marc
 


vamsi@...
 

Got the same error 

throw new IllegalArgumentException("Store manager class " + graph.getBackend().getStoreManagerClass() + "is not supported");


hadoopmarc@...
 

OK, thanks for confirming the stacktrace. You can report this behavior as an issue on https://github.com/JanusGraph/janusgraph/issues, referring to this thread. It is still not clear to me how this exception can occur, because the BigTable compatibility layer reuses the HBase backend, for which graph.getBackend().getStoreManager().getHadoopManager(); is available.

So, I am afraid there is no quick fix for your issue, unless you start debugging MapReduceIndexManagement for BigTable yourself. Maybe, simply reloading the graph is an option.


Best wishes,    Marc


vamsi.lingala@...
 

I am getting below errors while doing reindex and it stops the job.

Pls help me with this.

09:09:06.295 [Thread-12] ERROR o.j.d.k.scan.StandardScannerExecutor Unexpected error processing data: {}
org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:285) ~[janusgraph-core-0.5.1.jar:na]
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201) ~[janusgraph-core-0.5.1.jar:na]
	... 2 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-4 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893) ~[janusgraph-es-0.5.1.jar:na]
	... 4 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-4 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_275]
09:09:06.438 [Thread-13] ERROR o.j.d.k.scan.StandardScannerExecutor Unexpected error processing data: {}
org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:285) ~[janusgraph-core-0.5.1.jar:na]
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201) ~[janusgraph-core-0.5.1.jar:na]
	... 2 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893) ~[janusgraph-es-0.5.1.jar:na]
	... 4 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_275]
09:09:07.290 [Thread-10] ERROR o.j.d.k.scan.StandardScannerExecutor Unexpected error processing data: {}
org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:285) ~[janusgraph-core-0.5.1.jar:na]
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201) ~[janusgraph-core-0.5.1.jar:na]
	... 2 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893) ~[janusgraph-es-0.5.1.jar:na]
	... 4 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_275]
09:09:07.325 [Thread-14] ERROR o.j.d.k.scan.StandardScannerExecutor Unexpected error processing data: {}
org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:285) ~[janusgraph-core-0.5.1.jar:na]
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201) ~[janusgraph-core-0.5.1.jar:na]
	... 2 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-1 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893) ~[janusgraph-es-0.5.1.jar:na]
	... 4 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-1 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_275]
09:09:08.323 [Thread-11] ERROR o.j.d.k.scan.StandardScannerExecutor Unexpected error processing data: {}
org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:285) ~[janusgraph-core-0.5.1.jar:na]
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201) ~[janusgraph-core-0.5.1.jar:na]
	... 2 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893) ~[janusgraph-es-0.5.1.jar:na]
	... 4 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_275]
Exception in thread "Thread-12" org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206)
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118)
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:305)
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895)
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128)
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201)
	... 2 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-5 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893)
	... 4 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-5 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-13" org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206)
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118)
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:305)
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895)
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128)
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201)
	... 2 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-6 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893)
	... 4 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-6 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-10" org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206)
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118)
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:305)
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895)
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128)
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201)
	... 2 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-7 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893)
	... 4 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-7 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-14" org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206)
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118)
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:305)
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895)
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128)
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201)
	... 2 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-8 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893)
	... 4 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-8 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-11" org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206)
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118)
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:305)
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895)
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128)
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201)
	... 2 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-9 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893)
	... 4 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-9 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.lang.Thread.run(Thread.java:748)
09:18:46.489 [bigtable-rpc-retry-3] WARN  c.g.cloud.bigtable.grpc.io.Watchdog Found 3 stale streams and cancelled them
09:19:57.643 [Credentials-Refresh-8] INFO  c.g.c.b.g.io.OAuthCredentialsCache Refreshing the OAuth token


vamsi.lingala@...
 

I get above error while doing reindex using JanusGraphManagement


hadoopmarc@...
 

This trace says that your elasticsearch cluster becomes unresponsive. Check your elasticsearch cluster with requests like:

https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html

If the elasticsearch cluster does not recover by itself, restart it and monitor it closely during the next trial for index repair. The number of workers activated in the repair job may be too much for the size of your elasticsearch cluster.

Best wishes,    Marc


vamsi.lingala@...
 

Thanks a lot for you reply.

We changed our approach..we enabled the index and then ingested nodes..I can see data in elastic search but when I query I am getting timeout

==>------------------------------------------------------------------------------------------------
Vertex Index Name              | Type        | Unique    | Backing        | Key:           Status |
---------------------------------------------------------------------------------------------------
maidMixedIndex                 | Mixed       | false     | search         | maid:         ENABLED |
---------------------------------------------------------------------------------------------------
Edge Index (VCI) Name          | Type        | Unique    | Backing        | Key:           Status |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
Relation Index                 | Type        | Direction | Sort Key       | Order    |     Status |
---------------------------------------------------------------------------------------------------



gremlin> :remote config timeout 600000
==>Set remote timeout to 600000ms
gremlin> g.V().has("maid","8bcaa1a6-31df-4df4-b193-6b4165c4df8c").valueMap()
Evaluation exceeded the configured 'evaluationTimeout' threshold of 600000 ms or evaluation was otherwise cancelled directly for request [g.V().has("maid","8bcaa1a6-31df-4df4-b193-6b4165c4df8c").valueMap()] - try increasing the timeout with the :remote command
Type ':help' or ':h' for help.
Display stack trace? [yN]y
org.apache.tinkerpop.gremlin.jsr223.console.RemoteException: Evaluation exceeded the configured 'evaluationTimeout' threshold of 600000 ms or evaluation was otherwise cancelled directly for request [g.V().has("maid","8bcaa1a6-31df-4df4-b193-6b4165c4df8c").valueMap()] - try increasing the timeout with the :remote command
at org.apache.tinkerpop.gremlin.console.jsr223.DriverRemoteAcceptor.submit(DriverRemoteAcceptor.java:178)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:110)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:120)


hadoopmarc@...
 

Is this again for 4 billion vertices?  Do the gremlin server logs show whether the problem is with the index or with the backend?

Does this occur for any vertex? Try to retrieve the first few vertices from a table scan, hopefully these will be vertices with the maid property key:
g.V().limit(100).values('maid')

Then do
g.V().has("maid", ....).valueMap()
for the maid values found above.
Maybe this gives some hints about what is going wrong.

Best wishes,     Marc


vamsi.lingala@...
 
Edited

Thanks a lot for your reply..

I tried above solution but unfortunately it did not work.
 
On scanning through pod logs I came across below warning which led us to adding query.force-index=true in our config.
[gremlin-server-session-1] WARN  org.janusgraph.graphdb.transaction.StandardJanusGraphTx  - Query requires iterating over all vertices [(maid = xxxx]. For better performance, use indexes
 
Post adding and restarting our pods I came across another warning:
org.janusgraph.core.JanusGraphException: Could not find a suitable index to answer graph query and graph scans are disabled: [(~label = LABEL AND maid = XXX)]:VERTEX
 
So, the question rounds up to:
Is it compulsory for Janusgraph to have Composite(Internal Index) to query large graphs even though we have Mixed index in place??


hadoopmarc@...
 

Hi,

No, MixedIndex should be fine. Can you show the code lines that define the index for the maid property key? Possibly, the index is restricted to a specific label so you have to query:
g.V().has('product', 'maid', '45324j5nu8g5r83q89u53h89g')

See also:
https://docs.janusgraph.org/index-management/index-performance/#label-constraint

Best wishes,    Marc