1. janusgraph-users@lists.lfaidata.foundation
  2. Messages
   Date   
Re: Open instances

hadoopmarc@...
 

Hi,

Good that you noticed this!
Some explanation and the cure can be found at:

https://docs.janusgraph.org/advanced-topics/recovery/#janusgraph-instance-failure

Best wishes,    Marc

Open instances

vamsi.lingala@...
 
Edited

I can see there are many open instances and looks they are not getting closed automatically.

gremlin> mgmt.getOpenInstances() //all open instances
==>0acc001a144-janusgraph-5fbcff8c8b-9bb6v1
==>0acc001d142-janusgraph-5fbcff8c8b-2xgks1
==>0acc001e144-janusgraph-5fbcff8c8b-xzsdv1
==>0acc0023142-janusgraph-5fbcff8c8b-8sfr71
==>0acc0020143-janusgraph-5fbcff8c8b-dl6cw1
==>0acc001b143-janusgraph-5fbcff8c8b-4nskz1
==>0acc0123142-janusgraph-5fbcff8c8b-7vc491(current)
==>0acc0128142-janusgraph-5fbcff8c8b-82lw21
==>0acc012a142-janusgraph-5fbcff8c8b-cwl2h1
==>0acc001f144-janusgraph-5fbcff8c8b-265fx1
==>0acc0127142-janusgraph-5fbcff8c8b-bc8971
==>0acc001c142-janusgraph-5fbcff8c8b-jrb4q1
==>0acc0022143-janusgraph-5fbcff8c8b-qdbkt1
==>0acc0129143-janusgraph-5fbcff8c8b-58kk51
==>0acc0021142-janusgraph-5fbcff8c8b-skkcs1

and if we want to create or disable index we can't do until we close instances and some of the instance could be doing internal works like indexing etc and if we close them we face issues..any way to know what instances are doing?

Please help us
Re: Droping a graph delete all other graph indexes on Solr

@Tartar
 

After some investigation from my colleagues, the problem may come from the file SolrIndex.java on method clearStorage line 971

Re: Droping a graph delete all other graph indexes on Solr

@Tartar
 

Note: I tried to disable the graph2 index and remove it in Solr but problem persists.

Droping a graph delete all other graph indexes on Solr

@Tartar
 

Hello everyone,

I don't know if it's an issue or I have done something wrong but droping a graph delete all other indexes in solr.

How I create my graphs:

def map = new HashMap<String, Object>();
map.put(""gremlin.graph"", ""org.janusgraph.core.ConfiguredGraphFactory"");
map.put(""graph.graphname"", ""{name}"");
map.put(""schema.default"", ""none"");
map.put(""storage.backend"", ""inmemory"");
map.put(""storage.buffer-size"", 10000);
map.put(""storage.batch-loading"", true);
map.put(""storage.parallel-backend-ops"", true);
map.put(""storage.transactions"", false);
map.put(""ids.block-size"", 100000);
 
map.put(""index.solr.backend"", ""solr"");
map.put(""index.solr.max-result-set-size"", 100000);
map.put(""index.solr.solr.mode"", ""cloud"");
map.put(""index.solr.solr.zookeeper-url"", ""{_solrBackendService.GetZookeeperHostUrl()}"");
map.put(""index.solr.solr.http-urls"", ""{_solrBackendService.GetBackendUrl()}"");
map.put(""index.solr.solr.configset"", ""janusgraph-configset"");
map.put(""index.solr.solr.wait-searcher"", true);
 
ConfiguredGraphFactory.createConfiguration(new MapConfiguration(map));
ConfiguredGraphFactory.open(""{name}"");

In solr admin web page I check that my graph1 index has data.
In Gremlin console I check that my graph2 has only its index with:
.openManagement().getGraphIndexes(Vertex)
When I run ConfiguredGraphFactory.drop("graph2"), I don't have data anymore in my graph1 index in solr admin page.

Thank you for your help.

Re: reindex job is very slow on ElasticSearch and BigTable

vamsi.lingala@...
 
Edited

Thanks a lot for your reply..

I tried above solution but unfortunately it did not work.
 
On scanning through pod logs I came across below warning which led us to adding query.force-index=true in our config.
[gremlin-server-session-1] WARN  org.janusgraph.graphdb.transaction.StandardJanusGraphTx  - Query requires iterating over all vertices [(maid = xxxx]. For better performance, use indexes
 
Post adding and restarting our pods I came across another warning:
org.janusgraph.core.JanusGraphException: Could not find a suitable index to answer graph query and graph scans are disabled: [(~label = LABEL AND maid = XXX)]:VERTEX
 
So, the question rounds up to:
Is it compulsory for Janusgraph to have Composite(Internal Index) to query large graphs even though we have Mixed index in place??
Re: No results returned with duplicate Has steps in a vertex-search traversal

Patrick Streifel <prstreifel@...>
 

Hi,

We have a mixed Elasticsearch index that indexes every vertex property on our graph. 
For the above example, all of the fields are indexed as a "keyword" string in Elasticsearch. 
We sometimes get inconsistent behavior. For example if instead of querying by two keyword fields (FullName and PersonSurName), I instead query by one field with a "keyword" mapping type and another with a "text" mapping type, I get the desired result back. 
Again, I can see in the logs the correct records being returned by our Elasticsearch index, regardless of the example. It's just not getting returned by JanusGraph after that.
We have no composite indices on our graph. 

Thanks!

Re: reindex job is very slow on ElasticSearch and BigTable

hadoopmarc@...
 

Is this again for 4 billion vertices?  Do the gremlin server logs show whether the problem is with the index or with the backend?

Does this occur for any vertex? Try to retrieve the first few vertices from a table scan, hopefully these will be vertices with the maid property key:
g.V().limit(100).values('maid')

Then do
g.V().has("maid", ....).valueMap()
for the maid values found above.
Maybe this gives some hints about what is going wrong.

Best wishes,     Marc

Re: reindex job is very slow on ElasticSearch and BigTable

vamsi.lingala@...
 

Thanks a lot for you reply.

We changed our approach..we enabled the index and then ingested nodes..I can see data in elastic search but when I query I am getting timeout

==>------------------------------------------------------------------------------------------------
Vertex Index Name              | Type        | Unique    | Backing        | Key:           Status |
---------------------------------------------------------------------------------------------------
maidMixedIndex                 | Mixed       | false     | search         | maid:         ENABLED |
---------------------------------------------------------------------------------------------------
Edge Index (VCI) Name          | Type        | Unique    | Backing        | Key:           Status |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
Relation Index                 | Type        | Direction | Sort Key       | Order    |     Status |
---------------------------------------------------------------------------------------------------



gremlin> :remote config timeout 600000
==>Set remote timeout to 600000ms
gremlin> g.V().has("maid","8bcaa1a6-31df-4df4-b193-6b4165c4df8c").valueMap()
Evaluation exceeded the configured 'evaluationTimeout' threshold of 600000 ms or evaluation was otherwise cancelled directly for request [g.V().has("maid","8bcaa1a6-31df-4df4-b193-6b4165c4df8c").valueMap()] - try increasing the timeout with the :remote command
Type ':help' or ':h' for help.
Display stack trace? [yN]y
org.apache.tinkerpop.gremlin.jsr223.console.RemoteException: Evaluation exceeded the configured 'evaluationTimeout' threshold of 600000 ms or evaluation was otherwise cancelled directly for request [g.V().has("maid","8bcaa1a6-31df-4df4-b193-6b4165c4df8c").valueMap()] - try increasing the timeout with the :remote command
at org.apache.tinkerpop.gremlin.console.jsr223.DriverRemoteAcceptor.submit(DriverRemoteAcceptor.java:178)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:110)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:120)
Re: reindex job is very slow on ElasticSearch and BigTable

hadoopmarc@...
 

This trace says that your elasticsearch cluster becomes unresponsive. Check your elasticsearch cluster with requests like:

https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html

If the elasticsearch cluster does not recover by itself, restart it and monitor it closely during the next trial for index repair. The number of workers activated in the repair job may be too much for the size of your elasticsearch cluster.

Best wishes,    Marc

Re: reindex job is very slow on ElasticSearch and BigTable

vamsi.lingala@...
 

I get above error while doing reindex using JanusGraphManagement

Re: reindex job is very slow on ElasticSearch and BigTable

vamsi.lingala@...
 

I am getting below errors while doing reindex and it stops the job.

Pls help me with this.

09:09:06.295 [Thread-12] ERROR o.j.d.k.scan.StandardScannerExecutor Unexpected error processing data: {}
org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:285) ~[janusgraph-core-0.5.1.jar:na]
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201) ~[janusgraph-core-0.5.1.jar:na]
	... 2 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-4 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893) ~[janusgraph-es-0.5.1.jar:na]
	... 4 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-4 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_275]
09:09:06.438 [Thread-13] ERROR o.j.d.k.scan.StandardScannerExecutor Unexpected error processing data: {}
org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:285) ~[janusgraph-core-0.5.1.jar:na]
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201) ~[janusgraph-core-0.5.1.jar:na]
	... 2 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893) ~[janusgraph-es-0.5.1.jar:na]
	... 4 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_275]
09:09:07.290 [Thread-10] ERROR o.j.d.k.scan.StandardScannerExecutor Unexpected error processing data: {}
org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:285) ~[janusgraph-core-0.5.1.jar:na]
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201) ~[janusgraph-core-0.5.1.jar:na]
	... 2 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893) ~[janusgraph-es-0.5.1.jar:na]
	... 4 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_275]
09:09:07.325 [Thread-14] ERROR o.j.d.k.scan.StandardScannerExecutor Unexpected error processing data: {}
org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:285) ~[janusgraph-core-0.5.1.jar:na]
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201) ~[janusgraph-core-0.5.1.jar:na]
	... 2 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-1 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893) ~[janusgraph-es-0.5.1.jar:na]
	... 4 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-1 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_275]
09:09:08.323 [Thread-11] ERROR o.j.d.k.scan.StandardScannerExecutor Unexpected error processing data: {}
org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:285) ~[janusgraph-core-0.5.1.jar:na]
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201) ~[janusgraph-core-0.5.1.jar:na]
	... 2 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893) ~[janusgraph-es-0.5.1.jar:na]
	... 4 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_275]
Exception in thread "Thread-12" org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206)
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118)
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:305)
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895)
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128)
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201)
	... 2 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-5 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893)
	... 4 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-5 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-13" org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206)
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118)
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:305)
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895)
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128)
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201)
	... 2 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-6 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893)
	... 4 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-6 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-10" org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206)
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118)
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:305)
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895)
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128)
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201)
	... 2 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-7 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893)
	... 4 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-7 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-14" org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206)
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118)
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:305)
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895)
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128)
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201)
	... 2 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-8 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893)
	... 4 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-8 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-11" org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206)
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118)
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:305)
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895)
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128)
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201)
	... 2 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-9 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893)
	... 4 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-9 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.lang.Thread.run(Thread.java:748)
09:18:46.489 [bigtable-rpc-retry-3] WARN  c.g.cloud.bigtable.grpc.io.Watchdog Found 3 stale streams and cancelled them
09:19:57.643 [Credentials-Refresh-8] INFO  c.g.c.b.g.io.OAuthCredentialsCache Refreshing the OAuth token
Re: Janusgraph spark on yarn error

j2kupper@...
 

Thank you!

Sorry for my long time answer.
I am do some experiments with janusgraph.
Re: Create new node for each group of connected nodes

anjanisingh22@...
 

On Sat, Jan 30, 2021 at 10:42 PM, <hadoopmarc@...> wrote:
Thanks Marc for quick response. 
Regards,
Anjani

Re: Create new node for each group of connected nodes

hadoopmarc@...
 

Hi Anjani,

Your use case obviously comes down to an OLAP query. While JanusGraph provides InputFormat classes to use TInkerPop's SparkGraphComputer and HadoopGraph, many users have experienced problems with them, see e.g. the latest thread:
https://lists.lfaidata.foundation/g/janusgraph-users/topic/issues_with_controlling/80107845?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,80107845

If you would get OLAP with SparkGraphComputer running on your graph with sufficient performance, an additional advantage would be that you can apply TinkerPop's ConnectedComponentVertexProgram.

A safer way to go, not depending on the JanusGraph InputFormats, would be:
  • run an OLTP query writing all vertex id's to a file. This may take days, but it will also give you a baseline of how long a full tablescan takes and what parallellism you need to get a reasonable running time. Be sure to iterate the traversal and not keep all id's in memory.
  • Use the file with id's as input to a spark job that does for each vertex the gremlin query to get all connected vertices. If the starting vertex has the lowest id, then add the additional required vertex and edges (relying on detecting the new vertex is not safe on an eventually consistent backend). Each spark executor can instantiate its own embedded janusgraph instance and queries inside an executor are done in an OLTP way.
Best wishes,    Marc
Create new node for each group of connected nodes

anjanisingh22@...
 

Hi All,

We are using Janus graph 0.5.2 with Cassandra as storage and Elastic as search engine. We have 700M + nodes.
Nodes are already connected by edges.

We got a use case to add one more node for each group of connected nodes and then create edges between newly created node and exiting nodes. 
For ex, say

node A and B are connected by an edge.
node C , D and E are connected by an edge.

then,
create one node for A and B and creates edges between newly created node and existing nodes
create one node for C, D and E and creates edges between newly created node and existing nodes

I would appreciate to have suggestions to achieve this considering our huge graph size.

 

Thanks in advance.

Regards,
Anjani
Re: Recommended way to perform Schema / Data migration

hadoopmarc@...
 

Hi Nick,

What do you want to migrate? Vertex labels? Property keys? Did you take a look at:

https://docs.janusgraph.org/basics/schema/#changing-schema-elements

Best wishes,    Marc

Re: reindex job is very slow on ElasticSearch and BigTable

hadoopmarc@...
 

OK, thanks for confirming the stacktrace. You can report this behavior as an issue on https://github.com/JanusGraph/janusgraph/issues, referring to this thread. It is still not clear to me how this exception can occur, because the BigTable compatibility layer reuses the HBase backend, for which graph.getBackend().getStoreManager().getHadoopManager(); is available.

So, I am afraid there is no quick fix for your issue, unless you start debugging MapReduceIndexManagement for BigTable yourself. Maybe, simply reloading the graph is an option.

Best wishes,    Marc

Re: Connecting to Multiple Schemas using Java

hadoopmarc@...
 

Hi Vinayak,

The TinkerPop ref docs gives the following code fragment for connecting with the cluster method:

Cluster cluster = Cluster.open();
GraphTraversalSource g1 = traversal().withRemote(DriverRemoteConnection.using(cluster, "g1"));
GraphTraversalSource g2 = traversal().withRemote(DriverRemoteConnection.using(cluster, "g2"));
Is this what you tried (I do not see it in your question)?

Best wishes,    Marc

Re: Issues with controlling partitions when using Apache Spark

Mladen Marović
 

Thanks for the responses.

I'll create a github issue for this then, and also create a PR with the changes that fixed this issue for me, in case anyone finds it useful.

I'm also interested in doing the spark-cassandra-connector implementation, however it might take a while until I get around to it.
1101 - 1120 of 6678