Date   

Re: No results returned with duplicate Has steps in a vertex-search traversal

Boxuan Li
 

Just created a simple test case but couldn’t reproduce:

@Test
public void testDuplicateMixedIndexQuery() {
final PropertyKey name = makeKey("name", String.class);
final PropertyKey prop = makeKey("prop", String.class);
mgmt.buildIndex("mixed", Vertex.class).addKey(name, Mapping.STRING.asParameter()).buildMixedIndex(INDEX);
mgmt.buildIndex("mixed2", Vertex.class).addKey(prop, Mapping.STRING.asParameter()).buildMixedIndex(INDEX);
finishSchema();

tx.addVertex("name", "bob", "prop", "val");
tx.commit();

clopen(option(FORCE_INDEX_USAGE), true);
newTx();
assertTrue(tx.traversal().V().has("prop", "val").has("name", P.within("bob","alice")).hasNext());
assertTrue(tx.traversal().V().has("prop", "val").has("name", P.within("bob","alice")).has("name", P.within("bob","alice")).hasNext());
}

Would it be possible for you to narrow down the scope, e.g. removing other “has” steps in your query? It would be helpful if you could write a piece of minimal reproducible test code.

On Feb 4, 2021, at 3:43 AM, Patrick Streifel <prstreifel@...> wrote:

Hi,

We have a mixed Elasticsearch index that indexes every vertex property on our graph. 
For the above example, all of the fields are indexed as a "keyword" string in Elasticsearch. 
We sometimes get inconsistent behavior. For example if instead of querying by two keyword fields (FullName and PersonSurName), I instead query by one field with a "keyword" mapping type and another with a "text" mapping type, I get the desired result back. 
Again, I can see in the logs the correct records being returned by our Elasticsearch index, regardless of the example. It's just not getting returned by JanusGraph after that.
We have no composite indices on our graph. 

Thanks!


Re: Authentication All the Schema's

Vinayak Bali
 

Hi Marc,

Thank You for the update. Authentication to the graph system as a whole is also not working for me for the configurations shared earlier. If I don't pass credentials then still API is returning the results. Authentication to the graph system as a whole will also work for now till the future versions are released. Request you to guide me to accomplish it.

Thanks & Regards,
Vinayak

On Fri, Feb 5, 2021 at 1:31 PM <hadoopmarc@...> wrote:
Hi Vinayak,

No, this is not possible. TinkerPop/JanusGraph currently only support authentication to the graph system as a whole and do not support authorization. Later this year, the Apache TinkerPop 3.5.0 release will offer authorization, though, which will then also become available through a future JanusGraph release.

https://github.com/apache/tinkerpop/commit/61f7b8c08ac6a1232b460e100b3ff7c91ab4142d

Until then, you will have to use separate Gremlin Server instances.

Best wishes,    Marc


Re: Authentication All the Schema's

hadoopmarc@...
 

Hi Vinayak,

No, this is not possible. TinkerPop/JanusGraph currently only support authentication to the graph system as a whole and do not support authorization. Later this year, the Apache TinkerPop 3.5.0 release will offer authorization, though, which will then also become available through a future JanusGraph release.

https://github.com/apache/tinkerpop/commit/61f7b8c08ac6a1232b460e100b3ff7c91ab4142d

Until then, you will have to use separate Gremlin Server instances.

Best wishes,    Marc


Re: Open instances

rngcntr
 

We once had a short discussion about that over on the old mailing list. Sadly, we did not find a quick solution there. However, seeing more people at least confirms that it does not only occur due to a misconfiguration on our side.


Authentication All the Schema's

Vinayak Bali
 

Hi,

Working on a web application using Janusgraph. We are connecting to Janusgraph using API(Java). Need to configure authentication to all the schema in use. Configured the authentication by using the following document for reference.
I authentication is not working, we are getting a blank array as output using the API.
Property files are as follows:

gremlin-server.yaml

# Copyright 2019 JanusGraph Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

host: 0.0.0.0
port: 8182
scriptEvaluationTimeout: 30000
channelizer: org.apache.tinkerpop.gremlin.server.channel.WsAndHttpChannelizer
graphManager: org.janusgraph.graphdb.management.JanusGraphManager
graphs: {
  ConfigurationManagementGraph: conf/janusgraph-cql-configurationgraph.properties,
 graph: conf/graph.properties,
 graph1: conf/graph1.properties,
 graph2: conf/graph2.properties
}
authentication: {
  authenticator: org.janusgraph.graphdb.tinkerpop.gremlin.server.auth.SaslAndHMACAuthenticator,
  authenticationHandler: org.janusgraph.graphdb.tinkerpop.gremlin.server.handler.SaslAndHMACAuthenticationHandler,
  config: {
    defaultUsername: user,
    defaultPassword: password,
    hmacSecret: secret,
    credentialsDb: conf/janusgraph-credentials-server.properties
  }
}
scriptEngines: {
  gremlin-groovy: {
    plugins: { org.janusgraph.graphdb.tinkerpop.plugin.JanusGraphGremlinPlugin: {},
               org.apache.tinkerpop.gremlin.server.jsr223.GremlinServerGremlinPlugin: {},
               org.apache.tinkerpop.gremlin.tinkergraph.jsr223.TinkerGraphGremlinPlugin: {},
               org.apache.tinkerpop.gremlin.jsr223.ImportGremlinPlugin: {classImports: [java.lang.Math], methodImports: [java.lang.Math#*]},
               org.apache.tinkerpop.gremlin.jsr223.ScriptFileGremlinPlugin: {files: []}}}}
serializers:
  - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
  # Older serialization versions for backwards compatibility:
  - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
  - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoLiteMessageSerializerV1d0, config: {ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
  - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { serializeResultToString: true }}
  - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV2d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
  - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV1d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistryV1d0] }}
  - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistryV1d0] }}
processors:
  - { className: org.apache.tinkerpop.gremlin.server.op.session.SessionOpProcessor, config: { sessionTimeout: 28800000 }}
  - { className: org.apache.tinkerpop.gremlin.server.op.traversal.TraversalOpProcessor, config: { cacheExpirationTime: 600000, cacheMaxSize: 1000 }}
metrics: {
  consoleReporter: {enabled: true, interval: 180000},
  csvReporter: {enabled: true, interval: 180000, fileName: /tmp/gremlin-server-metrics.csv},
  jmxReporter: {enabled: true},
  slf4jReporter: {enabled: true, interval: 180000},
  gangliaReporter: {enabled: false, interval: 180000, addressingMode: MULTICAST},
  graphiteReporter: {enabled: false, interval: 180000}}
maxInitialLineLength: 4096
maxHeaderSize: 8192
maxChunkSize: 8192
maxContentLength: 65536
maxAccumulationBufferComponents: 1024
resultIterationBatchSize: 64
writeBufferLowWaterMark: 32768
writeBufferHighWaterMark: 65536

janusgraph-credentials-server.properties

gremlin.graph=org.janusgraph.core.JanusGraphFactory
storage.backend=cql
storage.hostname=127.0.0.1
storage.cql.keyspace=authentication
cache.db-cache = true
cache.db-cache-clean-wait = 20
cache.db-cache-time = 180000
cache.db-cache-size = 0.25
index.search.backend=elasticsearch
index.search.hostname=127.0.0.1

empty-sample.groovy

// Copyright 2019 JanusGraph Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//      http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

// an init script that returns a Map allows explicit setting of global bindings.
def globals = [:]

// defines a sample LifeCycleHook that prints some output to the Gremlin Server console.
// note that the name of the key in the "global" map is unimportant.
globals << [hook : [
        onStartUp: { ctx ->
            ctx.logger.info("Executed once at startup of Gremlin Server.")
        },
        onShutDown: { ctx ->
            ctx.logger.info("Executed once at shutdown of Gremlin Server.")
        }
] as LifeCycleHook]

// define the default TraversalSource to bind queries to - this one will be named "g".
graph=JanusGraphFactory.open('conf/graph.properties')
graph1=JanusGraphFactory.open('conf/graph1.properties')
graph2=JanusGraphFactory.open('conf/graph2.properties')
globals << [ g : graph.traversal(), g1 : graph1.traversal(), g2:graph2.traversal() ]

I need to secure each and every schema. 
For example: Consider A,B,C,D,E,F  as users and graph, graph1 and graph2 as 3 schemas.
Then,
A has access to the graph,graph1
B only graph
Call the schema.
so on for all the users.
Request you to share your experience and Feedback.

Thanks & Regards,
Vinayak


Re: Issues with controlling partitions when using Apache Spark

Mladen Marović
 

Just a quick info: I opened an issue for this and did some additional research. See https://github.com/JanusGraph/janusgraph/issues/2420 for more details.


Re: reindex job is very slow on ElasticSearch and BigTable

hadoopmarc@...
 

Hi,

No, MixedIndex should be fine. Can you show the code lines that define the index for the maid property key? Possibly, the index is restricted to a specific label so you have to query:
g.V().has('product', 'maid', '45324j5nu8g5r83q89u53h89g')

See also:
https://docs.janusgraph.org/index-management/index-performance/#label-constraint

Best wishes,    Marc


Re: Open instances

hadoopmarc@...
 

Hi,

Good that you noticed this!
Some explanation and the cure can be found at:

https://docs.janusgraph.org/advanced-topics/recovery/#janusgraph-instance-failure

Best wishes,    Marc


Open instances

vamsi.lingala@...
 
Edited

I can see there are many open instances and looks they are not getting closed automatically.

gremlin> mgmt.getOpenInstances() //all open instances
==>0acc001a144-janusgraph-5fbcff8c8b-9bb6v1
==>0acc001d142-janusgraph-5fbcff8c8b-2xgks1
==>0acc001e144-janusgraph-5fbcff8c8b-xzsdv1
==>0acc0023142-janusgraph-5fbcff8c8b-8sfr71
==>0acc0020143-janusgraph-5fbcff8c8b-dl6cw1
==>0acc001b143-janusgraph-5fbcff8c8b-4nskz1
==>0acc0123142-janusgraph-5fbcff8c8b-7vc491(current)
==>0acc0128142-janusgraph-5fbcff8c8b-82lw21
==>0acc012a142-janusgraph-5fbcff8c8b-cwl2h1
==>0acc001f144-janusgraph-5fbcff8c8b-265fx1
==>0acc0127142-janusgraph-5fbcff8c8b-bc8971
==>0acc001c142-janusgraph-5fbcff8c8b-jrb4q1
==>0acc0022143-janusgraph-5fbcff8c8b-qdbkt1
==>0acc0129143-janusgraph-5fbcff8c8b-58kk51
==>0acc0021142-janusgraph-5fbcff8c8b-skkcs1

and if we want to create or disable index we can't do until we close instances and some of the instance could be doing internal works like indexing etc and if we close them we face issues..any way to know what instances are doing?

Please help us


Re: Droping a graph delete all other graph indexes on Solr

@Tartar
 

After some investigation from my colleagues, the problem may come from the file SolrIndex.java on method clearStorage line 971


Re: Droping a graph delete all other graph indexes on Solr

@Tartar
 

Note: I tried to disable the graph2 index and remove it in Solr but problem persists.


Droping a graph delete all other graph indexes on Solr

@Tartar
 

Hello everyone,

I don't know if it's an issue or I have done something wrong but droping a graph delete all other indexes in solr.

How I create my graphs:

def map = new HashMap<String, Object>();
map.put(""gremlin.graph"", ""org.janusgraph.core.ConfiguredGraphFactory"");
map.put(""graph.graphname"", ""{name}"");
map.put(""schema.default"", ""none"");
map.put(""storage.backend"", ""inmemory"");
map.put(""storage.buffer-size"", 10000);
map.put(""storage.batch-loading"", true);
map.put(""storage.parallel-backend-ops"", true);
map.put(""storage.transactions"", false);
map.put(""ids.block-size"", 100000);
 
map.put(""index.solr.backend"", ""solr"");
map.put(""index.solr.max-result-set-size"", 100000);
map.put(""index.solr.solr.mode"", ""cloud"");
map.put(""index.solr.solr.zookeeper-url"", ""{_solrBackendService.GetZookeeperHostUrl()}"");
map.put(""index.solr.solr.http-urls"", ""{_solrBackendService.GetBackendUrl()}"");
map.put(""index.solr.solr.configset"", ""janusgraph-configset"");
map.put(""index.solr.solr.wait-searcher"", true);
 
ConfiguredGraphFactory.createConfiguration(new MapConfiguration(map));
ConfiguredGraphFactory.open(""{name}"");

In solr admin web page I check that my graph1 index has data.
In Gremlin console I check that my graph2 has only its index with:
.openManagement().getGraphIndexes(Vertex)
When I run ConfiguredGraphFactory.drop("graph2"), I don't have data anymore in my graph1 index in solr admin page.

Thank you for your help.


Re: reindex job is very slow on ElasticSearch and BigTable

vamsi.lingala@...
 
Edited

Thanks a lot for your reply..

I tried above solution but unfortunately it did not work.
 
On scanning through pod logs I came across below warning which led us to adding query.force-index=true in our config.
[gremlin-server-session-1] WARN  org.janusgraph.graphdb.transaction.StandardJanusGraphTx  - Query requires iterating over all vertices [(maid = xxxx]. For better performance, use indexes
 
Post adding and restarting our pods I came across another warning:
org.janusgraph.core.JanusGraphException: Could not find a suitable index to answer graph query and graph scans are disabled: [(~label = LABEL AND maid = XXX)]:VERTEX
 
So, the question rounds up to:
Is it compulsory for Janusgraph to have Composite(Internal Index) to query large graphs even though we have Mixed index in place??


Re: No results returned with duplicate Has steps in a vertex-search traversal

Patrick Streifel <prstreifel@...>
 

Hi,

We have a mixed Elasticsearch index that indexes every vertex property on our graph. 
For the above example, all of the fields are indexed as a "keyword" string in Elasticsearch. 
We sometimes get inconsistent behavior. For example if instead of querying by two keyword fields (FullName and PersonSurName), I instead query by one field with a "keyword" mapping type and another with a "text" mapping type, I get the desired result back. 
Again, I can see in the logs the correct records being returned by our Elasticsearch index, regardless of the example. It's just not getting returned by JanusGraph after that.
We have no composite indices on our graph. 

Thanks!


Re: reindex job is very slow on ElasticSearch and BigTable

hadoopmarc@...
 

Is this again for 4 billion vertices?  Do the gremlin server logs show whether the problem is with the index or with the backend?

Does this occur for any vertex? Try to retrieve the first few vertices from a table scan, hopefully these will be vertices with the maid property key:
g.V().limit(100).values('maid')

Then do
g.V().has("maid", ....).valueMap()
for the maid values found above.
Maybe this gives some hints about what is going wrong.

Best wishes,     Marc


Re: reindex job is very slow on ElasticSearch and BigTable

vamsi.lingala@...
 

Thanks a lot for you reply.

We changed our approach..we enabled the index and then ingested nodes..I can see data in elastic search but when I query I am getting timeout

==>------------------------------------------------------------------------------------------------
Vertex Index Name              | Type        | Unique    | Backing        | Key:           Status |
---------------------------------------------------------------------------------------------------
maidMixedIndex                 | Mixed       | false     | search         | maid:         ENABLED |
---------------------------------------------------------------------------------------------------
Edge Index (VCI) Name          | Type        | Unique    | Backing        | Key:           Status |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
Relation Index                 | Type        | Direction | Sort Key       | Order    |     Status |
---------------------------------------------------------------------------------------------------



gremlin> :remote config timeout 600000
==>Set remote timeout to 600000ms
gremlin> g.V().has("maid","8bcaa1a6-31df-4df4-b193-6b4165c4df8c").valueMap()
Evaluation exceeded the configured 'evaluationTimeout' threshold of 600000 ms or evaluation was otherwise cancelled directly for request [g.V().has("maid","8bcaa1a6-31df-4df4-b193-6b4165c4df8c").valueMap()] - try increasing the timeout with the :remote command
Type ':help' or ':h' for help.
Display stack trace? [yN]y
org.apache.tinkerpop.gremlin.jsr223.console.RemoteException: Evaluation exceeded the configured 'evaluationTimeout' threshold of 600000 ms or evaluation was otherwise cancelled directly for request [g.V().has("maid","8bcaa1a6-31df-4df4-b193-6b4165c4df8c").valueMap()] - try increasing the timeout with the :remote command
at org.apache.tinkerpop.gremlin.console.jsr223.DriverRemoteAcceptor.submit(DriverRemoteAcceptor.java:178)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:110)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:120)


Re: reindex job is very slow on ElasticSearch and BigTable

hadoopmarc@...
 

This trace says that your elasticsearch cluster becomes unresponsive. Check your elasticsearch cluster with requests like:

https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html

If the elasticsearch cluster does not recover by itself, restart it and monitor it closely during the next trial for index repair. The number of workers activated in the repair job may be too much for the size of your elasticsearch cluster.

Best wishes,    Marc


Re: reindex job is very slow on ElasticSearch and BigTable

vamsi.lingala@...
 

I get above error while doing reindex using JanusGraphManagement


Re: reindex job is very slow on ElasticSearch and BigTable

vamsi.lingala@...
 

I am getting below errors while doing reindex and it stops the job.

Pls help me with this.

09:09:06.295 [Thread-12] ERROR o.j.d.k.scan.StandardScannerExecutor Unexpected error processing data: {}
org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:285) ~[janusgraph-core-0.5.1.jar:na]
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201) ~[janusgraph-core-0.5.1.jar:na]
	... 2 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-4 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893) ~[janusgraph-es-0.5.1.jar:na]
	... 4 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-4 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_275]
09:09:06.438 [Thread-13] ERROR o.j.d.k.scan.StandardScannerExecutor Unexpected error processing data: {}
org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:285) ~[janusgraph-core-0.5.1.jar:na]
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201) ~[janusgraph-core-0.5.1.jar:na]
	... 2 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893) ~[janusgraph-es-0.5.1.jar:na]
	... 4 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_275]
09:09:07.290 [Thread-10] ERROR o.j.d.k.scan.StandardScannerExecutor Unexpected error processing data: {}
org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:285) ~[janusgraph-core-0.5.1.jar:na]
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201) ~[janusgraph-core-0.5.1.jar:na]
	... 2 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893) ~[janusgraph-es-0.5.1.jar:na]
	... 4 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_275]
09:09:07.325 [Thread-14] ERROR o.j.d.k.scan.StandardScannerExecutor Unexpected error processing data: {}
org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:285) ~[janusgraph-core-0.5.1.jar:na]
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201) ~[janusgraph-core-0.5.1.jar:na]
	... 2 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-1 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893) ~[janusgraph-es-0.5.1.jar:na]
	... 4 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-1 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_275]
09:09:08.323 [Thread-11] ERROR o.j.d.k.scan.StandardScannerExecutor Unexpected error processing data: {}
org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:285) ~[janusgraph-core-0.5.1.jar:na]
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128) ~[janusgraph-core-0.5.1.jar:na]
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201) ~[janusgraph-core-0.5.1.jar:na]
	... 2 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397) ~[janusgraph-es-0.5.1.jar:na]
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893) ~[janusgraph-es-0.5.1.jar:na]
	... 4 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_275]
Exception in thread "Thread-12" org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206)
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118)
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:305)
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895)
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128)
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201)
	... 2 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-5 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893)
	... 4 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-5 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-13" org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206)
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118)
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:305)
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895)
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128)
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201)
	... 2 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-6 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893)
	... 4 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-6 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-10" org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206)
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118)
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:305)
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895)
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128)
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201)
	... 2 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-7 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893)
	... 4 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-7 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-14" org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206)
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118)
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:305)
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895)
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128)
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201)
	... 2 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-8 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893)
	... 4 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-8 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-11" org.janusgraph.core.JanusGraphException: Unknown exception while executing index operation
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:206)
	at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118)
	at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:305)
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Unknown exception while executing index operation
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.convert(ElasticSearchIndex.java:447)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:895)
	at org.janusgraph.diskstorage.indexing.IndexTransaction.restore(IndexTransaction.java:128)
	at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:201)
	... 2 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-9 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:483)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:474)
	at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:397)
	at org.janusgraph.diskstorage.es.ElasticSearchIndex.restore(ElasticSearchIndex.java:893)
	... 4 more
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-9 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.lang.Thread.run(Thread.java:748)
09:18:46.489 [bigtable-rpc-retry-3] WARN  c.g.cloud.bigtable.grpc.io.Watchdog Found 3 stale streams and cancelled them
09:19:57.643 [Credentials-Refresh-8] INFO  c.g.c.b.g.io.OAuthCredentialsCache Refreshing the OAuth token


Re: Janusgraph spark on yarn error

j2kupper@...
 

Thank you!

Sorry for my long time answer.
I am do some experiments with janusgraph.

1081 - 1100 of 6665