Re: Queries with negated text predicates fail with lucene
toom@...
Hi Marc,
IMHO, an index should not prevent a query to work. Moreover the result of a query should not depends of backends (storage and index). If an index backend cannot process a predicate, the predicate should be be executed as if index wasn't present.
To clarify, below is a sample of code. The same query works without index (line 13) and fails with index (line 31).
Regards,
Toom.
IMHO, an index should not prevent a query to work. Moreover the result of a query should not depends of backends (storage and index). If an index backend cannot process a predicate, the predicate should be be executed as if index wasn't present.
To clarify, below is a sample of code. The same query works without index (line 13) and fails with index (line 31).
1 // create schema
2 mgmt = graph.openManagement()
3 mgmt.makePropertyKey('string').dataType(String.class).cardinality(Cardinality.SINGLE).make()
4 mgmt.makeVertexLabel('data').make()
5 mgmt.commit()
6
7 // add data
8 g.addV('data').property('string', 'foo')
9 ==>v[4120]
10 g.addV('data').property('string', 'bar')
11 ==>v[4312]
12
13 g.V().hasLabel('data').has('string', textNotContains('bar'))
14 WARN org.janusgraph.graphdb.transaction.StandardJanusGraphTx - Query requires iterating over all vertices [(~label = data AND string textNotContains bar)]. For better performance, use indexes
15 ==>v[4120]
16
17 // add indexe with lucene backend
18 mgmt = graph.openManagement()
19 string = mgmt.getPropertyKey("string")
20 mgmt.buildIndex('myindex', Vertex.class).addKey(string, Mapping.TEXTSTRING.asParameter()).buildMixedIndex("search")
21 mgmt.commit()
22
23 // Wait the indexes
24 ManagementSystem.awaitGraphIndexStatus(graph, 'myindex').call()
25
26 // Reindex data
27 mgmt = graph.openManagement()
28 mgmt.updateIndex(mgmt.getGraphIndex("myindex"), SchemaAction.REINDEX).get()
29 mgmt.commit()
30
31 g.V().hasLabel('data').has('string', textNotContains('bar'))
32 Could not call index
Regards,
Toom.
Re: Potential transaction issue (JG 0.6.0)
Charles <dostanian@...>
I have also encountered this problem, but I have found a reliable way to reproduce.
The following query works perfectly in the gremlin console (both server and client from Janusgraph 0.6.0 distribution)
gremlin> g.V().has("COMPANY", "companyId", 44507).out("EMPLOYS").has("status", "APPROVED").skip(0).limit(10).elementMap("workerId")
gremlin> g.V().has("COMPANY", "companyId", 44507).out("EMPLOYS").has("status", "APPROVED").order().by("lastName").by("firstName").skip(0).limit(10).elementMap("workerId")
In Java this query fails with the same exception as in your trace, if I pass offset = 0 (zero) it seems to work sometimes fiddling with the offset (3,5,10 and so on)
return traversal.V().has(VertexType.COMPANY.name(), CompanyWrapper.PROP_COMPANY_ID, companyId)
.out(EdgeType.EMPLOYS.name())
.has(WorkerWrapper.PROP_STATUS, WORKER_STATUS_APPROVED)
.skip(offset)
.limit(limit)
.elementMap(properties)
.toStream()
.map(WorkerWrapper::of);
To make the query succeed I have to either remove the skip() and limit() or order the results before skipping and limiting i.e.
return traversal.V().has(VertexType.COMPANY.name(), CompanyWrapper.PROP_COMPANY_ID, companyId)
.out(EdgeType.EMPLOYS.name())
.has(WorkerWrapper.PROP_STATUS, WORKER_STATUS_APPROVED)
.order()
.by(WorkerWrapper.PROP_LAST_NAME).by(WorkerWrapper.PROP_FIRST_NAME)
.skip(offset)
.limit(limit)
.elementMap(properties)
.toStream()
.map(WorkerWrapper::of);
Quite a number of my queries depends on this style of code, start with a known node, traverse edges, has clause and then skip and limit.
9674 [gremlin-server-exec-2] WARN org.apache.tinkerpop.gremlin.server.op.traversal.TraversalOpProcessor - Exception processing a Traversal on iteration for request [6be9a75f-fa5d-4ef1-a61e-7d133bca33c8].
2021-10-15T21:32:09.932504000Z java.lang.NullPointerException
2021-10-15T21:32:09.932569000Z at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.getInternalVertex(StandardJanusGraphTx.java:508)
2021-10-15T21:32:09.932611000Z at org.janusgraph.graphdb.query.vertex.VertexLongList.get(VertexLongList.java:72)
2021-10-15T21:32:09.932652000Z at org.janusgraph.graphdb.query.vertex.VertexLongList$1.next(VertexLongList.java:144)
2021-10-15T21:32:09.932697000Z at org.janusgraph.graphdb.query.vertex.VertexLongList$1.next(VertexLongList.java:131)
2021-10-15T21:32:09.932734000Z at org.apache.tinkerpop.gremlin.process.traversal.step.map.FlatMapStep.processNextStart(FlatMapStep.java:45)
2021-10-15T21:32:09.932774000Z at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:150)
2021-10-15T21:32:09.932812000Z at org.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.next(ExpandableStepIterator.java:55)
2021-10-15T21:32:09.932852000Z at org.apache.tinkerpop.gremlin.process.traversal.step.filter.FilterStep.processNextStart(FilterStep.java:37)
2021-10-15T21:32:09.932890000Z at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:150)
2021-10-15T21:32:09.932927000Z at org.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.next(ExpandableStepIterator.java:55)
2021-10-15T21:32:09.932961000Z at org.apache.tinkerpop.gremlin.process.traversal.step.filter.FilterStep.processNextStart(FilterStep.java:37)
2021-10-15T21:32:09.932995000Z at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:150)
2021-10-15T21:32:09.933033000Z at org.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.next(ExpandableStepIterator.java:55)
2021-10-15T21:32:09.933068000Z at org.apache.tinkerpop.gremlin.process.traversal.step.map.ScalarMapStep.processNextStart(ScalarMapStep.java:39)
2021-10-15T21:32:09.933109000Z at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:150)
2021-10-15T21:32:09.933143000Z at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:222)
2021-10-15T21:32:09.933188000Z at org.apache.tinkerpop.gremlin.server.util.TraverserIterator.fillBulker(TraverserIterator.java:69)
2021-10-15T21:32:09.933228000Z at org.apache.tinkerpop.gremlin.server.util.TraverserIterator.hasNext(TraverserIterator.java:56)
2021-10-15T21:32:09.933265000Z at org.apache.tinkerpop.gremlin.server.op.traversal.TraversalOpProcessor.handleIterator(TraversalOpProcessor.java:410)
2021-10-15T21:32:09.933299000Z at org.apache.tinkerpop.gremlin.server.op.traversal.TraversalOpProcessor.lambda$iterateBytecodeTraversal$0(TraversalOpProcessor
Re: Queries with negated text predicates fail with lucene
hadoopmarc@...
Hi Toom,
See, https://docs.janusgraph.org/index-backend/text-search/#full-text-search_1
Indeed, the negative text predicates are only available to Elasticsearch (and, apparently as you say, to the CompositeIndex).
Best wishes, Marc
See, https://docs.janusgraph.org/index-backend/text-search/#full-text-search_1
Indeed, the negative text predicates are only available to Elasticsearch (and, apparently as you say, to the CompositeIndex).
Best wishes, Marc
Queries with negated text predicates fail with lucene
toom@...
Hi,
With JanusGraph 0.6.0 and Lucene index backend, queries fail if they contain predicate like textNotPrefix, textNotContains:
If ElasticSearch is used or if there is no index backend, the same query work.
I'm not sure Lucene index can be used for negated queries but the queries should not fails. How can I transform my query to make it work ?
Regards,
Toom.
With JanusGraph 0.6.0 and Lucene index backend, queries fail if they contain predicate like textNotPrefix, textNotContains:
java.lang.IllegalArgumentException: Relation is not supported for string value: textNotPrefix
at org.janusgraph.diskstorage.lucene.LuceneIndex.convertQuery(LuceneIndex.java:814)
at org.janusgraph.diskstorage.lucene.LuceneIndex.convertQuery(LuceneIndex.java:864)
at org.janusgraph.diskstorage.lucene.LuceneIndex.query(LuceneIndex.java:593)
at org.janusgraph.diskstorage.indexing.IndexTransaction.queryStream(IndexTransaction.java:110)
If ElasticSearch is used or if there is no index backend, the same query work.
I'm not sure Lucene index can be used for negated queries but the queries should not fails. How can I transform my query to make it work ?
Regards,
Toom.
potential memory leak
Vivek Singh Raghuwanshi
Hi Team,
--
We are facing some issues with the Janusgraph 0.5.3, we captured heap dumps and find memory leaks.
Can you please check if this is a leak suspect?
ViVek Raghuwanshi
Mobile +1-847-848-7388
Mobile +1-847-848-7388
Google Number +1-707-847-8481
http://in.linkedin.com/in/vivekraghuwanshi
http://in.linkedin.com/in/vivekraghuwanshi
Re: Query performance with range
hadoopmarc@...
Hi Claudio,
Paging with range can only work with a vertex centric index, otherwise the vertex table is scanned for every page. If you just want all results, the alternative is to forget about the range() step and iterate over the query result.
Marc
Paging with range can only work with a vertex centric index, otherwise the vertex table is scanned for every page. If you just want all results, the alternative is to forget about the range() step and iterate over the query result.
Marc
Re: Thread goes into Waiting state forever
hadoopmarc@...
Hi Tanroop,
Does the problem also occur if you replace v() with V().limit(1) inside your query?
If not, at what result size does your issue start to occur?
Btw, your post has a typo: "user" and "uidx" should be the same, but I assume it stems from anonymizing your query.
Marc
Does the problem also occur if you replace v() with V().limit(1) inside your query?
If not, at what result size does your issue start to occur?
Btw, your post has a typo: "user" and "uidx" should be the same, but I assume it stems from anonymizing your query.
Marc
Query performance with range
Claudio Fumagalli
Hi,
I have some performance issue extacting nodes attached to a node with pagination
I have a simple graph with CompositeIndex on property name (Please find schema definition in attachments).
The graph has three a 3 genre nodes:
Our goal is given a genre to extract the 20K movies attached to "action" that has value="a", this should be done iteratively limiting the chunk of data extracted at each execution (e.g. We paginate the query using range).
I'm using janus 0.6.0 with cassandra 3.11.6. Please find attached the docker compose I've used to create the janus+cassandra environment and also the janus and gremlin configurations.
This is the query that we use to extract a page:
g.V().has("name", "action").to(Direction.IN, "has_genre").has("value", "a").range(skip, limit).valueMap("id").next();
Here the results of the extraction with different page size:
We have verified that activating db-cache (cache.db-cache=true) has a huge impact on perfomance but this is not easly applicable on our real scenario because we have multiple janus nodes (to support the scaling of the system) and with the cache active we have the risk of read stale data (The data are updated frequently and changes must be read by other services in our processing pipeline).
I have some performance issue extacting nodes attached to a node with pagination
I have a simple graph with CompositeIndex on property name (Please find schema definition in attachments).
The graph has three a 3 genre nodes:
- "action" node has 20K attached movies with value="a" and 20K attached movies with value="b".
- "drama" node has 10K attached movies with value="b"
- "comedy" node has 10K attached movies with value="c"
Our goal is given a genre to extract the 20K movies attached to "action" that has value="a", this should be done iteratively limiting the chunk of data extracted at each execution (e.g. We paginate the query using range).
I'm using janus 0.6.0 with cassandra 3.11.6. Please find attached the docker compose I've used to create the janus+cassandra environment and also the janus and gremlin configurations.
This is the query that we use to extract a page:
g.V().has("name", "action").to(Direction.IN, "has_genre").has("value", "a").range(skip, limit).valueMap("id").next();
Here the results of the extraction with different page size:
- page size 100 read 200 pages in 591453 ms - average elapsed per page 2957.265 ms - min 284 ms - max 11618 ms
- page size 1000 read 20 pages in 62293 ms - average elapsed per page 3114.65 ms - min 632 ms - max 9712 ms
This is the profile of the query for the last chunk:
It seems that the condition has("value", "a") is evaluated reading each of the attached nodes one by one and than evaluating the filter, is this the expected behaviour and performance? Is there any possible optimization in the interaction between Janus and Cassandra (For example read attached nodes in bulk)?
gremlin> g.V().has("name", "action").to(Direction.IN, "has_genre").has("value", "a").range(19900, 20000).valueMap("id").profile();
==>Traversal Metrics
Step Count Traversers Time (ms) % Dur
=============================================================================================================
JanusGraphStep([],[name.eq(action)]) 1 1 0.904 0.01
constructGraphCentricQuery 0.169
GraphCentricQuery 0.591
\_condition=(name = action)
\_orders=[]
\_isFitted=true
\_isOrdered=true
\_query=multiKSQ[1]
\_index=ByName
backend-query 1 0.460
\_query=ByName:multiKSQ[1]
JanusGraphMultiQueryStep 1 1 0.059 0.00
JanusGraphVertexStep(IN,[has_genre],vertex) 40000 40000 240.729 2.31
\_condition=type[has_genre]
\_orders=[]
\_isFitted=true
\_isOrdered=true
\_query=has_genre:SliceQuery[0x71E1,0x71E2)
\_multi=true
\_vertices=1
optimization 0.019
backend-query 40000 86.565
\_query=has_genre:SliceQuery[0x71E1,0x71E2)
HasStep([value.eq(a)]) 20000 20000 10157.815 97.51
RangeGlobalStep(19900,20000) 100 100 15.166 0.15
PropertyMapStep([id],value) 100 100 2.791 0.03
>TOTAL - - 10417.467 -
It seems that the condition has("value", "a") is evaluated reading each of the attached nodes one by one and than evaluating the filter, is this the expected behaviour and performance? Is there any possible optimization in the interaction between Janus and Cassandra (For example read attached nodes in bulk)?
We have verified that activating db-cache (cache.db-cache=true) has a huge impact on perfomance but this is not easly applicable on our real scenario because we have multiple janus nodes (to support the scaling of the system) and with the cache active we have the risk of read stale data (The data are updated frequently and changes must be read by other services in our processing pipeline).
Thank you
Re: GraphTraversal Thread Stuck
ssbothe3@...
Hi
You are suggesting above experiment for isolating the issue right ?
I thought about not using the CQL executor service but we are in primary stage have not done any workload tests to figure out the correct CQL driver config params.
So will prefer to have a safety net of executor service which will prevent too many parallel call to CQL driver.
And looks like someone also faced similar issue.
Thanks,
Sujay Bothe
Re: Thread goes into Waiting state forever
ssbothe3@...
Hi Tanroop,
I have also faced same issue and have posted a query about it on this channel 'GraphTraversal Thread Stuck'.
Did you found the root cause of above issue ?
Thanks,
Sujay Bothe
Re: Janusgraph upgrade 0.5.2 --> 0.6.0 | CQL issue
hadoopmarc@...
Hi Pawan,
OK, I did a small upgrade session myself and did not encounter any issues (apart from having to set numTokens: 4 in conf/cassandra.yaml).
This is what I did:
Best wishes, Marc
OK, I did a small upgrade session myself and did not encounter any issues (apart from having to set numTokens: 4 in conf/cassandra.yaml).
This is what I did:
- Start from a fresh janusgraph-full-0.5.3 distribution (so, with an empty db folder)
- Run bin/janusgraph.sh start
- Run bin/gremlin.sh and
graph = JanusGraphFactory.open('conf/janusgraph-cql-es.properties')
GraphOfTheGodsFactory.load(graph)
:q - Run bin/janusgraph.sh stop
- Start from a fresh janusgraph-full-0.6.0 distribution (so, with an empty db folder)
- Copy the db folder from janusgraph-full-0.5.3/db to janusgraph-full-0.6.0
- Set set numTokens: 4 in conf/cassandra.yaml
- Run bin/janusgraph.sh start
- Run bin/gremlin.sh and
graph = JanusGraphFactory.open('conf/janusgraph-cql-es.properties')
graph.traversal().V()
Best wishes, Marc
Re: JanusGraph custom types
schwartz@...
I've seen this - https://docs.janusgraph.org/advanced-topics/serializer/
But an example would greatly help (btw, we develop mostly in Python).
But an example would greatly help (btw, we develop mostly in Python).
JanusGraph custom types
Hi!
Is there a way to add custom property types to JanusGraph (similar to RelationIdentifier / Geoshape) to be later used by the
GraphSON serializer? Ideally without re-building JanusGraph.
Due to some downstream limitations, we sometimes can't serialize Python datetime objects. It would be great if we could somehow add a custom step such that .values("date-field").as_iso() would return the field in ISO 8601.
Many thanks!
Is there a way to add custom property types to JanusGraph (similar to RelationIdentifier / Geoshape) to be later used by the
GraphSON serializer? Ideally without re-building JanusGraph.
Due to some downstream limitations, we sometimes can't serialize Python datetime objects. It would be great if we could somehow add a custom step such that .values("date-field").as_iso() would return the field in ISO 8601.
Many thanks!
Re: Janusgraph upgrade 0.5.2 --> 0.6.0 | CQL issue
Pawan Shriwas
Hi Marc,
I have also tried with "graph.allow-upgrade=true" and added "storage.cql.request-timeout" as well but am still facing the same issue.
Please check the attached janusgraph server log for your reference.
janusgraph version 0.6.0
Cassandra details - Cassandra 3.9 Snapshot | CQL 3.4.2
Thanks,
Pawan
On Fri, Oct 8, 2021 at 8:52 PM <hadoopmarc@...> wrote:
Hi Pawan,
It seems I miss the practical experience, but it seems you have to set graph.allow-upgrade=true (although it is not clear whether this applies to all version changes or only to "storage version" changes)
https://docs.janusgraph.org/v0.6/configs/configuration-reference/
If this is the solution, the error message is certainly not helpful and the upgrade docs incomplete.
Best wishes, Marc
--
Thanks & Regard
PAWAN SHRIWAS
PAWAN SHRIWAS
Re: Janusgraph upgrade 0.5.2 --> 0.6.0 | CQL issue
hadoopmarc@...
Hi Pawan,
It seems I miss the practical experience, but it seems you have to set graph.allow-upgrade=true (although it is not clear whether this applies to all version changes or only to "storage version" changes)
https://docs.janusgraph.org/v0.6/configs/configuration-reference/
If this is the solution, the error message is certainly not helpful and the upgrade docs incomplete.
Best wishes, Marc
It seems I miss the practical experience, but it seems you have to set graph.allow-upgrade=true (although it is not clear whether this applies to all version changes or only to "storage version" changes)
https://docs.janusgraph.org/v0.6/configs/configuration-reference/
If this is the solution, the error message is certainly not helpful and the upgrade docs incomplete.
Best wishes, Marc
Re: Janusgraph upgrade 0.5.2 --> 0.6.0 | CQL issue
Pawan Shriwas
Hi Marc,
Thanks for your reply,
But nothing was changed on the cassandra cluster part, Keyspace is already there with graph data and it was working before the janusgraph version upgrade. I don't know but it seems new version(0.6.0) creates a keyspace again while that keyspace is already there and available in DB and has access to that user.
Thanks,
Pawan
On Fri, Oct 8, 2021 at 12:07 PM <hadoopmarc@...> wrote:
Hi Pawan,
Caused by: com.datastax.oss.driver.api.core.servererrors.UnauthorizedException: Unauthorized. User graph_user has no CREATE permission on <all keyspaces> or any of its parentsThis part of your stacktrace suggests that something changed on your Cassandra cluster or with the graph_user. So, check the CREATE permission, with your admin or try to create a keyspace with nodetool.
CREATE KEYSPACE IF NOT EXISTS graphDataTable WITH replication={'replication_factor':1,'class':'SimpleStrategy'}
Also note that a few changes in cql properties had to be made (but probably not related):
https://docs.janusgraph.org/v0.6/changelog/
Best wishes, Marc
--
Thanks & Regard
PAWAN SHRIWAS
PAWAN SHRIWAS
Re: Janusgraph upgrade 0.5.2 --> 0.6.0 | CQL issue
hadoopmarc@...
Hi Pawan,
Also note that a few changes in cql properties had to be made (but probably not related):
https://docs.janusgraph.org/v0.6/changelog/
Best wishes, Marc
Caused by: com.datastax.oss.driver.api.core.servererrors.UnauthorizedException: Unauthorized. User graph_user has no CREATE permission on <all keyspaces> or any of its parentsThis part of your stacktrace suggests that something changed on your Cassandra cluster or with the graph_user. So, check the CREATE permission, with your admin or try to create a keyspace with nodetool.
CREATE KEYSPACE IF NOT EXISTS graphDataTable WITH replication={'replication_factor':1,'class':'SimpleStrategy'}
Also note that a few changes in cql properties had to be made (but probably not related):
https://docs.janusgraph.org/v0.6/changelog/
Best wishes, Marc
Re: GraphTraversal Thread Stuck
Boxuan Li
Hi Sujay,
I am not sure about the root cause (it might be a JanusGraph bug or a Datastax CQL driver bug), but you could try the JanusGraph 0.6.0 version and disable the `storage.cql.executor-service.enabled` option (https://docs.janusgraph.org/configs/configuration-reference/#storagecqlexecutor-service), which does not use an internal thread pool for CQLStoreManager as opposed to 0.5.3. If the problem still exists, I would argue it is more likely to be a bug with the Datastax CQL driver.
Best,
Boxuan
I am not sure about the root cause (it might be a JanusGraph bug or a Datastax CQL driver bug), but you could try the JanusGraph 0.6.0 version and disable the `storage.cql.executor-service.enabled` option (https://docs.janusgraph.org/configs/configuration-reference/#storagecqlexecutor-service), which does not use an internal thread pool for CQLStoreManager as opposed to 0.5.3. If the problem still exists, I would argue it is more likely to be a bug with the Datastax CQL driver.
Best,
Boxuan
GraphTraversal Thread Stuck
Sujay Bothe <ssbothe3@...>
Hello,
Janus version - 0.5.3
Cassandra version - 3.11.4
I am facing one issue where the GraphTraversal hasNext() call got stuck .
The thread from which the Traversal was invoked is still stuck and below is the stackTrace for it.
"MYTHREAD" #80 prio=5 os_prio=0 tid=0x00007f74f82f9000 nid=0x1ab9 in Object.wait() [0x00007f746a8ec000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at io.vavr.concurrent.FutureImpl$$Lambda$226/250860313.run(Unknown Source)
at io.vavr.control.Try.run(Try.java:105)
at io.vavr.concurrent.FutureImpl.await(FutureImpl.java:114)
- locked <0x00000000c42f2e60> (a java.lang.Object)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.interruptibleWait(CQLKeyColumnValueStore.java:308)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:289)
at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:76)
at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:97)
at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:94)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158)
at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration.get(KCVSConfiguration.java:94)
at org.janusgraph.graphdb.tinkerpop.JanusGraphVariables.get(JanusGraphVariables.java:46)
at MyObjectStrategy.apply(MyObjectStrategy.java:411)
at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversalStrategies.applyStrategies(DefaultTraversalStrategies.java:88)
at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.applyStrategies(DefaultTraversal.java:124)
at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:196)
at MyTraversal.get(MyTraversal.java:300)
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at io.vavr.concurrent.FutureImpl$$Lambda$226/250860313.run(Unknown Source)
at io.vavr.control.Try.run(Try.java:105)
at io.vavr.concurrent.FutureImpl.await(FutureImpl.java:114)
- locked <0x00000000c42f2e60> (a java.lang.Object)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.interruptibleWait(CQLKeyColumnValueStore.java:308)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:289)
at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:76)
at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:97)
at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:94)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158)
at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration.get(KCVSConfiguration.java:94)
at org.janusgraph.graphdb.tinkerpop.JanusGraphVariables.get(JanusGraphVariables.java:46)
at MyObjectStrategy.apply(MyObjectStrategy.java:411)
at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversalStrategies.applyStrategies(DefaultTraversalStrategies.java:88)
at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.applyStrategies(DefaultTraversal.java:124)
at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:196)
at MyTraversal.get(MyTraversal.java:300)
As you can see above, the CQLKeyColumnValueStore is waiting on the future object.
It has been stuck for a couple of days now.
I went through the code of CQLKeyColumnValueStore.java of JanusGraph.
It is executing the sliceQuery using the CQLStoreManager.java's executorService threadpool.
The thread names for above thread pool starts with 'CQLStoreManager'
I checked the state of all 20 CQLStoreManager threads in the my java process and all of them are in PARKING state
CQLStoreManager[00]" #191278 daemon prio=5 os_prio=0 tid=0x00007f73f837e000 nid=0x3f0a waiting on condition [0x00007f73cd46c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c3c26ff8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c3c26ff8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
So I am not able to understand that what happened to the CQLStoreManager thread which was suppose to update the future object on which
CQLKeyColumnValueStore.interruptibleWait(CQLKeyColumnValueStore.java:308) is waiting.One more additional information that I want to share is that this has happened around the time at which one of the cassandra nodes was disconnected
and the traversal was failing with TemporaryBackendException (Enough replicas not available) .
and the traversal was failing with TemporaryBackendException (Enough replicas not available) .
But afterwards the node got added to the cluster and quorum was reached , still the thread remained stuck.
Can someone please help here ?
Do let me know if any additional information is needed.
Thanks,
Sujay Bothe
Janusgraph upgrade 0.5.2 --> 0.6.0 | CQL issue
Pawan Shriwas
Hi Everyone,
I am facing one issue with janusgraph version 0.6.0 where I am upgrading my janusgraph console/server/embedded from 0.5.2 to 0.6.0 version. Please see below stacktrace for the same.
Please note that the same graph configuration was working on 0.5.2 version.
Suggest how can i recover it.
Stacktrack -
5858 [main] WARN org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager - Graph [graph] configured at [/etc/opt/janusgraph/janusgraph.properties] could not be instantiated and will not be available in Gremlin Server. GraphFactory message: GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:84)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:72)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:106)
at org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager.addGraph(DefaultGraphManager.java:63)
at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
at org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager.<init>(DefaultGraphManager.java:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:84)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:124)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:87)
at org.janusgraph.graphdb.server.JanusGraphServer.start(JanusGraphServer.java:85)
at org.janusgraph.graphdb.server.JanusGraphServer.main(JanusGraphServer.java:53)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:80)
... 14 more
Caused by: java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cql.CQLStoreManager
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:79)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:525)
at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:489)
at org.janusgraph.graphdb.configuration.builder.GraphDatabaseConfigurationBuilder.build(GraphDatabaseConfigurationBuilder.java:64)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:176)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:147)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:127)
... 19 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:73)
... 25 more
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Couldn't initialize CQLStoreManager
at org.janusgraph.diskstorage.cql.CQLStoreManager.<init>(CQLStoreManager.java:155)
at org.janusgraph.diskstorage.cql.CQLStoreManager.<init>(CQLStoreManager.java:116)
... 30 more
Caused by: com.datastax.oss.driver.api.core.servererrors.UnauthorizedException: Unauthorized. User graph_user has no CREATE permission on <all keyspaces> or any of its parents
CREATE KEYSPACE IF NOT EXISTS graphDataTable WITH replication={'replication_factor':1,'class':'SimpleStrategy'}
^^^^^^
(ql error -4)
at com.datastax.oss.driver.api.core.servererrors.UnauthorizedException.copy(UnauthorizedException.java:49)
at com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149)
at com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:53)
at com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:30)
at com.datastax.oss.driver.internal.core.session.DefaultSession.execute(DefaultSession.java:230)
at com.datastax.oss.driver.api.core.cql.SyncCqlSession.execute(SyncCqlSession.java:54)
at org.janusgraph.diskstorage.cql.CQLStoreManager.initializeKeyspace(CQLStoreManager.java:191)
at org.janusgraph.diskstorage.cql.CQLStoreManager.<init>(CQLStoreManager.java:142)
... 31 more
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:84)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:72)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:106)
at org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager.addGraph(DefaultGraphManager.java:63)
at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
at org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager.<init>(DefaultGraphManager.java:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:84)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:124)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:87)
at org.janusgraph.graphdb.server.JanusGraphServer.start(JanusGraphServer.java:85)
at org.janusgraph.graphdb.server.JanusGraphServer.main(JanusGraphServer.java:53)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:80)
... 14 more
Caused by: java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cql.CQLStoreManager
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:79)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:525)
at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:489)
at org.janusgraph.graphdb.configuration.builder.GraphDatabaseConfigurationBuilder.build(GraphDatabaseConfigurationBuilder.java:64)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:176)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:147)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:127)
... 19 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:73)
... 25 more
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Couldn't initialize CQLStoreManager
at org.janusgraph.diskstorage.cql.CQLStoreManager.<init>(CQLStoreManager.java:155)
at org.janusgraph.diskstorage.cql.CQLStoreManager.<init>(CQLStoreManager.java:116)
... 30 more
Caused by: com.datastax.oss.driver.api.core.servererrors.UnauthorizedException: Unauthorized. User graph_user has no CREATE permission on <all keyspaces> or any of its parents
CREATE KEYSPACE IF NOT EXISTS graphDataTable WITH replication={'replication_factor':1,'class':'SimpleStrategy'}
^^^^^^
(ql error -4)
at com.datastax.oss.driver.api.core.servererrors.UnauthorizedException.copy(UnauthorizedException.java:49)
at com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149)
at com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:53)
at com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:30)
at com.datastax.oss.driver.internal.core.session.DefaultSession.execute(DefaultSession.java:230)
at com.datastax.oss.driver.api.core.cql.SyncCqlSession.execute(SyncCqlSession.java:54)
at org.janusgraph.diskstorage.cql.CQLStoreManager.initializeKeyspace(CQLStoreManager.java:191)
at org.janusgraph.diskstorage.cql.CQLStoreManager.<init>(CQLStoreManager.java:142)
... 31 more
Thanks,
Pawan