janusgraph-users@lists.lfaidata.foundation

JanusGraph 0.5.2 and BigTable

Assaf Schwartz <schw...@...>

#5433

Hi All,

I'm experiencing an issues with running JanusGraph (on top of GKE) against BigTable.

This is the general setup description:

We are using a single node BigTable cluster (for development / integration purposes) with the vanilla 0.5.2 docker.
Indexing is configured to be done with ES (also running on GKE)
JanusGraph is configured through environment variables:
- janusgraph.storage.backend=hbase
- janusgraph.storage.hbase.ext.hbase.client.connection.impl=com.google.cloud.bigtable.hbase2_x.BigtableConnection
- janusgraph.storage.hbase.ext.google.bigtable.project.id=<project_id>
- janusgraph.storage.hbase.ext.google.bigtable.instance.id=<bt>
Interaction with JanusGraph are done only through a single gRPC server that is running gremlin-python, let's call it DB-SERVER.
Last time we've done testing against BT was with version 0.4.1 of JanusGraph, precompiled to support HBase1.
All of our components communicate via gRPC.

Description of the problem:

The DB-SERVER creates a Vertex i, generate some XML to represent work to be done, and sends it to another service for processing, let's call in ORCHESTRATOR.
The ORCHESTRATOR generates two properties, w and r (local identifiers) and sends them back to the DB-SERVER, so they will be set as properties on Vertex i. These two properties are also mixed String indexes.
After setting the properties, DB-SERVER will ack ORCHESTRATOR, which will start processing. As part of the processing, ORCHESTRATOR will send updates back to the DB-SERVER using w and r.
On getting these updates DB-SERVER, it will try looking up Vertex i based on w and r, like so:
g.V().has("r", <some_r>).has("w", <some_w>).next()
At that point, a null / None is returned as the traversal fails to find Vertex i.
Trying the same traversal in a separate console (python and gremlin) does fetch the vertex. Since it's a single instance cluster, I ruled out any eventual consistency issues.

I'm not sure if it's a regression introduced after 0.4.1.

I've also validated that db-caching is turned off.

Help! :)

Many thanks in advance,

Assaf

Re: Use index for sorting

toom <to...@...>

#5432

Thank you very much, I'll try that.

Toom.

Show quoted text

On Thursday, December 17, 2020 at 8:28:52 AM UTC+1 ow...@... wrote:

We use special strategy that rewrite order comparator for supporting missing values https://gist.github.com/mad/6bef960d51d7a0d89a8c310851023303

пятница, 4 декабря 2020 г. в 09:59:13 UTC+3, t...@...:
The problem of using custom value for null is that we need to choose a value for each data type, and hope that nobody will try to use this particular value. I suppose it is feasible for data type like string, date or double but not for boolean.

Toom.
On Friday, December 4, 2020 at 2:22:02 AM UTC+1 li...@... wrote:
No, null support is an optional feature for graph providers. JanusGraph does not allow null value and I don’t think it will be supported (in near future).

Apart from the solution suggested by Marc, is it possible for you to come up with some custom value to represent null?

Best regards,
Boxuan

「toom <t...@...>」在 2020年12月4日週五，上午3:17 寫道：

Hi Marc,

Thank you for your response.
If I understand correctly, with TinkerPop 3.5 I will be able to sort on property with missing values. It is a good news.
Do you know it JanusGraph 0.6.0 will be release with that version ?

Regarding the impact of the step order on index use, I wrote a strategy [1] that put HasStep and OrderStep before FilterStep if they follow a GraphStep.

Best regards,

Toom.

[1] https://gist.github.com/To-om/e7e406d85dcdd65bc2a533960b50a1d2
On Thursday, December 3, 2020 at 8:17:27 AM UTC+1 HadoopMarc wrote:
Hi Toom,

No solution, but the exception that you mention comes from TinkerPop:
https://github.com/apache/tinkerpop/blob/b4928e1262174a68c3dc1f3234d4340e266f8d98/docs/src/upgrade/release-3.5.x.asciidoc (not released yet)

Apparently, you want all selected vertices, including the ones with null properties, so I would wait for TinkerPop 3.5 and in the mean time use your own workaround for a single filter criterion and do the ordering outside gremlin for more complex sets of filtering criteria.

Best wishes, Marc

Op woensdag 2 december 2020 om 08:13:05 UTC+1 schreef t...@...:
Hello,

I'm using JanusGraph with Cassandra (0.5.2) and ElasticSearch.

I try to optimize my queries and use the mixed indexes as much as possible, in particular for sortings, but I have some difficulties:

It is not possible to sort by properties that can have missing values (or I get a "The property does not exist as the key has no associated value for the provided element"). Therefore I used ".order().by(coalesce(values('closingDate'), new Date()))" but in this case, the index is not used.

If there is only one sorting criterion, I probably can do something like:

g.inject(1).union(
g.V().hasLabel('Case').has('closingDate').order().by('closingDate'),
g.V().hasLabel('Case').hasNot('closingDate'))

But what is my best option if I want to use several criteria?

I also note that the FilterRankingStrategy strategy can have negative effect on performance when there are filters that don't use index. For example, the following query is faster without step reordering.

g.V().hasLabel('Case').has('closingDate').order().by('closingDate').filter(out('attachment').has('file'))

FilterRanking swaps order() and filter() steps and then index is not used for sorting.

Any help will be much appreciated.

Toom.

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/c6bcb878-bfc2-4d8f-a3ee-38cc214382bcn%40googlegroups.com.

Re: Use index for sorting

"ow...@gmail.com" <owner...@...>

#5431

We use special strategy that rewrite order comparator for supporting missing values https://gist.github.com/mad/6bef960d51d7a0d89a8c310851023303

пятница, 4 декабря 2020 г. в 09:59:13 UTC+3, t...@...:

Show quoted text

The problem of using custom value for null is that we need to choose a value for each data type, and hope that nobody will try to use this particular value. I suppose it is feasible for data type like string, date or double but not for boolean.

Toom.
On Friday, December 4, 2020 at 2:22:02 AM UTC+1 li...@... wrote:
No, null support is an optional feature for graph providers. JanusGraph does not allow null value and I don’t think it will be supported (in near future).

Apart from the solution suggested by Marc, is it possible for you to come up with some custom value to represent null?

Best regards,
Boxuan

「toom <t...@...>」在 2020年12月4日週五，上午3:17 寫道：

Hi Marc,

Thank you for your response.
If I understand correctly, with TinkerPop 3.5 I will be able to sort on property with missing values. It is a good news.
Do you know it JanusGraph 0.6.0 will be release with that version ?

Regarding the impact of the step order on index use, I wrote a strategy [1] that put HasStep and OrderStep before FilterStep if they follow a GraphStep.

Best regards,

Toom.

[1] https://gist.github.com/To-om/e7e406d85dcdd65bc2a533960b50a1d2
On Thursday, December 3, 2020 at 8:17:27 AM UTC+1 HadoopMarc wrote:
Hi Toom,

No solution, but the exception that you mention comes from TinkerPop:
https://github.com/apache/tinkerpop/blob/b4928e1262174a68c3dc1f3234d4340e266f8d98/docs/src/upgrade/release-3.5.x.asciidoc (not released yet)

Apparently, you want all selected vertices, including the ones with null properties, so I would wait for TinkerPop 3.5 and in the mean time use your own workaround for a single filter criterion and do the ordering outside gremlin for more complex sets of filtering criteria.

Best wishes, Marc

Op woensdag 2 december 2020 om 08:13:05 UTC+1 schreef t...@...:
Hello,

I'm using JanusGraph with Cassandra (0.5.2) and ElasticSearch.

I try to optimize my queries and use the mixed indexes as much as possible, in particular for sortings, but I have some difficulties:

It is not possible to sort by properties that can have missing values (or I get a "The property does not exist as the key has no associated value for the provided element"). Therefore I used ".order().by(coalesce(values('closingDate'), new Date()))" but in this case, the index is not used.

If there is only one sorting criterion, I probably can do something like:

g.inject(1).union(
g.V().hasLabel('Case').has('closingDate').order().by('closingDate'),
g.V().hasLabel('Case').hasNot('closingDate'))

But what is my best option if I want to use several criteria?

I also note that the FilterRankingStrategy strategy can have negative effect on performance when there are filters that don't use index. For example, the following query is faster without step reordering.

g.V().hasLabel('Case').has('closingDate').order().by('closingDate').filter(out('attachment').has('file'))

FilterRanking swaps order() and filter() steps and then index is not used for sorting.

Any help will be much appreciated.

Toom.

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/c6bcb878-bfc2-4d8f-a3ee-38cc214382bcn%40googlegroups.com.

Aggregating edges based on the source & target vertex attributes

vishnu gajendran <ggvis...@...>

#5430

Hello,

I request your help regarding the janus graph query which I am trying to construct. Let's consider the following graph where each vertex denotes a person and the edge between any two vertex denotes collaboration between them.

Vertices:

p1 = graph.addVertex('person')

p1.property('personId', 1)

p1.property('organization', "engineering")

p2 = graph.addVertex('person')

p2.property('personId', 2)

p2.property('organization', "sales")

p3 = graph.addVertex('person')

p3.property('personId', 3)

p3.property('organization', "marketing")

p4 = graph.addVertex('person')

p4.property('personId', 4)

p4.property('organization', "engineering")

Edges:

p1.addEdge('collaboration', p2, 'collaborationHours', 1)

p1.addEdge('collaboration', p3, 'collaborationHours', 2)

p2.addEdge('collaboration', p3, 'collaborationHours', 2)

p3.addEdge('collaboration', p4, ' collaborationHours', 2)

p4.addEdge('collaboration', p2, ' collaborationHours', 2)

Expected Result is the following table:

Organization1 Organization2 Total Collaboration Hours
Engineering Sales 4
Engineering Marketing 2
Sales Marketing 2
Marketing Engineering 2

Here, I am trying to aggregate the "person to person" graph into "organization to organization" graph. Does JanusGraph support such aggregation queries? If yes, can you please help me with the query for the same?

Thanks

Re: Condition where-inV-is does not work

HadoopMarc <bi...@...>

#5429

Hi Anton,

To compare objects you need the eq() predicate, not the is() step (see https://tinkerpop.apache.org/docs/current/reference/#a-note-on-predicates). Also note that JanusGraph uses RelationIdentifier objects as id's.

So, give the following a try (not tested):

g.V(110792).as('a').V(41099392).outE('LINK').where(inV().eq('a'))

Best wishes, Marc

Op woensdag 16 december 2020 om 10:03:33 UTC+1 schreef Anton Eroshenko:

Show quoted text

Thanks for reply.
In real query I don't have an ID, that's why I use traversal in where clause. Tried this one, but it does not work either:
g.V(41099392).outE('LINK').where(inV().hasId(V(110792).id()))

On Wednesday, December 16, 2020 at 1:45:50 AM UTC+7 Amiya wrote:

Try g.V(41099392).outE('LINK').where(inV().hasId(110792))

On Tuesday, 15 December 2020 at 20:45:40 UTC+5:30 Anton Eroshenko wrote:

Hi, JanusGraph users.
I try a simple query from TinkerPop docs with my JanusGraph installation, but it does not work as expected. How is it possible:
gremlin> g.V(41099392).outE('LINK').inV()
==>v[110792]
==>v[81993864]
gremlin> g.V(41099392).outE('LINK').where(inV().is(V(110792)))
gremlin>

The last query returns nothing...
How to filter out vertices by traversal? Appreciate any help

Re: addE doesn't create more than 1 edge

Anton Eroshenko <erosh...@...>

#5428

Thanks for your explanation. I got the idea, that "many" side of "one-to-many" relationship should go first when adding edges.

Show quoted text

On Wednesday, December 16, 2020 at 2:03:31 AM UTC+7 Amiya wrote:

Marc is correct about "It is not a bug: the addE() step only adds one edge per traverser".
I think query by Mark will create a self link.

Below query might work (not tested)

g.V().hasLabel('Activity').as('a').
V().has('person', 'id', within('p1', 'p2')).addE('LINK').from('a')
On Tuesday, 15 December 2020 at 20:36:56 UTC+5:30 HadoopMarc wrote:
Hi Anton,

It is not a bug: the addE() step only adds one edge per traverser. Also, using g.V() twice is an anti-pattern. Try it the other way around, something like (not tested):
g.V().hasLabel('person').has('id', within('p1', 'p2')).as('a').addE('LINK').from('a')

Now the addE() step runs on two traversers.

HTH, Marc

Op dinsdag 15 december 2020 om 12:22:56 UTC+1 schreef Anton Eroshenko:

I'm trying to link one vertex (let's say Activity) with two others (person), I expect to have two
edges in result, the request below creates only one.

gremlin> g.V().hasLabel('Activity')
==>v[40984624]
gremlin> g.V().hasLabel('person').has('id', within('p1', 'p2'))
==>v[40996896]
==>v[41037952]
gremlin> g.V().hasLabel('Activity').addE('LINK').to(g.V().hasLabel('person').has('id', within('p1', 'p2')))
==>e[oe5mu-oefxs-b0np-oepeo][40984624-RESP->40996896]

Is it a bug or I'm missing something?

Re: Condition where-inV-is does not work

Anton Eroshenko <erosh...@...>

#5427

Thanks for reply.

In real query I don't have an ID, that's why I use traversal in where clause. Tried this one, but it does not work either:

g.V(41099392).outE('LINK').where(inV().hasId(V(110792).id()))

Show quoted text

On Wednesday, December 16, 2020 at 1:45:50 AM UTC+7 Amiya wrote:

Try g.V(41099392).outE('LINK').where(inV().hasId(110792))

On Tuesday, 15 December 2020 at 20:45:40 UTC+5:30 Anton Eroshenko wrote:

Hi, JanusGraph users.
I try a simple query from TinkerPop docs with my JanusGraph installation, but it does not work as expected. How is it possible:
gremlin> g.V(41099392).outE('LINK').inV()
==>v[110792]
==>v[81993864]
gremlin> g.V(41099392).outE('LINK').where(inV().is(V(110792)))
gremlin>

The last query returns nothing...
How to filter out vertices by traversal? Appreciate any help

Re: How to improve traversal query performance

Manabu Kotani <smallcany...@...>

#5426

Hi Marc,

Thank you for your too much advice.

I will keep learning about Gremlin query for better performance.

Thanks a lot!

Manabu.

2020年12月11日金曜日 22:40:21 UTC+9 HadoopMarc:

Show quoted text

Hi Manabu,

Yes, providing an example graph works much better in exploring the problem space. I am afraid, though, that I did not find much that will help you out.
on a single machine with cassandra and using gremlin console with embedded janusgraph, the total query times stated by profile() deviated significantly from the experienced wall clock times even when everything had a cold start:
                                                                total profile (ms)               System.currentTimeMillis(ms)
repeat query, no query-batch             1113                                      1775
repeat query             622             1096
repeat query (warm caches)                 40                                         360
Did you do any wall clock query performance tests on your production system with warmed caches? Results might be better - or worse - than suggested by profile().
the values 5, 161, 9, 8,... under the repeat step add up to the number of touched edges (2828) in the graph. For this generated graph the number of traversers (2925) is dominated by this number of edges. Trying to bulk any intermediate results using sack will have little effect (in contrary to what I suggested earlier). From another perspective, you can check that the following query without any path references still results in the same number of 2925 traversers:
g.V().has('serial', within(startIds)).repeat(inE('assembled').outV()).emit().profile()
other people wanting to play with this graph should use the following line in Manabu's code:

columns = line.split(' ', -1)
So, concluding, there does not seem to be much you can do about the query: you simply want a large resultset from a traversal with multiple steps. Depending on the size of you graph, you could hod the graph in memory using the inmemory backend, or you could replace cassandra with cql and put on it on infrastructure with SSD storage. Of course, you could also precompute and store results, or split up the query with repeat().times(1), repeat().times(2), etc. for faster intermediate results.

Best wishes,    Marc

Op dinsdag 8 december 2020 om 08:56:03 UTC+1 schreef Manabu Kotani:
Hi Marc,

Profile outputs I tried.

1. g.V().has('serial',within('XXXXXX','YYYYYY')).inE('assembled').outV()
----------------------------------------------------------------------
gremlin> g.V().has('serial', within('1654145144','1648418968','1652445288','1654952168','1653379120','1654325440','1653383216','1658298568','1649680536','1649819672','1654964456','1649729552','1656103144','1655460032','1656111336','1654669360')).inE('assembled').outV().profile()
==>Traversal Metrics
Step Count Traversers Time (ms) % Dur
=============================================================================================================
JanusGraphStep([],[serial.within([1654145144, 1... 16 16 18.860 63.13
\_condition=((serial = 1654145144 OR serial = 1648418968 OR serial = 1652445288 OR serial = 1654952168 OR
serial = 1653379120 OR serial = 1654325440 OR serial = 1653383216 OR serial = 1658298568 OR se
rial = 1649680536 OR serial = 1649819672 OR serial = 1654964456 OR serial = 1649729552 OR seri
al = 1656103144 OR serial = 1655460032 OR serial = 1656111336 OR serial = 1654669360))
\_orders=[]
\_isFitted=true
\_isOrdered=true
\_query=multiKSQ[16]@2000
\_index=bySerial
optimization 0.058
optimization 0.694
backend-query 16 17.823
\_query=bySerial:multiKSQ[16]@2000
\_limit=2000
JanusGraphVertexStep(IN,[assembled],vertex) 73 73 11.016 36.87
\_condition=type[assembled]
\_orders=[]
\_isFitted=true
\_isOrdered=true
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
\_multi=true
\_vertices=16
optimization 0.205
backend-query 73 9.332
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
>TOTAL - - 29.877 -

2. g.V().has('serial',within('XXXXXX','YYYYYY')).as('a').in('assembled').inE('assembled').where(outV(), neq('a')).outV()            // query not tested
----------------------------------------------------------------------
gremlin> g.V().has('serial', within('1654145144','1648418968','1652445288','1654952168','1653379120','1654325440','1653383216','1658298568','1649680536','1649819672','1654964456','1649729552','1656103144','1655460032','1656111336','1654669360')).as('a').in('assembled').inE('assembled').where(outV().is(neq('a'))).outV().profile()
==>Traversal Metrics
Step Count Traversers Time (ms) % Dur
=============================================================================================================
JanusGraphStep([],[serial.within([1654145144, 1... 16 16 19.980 26.52
\_condition=((serial = 1654145144 OR serial = 1648418968 OR serial = 1652445288 OR serial = 1654952168 OR
serial = 1653379120 OR serial = 1654325440 OR serial = 1653383216 OR serial = 1658298568 OR se
rial = 1649680536 OR serial = 1649819672 OR serial = 1654964456 OR serial = 1649729552 OR seri
al = 1656103144 OR serial = 1655460032 OR serial = 1656111336 OR serial = 1654669360))
\_orders=[]
\_isFitted=true
\_isOrdered=true
\_query=multiKSQ[16]@2000
\_index=bySerial
optimization 0.026
optimization 0.588
backend-query 16 18.813
\_query=bySerial:multiKSQ[16]@2000
\_limit=2000
JanusGraphVertexStep(IN,[assembled],vertex) 73 73 6.521 8.66
\_condition=type[assembled]
\_orders=[]
\_isFitted=true
\_isOrdered=true
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
\_multi=true
\_vertices=16
optimization 0.154
backend-query 73 5.310
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
JanusGraphVertexStep(IN,[assembled],edge) 2578 2578 20.170 26.77
\_condition=type[assembled]
\_orders=[]
\_isFitted=true
\_isOrdered=true
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
\_multi=true
\_vertices=59
optimization 0.032
backend-query 2578 10.266
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
TraversalFilterStep([EdgeVertexStep(OUT), Profi... 2578 2578 21.824 28.97
EdgeVertexStep(OUT) 2578 2578 4.776
IsStep(neq(a)) 6.172
EdgeVertexStep(OUT) 2578 2578 6.842 9.08
>TOTAL - - 75.338 -

3. Results I want to get.
----------------------------------------------------------------------
g.V().has('serial', within('1654145144','1648418968','1652445288','1654952168','1653379120','1654325440','1653383216','1658298568','1649680536','1649819672','1654964456','1649729552','1656103144','1655460032','1656111336','1654669360')).as('a').repeat(inE('assembled').as('b').outV().as('c').simplePath()).emit().select('a').id().as('parent').select('b').values('work_date').as('work_date').select('c').values('serial').as('child').select('parent','child','work_date').order().by('parent').by('child').by('work_date').profile()
==>Traversal Metrics
Step Count Traversers Time (ms) % Dur
=============================================================================================================
JanusGraphStep([],[serial.within([1654145144, 1... 16 16 24.028 4.95
\_condition=((serial = 1654145144 OR serial = 1648418968 OR serial = 1652445288 OR serial = 1654952168 OR
serial = 1653379120 OR serial = 1654325440 OR serial = 1653383216 OR serial = 1658298568 OR se
rial = 1649680536 OR serial = 1649819672 OR serial = 1654964456 OR serial = 1649729552 OR seri
al = 1656103144 OR serial = 1655460032 OR serial = 1656111336 OR serial = 1654669360))
\_orders=[]
\_isFitted=true
\_isOrdered=true
\_query=multiKSQ[16]@2000
\_index=bySerial
optimization 0.074
optimization 1.256
backend-query 16 312.583
\_query=bySerial:multiKSQ[16]@2000
\_limit=2000
RepeatStep([JanusGraphVertexStep(IN,[assembled]... 2925 2925 272.924 56.26
JanusGraphVertexStep(IN,[assembled],edge)@[b] 2925 2925 223.728
\_condition=type[assembled]
\_orders=[]
\_isFitted=true
\_isOrdered=true
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
\_multi=true
\_vertices=9
optimization 0.203
backend-query 5 1.557
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.020
backend-query 161 2.356
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.125
backend-query 9 25.853
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.023
backend-query 8 2.168
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.024
backend-query 0 1.808
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.026
backend-query 5 1.354
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.024
backend-query 161 1.989
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.040
backend-query 9 3.490
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.026
backend-query 0 2.231
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.061
backend-query 5 1.877
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.027
backend-query 161 4.645
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.056
backend-query 10 2.554
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.073
backend-query 9 4.274
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.026
backend-query 0 1.199
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.026
backend-query 5 1.165
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.025
backend-query 161 8.010
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.032
backend-query 9 1.542
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.032
backend-query 4 5.402
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.038
backend-query 5 4.173
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.029
backend-query 161 4.113
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.056
backend-query 9 1.617
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.035
backend-query 0 1.517
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.044
backend-query 5 1.522
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.045
backend-query 161 1.985
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.029
backend-query 9 1.435
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.028
backend-query 0 1.034
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.021
backend-query 3 1.108
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.024
backend-query 161 1.785
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.036
backend-query 9 7.190
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.032
backend-query 8 12.321
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.044
backend-query 0 1.926
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.032
backend-query 5 1.782
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.028
backend-query 161 3.398
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.031
backend-query 9 1.412
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.026
backend-query 0 1.212
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.035
backend-query 5 1.283
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.028
backend-query 161 2.149
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.035
backend-query 9 1.415
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.025
backend-query 1 1.214
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.025
backend-query 3 1.313
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.027
backend-query 161 2.004
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.038
backend-query 9 8.265
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.031
backend-query 0 1.718
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.028
backend-query 5 1.489
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.029
backend-query 161 2.066
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.029
backend-query 9 1.361
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.024
backend-query 2 1.454
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.026
backend-query 5 1.234
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.025
backend-query 161 1.819
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.027
backend-query 9 1.361
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.026
backend-query 0 1.136
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.028
backend-query 5 1.265
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.029
backend-query 161 10.425
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.040
backend-query 9 2.437
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.050
backend-query 0 1.462
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.049
backend-query 5 2.208
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.028
backend-query 163 2.415
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.035
backend-query 9 1.252
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.027
backend-query 0 1.164
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.032
backend-query 4 1.335
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.028
backend-query 161 1.944
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.028
backend-query 9 1.473
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.026
backend-query 0 1.114
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.025
backend-query 3 1.279
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.025
backend-query 161 1.867
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.033
backend-query 9 1.463
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.027
backend-query 0 1.169
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
EdgeVertexStep(OUT)@[c] 2925 2925 7.733
PathFilterStep(simple) 2925 2925 10.508
JanusGraphMultiQueryStep(RepeatEndStep) 2925 2925 14.827
RepeatEndStep 2925 2925 9.754
SelectOneStep(last,a) 2925 2925 8.340 1.72
IdStep@[parent] 2925 2925 7.347 1.51
SelectOneStep(last,b) 2925 2925 8.690 1.79
JanusGraphPropertiesStep([work_date],value)@[wo... 2925 2925 35.051 7.22
SelectOneStep(last,c) 2925 2925 9.512 1.96
JanusGraphPropertiesStep([serial],value)@[child] 2925 2925 79.337 16.35
\_condition=type[serial]
\_orders=[]
\_isFitted=true
\_isOrdered=true
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@811c505d
\_multi=true
\_vertices=302
optimization 0.044
backend-query 302 53.962
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@811c505d
SelectStep(last,[parent, child, work_date]) 2925 2925 10.705 2.21
OrderGlobalStep([[value(parent), asc], [value(c... 2925 2925 29.210 6.02
>TOTAL - - 485.149 -

Best regards,
Manabu

2020年11月26日木曜日 16:07:27 UTC+9 HadoopMarc:
Hi Manabu,

OK, I think for this graph structure your initial query is fine for getting the right output results. Still, to better understand the impact on the performance of using sack() you might want to split up your query during experimentation:
g.V().has('serial',within('XXXXXX','YYYYYY')).inE('assembled').outV()
g.V().has('serial',within('XXXXXX','YYYYYY')).as('a').in('assembled').inE('assembled').where(outV(), neq('a')).outV()            // query not tested

Note that I did not use the SimplePath() step, but because it probably precludes the bulking after adding appropriate sack() steps.
If you want others to step in for getting the sack() steps right, please provide the gremlin steps to create your sample graph and the query you have already tried with query output and profile() output.

Best wishes,      Marc

Op donderdag 26 november 2020 om 00:29:33 UTC+1 schreef Manabu Kotani:
Hi Marc,

Sorry, I forgot an attachment (Image of tree structure).

Relationships between vertices and edges are like below.
(Label:item, serial:A)<--[Label:assembled, work_date:2020-11-24]--(Label:item, serial:B)
(Label:item, serial:A)<--[Label:assembled, work_date:2020-11-25]--(Label:item, serial:C)
(Label:item, serial:B)<--[Label:assembled, work_date:2020-11-23]--(Label:item, serial:D)
(Label:item, serial:B)<--[Label:assembled, work_date:2020-11-22]--(Label:item, serial:E)
(Label:item, serial:C)<--[Label:assembled, work_date:2020-11-21]--(Label:item, serial:E)
(Label:item, serial:C)<--[Label:assembled, work_date:2020-11-20]--(Label:item, serial:F)

Best regards,
Manabu

2020年11月25日水曜日 19:55:54 UTC+9 HadoopMarc:
Hi Manabu,

What edge are present between vertices A,B,C,D,E,F?

If there are only edges A-B, A-C, A-D, A-E, A-F, you do not need repeat().

Best wishes,    Marc

Op woensdag 25 november 2020 om 09:22:34 UTC+1 schreef Manabu Kotani:
Hi Marc,

Thank you for your quick reply.

Sorry for the lack of my explanation.
I have a graph like below. (There are 3 levels in this figure, but not necessarily 3 levels.)

When query by "A" for property "serial", then I would like to get results like these.
1. A, B, 2020-11-24
2. A, C, 2020-11-25
3. A, D, 2020-11-23
4. A, E, 2020-11-21
5. A, E, 2020-11-22
6. A, F, 2020-11-20

In this situation, how shoud I use until() step?

Sorry for my low comprehension, I've just started to learn Gremlin.

Best regards,
Manabu

2020年11月25日水曜日 15:46:35 UTC+9 HadoopMarc:
Hi Manabu,

repeat()/simplePath()/emit() can have valid uses, although normally you combine it with the times() or until() step to limit the number of repeats. The profile from your query suggests that the repeat step never takes effect, that is, each traversal takes only a single step from parent to child. The repeat step is not wrong in itself, but if it is not necessary you do not want it to be there if you do not know its impact on performance.

Best wishes,    Marc

Op dinsdag 24 november 2020 om 08:43:36 UTC+1 schreef Manabu Kotani:
Hi Marc,

Thank you for your reply.

I'm reading ref docs that you referred about sack()/barrier(). But, I've not able to understand yet.

One question.
What means this you advised? repeat()/simplePath()/emit() steps should not be used?

in the current traversal the repeat(), simplePath() and emit() steps have no effect if the children do not assemble children themselves. So you can leave these steps out for clarity and to be sure that they do not influence the janusgraph execution plan
Best regards,
Manabu

2020年11月21日土曜日 20:13:28 UTC+9 HadoopMarc:
Hi Manabu,

Without knowing if the considerations below will really help, you may try the following:
in the current traversal the repeat(), simplePath() and emit() steps have no effect if the children do not assemble children themselves. So you can leave these steps out for clarity and to be sure that they do not influence the janusgraph execution plan
you may want to study the profile(), sack() and barrier() steps in the TinkerPop ref docs:
https://tinkerpop.apache.org/docs/3.4.8/reference/#profile-step
https://tinkerpop.apache.org/docs/3.4.8/reference/#sack-step
https://tinkerpop.apache.org/docs/3.4.8/reference/#barrier-step
https://kelvinlawrence.net/book/Gremlin-Graph-Guide.html#lambdasack
https://groups.google.com/g/gremlin-users/c/yGs1rv8YXRw/m/Y_iKg15KAwAJ
https://groups.google.com/g/gremlin-users/c/Z7WqhtHvhRc/m/SAbRRMvNBQAJ
Because of your use of the as()/select() mechanism the traversers cannot be bulked and the number of traversers (20669) equals the path count (20669). I am not sure how bulking would affect the JanusGraph execution plan, but with some luck the number of backend calls decreases from 20669 to 1392 (the number of child vertices).
You can influence the bulking by using the sack() step instead of the as()/select() mechanism. You presently get 20669 (a,b,c) items out of your query. You have to rewrite that to get 1392 child vertices out of your query that each have a sack with (a, b) items. Not well covered in the TinkerPop docs is that the sack can be initialized with a Collection object and that the split and merge operators can take closures to act on such a Collection. E.g. you could structure the Sack as an ArrayList that gets (b, a) tuples or "b|a" strings appended during the traversal of a path.
Finally, note that the barrier() step and LazyBarrierStrategy also influence the bulking process.
Best wishes,    Marc

Op donderdag 19 november 2020 om 02:37:48 UTC+1 schreef Manabu Kotani:
Hi All,

I'm testing traversal query performance.
My query (please see below) takes about 1.8sec.

Is there solution for improve performance (faster than 1.8sec)?
I hope that takes less than 500ms.

1.Environment:
JanusGraph (0.5.2) + Cassandra (3.11.0) on Docker Desktop (Windows)

2.Schema:
------------------------------------------------------------------------------------------------
Vertex Label Name | Partitioned | Static |
---------------------------------------------------------------------------------------------------
item | false | false |
---------------------------------------------------------------------------------------------------
Edge Label Name | Directed | Unidirected | Multiplicity |
---------------------------------------------------------------------------------------------------
assembled | true | false | MULTI |
---------------------------------------------------------------------------------------------------
Property Key Name | Cardinality | Data Type |
---------------------------------------------------------------------------------------------------
serial | SINGLE | class java.lang.String |
work_date | SINGLE | class java.util.Date |
---------------------------------------------------------------------------------------------------
Vertex Index Name | Type | Unique | Backing | Key: Status |
---------------------------------------------------------------------------------------------------
bySerial | Composite | false | internalindex | serial: ENABLED |
---------------------------------------------------------------------------------------------------
Edge Index (VCI) Name | Type | Unique | Backing | Key: Status |
---------------------------------------------------------------------------------------------------
byWorkDate | Composite | false | internalindex | work_date: ENABLED |
---------------------------------------------------------------------------------------------------
Relation Index | Type | Direction | Sort Key | Order | Status |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------

3.Query:
g.V().has('serial',within('XXXXXX','YYYYYY',....<- 100 search keys).as('a')
.repeat(inE('assembled').as('b').outV().as('c').simplePath())
.emit()
.select('a').values('serial').as('parent')
.select('b').values('work_date').as('work_date')
.select('c').values('serial').as('child')
.select('parent','child','work_date')
.order().by('parent').by('child').by('work_date')
-----------------------------------------------------------------------------------------------------------

4.Query Profile:
==>Traversal Metrics
Step Count Traversers Time (ms) % Dur
=============================================================================================================
JanusGraphStep([],[serial.within([XXXXXX... 100 100 159.582 8.89
\_condition=((serial = XXXXXX OR serial = YYYYYY OR .... <- 100 search keys))
\_orders=[]
\_isFitted=true
\_isOrdered=true
\_query=multiKSQ[100]@2000
\_index=bySerial
optimization 0.018
optimization 6.744
backend-query 100 1074.225
\_query=bySerial:multiKSQ[100]@2000
\_limit=2000
RepeatStep([JanusGraphVertexStep(IN,[assembled]... 20669 20669 857.001 47.74
JanusGraphVertexStep(IN,[assembled],edge)@[b] 20669 20669 633.529
\_condition=type[assembled]
\_orders=[]
\_isFitted=true
\_isOrdered=true
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
\_multi=true
\_vertices=204
optimization 0.477
backend-query 228 2.076
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.150
backend-query 0 43.366
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.093
backend-query 229 1.978
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.107
backend-query 0 32.738
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.111
backend-query 229 1.577
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.107
backend-query 0 17.827
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.085
backend-query 229 1.517
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.108
backend-query 0 5.729
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.071
backend-query 228 1.993
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.083
backend-query 0 3.335
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.150
backend-query 229 1.890
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.137
backend-query 0 32.593
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.110
backend-query 229 2.253
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.069
backend-query 230 1.624
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.070
backend-query 0 12.797
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.116
backend-query 229 1.579
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.090
backend-query 0 5.764
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.107
backend-query 229 1.651
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.134
backend-query 0 22.327
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.074
backend-query 229 1.756
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.075
backend-query 0 11.145
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.069
backend-query 229 1.947
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.086
backend-query 0 3.727
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.100
backend-query 116 1.492
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.085
backend-query 0 27.159
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.132
backend-query 229 1.524
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.100
backend-query 0 7.173
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.075
backend-query 230 1.880
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.114
backend-query 0 3.696
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.085
backend-query 228 1.645
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.143
backend-query 0 2.924
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.105
backend-query 229 2.010
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.316
backend-query 0 3.806
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.095
backend-query 230 1.854
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.185
backend-query 229 1.936
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.099
backend-query 0 2.135
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.070
backend-query 231 1.479
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.067
backend-query 0 5.907
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.069
backend-query 1 1.129
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.109
backend-query 0 1.069
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.082
backend-query 231 1.245
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.072
backend-query 0 1.175
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.064
backend-query 229 1.308
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.078
backend-query 0 7.058
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.145
backend-query 231 1.655
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.115
backend-query 0 3.946
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.067
backend-query 117 1.231
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.063
backend-query 0 11.856
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.065
backend-query 230 1.606
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.072
backend-query 0 6.973
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.070
backend-query 229 1.445
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.088
backend-query 230 1.836
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.098
backend-query 0 2.552
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.088
backend-query 116 1.450
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.060
backend-query 0 4.072
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.062
backend-query 229 1.421
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.058
backend-query 0 2.342
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.058
backend-query 229 0.999
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.070
backend-query 0 1.847
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.063
backend-query 229 1.171
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.064
backend-query 0 0.999
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.051
backend-query 228 0.991
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.070
backend-query 0 2.107
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.215
backend-query 116 1.678
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.069
backend-query 229 1.578
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.081
backend-query 0 3.649
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.096
backend-query 229 1.619
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.066
backend-query 228 1.549
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.070
backend-query 116 1.610
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.154
backend-query 228 1.746
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.092
backend-query 0 2.958
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.093
backend-query 232 1.698
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.143
backend-query 229 1.719
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.081
backend-query 0 2.809
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.065
backend-query 229 1.410
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.082
backend-query 229 1.458
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.109
backend-query 228 1.651
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.066
backend-query 228 1.417
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.111
backend-query 117 1.536
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.188
backend-query 0 1.660
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.132
backend-query 229 2.361
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.110
backend-query 0 2.384
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.140
backend-query 229 1.680
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.067
backend-query 230 1.342
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.070
backend-query 0 3.129
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.118
backend-query 231 1.397
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.169
backend-query 0 5.665
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.114
backend-query 116 1.780
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.128
backend-query 0 2.316
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.108
backend-query 229 1.521
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.083
backend-query 231 1.508
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.074
backend-query 0 2.327
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.092
backend-query 116 1.509
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.285
backend-query 0 2.007
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.079
backend-query 116 1.245
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.134
backend-query 230 1.521
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.092
backend-query 1 1.278
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.064
backend-query 0 1.104
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.076
backend-query 231 1.287
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.079
backend-query 229 1.768
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.098
backend-query 0 2.570
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.110
backend-query 116 1.489
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.061
backend-query 0 1.756
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.055
backend-query 229 1.133
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.060
backend-query 116 1.241
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.056
backend-query 0 2.435
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.056
backend-query 228 1.099
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.061
backend-query 0 1.017
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.080
backend-query 229 1.217
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.065
backend-query 230 1.448
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.065
backend-query 229 1.546
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.079
backend-query 230 1.955
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.165
backend-query 0 3.284
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.102
backend-query 229 1.936
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.081
backend-query 0 4.640
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.072
backend-query 229 1.384
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.062
backend-query 0 2.224
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.088
backend-query 116 1.419
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.069
backend-query 0 2.289
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.070
backend-query 231 1.474
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.071
backend-query 229 1.646
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.072
backend-query 0 1.408
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.068
backend-query 230 1.974
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.090
backend-query 229 1.923
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.151
backend-query 230 2.211
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.074
backend-query 230 1.234
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.059
backend-query 0 1.695
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.125
backend-query 230 1.199
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.064
backend-query 0 1.089
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.057
backend-query 116 1.807
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.085
backend-query 0 1.299
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.074
backend-query 228 1.397
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.081
backend-query 228 1.776
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.079
backend-query 0 1.980
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.101
backend-query 229 1.571
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.070
backend-query 231 1.483
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.064
backend-query 0 2.260
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.060
backend-query 230 1.471
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.070
backend-query 232 1.305
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.060
backend-query 229 1.246
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.063
backend-query 229 1.093
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.053
backend-query 229 1.420
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.062
backend-query 226 1.596
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.145
backend-query 0 2.730
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.059
backend-query 229 1.550
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.076
backend-query 231 1.622
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.058
backend-query 117 1.224
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.108
backend-query 0 2.025
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.062
backend-query 230 1.251
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.058
backend-query 230 1.223
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.068
backend-query 116 1.224
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.051
backend-query 0 0.937
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.045
backend-query 116 1.597
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.058
backend-query 228 1.595
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.063
backend-query 0 3.238
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.058
backend-query 229 1.573
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.078
backend-query 231 1.894
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.092
backend-query 230 1.717
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
optimization 0.061
backend-query 231 1.302
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@812bd43d
EdgeVertexStep(OUT)@[c] 20669 20669 39.223
PathFilterStep(simple) 20669 20669 44.905
JanusGraphMultiQueryStep(RepeatEndStep) 20669 20669 65.528
RepeatEndStep 20669 20669 39.443
SelectOneStep(last,a) 20669 20669 44.574 2.48
JanusGraphPropertiesStep([serial],value)@[parent] 20669 20669 92.515 5.15
\_condition=type[serial]
\_orders=[]
\_isFitted=true
\_isOrdered=true
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@811c505d
\_multi=true
\_vertices=100
optimization 0.090
backend-query 100 12.807
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@811c505d
SelectOneStep(last,b) 20669 20669 41.753 2.33
JanusGraphPropertiesStep([work_date],value)@[wo... 20669 20669 98.648 5.50
SelectOneStep(last,c) 20669 20669 41.674 2.32
JanusGraphPropertiesStep([serial],value)@[child] 20669 20669 246.094 13.71
\_condition=type[serial]
\_orders=[]
\_isFitted=true
\_isOrdered=true
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@811c505d
\_multi=true
\_vertices=1392
optimization 0.060
backend-query 1392 136.281
\_query=org.janusgraph.diskstorage.keycolumnvalue.SliceQuery@811c505d
SelectStep(last,[parent, child, work_date]) 20669 20669 49.139 2.74
OrderGlobalStep([[value(parent), asc], [value(c... 20669 20669 164.034 9.14
>TOTAL - - 1795.018 -
-----------------------------------------------------------------------------------------------------------

Sorry for my poor English.
Thanks,
Manabu

Re: addE doesn't create more than 1 edge

Amiya <amiyakr...@...>

#5424

Marc is correct about "It is not a bug: the addE() step only adds one edge per traverser".

I think query by Mark will create a self link.

Below query might work (not tested)

g.V().hasLabel('Activity').as('a').

V().has('person', 'id', within('p1', 'p2')).addE('LINK').from('a')

Show quoted text

On Tuesday, 15 December 2020 at 20:36:56 UTC+5:30 HadoopMarc wrote:

Hi Anton,

It is not a bug: the addE() step only adds one edge per traverser. Also, using g.V() twice is an anti-pattern. Try it the other way around, something like (not tested):
g.V().hasLabel('person').has('id', within('p1', 'p2')).as('a').addE('LINK').from('a')

Now the addE() step runs on two traversers.

HTH, Marc

Op dinsdag 15 december 2020 om 12:22:56 UTC+1 schreef Anton Eroshenko:

I'm trying to link one vertex (let's say Activity) with two others (person), I expect to have two
edges in result, the request below creates only one.

gremlin> g.V().hasLabel('Activity')
==>v[40984624]
gremlin> g.V().hasLabel('person').has('id', within('p1', 'p2'))
==>v[40996896]
==>v[41037952]
gremlin> g.V().hasLabel('Activity').addE('LINK').to(g.V().hasLabel('person').has('id', within('p1', 'p2')))
==>e[oe5mu-oefxs-b0np-oepeo][40984624-RESP->40996896]

Is it a bug or I'm missing something?

Re: Condition where-inV-is does not work

Amiya <amiyakr...@...>

#5423

Try g.V(41099392).outE('LINK').where(inV().hasId(110792))

Show quoted text

On Tuesday, 15 December 2020 at 20:45:40 UTC+5:30 Anton Eroshenko wrote:

Hi, JanusGraph users.
I try a simple query from TinkerPop docs with my JanusGraph installation, but it does not work as expected. How is it possible:
gremlin> g.V(41099392).outE('LINK').inV()
==>v[110792]
==>v[81993864]
gremlin> g.V(41099392).outE('LINK').where(inV().is(V(110792)))
gremlin>

The last query returns nothing...
How to filter out vertices by traversal? Appreciate any help

Re: addE doesn't create more than 1 edge

HadoopMarc <bi...@...>

#5422

Hi Anton,

It is not a bug: the addE() step only adds one edge per traverser. Also, using g.V() twice is an anti-pattern. Try it the other way around, something like (not tested):

g.V().hasLabel('person').has('id', within('p1', 'p2')).as('a').addE('LINK').from('a')

Now the addE() step runs on two traversers.

HTH, Marc

Op dinsdag 15 december 2020 om 12:22:56 UTC+1 schreef Anton Eroshenko:

Show quoted text

I'm trying to link one vertex (let's say Activity) with two others (person), I expect to have two
edges in result, the request below creates only one.

gremlin> g.V().hasLabel('Activity')
==>v[40984624]
gremlin> g.V().hasLabel('person').has('id', within('p1', 'p2'))
==>v[40996896]
==>v[41037952]
gremlin> g.V().hasLabel('Activity').addE('LINK').to(g.V().hasLabel('person').has('id', within('p1', 'p2')))
==>e[oe5mu-oefxs-b0np-oepeo][40984624-RESP->40996896]

Is it a bug or I'm missing something?

Condition where-inV-is does not work

Anton Eroshenko <erosh...@...>

#5421

Hi, JanusGraph users.

I try a simple query from TinkerPop docs with my JanusGraph installation, but it does not work as expected. How is it possible:

gremlin> g.V(41099392).outE('LINK').inV()
==>v[110792]
==>v[81993864]
gremlin> g.V(41099392).outE('LINK').where(inV().is(V(110792)))
gremlin>

The last query returns nothing...

How to filter out vertices by traversal? Appreciate any help

Re: Running OLAP on HBase with SparkGraphComputer fails with Error Container killed by YARN for exceeding memory limits

Evgeniy Ignatiev <yevgeniy...@...>

#5425

Oh, I recall that we once tried to debug the same issue with JanusGraph-Hbase, had clear supernodes in the graph. No attempts on repartitioning, including analyzing code of SparkGraphComputer and tinkering around thought to make it work for partitioned vertices etc. were successful - apparently using Cassandra (latest 3.x version at the time) didn't lead to OOM, but was noticeably slower than HBase when we used it with smaller graphs.

Best regards,
Evgenii Ignatev.

On 15.12.2020 07:07, Roy Yu wrote:

Show quoted text

Thanks Marc

On Friday, December 11, 2020 at 3:40:25 PM UTC+8 HadoopMarc wrote:

Hi Roy,

I think I would first check whether the skew is absent if you count the rows reading the HBase table directly from spark (so, without using janusgraph), e.g.:

https://stackoverflow.com/questions/42019905/how-to-use-newapihadooprdd-spark-in-java-to-read-hbase-data

If this works all right, than you know that somehow in janusgraph HBaseInputFormat the mappers do not get the right key ranges to read from.

I also though about the storage.hbase.region-count property of janusgraph-hbase. If you would specify this at 40 while creating the graph, janusgraph-hbase would create many small regions that will be compacted by HBase later on. But maybe this creates a different structure in the row keys that can be leveraged by the hbase.mapreduce.tableinput.mappers.per.region.

Best wishes, Marc

Op woensdag 9 december 2020 om 17:16:35 UTC+1 schreef Roy Yu:

Hi Marc,

The parameter hbase.mapreduce.tableinput.mappers.per.region can be effective. I set it to 40, and there are 40 tasks processing every region. But here comes the new promblem--the data skew. I use g.E().count() to count all the edges of the graph. During counting one region, one spark task containing all 2.6GB data, while other 39 tasks containing 0 data. The task failed again. I checked my data. There are some vertices which have more 1 million incident edges. So I tried to solve this promblem using vertex cut(https://docs.janusgraph.org/advanced-topics/partitioning/), my graph schema is something like [mgmt.makeVertexLabel('product').partition().make() ]. But when I using MR to load data to the new graph, it consumed more than 10 times when the attemp without using partition(), from the hbase table detail page, I found the data loading process was busy reading data from and writing data to the first region. The first region became the hot spot. I guess it relates to vertex ids. Could help me again?

On Tuesday, December 8, 2020 at 3:13:42 PM UTC+8 HadoopMarc wrote:

Hi Roy,

As I mentioned, I did not keep up with possibly new janusgraph-hbase features. From the HBase source, I see that HBase now has a "hbase.mapreduce.tableinput.mappers.per.region" config parameter.

https://github.com/apache/hbase/blob/rel/2.1.9/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java

It should not be too difficult to adapt the janusgraph HBaseInputFormat to leverage this feature (or maybe it even works without change???).

Best wishes,

Marc

Op dinsdag 8 december 2020 om 04:21:19 UTC+1 schreef Roy Yu:

you seem to run on cloud infra that reduces your requested 40 Gb to 33 Gb (see https://databricks.com/session_na20/running-apache-spark-on-kubernetes-best-practices-and-pitfalls). Fact of life.
---------------------

Sorry Marc I misled you. Error Message was generated when I set spark.executor.memory to 30G, when it failed, I increased spark.executor.memory to 40G, it failed either. I felt desperate and come here to ask for help

On Tuesday, December 8, 2020 at 10:35:19 AM UTC+8 Roy Yu wrote:

Hi Marc

Thanks for your immediate response.

I've tried to set spark.yarn.executor.memoryOverhead=10G and re-run the task, and it stilled failed. From the spark task UI, I saw 80% of processing time is Full GC time. As you said, 2.6GB(GZ compressed) exploding is my root cause. Now I'm trying to reduce my region size to 1GB, if that will still fail, I'm gonna config the hbase hfile not using compressed format.

This was my first time running janusgraph OLAP, and I think this is a common promblom, as HBase region size 2.6GB(compressed) is not large, 20GB is very common in our production. If the community dose not solve the promblem, the Janusgraph HBase based OLAP solution cannot be adopted by other companies either.

On Tuesday, December 8, 2020 at 12:40:40 AM UTC+8 HadoopMarc wrote:

Hi Roy,

There seem to be three things bothering you here:

you did not specify spark.yarn.executor.memoryOverhead, as the exception message says. Easily solved.

you seem to run on cloud infra that reduces your requested 40 Gb to 33 Gb (see https://databricks.com/session_na20/running-apache-spark-on-kubernetes-best-practices-and-pitfalls). Fact of life.

the janusgraph HBaseInputFormat use sentire HBase regions as hadoop partitions, which are fed into spark tasks. The 2.6Gb region size is for compressed binary data which explodes when expanded into java objects. This is your real problem.

I did not follow the latest status of janusgraph-hbase features for the HBaseInputFormat, but you have to somehow use spark with smaller partitions than an entire HBase region.

A long time ago, I had success with skipping the HBaseInputFormat and have spark executors connect to JanusGraph themselves. That is not a quick solution, though.

Best wishes,

Marc

Op maandag 7 december 2020 om 14:10:55 UTC+1 schreef Roy Yu:

Error message:
ExecutorLostFailure (executor 1 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 33.1 GB of 33 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714.

graph conifg:

spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:MaxGCPauseMillis=500 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/mnt/data_1/log/spark2/gc-spark%p.log

spark.executor.cores=1

spark.executor.memory=40960m

spark.executor.instances=3

Region info:

hdfs dfs -du -h /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc

67 134 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/.regioninfo

0 0 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/.tmp

2.6 G 5.1 G /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/e

0 0 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/f

0 0 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/g

0 0 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/h

0 0 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/i

0 0 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/l

0 0 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/m

0 0 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/recovered.edits

0 0 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/s

0 0 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/t

root@~$

Anybody who can help me?

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgra...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/46bcc3bb-9e66-4fb1-add0-22374909fb63n%40googlegroups.com.

addE doesn't create more than 1 edge

Anton Eroshenko <erosh...@...>

#5420

I'm trying to link one vertex (let's say Activity) with two others (person), I expect to have two

edges in result, the request below creates only one.

gremlin> g.V().hasLabel('Activity')
==>v[40984624]
gremlin> g.V().hasLabel('person').has('id', within('p1', 'p2'))
==>v[40996896]
==>v[41037952]
gremlin> g.V().hasLabel('Activity').addE('LINK').to(g.V().hasLabel('person').has('id', within('p1', 'p2')))
==>e[oe5mu-oefxs-b0np-oepeo][40984624-RESP->40996896]

Is it a bug or I'm missing something?

Re: Configuring Transaction Log feature

Sandeep Mishra <sandy...@...>

#5419

The code explains behavior. the api sets start time to null instead of Instant.now() hence different behaviour.

public LogProcessorBuilder setStartTimeNow() {
this.startTime = null;
return this;
}

Show quoted text

On Saturday, December 12, 2020 at 10:26:59 PM UTC+8 Sandeep Mishra wrote:

Pawan,
I was able to make your code work. the problem is "setStartTimeNow()"
Instead use setStartTime(Instant.now()) and test. It works. I am yet to explore difference between two api.
make sure to use a new logidentifier to test.

Regards,
Sandeep

On Wednesday, December 9, 2020 at 8:54:17 PM UTC+8 shr...@... wrote:
Hi Sandeep,

I think I have already added below line to indicate that it should pull the detail from now onwords in processor. Is it not working?

"setStartTimeNow()"

Is anyone other face the same thing in their java code?

Thanks,
Pawan

On Friday, 4 December 2020 at 16:22:51 UTC+5:30 sa...@... wrote:
pawan,
can you check for following in your logs Loaded unidentified ReadMarker start time...
seems your readmarker is starting from 1970. so it tries to read changes since then

Regards,
Sandeep
On Saturday, November 28, 2020 at 8:48:18 PM UTC+8 shr...@... wrote:
one correction to last post in below line.

JanusGraphTransaction tx = graph.buildTransaction().logIdentifier("TestLog").start();

On Saturday, 28 November 2020 at 18:16:09 UTC+5:30 Pawan Shriwas wrote:
Hi Sandeep,

Please see below java code and properties information which I am trying in local with Cassandra cql as backend. This code is not giving me the change log as event which I can get via gremlin console with same script and properties. Please let me know if anything needs to be modify here with code or properties.


package com.example.graph;

import org.janusgraph.core.JanusGraph;
import org.janusgraph.core.JanusGraphFactory;
import org.janusgraph.core.JanusGraphTransaction;
import org.janusgraph.core.JanusGraphVertex;
import org.janusgraph.core.log.ChangeProcessor;
import org.janusgraph.core.log.ChangeState;
import org.janusgraph.core.log.LogProcessorFramework;
import org.janusgraph.core.log.TransactionId;

public class TestLog {

public static void listenLogsEvent(){

JanusGraph graph = JanusGraphFactory.open("/home/ist/Downloads/IM/jgraphdb_local.properties");
LogProcessorFramework logProcessor = JanusGraphFactory.openTransactionLog(graph);

logProcessor.addLogProcessor("TestLog").
setProcessorIdentifier("TestLogCounter").
setStartTimeNow().
addProcessor(new ChangeProcessor(){
@Override
public void process(JanusGraphTransaction tx, TransactionId txId, ChangeState changeState) {
System.out.println("tx--"+tx.toString());
System.out.println("txId--"+txId.toString());
System.out.println("changeState--"+changeState.toString());
}
}).
build();

for(int i=0;i<=10;i++) {
System.out.println("going to add ="+i);
JanusGraphTransaction tx = graph.buildTransaction().logIdentifier("PawanTestLog").start();
JanusGraphVertex a = tx.addVertex("TimeL");
a.property("type", "HOLD");
a.property("serialNo", "XS31B4");
tx.commit();
System.out.println("Vertex committed ="+a.toString());
}
}

public static void main(String[] args) {
System.out.println("starting main");
listenLogsEvent();
}
}


gremlin.graph=org.janusgraph.core.JanusGraphFactory
graph.name=TestGraph
storage.backend = cql
storage.hostname = localhost
storage.cql.keyspace=janusgraphcql
query.fast-property = true
storage.lock.wait-time=10000
storage.batch-loading=true

Thanks in advance.

Thanks,
Pawan

On Saturday, 28 November 2020 at 16:19:20 UTC+5:30 sa...@... wrote:
Pawan,
Can you elaborate more on the program where your are trying to embed the script in?
Regards,
Sandeep

On Sat, 28 Nov 2020, 13:48 Pawan Shriwas, <shr...@...> wrote:
Hey Jason,

Same thing happen with my as well where above script work well in gremlin console but when we use it in java. we are not getting anything in process() section as callback. Could you help for the same.

On Wednesday, 7 February 2018 at 20:28:41 UTC+5:30 Jason Plurad wrote:
It means that it will use the 'storage.backend' value as the storage. See the code in GraphDatabaseConfiguration.java. It looks like your only choice is 'default', and it seems like the option is there for the future possibility to use a different backend.

The code in the docs seemed to work ok, other than a minor change in the setStartTime() parameters. You can cut and paste this code into the Gremlin Console to use with the prepackaged distribution.

import java.util.concurrent.atomic.*; import org.janusgraph.core.log.*; import java.util.concurrent.*; graph = JanusGraphFactory.open('conf/janusgraph-cassandra-es.properties'); totalHumansAdded = new AtomicInteger(0); totalGodsAdded = new AtomicInteger(0); logProcessor = JanusGraphFactory.openTransactionLog(graph); logProcessor.addLogProcessor("addedPerson"). setProcessorIdentifier("addedPersonCounter"). setStartTime(Instant.now()). addProcessor(new ChangeProcessor() { public void process(JanusGraphTransaction tx, TransactionId txId, ChangeState changeState) { for (v in changeState.getVertices(Change.ADDED)) { if (v.label().equals("human")) totalHumansAdded.incrementAndGet(); System.out.println("total humans = " + totalHumansAdded); } } }). addProcessor(new ChangeProcessor() { public void process(JanusGraphTransaction tx, TransactionId txId, ChangeState changeState) { for (v in changeState.getVertices(Change.ADDED)) { if (v.label().equals("god")) totalGodsAdded.incrementAndGet(); System.out.println("total gods = " + totalGodsAdded); } } }). build() tx = graph.buildTransaction().logIdentifier("addedPerson").start(); u = tx.addVertex(T.label, "human"); u.property("name", "proteros"); u.property("age", 36); tx.commit();

If you inspect the keyspace in Cassandra afterwards, you'll see that a separate table is created for "ulog_addedPerson".

Did you have some example code of what you are attempting?

On Wednesday, February 7, 2018 at 5:55:58 AM UTC-5, Sandeep Mishra wrote:
Hi Guys,

We are trying to used transaction log feature of Janusgraph, which is not working as expected.No callback is received at
public void process(JanusGraphTransaction janusGraphTransaction, TransactionId transactionId, ChangeState changeState) {

Janusgraph documentation says value for log.[X].backend is 'default'.
Not sure what exactly it means. does it mean HBase which is being used as backend for data.

Please let me know, if anyone has configured it.

Thanks and Regards,
Sandeep Mishra

--
You received this message because you are subscribed to a topic in the Google Groups "JanusGraph users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/janusgraph-users/JN4ZsB9_DMM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to janusgr...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/c4494822-6d82-4ba0-b769-bc49a4f80835n%40googlegroups.com.

Re: Property with multiple data types

HadoopMarc <bi...@...>

#5418

Hi Laura,

The JanusGraph storage backends can store many isolated graphs (see e.g. the storage.cql.keyspace configuration property). However, it is not possible to have edges between vertices from different graphs, so I guess this is not what you are looking for.

Your question is valid, and, by coincidence, it is currently discussed on the developer's list of Apache TinkerPop:

https://lists.apache.org/thread.html/rd1b6f842b806dd9bca18d91faced3db14ab6cf4e55c9d762b9657d5e%40%3Cdev.tinkerpop.apache.org%3E

Best wishes, Marc

Op maandag 14 december 2020 om 18:34:24 UTC+1 schreef Laura Morales:

Show quoted text

Maybe I'm completely wrong, but would I be right to say that "labels" are the equivalent of Java classes? Like, one label represents a Java class and graph properties represent a class properties? So, saying that a node has label L would be like saying that a certain Java object is of class C? (That's why there's only one label per node). I was picturing labels as arbitrary strings that are attached to a node.
Then in Java, classes are namespaced to avoid collision. If my analogy is true, what would be the equivalent of namespaces with Janus?

Sent: Monday, December 14, 2020 at 2:29 PM
From: "HadoopMarc" <b...@...>
To: "JanusGraph users" <janu...@...>
Subject: Re: Property with multiple data types

Hi Laura,

Things are a bit different than you ask:

a vertex has a single label onlya property key has a single datatype only, but it can be Object.class, see https://docs.janusgraph.org/basics/schema/#property-key-data-typeindices can have a label constraint, but these are not helpful if you want to mix the datatypes in a property for the same vertexI cannot predict well how the various janusgraph parts will behave when mixing up real integers and "string-integers" in a property key of the Object.class datatype. I guess that the gremlin traversals will have problems, while an indexing backend for MixedIndices probably can deal with it. The ref docs definitely advise to use the basic datatypes and spare yourself future headaches (so, unify the datatypes on ingestion).
Best wishes, Marc

Op maandag 14 december 2020 om 09:07:31 UTC+1 schreef Laura Morales:
Thank you Marc, I think it does indeed! If I understand correctly, I can use labels to "namespace" my nodes, or in other words as a way to identify subgraphs.
If I have a node with 2 labels instead, say label1 and label2, I can create 2 indices for the same node, right? That is an index for label1.age (Integer) and an index for label2.age (String), both indices containing the same node. In this scenario I should be allowed to add 2 types of properties to the same node, one containing an Integer and the other one containing a String. Then query by choosing a specific label. Does this work? Can I do something like this?

Sent: Monday, December 14, 2020 at 8:01 AM
From: "HadoopMarc" <b...@...>
To: "JanusGraph users" <janu...@...>
Subject: Re: Property with multiple data types

Hi Laura,

Good that you pay close attention to understanding indices in JanusGraph because they are essential to proper use. Does the following section of the ref docs answers your question?

https://docs.janusgraph.org/index-management/index-performance/#label-constraint[https://docs.janusgraph.org/index-management/index-performance/#label-constraint]

Best wishes, Marc

Op zondag 13 december 2020 om 16:30:19 UTC+1 schreef Laura Morales:I'm new to Janus and LPGs. I have a question after reading the Janus documentation. As far as I understand, edges labels as well as properties (for both nodes and edges) are indexed globally. What happens when I have a sufficiently large graph, that completely unrelated and separate nodes want to use a property called with the same name but that holds different data types? For example, a property called "age" could be used by some nodes with a Integer type (eg. "age": 23), but other nodes on the other far-side of my big graph might want/need/require to use a String type (eg. "age": "twenty-seven"). Is this configuration possible with Janus? Or do I *have to* use two different names such as age_int and age_string?

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@...[mailto:janusgr...@...].
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/0b84be68-3688-46fe-a104-32baef119e2an%40googlegroups.com[https://groups.google.com/d/msgid/janusgraph-users/0b84be68-3688-46fe-a104-32baef119e2an%40googlegroups.com?utm_medium=email&utm_source=footer][https://groups.google.com/d/msgid/janusgraph-users/0b84be68-3688-46fe-a104-32baef119e2an%40googlegroups.com%5Bhttps://groups.google.com/d/msgid/janusgraph-users/0b84be68-3688-46fe-a104-32baef119e2an%40googlegroups.com?utm_medium=email&utm_source=footer]].

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@...[mailto:janusgr...@...].
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/24ee1b44-3501-4d40-abef-b32aa345c959n%40googlegroups.com[https://groups.google.com/d/msgid/janusgraph-users/24ee1b44-3501-4d40-abef-b32aa345c959n%40googlegroups.com?utm_medium=email&utm_source=footer].

Re: Running OLAP on HBase with SparkGraphComputer fails with Error Container killed by YARN for exceeding memory limits

Roy Yu <7604...@...>

#5417

Thanks Marc

Show quoted text

On Friday, December 11, 2020 at 3:40:25 PM UTC+8 HadoopMarc wrote:

Hi Roy,

I think I would first check whether the skew is absent if you count the rows reading the HBase table directly from spark (so, without using janusgraph), e.g.:

https://stackoverflow.com/questions/42019905/how-to-use-newapihadooprdd-spark-in-java-to-read-hbase-data

If this works all right, than you know that somehow in janusgraph HBaseInputFormat the mappers do not get the right key ranges to read from.

I also though about the storage.hbase.region-count property of janusgraph-hbase. If you would specify this at 40 while creating the graph, janusgraph-hbase would create many small regions that will be compacted by HBase later on. But maybe this creates a different structure in the row keys that can be leveraged by the hbase.mapreduce.tableinput.mappers.per.region.

Best wishes, Marc

Op woensdag 9 december 2020 om 17:16:35 UTC+1 schreef Roy Yu:
Hi Marc,

The parameter hbase.mapreduce.tableinput.mappers.per.region can be effective. I set it to 40, and there are 40 tasks processing every region. But here comes the new promblem--the data skew. I use g.E().count() to count all the edges of the graph. During counting one region, one spark task containing all 2.6GB data, while other 39 tasks containing 0 data. The task failed again. I checked my data. There are some vertices which have more 1 million incident edges. So I tried to solve this promblem using vertex cut(https://docs.janusgraph.org/advanced-topics/partitioning/), my graph schema is something like [mgmt.makeVertexLabel('product').partition().make() ]. But when I using MR to load data to the new graph, it consumed more than 10 times when the attemp without using partition(), from the hbase table detail page, I found the data loading process was busy reading data from and writing data to the first region. The first region became the hot spot. I guess it relates to vertex ids. Could help me again?

On Tuesday, December 8, 2020 at 3:13:42 PM UTC+8 HadoopMarc wrote:
Hi Roy,

As I mentioned, I did not keep up with possibly new janusgraph-hbase features. From the HBase source, I see that HBase now has a "hbase.mapreduce.tableinput.mappers.per.region" config parameter.

https://github.com/apache/hbase/blob/rel/2.1.9/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java

It should not be too difficult to adapt the janusgraph HBaseInputFormat to leverage this feature (or maybe it even works without change???).

Best wishes,

Marc

Op dinsdag 8 december 2020 om 04:21:19 UTC+1 schreef Roy Yu:
you seem to run on cloud infra that reduces your requested 40 Gb to 33 Gb (see https://databricks.com/session_na20/running-apache-spark-on-kubernetes-best-practices-and-pitfalls). Fact of life.
---------------------
Sorry Marc I misled you. Error Message was generated when I set spark.executor.memory to 30G, when it failed, I increased spark.executor.memory to 40G, it failed either. I felt desperate and come here to ask for help
On Tuesday, December 8, 2020 at 10:35:19 AM UTC+8 Roy Yu wrote:
Hi Marc

Thanks for your immediate response.
I've tried to set spark.yarn.executor.memoryOverhead=10G and re-run the task, and it stilled failed. From the spark task UI, I saw 80% of processing time is Full GC time. As you said, 2.6GB(GZ compressed) exploding is my root cause. Now I'm trying to reduce my region size to 1GB, if that will still fail, I'm gonna config the hbase hfile not using compressed format.
This was my first time running janusgraph OLAP, and I think this is a common promblom, as HBase region size 2.6GB(compressed) is not large, 20GB is very common in our production. If the community dose not solve the promblem, the Janusgraph HBase based OLAP solution cannot be adopted by other companies either.

On Tuesday, December 8, 2020 at 12:40:40 AM UTC+8 HadoopMarc wrote:
Hi Roy,

There seem to be three things bothering you here:
you did not specify spark.yarn.executor.memoryOverhead, as the exception message says. Easily solved.
you seem to run on cloud infra that reduces your requested 40 Gb to 33 Gb (see https://databricks.com/session_na20/running-apache-spark-on-kubernetes-best-practices-and-pitfalls). Fact of life.
the janusgraph HBaseInputFormat use sentire HBase regions as hadoop partitions, which are fed into spark tasks. The 2.6Gb region size is for compressed binary data which explodes when expanded into java objects. This is your real problem.
I did not follow the latest status of janusgraph-hbase features for the HBaseInputFormat, but you have to somehow use spark with smaller partitions than an entire HBase region.
A long time ago, I had success with skipping the HBaseInputFormat and have spark executors connect to JanusGraph themselves. That is not a quick solution, though.

Best wishes,

Marc

Op maandag 7 december 2020 om 14:10:55 UTC+1 schreef Roy Yu:
Error message:
ExecutorLostFailure (executor 1 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 33.1 GB of 33 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714.

graph conifg:
spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:MaxGCPauseMillis=500 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/mnt/data_1/log/spark2/gc-spark%p.log
spark.executor.cores=1
spark.executor.memory=40960m
spark.executor.instances=3

Region info:
hdfs dfs -du -h /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc
67 134 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/.regioninfo
0 0 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/.tmp
2.6 G 5.1 G /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/e
0 0 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/f
0 0 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/g
0 0 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/h
0 0 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/i
0 0 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/l
0 0 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/m
0 0 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/recovered.edits
0 0 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/s
0 0 /apps/hbase/data/data/default/ky415/f069fafb3ee51d6a2e5bc2377b468bcc/t
root@~$

Anybody who can help me?

Re: Centric Indexes failing to support all conditions for better performance.

chrism <cmil...@...>

#5414

Thank you, looking forward to have profile() with such information added.
Cheers, CM

Show quoted text

On Tuesday, December 15, 2020 at 2:26:02 AM UTC+11 Boxuan Li wrote:

Hi Christopher,

I don't have any workaround in mind except testing and comparing query latencies.

I have created https://github.com/JanusGraph/janusgraph/issues/2283 which hopefully can be addressed before the next release. That being said, there is no planned date for the next release yet.

Btw as I mentioned earlier, if you use "hasNot" it almost never leverages index - no matter if it's a mixed or composite or vertex-centric index.

Best regards,
Boxuan

chrism在 2020年12月14日星期一上午11:56:57 [UTC+8]寫道：
Thank you Boxuan Li,

It is obvious that your are an expert, is any other way apart of isFitted=true to know that index is used or not?
(It may be even debugging JanusGraph server or Cassandra)

We need to construct Gremlin query, to utilize these indexes in full, and always,... problem is just what to type,
as our implementation requires more complicated than above conditions to match, using above as sample it would be:
(rating >= value AND time < value) OR HasNot( time ) - means that "time" was not specified.
What is visible from profile() is that we cannot use coalesce() or or() steps, and trying all kind of workarounds
cannot be verified easily having isFitted=false and no other "good" indication of using indexes.

Cheers, Christopher

On Sunday, December 13, 2020 at 7:24:13 PM UTC+11 li...@... wrote:
Hi Christopher,

isFitted = true basically means no in-memory filtering is needed. If you see isFitted = false, it does not necessarily mean vertex-centric indexes are not used. It could be the case that some vertex-centric index is used, but further in-memory filtering is still needed.
If you see isFitted = false, it does not necessarily mean any index is used. It could be the case that you are fetching all edges of a given vertex.

I totally understand your confusion because the documentation does not explain how the vertex-centric index is built. In JanusGraph, vertices and edges are stored in the “edgestore” store, while composite indexes are stored in the “graphindex” store. Mixed indexes
are stored in external index store like Elasticsearch. This might be a bit counter-intuitive, but vertex-centric indexes are stored in the “edgestore” store. Recall how edges are stored (https://docs.janusgraph.org/advanced-topics/data-model/#individual-edge-layout):

Roughly speaking, If you don’t have any vertex-centric index, then your edge is stored once for one endpoint. If you have one vertex-centric index, then applicable edges are stored twice. If you have two vertex-centric indexes, then applicable edges are stored
three times… These edges, although seemingly duplicate, have different “sort key”s which conform to corresponding vertex-centric indexes. Let’s say you have built an “battlesByRating” vertex-centric index based on the property “rating”, then apart from the
ordinary edge, JanusGraph creates an additional edge whose “sort key” is the rating value. Because the “column” is sorted in the underlying data storage (e.g. “column” in JanusGraph model is mapped to “clustering column” in Cassandra), you essentially gain
the ability to search an index by “rating” value/range.

What happens when your vertex-centric index has two properties like the following?

> mgmt.buildEdgeIndex(battled, 'battlesByRatingAndTime', Direction.OUT, Order.asc, rating, time)

Now your “sort key” is a combination of “rating” and “time” (note “rating” comes before “time”). Under this vertex-centric index, “sort key”s look like this:

(rating=1, time=2), (rating=1, time=3), (rating=2, time=1), (rating=2, time=5), (rating=4, time=2), …

This explains why isFitted = true when your query is has('rating', 5.0).has('time', inside(10, 50)) but not when your query is has(’time', 5.0).has(‘rating', inside(10, 50)).Again, note that isFitted = false does not necessarily
mean your query is not optimized by vertex-centric index. I think the profiler shall be improved to state whether and which vertex-centric index is used.

I am not quite sure about the case b) you mentioned. Seems it’s a design consideration but right now I cannot tell why it is there.

“hasNot" almost never uses indexes because JanusGraph cannot index something that does not exist. (Note that “null” value is not valid in JanusGraph).

Hope this helps.

Best regards,
Boxuan

On Dec 10, 2020, at 11:01 AM, chrism <cm...@...> wrote:

JanusGraph documentation: https://docs.janusgraph.org/index-management/index-performance/
is describing usage of Vertex Centrix Index [edge=battled + properties=(rating,time)]
g.V(h).outE('battled').has('rating', 5.0).has('time', inside(10, 50)).inV()

From my understanding profile() of above is reporting \_isFitted=true
to indicate that backend-query delivered all results as conditions:
\_condition=(rating = 0.5 AND time > 10 AND time < 50 AND type[battled])

Two things are obvious from above: centric index is supporting multiple property keys, and equality and range/interval constraints.
However isFitted is false for all kind of conditions or combinations which are not really breaking the above rules, still in range constraints:

a) g.V(h).outE('battled').has('rating',lt(5.0)).has('time', inside(10, 50)).inV() // P.lt used for first key
b) g.V(h).outE('battled').has('rating',gt(5.0)) // P.gt used
c) g.V(h).outE('battled').or( hasNot('rating'), has('rating',eq(5.0)) ) // OrStep() used

Even b) can be "fitted" by has('rating',inside(5.0,Long.MAX_VALUE))
all that is very confusing, and probably not working as expected, what I am doing wrong?
as from my experience only one property key can be used for query conditions and using index, the second is ignored.

Having isFitted=false is not really improving performance, from my understanding,
when one only condition allows to get most of my edges and is asking to filter them in memory, as this is stated by implementation of BasicVertexCentricQueryBuilder.java.
Are there limitations not described in the JG doco? It is a glitch?

Can you offer explanation how to utilize Centric Indexes for edges in full support?

Christopher

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/f8fb537e-216a-462d-928b-ac906eb707a3n%40googlegroups.com.

Re: Property with multiple data types

Laura Morales <laur...@...>

#5416

Maybe I'm completely wrong, but would I be right to say that "labels" are the equivalent of Java classes? Like, one label represents a Java class and graph properties represent a class properties? So, saying that a node has label L would be like saying that a certain Java object is of class C? (That's why there's only one label per node). I was picturing labels as arbitrary strings that are attached to a node.
Then in Java, classes are namespaced to avoid collision. If my analogy is true, what would be the equivalent of namespaces with Janus?

Sent: Monday, December 14, 2020 at 2:29 PM
From: "HadoopMarc" <bi...@...>
To: "JanusGraph users" <janusgra...@...>
Subject: Re: Property with multiple data types

Hi Laura,

Things are a bit different than you ask:

a vertex has a single label onlya property key has a single datatype only, but it can be Object.class, see https://docs.janusgraph.org/basics/schema/#property-key-data-typeindices can have a label constraint, but these are not helpful if you want to mix the datatypes in a property for the same vertexI cannot predict well how the various janusgraph parts will behave when mixing up real integers and "string-integers" in a property key of the Object.class datatype. I guess that the gremlin traversals will have problems, while an indexing backend for MixedIndices probably can deal with it. The ref docs definitely advise to use the basic datatypes and spare yourself future headaches (so, unify the datatypes on ingestion).
Best wishes,    Marc

Op maandag 14 december 2020 om 09:07:31 UTC+1 schreef Laura Morales:
Thank you Marc, I think it does indeed! If I understand correctly, I can use labels to "namespace" my nodes, or in other words as a way to identify subgraphs.
If I have a node with 2 labels instead, say label1 and label2, I can create 2 indices for the same node, right? That is an index for label1.age (Integer) and an index for label2.age (String), both indices containing the same node. In this scenario I should be allowed to add 2 types of properties to the same node, one containing an Integer and the other one containing a String. Then query by choosing a specific label. Does this work? Can I do something like this?

Sent: Monday, December 14, 2020 at 8:01 AM
From: "HadoopMarc" <b...@...>
To: "JanusGraph users" <janu...@...>
Subject: Re: Property with multiple data types

Hi Laura,

Good that you pay close attention to understanding indices in JanusGraph because they are essential to proper use. Does the following section of the ref docs answers your question?

https://docs.janusgraph.org/index-management/index-performance/#label-constraint[https://docs.janusgraph.org/index-management/index-performance/#label-constraint]

Best wishes,    Marc 

Op zondag 13 december 2020 om 16:30:19 UTC+1 schreef Laura Morales:I'm new to Janus and LPGs. I have a question after reading the Janus documentation. As far as I understand, edges labels as well as properties (for both nodes and edges) are indexed globally. What happens when I have a sufficiently large graph, that completely unrelated and separate nodes want to use a property called with the same name but that holds different data types? For example, a property called "age" could be used by some nodes with a Integer type (eg. "age": 23), but other nodes on the other far-side of my big graph might want/need/require to use a String type (eg. "age": "twenty-seven"). Is this configuration possible with Janus? Or do I *have to* use two different names such as age_int and age_string?

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@...[mailto:janusgr...@...].
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/0b84be68-3688-46fe-a104-32baef119e2an%40googlegroups.com[https://groups.google.com/d/msgid/janusgraph-users/0b84be68-3688-46fe-a104-32baef119e2an%40googlegroups.com?utm_medium=email&utm_source=footer][https://groups.google.com/d/msgid/janusgraph-users/0b84be68-3688-46fe-a104-32baef119e2an%40googlegroups.com%5Bhttps://groups.google.com/d/msgid/janusgraph-users/0b84be68-3688-46fe-a104-32baef119e2an%40googlegroups.com?utm_medium=email&utm_source=footer]].

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgra...@...[mailto:janusgra...@...].
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/24ee1b44-3501-4d40-abef-b32aa345c959n%40googlegroups.com[https://groups.google.com/d/msgid/janusgraph-users/24ee1b44-3501-4d40-abef-b32aa345c959n%40googlegroups.com?utm_medium=email&utm_source=footer].

Re: Centric Indexes failing to support all conditions for better performance.

Boxuan Li <libox...@...>

#5410

Hi Christopher,

I don't have any workaround in mind except testing and comparing query latencies.

I have created https://github.com/JanusGraph/janusgraph/issues/2283 which hopefully can be addressed before the next release. That being said, there is no planned date for the next release yet.

Btw as I mentioned earlier, if you use "hasNot" it almost never leverages index - no matter if it's a mixed or composite or vertex-centric index.

Best regards,

Boxuan

chrism在 2020年12月14日星期一上午11:56:57 [UTC+8]寫道：

Show quoted text

Thank you Boxuan Li,

It is obvious that your are an expert, is any other way apart of isFitted=true to know that index is used or not?
(It may be even debugging JanusGraph server or Cassandra)

We need to construct Gremlin query, to utilize these indexes in full, and always,... problem is just what to type,
as our implementation requires more complicated than above conditions to match, using above as sample it would be:
(rating >= value AND time < value) OR HasNot( time ) - means that "time" was not specified.
What is visible from profile() is that we cannot use coalesce() or or() steps, and trying all kind of workarounds
cannot be verified easily having isFitted=false and no other "good" indication of using indexes.

Cheers, Christopher

On Sunday, December 13, 2020 at 7:24:13 PM UTC+11 li...@... wrote:
Hi Christopher,

isFitted = true basically means no in-memory filtering is needed. If you see isFitted = false, it does not necessarily mean vertex-centric indexes are not used. It could be the case that some vertex-centric index is used, but further in-memory filtering is still needed.
If you see isFitted = false, it does not necessarily mean any index is used. It could be the case that you are fetching all edges of a given vertex.

I totally understand your confusion because the documentation does not explain how the vertex-centric index is built. In JanusGraph, vertices and edges are stored in the “edgestore” store, while composite indexes are stored in the “graphindex” store. Mixed indexes
are stored in external index store like Elasticsearch. This might be a bit counter-intuitive, but vertex-centric indexes are stored in the “edgestore” store. Recall how edges are stored (https://docs.janusgraph.org/advanced-topics/data-model/#individual-edge-layout):

Roughly speaking, If you don’t have any vertex-centric index, then your edge is stored once for one endpoint. If you have one vertex-centric index, then applicable edges are stored twice. If you have two vertex-centric indexes, then applicable edges are stored
three times… These edges, although seemingly duplicate, have different “sort key”s which conform to corresponding vertex-centric indexes. Let’s say you have built an “battlesByRating” vertex-centric index based on the property “rating”, then apart from the
ordinary edge, JanusGraph creates an additional edge whose “sort key” is the rating value. Because the “column” is sorted in the underlying data storage (e.g. “column” in JanusGraph model is mapped to “clustering column” in Cassandra), you essentially gain
the ability to search an index by “rating” value/range.

What happens when your vertex-centric index has two properties like the following?

> mgmt.buildEdgeIndex(battled, 'battlesByRatingAndTime', Direction.OUT, Order.asc, rating, time)

Now your “sort key” is a combination of “rating” and “time” (note “rating” comes before “time”). Under this vertex-centric index, “sort key”s look like this:

(rating=1, time=2), (rating=1, time=3), (rating=2, time=1), (rating=2, time=5), (rating=4, time=2), …

This explains why isFitted = true when your query is has('rating', 5.0).has('time', inside(10, 50)) but not when your query is has(’time', 5.0).has(‘rating', inside(10, 50)).Again, note that isFitted = false does not necessarily
mean your query is not optimized by vertex-centric index. I think the profiler shall be improved to state whether and which vertex-centric index is used.

I am not quite sure about the case b) you mentioned. Seems it’s a design consideration but right now I cannot tell why it is there.

“hasNot" almost never uses indexes because JanusGraph cannot index something that does not exist. (Note that “null” value is not valid in JanusGraph).

Hope this helps.

Best regards,
Boxuan

On Dec 10, 2020, at 11:01 AM, chrism <cm...@...> wrote:

JanusGraph documentation: https://docs.janusgraph.org/index-management/index-performance/
is describing usage of Vertex Centrix Index [edge=battled + properties=(rating,time)]
g.V(h).outE('battled').has('rating', 5.0).has('time', inside(10, 50)).inV()

From my understanding profile() of above is reporting \_isFitted=true
to indicate that backend-query delivered all results as conditions:
\_condition=(rating = 0.5 AND time > 10 AND time < 50 AND type[battled])

Two things are obvious from above: centric index is supporting multiple property keys, and equality and range/interval constraints.
However isFitted is false for all kind of conditions or combinations which are not really breaking the above rules, still in range constraints:

a) g.V(h).outE('battled').has('rating',lt(5.0)).has('time', inside(10, 50)).inV() // P.lt used for first key
b) g.V(h).outE('battled').has('rating',gt(5.0)) // P.gt used
c) g.V(h).outE('battled').or( hasNot('rating'), has('rating',eq(5.0)) ) // OrStep() used

Even b) can be "fitted" by has('rating',inside(5.0,Long.MAX_VALUE))
all that is very confusing, and probably not working as expected, what I am doing wrong?
as from my experience only one property key can be used for query conditions and using index, the second is ignored.

Having isFitted=false is not really improving performance, from my understanding,
when one only condition allows to get most of my edges and is asking to filter them in memory, as this is stated by implementation of BasicVertexCentricQueryBuilder.java.
Are there limitations not described in the JG doco? It is a glitch?

Can you offer explanation how to utilize Centric Indexes for edges in full support?

Christopher

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/f8fb537e-216a-462d-928b-ac906eb707a3n%40googlegroups.com.