Re: Issue with union() step when query.batch is set to true


Ted Wilmes <twi...@...>
 

Hi Marc,
Thanks for digging into this and trying it out. I should have added a bit more context in my previous post. The code I recently merged is an intermediate step towards full multiquery usage. Previously, we had a hole in our test suite that did not run the TinkerPop tests with query.batch enabled. These tests are now run, and the JanusGraphLocalQueryOptimizerStrategy will only enable multiquery on currently compatible steps. The next step towards multiquery nirvana is to get it working with the other steps. In light of this, I have entered https://github.com/JanusGraph/janusgraph/issues/1044 to track progress on this effort.

--Ted

On Tuesday, May 1, 2018 at 3:22:24 AM UTC-5, HadoopMarc wrote:
For information, a lookup in the PR shows:

private static final List<Class<? extends Step>> MULTIQUERY_INCOMPATIBLE_STEPS =
    Arrays.asList(RepeatStep.class, MatchStep.class, BranchStep.class, OptionalStep.class);

In other words, in this stage the examples that did not work for me are no longer executed as a multiquery. AFAICS the use of multiquery is not visible from the explain() step, so it might be useful to issue a WARN log message when people hit one of the MULTIQUERY_INCOMPATIBLE_STEPS while query.batch is set to true.

Cheers,    Marc



Op dinsdag 1 mei 2018 10:01:39 UTC+2 schreef HadoopMarc:
Hi Ted,

Good job, both my union() and emit() examples worked on the latest master branch!

Cheers,   Marc

Op maandag 30 april 2018 14:08:23 UTC+2 schreef HadoopMarc:

Probably related with the earlier posted issue on emit(): same kind of error with the union() step when query.batch is set to true.

My properties file:

# JanusGraph configuration sample: BerkeleyDB Java Edition
#
# This file opens a BDB JE instance in the directory
# /path/to/this/file/../db/berkeley.

# The primary persistence provider used by JanusGraph.  This is required.
# It should be set one of JanusGraph's built-in shorthand names for its
# standard storage backends (shorthands: berkeleyje, cassandrathrift,
# cassandra, astyanax, embeddedcassandra, cql, hbase, inmemory) or to the
# full package and classname of a custom/third-party StoreManager
# implementation.
#
# Default:    (no default value)
# Data Type:  String
# Mutability: LOCAL
storage
.backend=berkeleyje

# Storage directory for those storage backends that require local storage.
#
# Default:    (no default value)
# Data Type:  String
# Mutability: LOCAL
storage
.directory=../db/multiquery

query
.batch=true



The query below does not really make sense, but is a stripped down version of my originally failing query.

The console output for query.batch=true:

gremlin> graph = JanusGraphFactory.open('JGWORK/janusgraph-berkeleyje-multiquery.properties')
==>standardjanusgraph[berkeleyje:/tera/lib/janusgraph-0.2.0-hadoop2/JGWORK/../db/multiquery]
gremlin
> g = graph.traversal()
==>graphtraversalsource[standardjanusgraph[berkeleyje:/tera/lib/janusgraph-0.2.0-hadoop2/JGWORK/../db/multiquery], standard]
gremlin
> g.V().valueMap(true)
13:55:10 WARN  org.janusgraph.graphdb.transaction.StandardJanusGraphTx  - Query requires iterating over all vertices [()]. For better performance, use indexes
==>[id:4096,name:[alcmene],label:human,age:[45]]
==>[id:4128,name:[saturn],label:titan,age:[10000]]
==>[id:8224,name:[jupiter],label:god,age:[5000]]
==>[id:12320,name:[neptune],label:god,age:[4500]]
==>[id:4152,name:[hydra],label:monster]
==>[id:4184,name:[sky],label:location]
==>[id:8280,name:[nemean],label:monster]
==>[id:12376,name:[tartarus],label:location]
==>[id:4208,name:[hercules],label:demigod,age:[30]]
==>[id:8304,name:[cerberus],label:monster]
==>[id:4264,name:[sea],label:location]
==>[id:8360,name:[pluto],label:god,age:[4000]]
gremlin
> g.V(8280,12320).valueMap()
==>[name:[nemean]]
==>[name:[neptune],age:[4500]]
gremlin
> g.V(8280).union(bothE(), outE())
==>e[3ym-38w-7x1-6e0][4208-battled->8280]
gremlin
> g.V(12320).union(bothE(), outE())
==>e[6pw-9i8-9hx-3ag][12320-lives->4264]
==>e[744-9i8-b2t-6cg][12320-brother->8224]
==>e[7ic-9i8-b2t-6g8][12320-brother->8360]
==>e[5xg-6cg-b2t-9i8][8224-brother->12320]
==>e[3kl-6g8-b2t-9i8][8360-brother->12320]
==>e[6pw-9i8-9hx-3ag][12320-lives->4264]
==>e[744-9i8-b2t-6cg][12320-brother->8224]
==>e[7ic-9i8-b2t-6g8][12320-brother->8360]
gremlin
> g.V(8280,12320).union(bothE(), outE())
==>e[3ym-38w-7x1-6e0][4208-battled->8280]
java
.lang.NullPointerException
Type ':help' or ':h' for help.
Display stack trace? [yN]y
java
.lang.NullPointerException
    at org
.janusgraph.graphdb.tinkerpop.optimize.JanusGraphVertexStep.flatMap(JanusGraphVertexStep.java:109)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.map.FlatMapStep.processNextStart(FlatMapStep.java:49)
    at org
.janusgraph.graphdb.tinkerpop.optimize.JanusGraphVertexStep.processNextStart(JanusGraphVertexStep.java:102)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.next(ExpandableStepIterator.java:50)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.ComputerAwareStep$EndStep.processNextStart(ComputerAwareStep.java:76)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
    at org
.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:192)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.branch.BranchStep.standardAlgorithm(BranchStep.java:94)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.ComputerAwareStep.processNextStart(ComputerAwareStep.java:46)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
    at org
.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:192)
    at org
.apache.tinkerpop.gremlin.console.Console$_closure3.doCall(Console.groovy:212)
    at sun
.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
    at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java
.lang.reflect.Method.invoke(Method.java:498)
    at org
.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy
.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at org
.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
    at groovy
.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at org
.codehaus.groovy.tools.shell.Groovysh.setLastResult(Groovysh.groovy:447)
    at org
.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:191)
    at org
.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
    at sun
.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
    at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java
.lang.reflect.Method.invoke(Method.java:498)
    at org
.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy
.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy
.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
    at org
.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:72)
    at org
.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
    at org
.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:95)
    at org
.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
    at sun
.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
    at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java
.lang.reflect.Method.invoke(Method.java:498)
    at org
.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy
.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy
.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
    at org
.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:124)
    at org
.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
    at org
.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
    at sun
.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun
.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java
.lang.reflect.Method.invoke(Method.java:498)
    at org
.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy
.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy
.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
    at org
.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:83)
    at org
.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
    at org
.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:166)
    at org
.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
    at org
.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:478)
gremlin
>




The console output for query.batch=false:

gremlin> graph = JanusGraphFactory.open('conf/janusgraph-berkeleyje.properties')
==>standardjanusgraph[berkeleyje:/tera/lib/janusgraph-0.2.0-hadoop2/conf/../db/berkeley]
gremlin
> g=graph.traversal()
==>graphtraversalsource[standardjanusgraph[berkeleyje:/tera/lib/janusgraph-0.2.0-hadoop2/conf/../db/berkeley], standard]
gremlin
> g.V()
13:44:27 WARN  org.janusgraph.graphdb.transaction.StandardJanusGraphTx  - Query requires iterating over all vertices [()]. For better performance, use indexes
==>v[4144]
==>v[4184]
==>v[8280]
==>v[12376]
==>v[4208]
==>v[8304]
==>v[12400]
==>v[4232]
==>v[8328]
==>v[4256]
==>v[4272]
==>v[4296]
gremlin
> g.V(4144, 4208).valueMap()
==>[name:[nemean]]
==>[name:[neptune],age:[4500]]
gremlin
> g.V(4144).union(bothE(), outE())
==>e[2s6-3ao-7x1-374][4272-battled->4144]
gremlin
> g.V(4208).union(bothE(), outE())
==>e[55a-38w-9hx-39k][4208-lives->4232]
==>e[5ji-38w-b2t-6fc][4208-brother->8328]
==>e[5xq-38w-b2t-9kg][4208-brother->12400]
==>e[3yp-6fc-b2t-38w][8328-brother->4208]
==>e[6q6-9kg-b2t-38w][12400-brother->4208]
==>e[55a-38w-9hx-39k][4208-lives->4232]
==>e[5ji-38w-b2t-6fc][4208-brother->8328]
==>e[5xq-38w-b2t-9kg][4208-brother->12400]
gremlin
> g.V(4144, 4208).union(bothE(), outE())
==>e[2s6-3ao-7x1-374][4272-battled->4144]
==>e[55a-38w-9hx-39k][4208-lives->4232]
==>e[5ji-38w-b2t-6fc][4208-brother->8328]
==>e[5xq-38w-b2t-9kg][4208-brother->12400]
==>e[3yp-6fc-b2t-38w][8328-brother->4208]
==>e[6q6-9kg-b2t-38w][12400-brother->4208]
==>e[55a-38w-9hx-39k][4208-lives->4232]
==>e[5ji-38w-b2t-6fc][4208-brother->8328]
==>e[5xq-38w-b2t-9kg][4208-brother->12400]
gremlin
>



Cheers,     Marc

Join janusgraph-users@lists.lfaidata.foundation to automatically receive all group messages.