Date   

Re: Problems encountered when using hbase as a backend store

HadoopMarc <bi...@...>
 

Hi ...,

The conf/janusgraph-hbase.properties file from the JanusGraph distribution assumes you have a HBase server running on localhost. Is that assumption valid on your system?

Cheers,    Marc

Op vrijdag 4 mei 2018 23:35:17 UTC+2 schreef Zeph:

Did you create the table in HBase?


On Friday, May 4, 2018 at 10:13:16 AM UTC-4, liduoXu wrote:
Problems encountered when using hbase as a backend store, whether it is using JavaAPI connection or using gremlin client connection, all errors, can help solve it, thank you.
Here is the error log:
gremlin> JanusGraphFactory.open('conf/janusgraph-hbase.properties')
14:44:13 WARN  org.janusgraph.diskstorage.hbase.HBaseStoreManager  - Unexpected exception during getDeployment()
java.lang.RuntimeException: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:364)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getFeatures(HBaseStoreManager.java:403)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1377)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:80)
at org.janusgraph.core.JanusGraphFactory$open.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at groovysh_evaluate.run(groovysh_evaluate:3)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:70)
at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:191)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:72)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:83)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:166)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:478)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:739)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getLocalKeyPartition(HBaseStoreManager.java:524)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:361)
... 54 more
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:319)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:797)
at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:406)
at org.janusgraph.diskstorage.hbase.HBaseAdmin1_0.tableExists(HBaseAdmin1_0.java:111)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:716)
... 56 more
14:45:12 WARN  org.janusgraph.diskstorage.hbase.HBaseStoreManager  - Unexpected exception during getDeployment()
java.lang.RuntimeException: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:364)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getFeatures(HBaseStoreManager.java:403)
at org.janusgraph.diskstorage.Backend.getStandaloneGlobalConfiguration(Backend.java:442)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1378)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:80)
at org.janusgraph.core.JanusGraphFactory$open.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at groovysh_evaluate.run(groovysh_evaluate:3)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:70)
at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:191)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:72)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:83)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:166)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:478)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:739)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getLocalKeyPartition(HBaseStoreManager.java:524)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:361)
... 55 more
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:319)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:797)
at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:406)
at org.janusgraph.diskstorage.hbase.HBaseAdmin1_0.tableExists(HBaseAdmin1_0.java:111)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:716)
... 57 more


Re: JanusGraph using spark on yarn issue

HadoopMarc <bi...@...>
 

Hi Jiang,

You can first check my blog on this subject to check whether you missed configurations.

Also, beware of the issue of missing hadoop jars in the JanusGraph-0.2.0 binary distribution files.

HTH,    Marc

Op zaterdag 5 mei 2018 11:18:24 UTC+2 schreef jiang xingkai:

Hi All
  
       I am try to use janusgraph 0.2.0 and spark 1.6.1 on yarn to compute g.V().count()
       It got the error message :GetNewApplicationRequestProto cannot be cast to com.google.protobuf.Message
  

 gremlin
> g.V().count()
16:44:07 ERROR org.apache.spark.SparkContext  - Error initializing SparkContext.
java
.lang.ClassCastException: org.apache.hadoop.yarn.proto.YarnServiceProtos$GetNewApplicationRequestProto cannot be cast to com.google.protobuf.Message
        at org
.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:226)
        at com
.sun.proxy.$Proxy22.getNewApplication(Unknown Source)
        at org
.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getNewApplication(ApplicationClientProtocolPBClientImpl.java:193)
        at sun
.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun
.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java
.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com
.sun.proxy.$Proxy23.getNewApplication(Unknown Source)
        at org
.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNewApplication(YarnClientImpl.java:191)
        at org
.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:199)
        at org
.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:132)
        at org
.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
        at org
.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
        at org
.apache.spark.SparkContext.<init>(SparkContext.scala:530)
        at org
.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2281)
        at org
.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
        at org
.apache.tinkerpop.gremlin.spark.structure.Spark.create(Spark.java:52)
        at org
.apache.tinkerpop.gremlin.spark.structure.Spark.create(Spark.java:60)
        at org
.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer.lambda$submitWithExecutor$0(SparkGraphComputer.java:193)
        at java
.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java
.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java
.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java
.lang.Thread.run(Thread.java:748)
16:44:08 WARN  org.apache.spark.metrics.MetricsSystem  - Stopping a MetricsSystem that is not running
java
.lang.ClassCastException: org.apache.hadoop.yarn.proto.YarnServiceProtos$GetNewApplicationRequestProto cannot be cast to com.google.protobuf.Message
Type ':help' or ':h' for help.

  SparkGraphComputer Configuaration
spark.master=yarn-client
spark
.driver.memory=2g
spark
.serializer=org.apache.spark.serializer.KryoSerializer
spark
.executor.extraClassPath=/opt/janusgraph/lib/*:/opt/hadoop/etc/hadoop:/opt/spark/conf

Can anyone help me solve this problem?


JanusGraph using spark on yarn issue

jiang xingkai <xingka...@...>
 

Hi All
  
       I am try to use janusgraph 0.2.0 and spark 1.6.1 on yarn to compute g.V().count()
       It got the error message :GetNewApplicationRequestProto cannot be cast to com.google.protobuf.Message
  

 gremlin
> g.V().count()
16:44:07 ERROR org.apache.spark.SparkContext  - Error initializing SparkContext.
java
.lang.ClassCastException: org.apache.hadoop.yarn.proto.YarnServiceProtos$GetNewApplicationRequestProto cannot be cast to com.google.protobuf.Message
        at org
.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:226)
        at com
.sun.proxy.$Proxy22.getNewApplication(Unknown Source)
        at org
.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getNewApplication(ApplicationClientProtocolPBClientImpl.java:193)
        at sun
.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun
.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java
.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com
.sun.proxy.$Proxy23.getNewApplication(Unknown Source)
        at org
.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNewApplication(YarnClientImpl.java:191)
        at org
.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:199)
        at org
.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:132)
        at org
.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
        at org
.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
        at org
.apache.spark.SparkContext.<init>(SparkContext.scala:530)
        at org
.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2281)
        at org
.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
        at org
.apache.tinkerpop.gremlin.spark.structure.Spark.create(Spark.java:52)
        at org
.apache.tinkerpop.gremlin.spark.structure.Spark.create(Spark.java:60)
        at org
.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer.lambda$submitWithExecutor$0(SparkGraphComputer.java:193)
        at java
.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java
.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java
.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java
.lang.Thread.run(Thread.java:748)
16:44:08 WARN  org.apache.spark.metrics.MetricsSystem  - Stopping a MetricsSystem that is not running
java
.lang.ClassCastException: org.apache.hadoop.yarn.proto.YarnServiceProtos$GetNewApplicationRequestProto cannot be cast to com.google.protobuf.Message
Type ':help' or ':h' for help.

  SparkGraphComputer Configuaration
spark.master=yarn-client
spark
.driver.memory=2g
spark
.serializer=org.apache.spark.serializer.KryoSerializer
spark
.executor.extraClassPath=/opt/janusgraph/lib/*:/opt/hadoop/etc/hadoop:/opt/spark/conf

Can anyone help me solve this problem?


Re: Problems encountered when using hbase as a backend store

Zeph <ashley...@...>
 

Did you create the table in HBase?


On Friday, May 4, 2018 at 10:13:16 AM UTC-4, liduoXu wrote:
Problems encountered when using hbase as a backend store, whether it is using JavaAPI connection or using gremlin client connection, all errors, can help solve it, thank you.
Here is the error log:
gremlin> JanusGraphFactory.open('conf/janusgraph-hbase.properties')
14:44:13 WARN  org.janusgraph.diskstorage.hbase.HBaseStoreManager  - Unexpected exception during getDeployment()
java.lang.RuntimeException: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:364)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getFeatures(HBaseStoreManager.java:403)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1377)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:80)
at org.janusgraph.core.JanusGraphFactory$open.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at groovysh_evaluate.run(groovysh_evaluate:3)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:70)
at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:191)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:72)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:83)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:166)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:478)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:739)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getLocalKeyPartition(HBaseStoreManager.java:524)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:361)
... 54 more
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:319)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:797)
at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:406)
at org.janusgraph.diskstorage.hbase.HBaseAdmin1_0.tableExists(HBaseAdmin1_0.java:111)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:716)
... 56 more
14:45:12 WARN  org.janusgraph.diskstorage.hbase.HBaseStoreManager  - Unexpected exception during getDeployment()
java.lang.RuntimeException: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:364)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getFeatures(HBaseStoreManager.java:403)
at org.janusgraph.diskstorage.Backend.getStandaloneGlobalConfiguration(Backend.java:442)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1378)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:80)
at org.janusgraph.core.JanusGraphFactory$open.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at groovysh_evaluate.run(groovysh_evaluate:3)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:70)
at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:191)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:72)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:83)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:166)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:478)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:739)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getLocalKeyPartition(HBaseStoreManager.java:524)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:361)
... 55 more
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:319)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:797)
at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:406)
at org.janusgraph.diskstorage.hbase.HBaseAdmin1_0.tableExists(HBaseAdmin1_0.java:111)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:716)
... 57 more


Support for very large numbers

peter.f...@...
 

Hi, I'm very new to JanusGraph.

I have checked online what data types are supported and could not find whether large numbers (i.e. up to 256bit) are supported.
I found a GitHub Issue suggesting "Support BigDecimal as a native type", but I guess that can take some time.

Is there a way to work with large numbers in JanusGraph? (even if not natively?)


Problems encountered when using hbase as a backend store

liduoXu <10030...@...>
 

Problems encountered when using hbase as a backend store, whether it is using JavaAPI connection or using gremlin client connection, all errors, can help solve it, thank you.
Here is the error log:
gremlin> JanusGraphFactory.open('conf/janusgraph-hbase.properties')
14:44:13 WARN  org.janusgraph.diskstorage.hbase.HBaseStoreManager  - Unexpected exception during getDeployment()
java.lang.RuntimeException: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:364)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getFeatures(HBaseStoreManager.java:403)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1377)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:80)
at org.janusgraph.core.JanusGraphFactory$open.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at groovysh_evaluate.run(groovysh_evaluate:3)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:70)
at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:191)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:72)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:95)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:124)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:83)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:166)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:478)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:739)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getLocalKeyPartition(HBaseStoreManager.java:524)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:361)
... 54 more
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:319)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:797)
at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:406)
at org.janusgraph.diskstorage.hbase.HBaseAdmin1_0.tableExists(HBaseAdmin1_0.java:111)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:716)
... 56 more
14:45:12 WARN  org.janusgraph.diskstorage.hbase.HBaseStoreManager  - Unexpected exception during getDeployment()
java.lang.RuntimeException: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:364)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getFeatures(HBaseStoreManager.java:403)
at org.janusgraph.diskstorage.Backend.getStandaloneGlobalConfiguration(Backend.java:442)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1378)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:80)
at org.janusgraph.core.JanusGraphFactory$open.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at groovysh_evaluate.run(groovysh_evaluate:3)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:70)
at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:191)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:72)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:95)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:124)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:83)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:166)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:478)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:739)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getLocalKeyPartition(HBaseStoreManager.java:524)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:361)
... 55 more
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:319)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:797)
at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:406)
at org.janusgraph.diskstorage.hbase.HBaseAdmin1_0.tableExists(HBaseAdmin1_0.java:111)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:716)
... 57 more


Re: Unexpected behaviour by JanusGraph

Vaibhav vasant <vaibhavv...@...>
 

This issue seems to have been referenced in https://github.com/JanusGraph/janusgraph/issues/500
Quoting it "There are no transaction-level cache invalidation or expiration times expressed or implied that I can see. There is cache.tx-cache-size. There's a min tx-cache-size of 100 due to thinkaurelius/titan#893 . The tx appears to pick up new vertices (I did not try indexed values) but it does not recognize new or mutated properties on those vertices once cached". I am assuming since i am trying to mutate the property of source vertex by adding the new edge to it. So sometimes it is reading from cache and sometimes from storage. And closing the connection solves the problem. Still i don't think that closing the connection should be only solution as it is expensive reopening it. There should be a better way to clear transaction cache.


On Thursday, May 3, 2018 at 6:28:56 PM UTC+5:30, Vaibhav vasant wrote:
Hello,

I have a static JanusGraph object , so that it just needs to be connect to its storage back-end only once.
We have a use case where we create vertex and edge using Java.
The code for adding Vertex is : -
Class JanusGraphAdder {
static JanusGraph graph;

void addVertex()  {
Vertex vertex = graph.addVertex();
graph.tx().commit();
}

void addEdge(Vertex v) {
  v.addEdge("label",destination vertex);
graph.tx().commit();
}
}
The unexpected behavior comes up while trying to retrieve the edges of source vertex, sometimes it retrieves the updated data, sometimes it retrieves the data prior to addition of vertex. This behavior is not reproduced once i include graph.close() after committing. 
I thought it is reading stale data because of cache, so i removed database level cache and also transaction level cache i set the size as 0. Still the problem is persisting. Any idea?




Re: GetNewApplicationRequestProto cannot be cast to com.google.protobuf Message

jiang xingkai <xingka...@...>
 

Hi,have you solved the problem?

在 2018年4月9日星期一 UTC+8下午4:09:07,tan...@...写道:

While i change to spark 1.6.3 and hadoop 2.6.5, this problem still exists!look forward to your help!
在 2018年4月9日星期一 UTC+8上午10:04:26,Jason Plurad写道:
Check the version compatibility matrix. JanusGraph 0.2.0 is not compatible with Spark 2.11.
http://docs.janusgraph.org/latest/version-compat.html


On Sunday, April 8, 2018 at 9:38:48 PM UTC-4, tangshouzhang3 wrote:
When i use JanusGraph2.0 and spark2.11 on yarn to computer e.g.:
gremlin> graph = GraphFactory.open('conf/hadoop-graph/hadoop-load.properties')
==>hadoopgraph[gryoinputformat->nulloutputformat]
gremlin
> blvp = BulkLoaderVertexProgram.build().writeGraph('conf/janusgraph-cassandra.properties').create(graph)
==>BulkLoaderVertexProgram[bulkLoader=IncrementalBulkLoader,vertexIdProperty=bulkLoader.vertex.id,userSuppliedIds=false,keepOriginalIds=true,batchSize=0]
gremlin
> graph.compute(SparkGraphComputer).program(blvp).submit().get()

it throws an exception:
08:57:03 ERROR org.apache.spark.SparkContext  - Error initializing SparkContext.
java
.lang.ClassCastException: org.apache.hadoop.yarn.proto.YarnServiceProtos$GetNewApplicationRequestProto cannot be cast to com.google.protobuf.Message
        at org
.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:226)
        at com
.sun.proxy.$Proxy23.getNewApplication(Unknown Source)

Can anyone help me solve this problem?


Re: full-text via all properties

Ted Wilmes <twi...@...>
 

To do that, you'll need to add all of your properties into the mixed index. See here for more details: http://docs.janusgraph.org/latest/indexes.html#index-mixed.

Here's a small sample from that section where two properties are added:

mgmt.buildIndex('nameAndAge', Vertex.class).addKey(name).addKey(age).buildMixedIndex("search")

--Ted


On Thursday, May 3, 2018 at 8:58:08 AM UTC-5, 10951...@... wrote:
Hi
  We used janusgraph recently and adpoted Elasticsearch as its index backend.  A function such as  g.V().has(propertyKey , textContains(text)) provides me a very excellent full-text search on a specified propertyKey.
But I want a full-text search via all properties rather than a specified  propertyKey. Is JanusGraph satisfied my hope?


Re: Unexpected behaviour by JanusGraph

Vaibhav vasant <vaibhavv...@...>
 

Also after adding edge i set it's properties.

void addEdge(Vertex v) {
  Edge edge = v.addEdge("label",destination vertex);
edge.proprty("key","value");
graph.tx().commit();
}


On Thursday, May 3, 2018 at 6:28:56 PM UTC+5:30, Vaibhav vasant wrote:
Hello,

I have a static JanusGraph object , so that it just needs to be connect to its storage back-end only once.
We have a use case where we create vertex and edge using Java.
The code for adding Vertex is : -
Class JanusGraphAdder {
static JanusGraph graph;

void addVertex()  {
Vertex vertex = graph.addVertex();
graph.tx().commit();
}

void addEdge(Vertex v) {
  v.addEdge("label",destination vertex);
graph.tx().commit();
}
}
The unexpected behavior comes up while trying to retrieve the edges of source vertex, sometimes it retrieves the updated data, sometimes it retrieves the data prior to addition of vertex. This behavior is not reproduced once i include graph.close() after committing. 
I thought it is reading stale data because of cache, so i removed database level cache and also transaction level cache i set the size as 0. Still the problem is persisting. Any idea?




Unexpected behaviour by JanusGraph

Vaibhav vasant <vaibhavv...@...>
 

Hello,

I have a static JanusGraph object , so that it just needs to be connect to its storage back-end only once.
We have a use case where we create vertex and edge using Java.
The code for adding Vertex is : -
Class JanusGraphAdder {
static JanusGraph graph;

void addVertex()  {
Vertex vertex = graph.addVertex();
graph.tx().commit();
}

void addEdge(Vertex v) {
  v.addEdge("label",destination vertex);
graph.tx().commit();
}
}
The unexpected behavior comes up while trying to retrieve the edges of source vertex, sometimes it retrieves the updated data, sometimes it retrieves the data prior to addition of vertex. This behavior is not reproduced once i include graph.close() after committing. 
I thought it is reading stale data because of cache, so i removed database level cache and also transaction level cache i set the size as 0. Still the problem is persisting. Any idea?




Backup with indexing data

shrikant pachauri <sk.pa...@...>
 

Hi

I am currently using cassandra as storage backend and lucemne as indexing backend. Suppose I have to run some experiments on the inserted data so i want to take backup of the graph data along with the indexing data. I have a directory named searchIndex in DB folder of janus graph.

So any help how we can acheive this duplicationg the db folder does not seems to work properly.

Regards,
Shrikant 


full-text via all properties

10951...@...
 

Hi
  We used janusgraph recently and adpoted Elasticsearch as its index backend.  A function such as  g.V().has(propertyKey , textContains(text)) provides me a very excellent full-text search on a specified propertyKey.
But I want a full-text search via all properties rather than a specified  propertyKey. Is JanusGraph satisfied my hope?


Re: Creating the database

shapeshifter <shapesh...@...>
 

I totally agree with this, and some of what the OP said too.

While it's not a perfect analogue, any book on (my/ms)sql (or similar) would have entire chapters early on about creating and connecting to databases (graphs).  All the info I find on graph databases is either about traversals, connecting to DBs/indexers, or very low-level stuff for advanced users.  I've spent more time on this than I care to admit (PAID WORK time, not spare time) and most of that time is going around in circles.

It's almost like everyone has their server of choice (janus or whatever) set up using whichever single graph the server is set up with.  I have managed to create a new graph via the console, and have altered the config to get a traversal source from it on startup, but this is hardly dynamic. I want to be able to created named, empty graphs on demand, and select them arbitrarily, both in and outside the console.

While Kelvin's online book is excellent, he also confuses me by showing how to open a graph in Janus, but not showing anything about naming or persistence (how would I get access this new graph in the future? How do I get back to the air-routes graph again that was also opened, populated, but seemingly unnamed? That was assigned to graph too). ConfiguredGraphFactory appears to be the only way to do this(?)

Mind blown.

I note the link to the graph of the gods setup source code. This could be useful, but I'm using gremlin.net which doesn't seem to have Factory classes and only lets me connect to existing traversal sources (which I have to create manually on the console etc etc). I'm wondering if I'd be better off running the server as a REST API if I could use it for open/create commands as well as traversals.

I'd like to end on a positive note, so I'll say that in fairness, once I have connected to a graph, I have no issues with traversals, and there seems plenty of info on these.


On Thursday, 22 March 2018 15:53:38 UTC, Kaidong Wu wrote:
Hi Rohit, I am also new to Graph database, and I have the same feeling like you do, Janusgraph doc is having too limited details about the operations (creation, loading, editing, deleting database)

Kevin


How to replay a transaction log from the begining

Sandeep Mishra <sandy...@...>
 

Hi All,

I have created a transaction log in hbase as backend using following code.

JanusGraphTransaction tx = graph.buildTransaction().logIdentifier("addedPerson").start();

JanusGraphVertex u = tx.addVertex("human");
u.property("name", "proteros");
u.property("age", 36);

JanusGraphVertex u1 = tx.addVertex("human");
u.property("name", "sandeep");
u.property("age", 34);
tx.commit();


and trying to replay it using below code.

LogProcessorFramework logProcessor1 = JanusGraphFactory.openTransactionLog(graph);
logProcessor1.addLogProcessor("addedPerson").setStartTime(Instant.EPOCH).setProcessorIdentifier("addedPersonProcessor").setStartTime(Instant.EPOCH).addProcessor(new ChangeProcessor() {
@Override
public void process(JanusGraphTransaction janusGraphTransaction, TransactionId transactionId, ChangeState changeState) {
System.out.println("processing old log");
for (Vertex v : changeState.getVertices(Change.ADDED)) {
System.out.println("vertex " + v.label());
if (v.label().equals("human")) totalHumansAdded.incrementAndGet();
}
}
}).build();

However, my logProcessor doesn't process any previously saved vertices and only detect changes which are performed in after the LogProcessor has been built.

I tried setting up Start Time to INSTANT.EPOCH, which as per documentation should make log processing to start from BEGINING but no help.
The only difference is see in logs  is the MessagePuller is started from EPOCH.

22:59:19,039  INFO KCVSLog:744 - Loaded unidentified ReadMarker start time 2018-05-01T14:59:19.023Z into org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller@57db2b13
22:59:19,054  INFO KCVSLog:748 - Loaded indentified ReadMarker start time 1970-01-01T02:18:20Z into org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller@2f953efd
22:59:19,054  INFO KCVSLog:748 - Loaded indentified ReadMarker start time 1970-01-01T02:18:20Z into org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller@14bdbc74
22:59:19,054  INFO KCVSLog:748 - Loaded indentified ReadMarker start time 1970-01-01T02:18:20Z into org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller@5a7fe64f
22:59:19,054  INFO KCVSLog:748 - Loaded indentified ReadMarker start time 1970-01-01T02:18:20Z into org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller@41330d4f

Another point to note is, an vertex inserted using another Java Instance is detected by logprocessor instances running in any other Java Instance.

Looks like its a bug as it doesn't behave as per documentation.
When a log processor is built against a particular log, such as the addedPerson log in the example above, it will start reading transactional change records from the log immediately upon successful construction and initialization up to the head of the log. The start time specified in the builder marks the time point in the log where the log processor will start reading records. Optionally, one can specify an identifier for the log processor in the builder. The log processor will use the identifier to regularly persist its state of processing, i.e. it will maintain a marker on the last read log record. If the log processor is later restarted with the same identifier, it will continue reading from the last read record. This is particularly useful when the log processor is supposed to run for long periods of time and is therefore likely to fail. In such failure situations, the log processor can simply be restarted with the same identifier. It must be ensured that log processor identifiers are unique in a JanusGraph cluster in order to avoid conflicts on the persisted read markers.



Any comments???

Regards,
Sandeep


Re: Issue with union() step when query.batch is set to true

Ted Wilmes <twi...@...>
 

Hi Marc,
Thanks for digging into this and trying it out. I should have added a bit more context in my previous post. The code I recently merged is an intermediate step towards full multiquery usage. Previously, we had a hole in our test suite that did not run the TinkerPop tests with query.batch enabled. These tests are now run, and the JanusGraphLocalQueryOptimizerStrategy will only enable multiquery on currently compatible steps. The next step towards multiquery nirvana is to get it working with the other steps. In light of this, I have entered https://github.com/JanusGraph/janusgraph/issues/1044 to track progress on this effort.

--Ted

On Tuesday, May 1, 2018 at 3:22:24 AM UTC-5, HadoopMarc wrote:
For information, a lookup in the PR shows:

private static final List<Class<? extends Step>> MULTIQUERY_INCOMPATIBLE_STEPS =
    Arrays.asList(RepeatStep.class, MatchStep.class, BranchStep.class, OptionalStep.class);

In other words, in this stage the examples that did not work for me are no longer executed as a multiquery. AFAICS the use of multiquery is not visible from the explain() step, so it might be useful to issue a WARN log message when people hit one of the MULTIQUERY_INCOMPATIBLE_STEPS while query.batch is set to true.

Cheers,    Marc



Op dinsdag 1 mei 2018 10:01:39 UTC+2 schreef HadoopMarc:
Hi Ted,

Good job, both my union() and emit() examples worked on the latest master branch!

Cheers,   Marc

Op maandag 30 april 2018 14:08:23 UTC+2 schreef HadoopMarc:

Probably related with the earlier posted issue on emit(): same kind of error with the union() step when query.batch is set to true.

My properties file:

# JanusGraph configuration sample: BerkeleyDB Java Edition
#
# This file opens a BDB JE instance in the directory
# /path/to/this/file/../db/berkeley.

# The primary persistence provider used by JanusGraph.  This is required.
# It should be set one of JanusGraph's built-in shorthand names for its
# standard storage backends (shorthands: berkeleyje, cassandrathrift,
# cassandra, astyanax, embeddedcassandra, cql, hbase, inmemory) or to the
# full package and classname of a custom/third-party StoreManager
# implementation.
#
# Default:    (no default value)
# Data Type:  String
# Mutability: LOCAL
storage
.backend=berkeleyje

# Storage directory for those storage backends that require local storage.
#
# Default:    (no default value)
# Data Type:  String
# Mutability: LOCAL
storage
.directory=../db/multiquery

query
.batch=true



The query below does not really make sense, but is a stripped down version of my originally failing query.

The console output for query.batch=true:

gremlin> graph = JanusGraphFactory.open('JGWORK/janusgraph-berkeleyje-multiquery.properties')
==>standardjanusgraph[berkeleyje:/tera/lib/janusgraph-0.2.0-hadoop2/JGWORK/../db/multiquery]
gremlin
> g = graph.traversal()
==>graphtraversalsource[standardjanusgraph[berkeleyje:/tera/lib/janusgraph-0.2.0-hadoop2/JGWORK/../db/multiquery], standard]
gremlin
> g.V().valueMap(true)
13:55:10 WARN  org.janusgraph.graphdb.transaction.StandardJanusGraphTx  - Query requires iterating over all vertices [()]. For better performance, use indexes
==>[id:4096,name:[alcmene],label:human,age:[45]]
==>[id:4128,name:[saturn],label:titan,age:[10000]]
==>[id:8224,name:[jupiter],label:god,age:[5000]]
==>[id:12320,name:[neptune],label:god,age:[4500]]
==>[id:4152,name:[hydra],label:monster]
==>[id:4184,name:[sky],label:location]
==>[id:8280,name:[nemean],label:monster]
==>[id:12376,name:[tartarus],label:location]
==>[id:4208,name:[hercules],label:demigod,age:[30]]
==>[id:8304,name:[cerberus],label:monster]
==>[id:4264,name:[sea],label:location]
==>[id:8360,name:[pluto],label:god,age:[4000]]
gremlin
> g.V(8280,12320).valueMap()
==>[name:[nemean]]
==>[name:[neptune],age:[4500]]
gremlin
> g.V(8280).union(bothE(), outE())
==>e[3ym-38w-7x1-6e0][4208-battled->8280]
gremlin
> g.V(12320).union(bothE(), outE())
==>e[6pw-9i8-9hx-3ag][12320-lives->4264]
==>e[744-9i8-b2t-6cg][12320-brother->8224]
==>e[7ic-9i8-b2t-6g8][12320-brother->8360]
==>e[5xg-6cg-b2t-9i8][8224-brother->12320]
==>e[3kl-6g8-b2t-9i8][8360-brother->12320]
==>e[6pw-9i8-9hx-3ag][12320-lives->4264]
==>e[744-9i8-b2t-6cg][12320-brother->8224]
==>e[7ic-9i8-b2t-6g8][12320-brother->8360]
gremlin
> g.V(8280,12320).union(bothE(), outE())
==>e[3ym-38w-7x1-6e0][4208-battled->8280]
java
.lang.NullPointerException
Type ':help' or ':h' for help.
Display stack trace? [yN]y
java
.lang.NullPointerException
    at org
.janusgraph.graphdb.tinkerpop.optimize.JanusGraphVertexStep.flatMap(JanusGraphVertexStep.java:109)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.map.FlatMapStep.processNextStart(FlatMapStep.java:49)
    at org
.janusgraph.graphdb.tinkerpop.optimize.JanusGraphVertexStep.processNextStart(JanusGraphVertexStep.java:102)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.next(ExpandableStepIterator.java:50)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.ComputerAwareStep$EndStep.processNextStart(ComputerAwareStep.java:76)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
    at org
.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:192)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.branch.BranchStep.standardAlgorithm(BranchStep.java:94)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.ComputerAwareStep.processNextStart(ComputerAwareStep.java:46)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
    at org
.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:192)
    at org
.apache.tinkerpop.gremlin.console.Console$_closure3.doCall(Console.groovy:212)
    at sun
.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
    at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java
.lang.reflect.Method.invoke(Method.java:498)
    at org
.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy
.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at org
.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
    at groovy
.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at org
.codehaus.groovy.tools.shell.Groovysh.setLastResult(Groovysh.groovy:447)
    at org
.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:191)
    at org
.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
    at sun
.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
    at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java
.lang.reflect.Method.invoke(Method.java:498)
    at org
.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy
.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy
.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
    at org
.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:72)
    at org
.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
    at org
.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:95)
    at org
.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
    at sun
.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
    at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java
.lang.reflect.Method.invoke(Method.java:498)
    at org
.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy
.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy
.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
    at org
.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:124)
    at org
.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
    at org
.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
    at sun
.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun
.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java
.lang.reflect.Method.invoke(Method.java:498)
    at org
.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy
.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy
.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
    at org
.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:83)
    at org
.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
    at org
.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:166)
    at org
.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
    at org
.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:478)
gremlin
>




The console output for query.batch=false:

gremlin> graph = JanusGraphFactory.open('conf/janusgraph-berkeleyje.properties')
==>standardjanusgraph[berkeleyje:/tera/lib/janusgraph-0.2.0-hadoop2/conf/../db/berkeley]
gremlin
> g=graph.traversal()
==>graphtraversalsource[standardjanusgraph[berkeleyje:/tera/lib/janusgraph-0.2.0-hadoop2/conf/../db/berkeley], standard]
gremlin
> g.V()
13:44:27 WARN  org.janusgraph.graphdb.transaction.StandardJanusGraphTx  - Query requires iterating over all vertices [()]. For better performance, use indexes
==>v[4144]
==>v[4184]
==>v[8280]
==>v[12376]
==>v[4208]
==>v[8304]
==>v[12400]
==>v[4232]
==>v[8328]
==>v[4256]
==>v[4272]
==>v[4296]
gremlin
> g.V(4144, 4208).valueMap()
==>[name:[nemean]]
==>[name:[neptune],age:[4500]]
gremlin
> g.V(4144).union(bothE(), outE())
==>e[2s6-3ao-7x1-374][4272-battled->4144]
gremlin
> g.V(4208).union(bothE(), outE())
==>e[55a-38w-9hx-39k][4208-lives->4232]
==>e[5ji-38w-b2t-6fc][4208-brother->8328]
==>e[5xq-38w-b2t-9kg][4208-brother->12400]
==>e[3yp-6fc-b2t-38w][8328-brother->4208]
==>e[6q6-9kg-b2t-38w][12400-brother->4208]
==>e[55a-38w-9hx-39k][4208-lives->4232]
==>e[5ji-38w-b2t-6fc][4208-brother->8328]
==>e[5xq-38w-b2t-9kg][4208-brother->12400]
gremlin
> g.V(4144, 4208).union(bothE(), outE())
==>e[2s6-3ao-7x1-374][4272-battled->4144]
==>e[55a-38w-9hx-39k][4208-lives->4232]
==>e[5ji-38w-b2t-6fc][4208-brother->8328]
==>e[5xq-38w-b2t-9kg][4208-brother->12400]
==>e[3yp-6fc-b2t-38w][8328-brother->4208]
==>e[6q6-9kg-b2t-38w][12400-brother->4208]
==>e[55a-38w-9hx-39k][4208-lives->4232]
==>e[5ji-38w-b2t-6fc][4208-brother->8328]
==>e[5xq-38w-b2t-9kg][4208-brother->12400]
gremlin
>



Cheers,     Marc


Re: Issue with union() step when query.batch is set to true

HadoopMarc <bi...@...>
 

For information, a lookup in the PR shows:

private static final List<Class<? extends Step>> MULTIQUERY_INCOMPATIBLE_STEPS =
    Arrays.asList(RepeatStep.class, MatchStep.class, BranchStep.class, OptionalStep.class);

In other words, in this stage the examples that did not work for me are no longer executed as a multiquery. AFAICS the use of multiquery is not visible from the explain() step, so it might be useful to issue a WARN log message when people hit one of the MULTIQUERY_INCOMPATIBLE_STEPS while query.batch is set to true.

Cheers,    Marc



Op dinsdag 1 mei 2018 10:01:39 UTC+2 schreef HadoopMarc:

Hi Ted,

Good job, both my union() and emit() examples worked on the latest master branch!

Cheers,   Marc

Op maandag 30 april 2018 14:08:23 UTC+2 schreef HadoopMarc:

Probably related with the earlier posted issue on emit(): same kind of error with the union() step when query.batch is set to true.

My properties file:

# JanusGraph configuration sample: BerkeleyDB Java Edition
#
# This file opens a BDB JE instance in the directory
# /path/to/this/file/../db/berkeley.

# The primary persistence provider used by JanusGraph.  This is required.
# It should be set one of JanusGraph's built-in shorthand names for its
# standard storage backends (shorthands: berkeleyje, cassandrathrift,
# cassandra, astyanax, embeddedcassandra, cql, hbase, inmemory) or to the
# full package and classname of a custom/third-party StoreManager
# implementation.
#
# Default:    (no default value)
# Data Type:  String
# Mutability: LOCAL
storage
.backend=berkeleyje

# Storage directory for those storage backends that require local storage.
#
# Default:    (no default value)
# Data Type:  String
# Mutability: LOCAL
storage
.directory=../db/multiquery

query
.batch=true



The query below does not really make sense, but is a stripped down version of my originally failing query.

The console output for query.batch=true:

gremlin> graph = JanusGraphFactory.open('JGWORK/janusgraph-berkeleyje-multiquery.properties')
==>standardjanusgraph[berkeleyje:/tera/lib/janusgraph-0.2.0-hadoop2/JGWORK/../db/multiquery]
gremlin
> g = graph.traversal()
==>graphtraversalsource[standardjanusgraph[berkeleyje:/tera/lib/janusgraph-0.2.0-hadoop2/JGWORK/../db/multiquery], standard]
gremlin
> g.V().valueMap(true)
13:55:10 WARN  org.janusgraph.graphdb.transaction.StandardJanusGraphTx  - Query requires iterating over all vertices [()]. For better performance, use indexes
==>[id:4096,name:[alcmene],label:human,age:[45]]
==>[id:4128,name:[saturn],label:titan,age:[10000]]
==>[id:8224,name:[jupiter],label:god,age:[5000]]
==>[id:12320,name:[neptune],label:god,age:[4500]]
==>[id:4152,name:[hydra],label:monster]
==>[id:4184,name:[sky],label:location]
==>[id:8280,name:[nemean],label:monster]
==>[id:12376,name:[tartarus],label:location]
==>[id:4208,name:[hercules],label:demigod,age:[30]]
==>[id:8304,name:[cerberus],label:monster]
==>[id:4264,name:[sea],label:location]
==>[id:8360,name:[pluto],label:god,age:[4000]]
gremlin
> g.V(8280,12320).valueMap()
==>[name:[nemean]]
==>[name:[neptune],age:[4500]]
gremlin
> g.V(8280).union(bothE(), outE())
==>e[3ym-38w-7x1-6e0][4208-battled->8280]
gremlin
> g.V(12320).union(bothE(), outE())
==>e[6pw-9i8-9hx-3ag][12320-lives->4264]
==>e[744-9i8-b2t-6cg][12320-brother->8224]
==>e[7ic-9i8-b2t-6g8][12320-brother->8360]
==>e[5xg-6cg-b2t-9i8][8224-brother->12320]
==>e[3kl-6g8-b2t-9i8][8360-brother->12320]
==>e[6pw-9i8-9hx-3ag][12320-lives->4264]
==>e[744-9i8-b2t-6cg][12320-brother->8224]
==>e[7ic-9i8-b2t-6g8][12320-brother->8360]
gremlin
> g.V(8280,12320).union(bothE(), outE())
==>e[3ym-38w-7x1-6e0][4208-battled->8280]
java
.lang.NullPointerException
Type ':help' or ':h' for help.
Display stack trace? [yN]y
java
.lang.NullPointerException
    at org
.janusgraph.graphdb.tinkerpop.optimize.JanusGraphVertexStep.flatMap(JanusGraphVertexStep.java:109)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.map.FlatMapStep.processNextStart(FlatMapStep.java:49)
    at org
.janusgraph.graphdb.tinkerpop.optimize.JanusGraphVertexStep.processNextStart(JanusGraphVertexStep.java:102)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.next(ExpandableStepIterator.java:50)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.ComputerAwareStep$EndStep.processNextStart(ComputerAwareStep.java:76)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
    at org
.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:192)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.branch.BranchStep.standardAlgorithm(BranchStep.java:94)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.ComputerAwareStep.processNextStart(ComputerAwareStep.java:46)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
    at org
.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:192)
    at org
.apache.tinkerpop.gremlin.console.Console$_closure3.doCall(Console.groovy:212)
    at sun
.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
    at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java
.lang.reflect.Method.invoke(Method.java:498)
    at org
.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy
.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at org
.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
    at groovy
.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at org
.codehaus.groovy.tools.shell.Groovysh.setLastResult(Groovysh.groovy:447)
    at org
.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:191)
    at org
.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
    at sun
.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
    at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java
.lang.reflect.Method.invoke(Method.java:498)
    at org
.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy
.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy
.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
    at org
.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:72)
    at org
.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
    at org
.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:95)
    at org
.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
    at sun
.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
    at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java
.lang.reflect.Method.invoke(Method.java:498)
    at org
.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy
.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy
.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
    at org
.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:124)
    at org
.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
    at org
.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
    at sun
.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun
.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java
.lang.reflect.Method.invoke(Method.java:498)
    at org
.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy
.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy
.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
    at org
.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:83)
    at org
.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
    at org
.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:166)
    at org
.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
    at org
.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:478)
gremlin
>




The console output for query.batch=false:

gremlin> graph = JanusGraphFactory.open('conf/janusgraph-berkeleyje.properties')
==>standardjanusgraph[berkeleyje:/tera/lib/janusgraph-0.2.0-hadoop2/conf/../db/berkeley]
gremlin
> g=graph.traversal()
==>graphtraversalsource[standardjanusgraph[berkeleyje:/tera/lib/janusgraph-0.2.0-hadoop2/conf/../db/berkeley], standard]
gremlin
> g.V()
13:44:27 WARN  org.janusgraph.graphdb.transaction.StandardJanusGraphTx  - Query requires iterating over all vertices [()]. For better performance, use indexes
==>v[4144]
==>v[4184]
==>v[8280]
==>v[12376]
==>v[4208]
==>v[8304]
==>v[12400]
==>v[4232]
==>v[8328]
==>v[4256]
==>v[4272]
==>v[4296]
gremlin
> g.V(4144, 4208).valueMap()
==>[name:[nemean]]
==>[name:[neptune],age:[4500]]
gremlin
> g.V(4144).union(bothE(), outE())
==>e[2s6-3ao-7x1-374][4272-battled->4144]
gremlin
> g.V(4208).union(bothE(), outE())
==>e[55a-38w-9hx-39k][4208-lives->4232]
==>e[5ji-38w-b2t-6fc][4208-brother->8328]
==>e[5xq-38w-b2t-9kg][4208-brother->12400]
==>e[3yp-6fc-b2t-38w][8328-brother->4208]
==>e[6q6-9kg-b2t-38w][12400-brother->4208]
==>e[55a-38w-9hx-39k][4208-lives->4232]
==>e[5ji-38w-b2t-6fc][4208-brother->8328]
==>e[5xq-38w-b2t-9kg][4208-brother->12400]
gremlin
> g.V(4144, 4208).union(bothE(), outE())
==>e[2s6-3ao-7x1-374][4272-battled->4144]
==>e[55a-38w-9hx-39k][4208-lives->4232]
==>e[5ji-38w-b2t-6fc][4208-brother->8328]
==>e[5xq-38w-b2t-9kg][4208-brother->12400]
==>e[3yp-6fc-b2t-38w][8328-brother->4208]
==>e[6q6-9kg-b2t-38w][12400-brother->4208]
==>e[55a-38w-9hx-39k][4208-lives->4232]
==>e[5ji-38w-b2t-6fc][4208-brother->8328]
==>e[5xq-38w-b2t-9kg][4208-brother->12400]
gremlin
>



Cheers,     Marc


Re: Issue emit() step with query.batch=true

HadoopMarc <bi...@...>
 

 
Solved in this thread.

Op maandag 30 april 2018 10:29:41 UTC+2 schreef HadoopMarc:


Inspired by the recent discussions around the multiquery I tried a bit around with the query.batch=true setting and hit on the following issue:

gremlin> graph = JanusGraphFactory.open('JGWORK/janusgraph-berkeleyje-multiquery.properties')
==>standardjanusgraph[berkeleyje:/tera/lib/janusgraph-0.2.0-hadoop2/JGWORK/../db/multiquery]
gremlin
> g=graph.traversal()
==>graphtraversalsource[standardjanusgraph[berkeleyje:/tera/lib/janusgraph-0.2.0-hadoop2/JGWORK/../db/multiquery], standard]
gremlin
>  GraphOfTheGodsFactory.loadWithoutMixedIndex(graph,true)
==>null
gremlin
> g.V()
10:10:43 WARN  org.janusgraph.graphdb.transaction.StandardJanusGraphTx  - Query requires iterating over all vertices [()]. For better performance, use indexes
==>v[4096]
==>v[4128]
==>v[8224]
==>v[12320]
==>v[4152]
==>v[4184]
==>v[8280]
==>v[12376]
==>v[4208]
==>v[8304]
==>v[4264]
==>v[8360]
gremlin
> g.V(4096).bothE()
==>e[3ke-38w-74l-35s][4208-mother->4096]
gremlin
> g.V(4096).emit().repeat(both()).times(1)
==>v[4096]
java
.lang.NullPointerException
Type ':help' or ':h' for help.
Display stack trace? [yN]y
java
.lang.NullPointerException
    at org
.janusgraph.graphdb.tinkerpop.optimize.JanusGraphVertexStep.flatMap(JanusGraphVertexStep.java:109)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.map.FlatMapStep.processNextStart(FlatMapStep.java:49)
    at org
.janusgraph.graphdb.tinkerpop.optimize.JanusGraphVertexStep.processNextStart(JanusGraphVertexStep.java:102)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.next(ExpandableStepIterator.java:50)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.branch.RepeatStep$RepeatEndStep.standardAlgorithm(RepeatStep.java:274)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.ComputerAwareStep.processNextStart(ComputerAwareStep.java:46)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.branch.RepeatStep.standardAlgorithm(RepeatStep.java:185)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.ComputerAwareStep.processNextStart(ComputerAwareStep.java:46)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
    at org
.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:192)
    at org
.apache.tinkerpop.gremlin.console.Console$_closure3.doCall(Console.groovy:212)
    at sun
.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
    at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java
.lang.reflect.Method.invoke(Method.java:498)
    at org
.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy
.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at org
.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
    at groovy
.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at org
.codehaus.groovy.tools.shell.Groovysh.setLastResult(Groovysh.groovy:447)
    at org
.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:191)
    at org
.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
    at sun
.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
    at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java
.lang.reflect.Method.invoke(Method.java:498)
    at org
.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy
.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy
.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
    at org
.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:72)
    at org
.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
    at org
.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:95)
    at org
.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
    at sun
.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
    at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java
.lang.reflect.Method.invoke(Method.java:498)
    at org
.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy
.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy
.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
    at org
.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:124)
    at org
.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
    at org
.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
    at sun
.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun
.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java
.lang.reflect.Method.invoke(Method.java:498)
    at org
.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy
.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy
.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
    at org
.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:83)
    at org
.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
    at org
.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:166)
    at org
.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
    at org
.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:478)
gremlin> g.V(4096).repeat(both()).times(1)
==>v[4208]
gremlin
> gremlin> g.V(4096).repeat(both()).times(2)
==>v[8224]
==>v[4096]
==>v[4152]
==>v[8280]
==>v[8304]

It seems the query.batch=true does not swallow the emit() step.

My janusgraph-berkeleyje-multiquery.properties file is just the conf/janusgraph-berkeleyje.properties file from the distribution with the query.batch=true line added.

Cheers,    Marc


Re: Issue with union() step when query.batch is set to true

HadoopMarc <bi...@...>
 

Hi Ted,

Good job, both my union() and emit() examples worked on the latest master branch!

Cheers,   Marc

Op maandag 30 april 2018 14:08:23 UTC+2 schreef HadoopMarc:


Probably related with the earlier posted issue on emit(): same kind of error with the union() step when query.batch is set to true.

My properties file:

# JanusGraph configuration sample: BerkeleyDB Java Edition
#
# This file opens a BDB JE instance in the directory
# /path/to/this/file/../db/berkeley.

# The primary persistence provider used by JanusGraph.  This is required.
# It should be set one of JanusGraph's built-in shorthand names for its
# standard storage backends (shorthands: berkeleyje, cassandrathrift,
# cassandra, astyanax, embeddedcassandra, cql, hbase, inmemory) or to the
# full package and classname of a custom/third-party StoreManager
# implementation.
#
# Default:    (no default value)
# Data Type:  String
# Mutability: LOCAL
storage
.backend=berkeleyje

# Storage directory for those storage backends that require local storage.
#
# Default:    (no default value)
# Data Type:  String
# Mutability: LOCAL
storage
.directory=../db/multiquery

query
.batch=true



The query below does not really make sense, but is a stripped down version of my originally failing query.

The console output for query.batch=true:

gremlin> graph = JanusGraphFactory.open('JGWORK/janusgraph-berkeleyje-multiquery.properties')
==>standardjanusgraph[berkeleyje:/tera/lib/janusgraph-0.2.0-hadoop2/JGWORK/../db/multiquery]
gremlin
> g = graph.traversal()
==>graphtraversalsource[standardjanusgraph[berkeleyje:/tera/lib/janusgraph-0.2.0-hadoop2/JGWORK/../db/multiquery], standard]
gremlin
> g.V().valueMap(true)
13:55:10 WARN  org.janusgraph.graphdb.transaction.StandardJanusGraphTx  - Query requires iterating over all vertices [()]. For better performance, use indexes
==>[id:4096,name:[alcmene],label:human,age:[45]]
==>[id:4128,name:[saturn],label:titan,age:[10000]]
==>[id:8224,name:[jupiter],label:god,age:[5000]]
==>[id:12320,name:[neptune],label:god,age:[4500]]
==>[id:4152,name:[hydra],label:monster]
==>[id:4184,name:[sky],label:location]
==>[id:8280,name:[nemean],label:monster]
==>[id:12376,name:[tartarus],label:location]
==>[id:4208,name:[hercules],label:demigod,age:[30]]
==>[id:8304,name:[cerberus],label:monster]
==>[id:4264,name:[sea],label:location]
==>[id:8360,name:[pluto],label:god,age:[4000]]
gremlin
> g.V(8280,12320).valueMap()
==>[name:[nemean]]
==>[name:[neptune],age:[4500]]
gremlin
> g.V(8280).union(bothE(), outE())
==>e[3ym-38w-7x1-6e0][4208-battled->8280]
gremlin
> g.V(12320).union(bothE(), outE())
==>e[6pw-9i8-9hx-3ag][12320-lives->4264]
==>e[744-9i8-b2t-6cg][12320-brother->8224]
==>e[7ic-9i8-b2t-6g8][12320-brother->8360]
==>e[5xg-6cg-b2t-9i8][8224-brother->12320]
==>e[3kl-6g8-b2t-9i8][8360-brother->12320]
==>e[6pw-9i8-9hx-3ag][12320-lives->4264]
==>e[744-9i8-b2t-6cg][12320-brother->8224]
==>e[7ic-9i8-b2t-6g8][12320-brother->8360]
gremlin
> g.V(8280,12320).union(bothE(), outE())
==>e[3ym-38w-7x1-6e0][4208-battled->8280]
java
.lang.NullPointerException
Type ':help' or ':h' for help.
Display stack trace? [yN]y
java
.lang.NullPointerException
    at org
.janusgraph.graphdb.tinkerpop.optimize.JanusGraphVertexStep.flatMap(JanusGraphVertexStep.java:109)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.map.FlatMapStep.processNextStart(FlatMapStep.java:49)
    at org
.janusgraph.graphdb.tinkerpop.optimize.JanusGraphVertexStep.processNextStart(JanusGraphVertexStep.java:102)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.next(ExpandableStepIterator.java:50)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.ComputerAwareStep$EndStep.processNextStart(ComputerAwareStep.java:76)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
    at org
.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:192)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.branch.BranchStep.standardAlgorithm(BranchStep.java:94)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.ComputerAwareStep.processNextStart(ComputerAwareStep.java:46)
    at org
.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
    at org
.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:192)
    at org
.apache.tinkerpop.gremlin.console.Console$_closure3.doCall(Console.groovy:212)
    at sun
.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
    at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java
.lang.reflect.Method.invoke(Method.java:498)
    at org
.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy
.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at org
.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
    at groovy
.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at org
.codehaus.groovy.tools.shell.Groovysh.setLastResult(Groovysh.groovy:447)
    at org
.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:191)
    at org
.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
    at sun
.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
    at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java
.lang.reflect.Method.invoke(Method.java:498)
    at org
.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy
.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy
.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
    at org
.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:72)
    at org
.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
    at org
.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:95)
    at org
.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
    at sun
.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
    at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java
.lang.reflect.Method.invoke(Method.java:498)
    at org
.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy
.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy
.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
    at org
.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:124)
    at org
.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
    at org
.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
    at sun
.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun
.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java
.lang.reflect.Method.invoke(Method.java:498)
    at org
.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy
.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy
.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
    at org
.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
    at org
.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:83)
    at org
.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
    at org
.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:166)
    at org
.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
    at org
.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:478)
gremlin
>




The console output for query.batch=false:

gremlin> graph = JanusGraphFactory.open('conf/janusgraph-berkeleyje.properties')
==>standardjanusgraph[berkeleyje:/tera/lib/janusgraph-0.2.0-hadoop2/conf/../db/berkeley]
gremlin
> g=graph.traversal()
==>graphtraversalsource[standardjanusgraph[berkeleyje:/tera/lib/janusgraph-0.2.0-hadoop2/conf/../db/berkeley], standard]
gremlin
> g.V()
13:44:27 WARN  org.janusgraph.graphdb.transaction.StandardJanusGraphTx  - Query requires iterating over all vertices [()]. For better performance, use indexes
==>v[4144]
==>v[4184]
==>v[8280]
==>v[12376]
==>v[4208]
==>v[8304]
==>v[12400]
==>v[4232]
==>v[8328]
==>v[4256]
==>v[4272]
==>v[4296]
gremlin
> g.V(4144, 4208).valueMap()
==>[name:[nemean]]
==>[name:[neptune],age:[4500]]
gremlin
> g.V(4144).union(bothE(), outE())
==>e[2s6-3ao-7x1-374][4272-battled->4144]
gremlin
> g.V(4208).union(bothE(), outE())
==>e[55a-38w-9hx-39k][4208-lives->4232]
==>e[5ji-38w-b2t-6fc][4208-brother->8328]
==>e[5xq-38w-b2t-9kg][4208-brother->12400]
==>e[3yp-6fc-b2t-38w][8328-brother->4208]
==>e[6q6-9kg-b2t-38w][12400-brother->4208]
==>e[55a-38w-9hx-39k][4208-lives->4232]
==>e[5ji-38w-b2t-6fc][4208-brother->8328]
==>e[5xq-38w-b2t-9kg][4208-brother->12400]
gremlin
> g.V(4144, 4208).union(bothE(), outE())
==>e[2s6-3ao-7x1-374][4272-battled->4144]
==>e[55a-38w-9hx-39k][4208-lives->4232]
==>e[5ji-38w-b2t-6fc][4208-brother->8328]
==>e[5xq-38w-b2t-9kg][4208-brother->12400]
==>e[3yp-6fc-b2t-38w][8328-brother->4208]
==>e[6q6-9kg-b2t-38w][12400-brother->4208]
==>e[55a-38w-9hx-39k][4208-lives->4232]
==>e[5ji-38w-b2t-6fc][4208-brother->8328]
==>e[5xq-38w-b2t-9kg][4208-brother->12400]
gremlin
>



Cheers,     Marc


Re: Running JanusGraph on resource-constrained devices

Ted Wilmes <twi...@...>
 

I think you could make that work. You could use the Berkeley DB backend for storage. Would you be running it embedded in another Java app you'd have running on the device?

--Ted

On Monday, April 30, 2018 at 11:14:57 AM UTC-5, ncn...@... wrote:
I'm new to JanusGraph, hence please excuse me if my question doesn't make sense;

Can JanusGraph run on resource-constrained devices? Such devices would feature prominently in IoT applications. For example, consider a device with 2 GB RAM and 100 GB storage.

Thanks.

4481 - 4500 of 6663