Re: Compiling locally with gremlin-core 3.3.0-SNAPSHOT


sjudeng <sju...@...>
 

Hi Kedar,

I posted my full test procedure: https://gist.github.com/sjudeng/093687d5f435ddbf46ea1808fbc4b398

At the bottom of that page under "test OLAP traversal" there's the read-hbase.properties file (copied from read-cassandra.properties) that was used. Also note the common Spark configuration from the setup section that was also included in the file.

On Wednesday, June 21, 2017 at 11:43:46 PM UTC-5, Kedar Mhaswade wrote:
Hi Sjudeng,

Ah, I missed that one. Now I am able to run SGC, but I am still seeing issues with class loading [1]. Am I to copy some jar files from somewhere to the lib-folder of JG workspace (local clone)?

Is it possible for you to share with me the properties file with which you initialize the graph?
My config file is [2].

Thanks,
Kedar

[1]

Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.YarnSparkHadoopUtil

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:348)

at org.apache.spark.util.Utils$.classForName(Utils.scala:228)

at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:413)

[2]

#

# Hadoop Graph Configuration

#

gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph

# with my changes

gremlin.hadoop.graphInputFormat=org.janusgraph.hadoop.formats.cassandra.Cassandra3InputFormat

gremlin.hadoop.graphOutputFormat=org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoOutputFormat


gremlin.hadoop.jarsInDistributedCache=true

gremlin.hadoop.inputLocation=none

gremlin.hadoop.outputLocation=output


#

# JanusGraph Cassandra InputFormat configuration

#

janusgraphmr.ioformat.conf.storage.backend=cassandrathrift

janusgraphmr.ioformat.conf.storage.hostname=<ip1, ip2...>

janusgraphmr.ioformat.conf.storage.port=9160

janusgraphmr.ioformat.conf.storage.cassandra.keyspace=janusgraph

# more config

# default for timeout: 10000 ms = 10 s

janusgraphmr.ioformat.conf.storage.connection-timeout=100000 


#

# Apache Cassandra InputFormat configuration

#

cassandra.input.partitioner.class=org.apache.cassandra.dht.Murmur3Partitioner


#

# SparkGraphComputer Configuration

#

spark.master=yarn

spark.serializer=org.apache.spark.serializer.KryoSerializer

spark.executor.instances=100


# from vtslab on GH

spark.app.name=uGraph

spark.ui.port=4050

spark.executor.memory=2g

spark.executorEnv.SPARK_CONF_DIR=/etc/spark/conf

spark.executorEnv.HADOOP_CONF_DIR=/etc/hadoop/conf



On Wed, Jun 21, 2017 at 3:05 PM, sjudeng <s...@...> wrote:
Hi Kedar,

I think you just need to load the spark plugin.

gremlin> :plugin use tinkerpop.spark

On Wednesday, June 21, 2017 at 3:26:15 PM UTC-5, Kedar Mhaswade wrote:
Hi sjudeng,

Is this update missing SparkGraphComputer? After building tp33 branch, I get the following on gremlin console:

gremlin> graph1 = GraphFactory.open('conf/hadoop-graph/read-cassandra.properties')

==>hadoopgraph[cassandrainputformat->gryooutputformat]

gremlin> olap = graph1.traversal().withComputer(SparkGraphComputer)

No such property: SparkGraphComputer for class: groovysh_evaluate

What am I missing? How were you able to run the queries with Spark/Yarn?


Regards,

Kedar


On Wed, Jun 21, 2017 at 10:01 AM, Kedar Mhaswade <k...@...> wrote:


On Wed, Jun 21, 2017 at 4:01 AM, sjudeng <s...@...> wrote:
In case there's interest the branch in https://github.com/sjudeng/janusgraph/tree/tp33 has some very early work on building JanusGraph with TinkerPop 3.3.0-SNAPSHOT. At this point it's compiling and I've tested OLAP traversals and BulkLoaderVertexProgram without issue using Spark 2.1 on Yarn (Cloudera) with some small test graphs in HBase. Currently only in memory TinkerPop unit test suites have been run and there are a handful of failures related to profiling, graph I/O involving geometries and graph computer. The couple graph computer test failures may just be issues with FulgoraGraphComputer (e.g. not something more general that might affect SparkGraphComputer) but I'm not sure yet. I'll be pushing more commits and likely squashing/rebasing along the way so keep that in mind if pulling.

Thanks sjudeng!

I will take a look at the changes and try out some graphs against a Cassandra-3 backend (which means it will need my PR). Is there any chance that this will become a part of JanusGraph 0.2.0 release?

Regards,
Kedar
 

--
You received this message because you are subscribed to the Google Groups "JanusGraph developer list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-de...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "JanusGraph developer list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-de...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Join janusgraph-dev@lists.lfaidata.foundation to automatically receive all group messages.