Hi,

I am exploring Janusgraph bulk load via SparkGraphComputer, janusgraph has been setup as plugin to tinkerpop server and console, with HBase as underlying storage and Elasticsearch as external index store.

I am running this setup on MapR cluster and had to recompile Janusgraph to resolve guava specific conflicts (shaded guava with relocation).

Next I am trying out the example BulkLoaderVertexProgram code provided in Chapter 33, It works fine till I have composite and vertex centric indexes in my schema, but as soon as I define mixed indexes and execute same code I end up with following exception in my Spark Job in stage 2 of job 1 -

java.lang.NoClassDefFoundError: Could not initialize class org.janusgraph.diskstorage.es.ElasticSearchIndex

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:264)

        at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:56)

        at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)

        at org.janusgraph.diskstorage.Backend.getIndexes(Backend.java:464)

        at org.janusgraph.diskstorage.Backend.<init>(Backend.java:149)

        at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1850)

        at org.janusgraph.graphdb.database.StandardJanusGraph.<init>(StandardJanusGraph.java:134)

I have verified that all janusgraph specific jars are in spark executor classpath and mixed indexes work fine with GraphOfGod example.

First I want to understand is it right path to use BulkLoaderVertexProgram be used to add mixed indexes? or should I upload the data and build indexes thereafter?

let me know if any additional info is required to dig deeper.

~mbaxi

The class org.janusgraph.diskstorage.es.ElasticSearchIndex is in janusgraph-es-0.1.1.jar. If you're getting a NoClassDefFoundError, there's really not much more we can tell you other than be completely certain that the jar is on the appropriate classpath. Did you add janusgraph-*.jar only or did you add all jars in the $JANUSGRAPH_HOME/lib directory?

On Tuesday, August 22, 2017 at 1:28:18 PM UTC-4, mystic m wrote:

Hi,

I am exploring Janusgraph bulk load via SparkGraphComputer, janusgraph has been setup as plugin to tinkerpop server and console, with HBase as underlying storage and Elasticsearch as external index store.

I am running this setup on MapR cluster and had to recompile Janusgraph to resolve guava specific conflicts (shaded guava with relocation).

Next I am trying out the example BulkLoaderVertexProgram code provided in Chapter 33, It works fine till I have composite and vertex centric indexes in my schema, but as soon as I define mixed indexes and execute same code I end up with following exception in my Spark Job in stage 2 of job 1 -

java.lang.NoClassDefFoundError: Could not initialize class org.janusgraph.diskstorage.es.ElasticSearchIndex

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:264)

        at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:56)

        at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)

        at org.janusgraph.diskstorage.Backend.getIndexes(Backend.java:464)

        at org.janusgraph.diskstorage.Backend.<init>(Backend.java:149)

        at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1850)

        at org.janusgraph.graphdb.database.StandardJanusGraph.<init>(StandardJanusGraph.java:134)

I have verified that all janusgraph specific jars are in spark executor classpath and mixed indexes work fine with GraphOfGod example.

First I want to understand is it right path to use BulkLoaderVertexProgram be used to add mixed indexes? or should I upload the data and build indexes thereafter?

let me know if any additional info is required to dig deeper.

~mbaxi

You are right Jason that ElasticSearchIndex  class is in janusgraph-es-0.1.1.jar, also this jar is available in SPARK_EXECUTOR_CLASSPATH on all the nodes, I can see all janusgraph specific jars (lib + plugin folder) in Spark UI Environment tab and also in Yarn logs it gets added to spark classpath.

I will add few more details about the customizations done in our environment if that helps

1. To enable integration with MapR-DB and mfs, replaced all hadoop/spark/hbase jars bundled with janusgraph plugin with MapR specific jars.
2. In order to make Bulk Load withSparkGraphComputer work (no mixed indexes), shaded guava plugin in janusgraph-core and janusgraph-hbase-core
3. Change #2 made Bulk Load run successfully but broke integration with ElasticSearch, even graph = JanusGraphFactory.open('conf/janusgraph-hbase-es.properties') failed with NoClassDefFoundError for ElasticSearchIndex class.
4. Reverting back to originally bundled jars resolves #3 but breaks Bulk Load
5. Next I changed janusgraph-hadoop-core pom.xml to comment the test scope for janusgraph-es, which fixed #3 and I was able to execute GraphOfGods example with mixed index, this fix still breaks the Bulk Load (even without mixed index in schema definition.

I know all of above information is too wide in scope to be covered in a single question/discussion, but what I can conclude is that there is some integration issue when we want to use Janusgraph + HBase + Spark  + ES together which needs to be addressed correctly.

I think guava specific conflicts are root to these issues and resolving those correctly is required, If you have any insights to fixing this, please let me know.

~mbaxi

On Wednesday, August 23, 2017 at 6:41:32 PM UTC+5:30, Jason Plurad wrote:

The class org.janusgraph.diskstorage.es.ElasticSearchIndex is in janusgraph-es-0.1.1.jar. If you're getting a NoClassDefFoundError, there's really not much more we can tell you other than be completely certain that the jar is on the appropriate classpath. Did you add janusgraph-*.jar only or did you add all jars in the $JANUSGRAPH_HOME/lib directory?

On Tuesday, August 22, 2017 at 1:28:18 PM UTC-4, mystic m wrote:

Hi,

I am exploring Janusgraph bulk load via SparkGraphComputer, janusgraph has been setup as plugin to tinkerpop server and console, with HBase as underlying storage and Elasticsearch as external index store.

I am running this setup on MapR cluster and had to recompile Janusgraph to resolve guava specific conflicts (shaded guava with relocation).

Next I am trying out the example BulkLoaderVertexProgram code provided in Chapter 33, It works fine till I have composite and vertex centric indexes in my schema, but as soon as I define mixed indexes and execute same code I end up with following exception in my Spark Job in stage 2 of job 1 -

java.lang.NoClassDefFoundError: Could not initialize class org.janusgraph.diskstorage.es.ElasticSearchIndex

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:264)

        at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:56)

        at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)

        at org.janusgraph.diskstorage.Backend.getIndexes(Backend.java:464)

        at org.janusgraph.diskstorage.Backend.<init>(Backend.java:149)

        at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1850)

        at org.janusgraph.graphdb.database.StandardJanusGraph.<init>(StandardJanusGraph.java:134)

I have verified that all janusgraph specific jars are in spark executor classpath and mixed indexes work fine with GraphOfGod example.

First I want to understand is it right path to use BulkLoaderVertexProgram be used to add mixed indexes? or should I upload the data and build indexes thereafter?

let me know if any additional info is required to dig deeper.

~mbaxi

Hi m

You might also try the approach I explained in (also discussed in another thread on this forum):

http://yaaics.blogspot.nl/2017/07/configuring-janusgraph-for-spark-yarn.html

Here I show that you do not need the hadoop/hbase/spark jars of your specific distribution. If you get rid of the MapR spark-assembly you do not need the guava shading. The guava shading might be the cause of the ES problems somehow.

HTH,     Marc

Op woensdag 23 augustus 2017 18:56:32 UTC+2 schreef mystic m:

You are right Jason that ElasticSearchIndex  class is in janusgraph-es-0.1.1.jar, also this jar is available in SPARK_EXECUTOR_CLASSPATH on all the nodes, I can see all janusgraph specific jars (lib + plugin folder) in Spark UI Environment tab and also in Yarn logs it gets added to spark classpath.

I will add few more details about the customizations done in our environment if that helps

1. To enable integration with MapR-DB and mfs, replaced all hadoop/spark/hbase jars bundled with janusgraph plugin with MapR specific jars.
2. In order to make Bulk Load withSparkGraphComputer work (no mixed indexes), shaded guava plugin in janusgraph-core and janusgraph-hbase-core
3. Change #2 made Bulk Load run successfully but broke integration with ElasticSearch, even graph = JanusGraphFactory.open('conf/janusgraph-hbase-es.properties') failed with NoClassDefFoundError for ElasticSearchIndex class.
4. Reverting back to originally bundled jars resolves #3 but breaks Bulk Load
5. Next I changed janusgraph-hadoop-core pom.xml to comment the test scope for janusgraph-es, which fixed #3 and I was able to execute GraphOfGods example with mixed index, this fix still breaks the Bulk Load (even without mixed index in schema definition.

I know all of above information is too wide in scope to be covered in a single question/discussion, but what I can conclude is that there is some integration issue when we want to use Janusgraph + HBase + Spark  + ES together which needs to be addressed correctly.

I think guava specific conflicts are root to these issues and resolving those correctly is required, If you have any insights to fixing this, please let me know.

~mbaxi

On Wednesday, August 23, 2017 at 6:41:32 PM UTC+5:30, Jason Plurad wrote:

The class org.janusgraph.diskstorage.es.ElasticSearchIndex is in janusgraph-es-0.1.1.jar. If you're getting a NoClassDefFoundError, there's really not much more we can tell you other than be completely certain that the jar is on the appropriate classpath. Did you add janusgraph-*.jar only or did you add all jars in the $JANUSGRAPH_HOME/lib directory?

On Tuesday, August 22, 2017 at 1:28:18 PM UTC-4, mystic m wrote:

Hi,

I am exploring Janusgraph bulk load via SparkGraphComputer, janusgraph has been setup as plugin to tinkerpop server and console, with HBase as underlying storage and Elasticsearch as external index store.

I am running this setup on MapR cluster and had to recompile Janusgraph to resolve guava specific conflicts (shaded guava with relocation).

Next I am trying out the example BulkLoaderVertexProgram code provided in Chapter 33, It works fine till I have composite and vertex centric indexes in my schema, but as soon as I define mixed indexes and execute same code I end up with following exception in my Spark Job in stage 2 of job 1 -

java.lang.NoClassDefFoundError: Could not initialize class org.janusgraph.diskstorage.es.ElasticSearchIndex

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:264)

        at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:56)

        at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)

        at org.janusgraph.diskstorage.Backend.getIndexes(Backend.java:464)

        at org.janusgraph.diskstorage.Backend.<init>(Backend.java:149)

        at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1850)

        at org.janusgraph.graphdb.database.StandardJanusGraph.<init>(StandardJanusGraph.java:134)

I have verified that all janusgraph specific jars are in spark executor classpath and mixed indexes work fine with GraphOfGod example.

First I want to understand is it right path to use BulkLoaderVertexProgram be used to add mixed indexes? or should I upload the data and build indexes thereafter?

let me know if any additional info is required to dig deeper.

~mbaxi