1. Messages
  2. [BLOG] Configuring JanusGraph for spark-yarn

Re: [BLOG] Configuring JanusGraph for spark-yarn

John Helmsen <john....@...>
 

HadoopMarc,

This sounds like this could be really good news, but please clarify something for me:

Tinkerpop 3.2.3 claims compatibility with only Spark 1.6.1, and currently JanusGraph-0.1.1 only supports up to Tinkerpop 3.2.3, so I assumed that JanusGraph would only support Spark 1.6.1.

Now I have two interpretations of your post that I need to have clarified:

1) You have made Spark 1.6.2 work (actually do computations) with JanusGraph-0.1.1.
2) There is a version of Spark 1.6.1 also on the cluster, and it is being called by JanusGraph-0.1.1 while Spark 1.6.2 is being ignored.

Either one is a workable option for me, but please elaborate so I am completely clear about what is happening.

On Tuesday, September 26, 2017 at 3:30:40 PM UTC-4, HadoopMarc wrote:

Hi John

The funny thing is, the recipe does not use the HDP Spark installation at all!  SparkGraphComputer creates a SparkContext and has Yarn start all the Spark machinery. So spark versions do not matter at all, though Spark 2.x requires some other config properties (see the recent PR's on github TinkerPop).

The only interaction with the cluster Spark is for the Spark History server, but I did not notice any problems between Spark 1.6.1 and Spark 1.6.2. See your cluster spark-defaults.xml for the history configs.

Have fun!

Marc



Op maandag 25 september 2017 23:17:53 UTC+2 schreef John Helmsen:
Marc,

Thank you so much for the help in getting Spark 1.6.1 to work with JanusGraph.  We've gotten good use out of it, but now we come to a crossroads.

Our customer wants us to deploy it on their cluster, but their cluster runs Spark 1.6.2.  I noticed that you confirmed the operation of the Spark-YARN-JanusGraph on a HDP 2.5 stack, which typically is running 1.6.2.  Does the setup that we've already gone through transfer to 1.6.2?  If there are problems, what could you anticipate that they might be?

On Thursday, July 6, 2017 at 4:15:37 AM UTC-4, HadoopMarc wrote:

Readers wanting to run OLAP queries on a real spark-yarn cluster might want to check my recent post:

http://yaaics.blogspot.nl/2017/07/configuring-janusgraph-for-spark-yarn.html

Regards,  Marc
Join {janusgraph-users@lists.lfaidata.foundation to automatically receive all group messages.