1. Janusgraph-Users
  2. Messages

Re: [BLOG] Configuring JanusGraph for spark-yarn

HadoopMarc <bi...@...>
 

Hi John

The funny thing is, the recipe does not use the HDP Spark installation at all!  SparkGraphComputer creates a SparkContext and has Yarn start all the Spark machinery. So spark versions do not matter at all, though Spark 2.x requires some other config properties (see the recent PR's on github TinkerPop).

The only interaction with the cluster Spark is for the Spark History server, but I did not notice any problems between Spark 1.6.1 and Spark 1.6.2. See your cluster spark-defaults.xml for the history configs.

Have fun!

Marc



Op maandag 25 september 2017 23:17:53 UTC+2 schreef John Helmsen:

Marc,

Thank you so much for the help in getting Spark 1.6.1 to work with JanusGraph.  We've gotten good use out of it, but now we come to a crossroads.

Our customer wants us to deploy it on their cluster, but their cluster runs Spark 1.6.2.  I noticed that you confirmed the operation of the Spark-YARN-JanusGraph on a HDP 2.5 stack, which typically is running 1.6.2.  Does the setup that we've already gone through transfer to 1.6.2?  If there are problems, what could you anticipate that they might be?

On Thursday, July 6, 2017 at 4:15:37 AM UTC-4, HadoopMarc wrote:

Readers wanting to run OLAP queries on a real spark-yarn cluster might want to check my recent post:

http://yaaics.blogspot.nl/2017/07/configuring-janusgraph-for-spark-yarn.html

Regards,  Marc
Join {janusgraph-users@lists.lfaidata.foundation to automatically receive all group messages.