Re: [BLOG] Configuring JanusGraph for spark-yarn
toggle quoted message Show quoted text
TinkerPop's spark-gremlin module depends on spark-1.6.1, so when you install spark-gremlin in the gremlin-console or when you add it to your maven project, the spark-core-1.6.1.jar is already on your classpath. The configs in my recipe make sure all deps are also available to the spark-1.6.1 exuctors and application master on a Hadoop alias Yarn cluster. The cluster's spark-1.6.2 jars are never loaded when gremlin-console is used as in my recipe.
Using spark-submit would put spark-1.6.2 on the various classpaths, which would probably also work if it did not cause version conflict between the TinkerPop dependencies and the Spark dependencies.
Also, I believe your implicit assumption that it would be bad practice to put spark-1.6.2 jars on the classpath of a spark-1.6.1 application is not valid. Spark-1.6.2 should support all API's that a Spark-1.6.1 application can depend on (minor version difference).
I hope this clarifies things, configuring complex JVM apps is not for the weak-hearty.
Op dinsdag 26 september 2017 21:58:45 UTC+2 schreef John Helmsen: