Re: [BLOG] Configuring JanusGraph for spark-yarn


Gariee <garim...@...>
 

Hi Marc, 

Request your help, I am runnign Janusgraph with maprdb as backend, I have successfuly been able to create GodofGraphs example on M7 as backend 

But when I am trying to execute the following where cluster is mapr and spark on yarn


plugin activated: tinkerpop.tinkergraph

gremlin> graph = GraphFactory.open('conf/hadoop-graph/hadoop-load.properties')

==>hadoopgraph[gryoinputformat->nulloutputformat]

gremlin>  g = graph.traversal(computer(SparkGraphComputer))

==>graphtraversalsource[hadoopgraph[gryoinputformat->nulloutputformat], sparkgraphcomputer]

gremlin> g.V().count()


hadoop-load.properties ( I tried all different combinations as commented below each time its the same error)


#

# SparkGraphComputer Configuration

#

spark.master=yarn-client


spark.yarn.queue=cmp

mapred.job.queue.name=cmp


#spark.driver.allowMultipleContexts=true

#spark.executor.memory=4g

#spark.ui.port=20000


spark.serializer=org.apache.spark.serializer.KryoSerializer

spark.yarn.appMasterEnv.CLASSPATH=$CLASSPATH:/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:/opt/mapr/lib/kvstore*.jar:/opt/mapr/lib/libprotodefs*.jar:/opt/mapr/lib/baseutils*.jar:/opt/mapr/lib/maprutil*.jar:/opt/mapr/lib/json-20080701.jar:/opt/mapr/lib/flexjson-2.1.jar 


#spark.executor.instances=10

#spark.executor.cores=2

#spark.executor.CoarseGrainedExecutorBackend.cores=2

#spark.executor.CoarseGrainedExecutorBackend.driver=FIXME

#spark.executor.CoarseGrainedExecutorBackend.stopping=false

#spark.streaming.stopGracefullyOnShutdown=true

#spark.yarn.driver.memoryOverhead=4g

#spark.yarn.executor.memoryOverhead=1024

#spark.yarn.am.extraJavaOptions=-Dhdp.version=2.3.0.0-2557


--------------------------------------------------------------------------------------------------------------

yarn log

------------------------------------------------------------------------------------------------------------

When the last command is executed, driver abruptly shuts down in  yarn container and shuts down the spark context too with following error from Yarn logs

Container: container_e27_1501284102300_47651_01_000008 on abcd.com_8039

LogType:stderr
Log Upload Time:Mon Jul 31 14:08:42 -0700 2017
LogLength:2441
Log Contents:
17/07/31 14:08:05 INFO executor.CoarseGrainedExecutorBackend: Registered signal handlers for [TERM, HUP, INT]
17/07/31 14:08:05 INFO spark.SecurityManager: Changing view acls to: cmphs
17/07/31 14:08:05 INFO spark.SecurityManager: Changing modify acls to: cmphs
17/07/31 14:08:05 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(cmphs); users with modify permissions: Set(cmphs)
17/07/31 14:08:06 INFO spark.SecurityManager: Changing view acls to: cmphs
17/07/31 14:08:06 INFO spark.SecurityManager: Changing modify acls to: cmphs
17/07/31 14:08:06 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(cmphs); users with modify permissions: Set(cmphs)
17/07/31 14:08:06 INFO slf4j.Slf4jLogger: Slf4jLogger started
17/07/31 14:08:06 INFO Remoting: Starting remoting
17/07/31 14:08:06 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkExecutorActorSystem@...:36376]
17/07/31 14:08:06 INFO util.Utils: Successfully started service 'sparkExecutorActorSystem' on port 36376.
17/07/31 14:08:06 INFO storage.DiskBlockManager: Created local directory at /tmp/hadoop-mapr/nm-local-dir/usercache/cmphs/appcache/application_1501284102300_47651/blockmgr-244e0062-016e-4402-85c9-69f2ab9ef9d2
17/07/31 14:08:06 INFO storage.MemoryStore: MemoryStore started with capacity 2.7 GB
17/07/31 14:08:06 INFO executor.CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@...:43768
17/07/31 14:08:06 INFO executor.CoarseGrainedExecutorBackend: Successfully registered with driver
17/07/31 14:08:06 INFO executor.Executor: Starting executor ID 7 on host abcd.com
17/07/31 14:08:06 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44208.
17/07/31 14:08:06 INFO netty.NettyBlockTransferService: Server created on 44208
17/07/31 14:08:06 INFO storage.BlockManagerMaster: Trying to register BlockManager
17/07/31 14:08:06 INFO storage.BlockManagerMaster: Registered BlockManager
17/07/31 14:08:41 INFO executor.CoarseGrainedExecutorBackend: Driver commanded a shutdown
17/07/31 14:08:41 INFO storage.MemoryStore: MemoryStore cleared
17/07/31 14:08:41 INFO storage.BlockManager: BlockManager stopped
17/07/31 14:08:41 INFO util.ShutdownHookManager: Shutdown hook called
End of LogType:stderr

LogType:stdout
Log Upload Time:Mon Jul 31 14:08:42 -0700 2017
LogLength:0
Log Contents:
End of LogType:stdout


I am stuck wit h this error , request help here.

Join {janusgraph-users@lists.lfaidata.foundation to automatically receive all group messages.