I am trying to execute the following where cluster is mapr and spark on yarn
graph=GraphFactory.open('conf/hadoop-graph/hadoop-load.properties') g = graph.traversal(computer(SparkGraphComputer)) g.V().count()
When the last command is executed, driver abruptly shuts down with with following trace from Yarn logs
Container: container_e27_1501284102300_47651_01_000008 on abcd.com_8039
LogType:stderr Log Upload Time:Mon Jul 31 14:08:42 -0700 2017 LogLength:2441 Log Contents: 17/07/31 14:08:05 INFO executor.CoarseGrainedExecutorBackend: Registered signal handlers for [TERM, HUP, INT] 17/07/31 14:08:05 INFO spark.SecurityManager: Changing view acls to: cmphs 17/07/31 14:08:05 INFO spark.SecurityManager: Changing modify acls to: cmphs 17/07/31 14:08:05 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(cmphs); users with modify permissions: Set(cmphs) 17/07/31 14:08:06 INFO spark.SecurityManager: Changing view acls to: cmphs 17/07/31 14:08:06 INFO spark.SecurityManager: Changing modify acls to: cmphs 17/07/31 14:08:06 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(cmphs); users with modify permissions: Set(cmphs) 17/07/31 14:08:06 INFO slf4j.Slf4jLogger: Slf4jLogger started 17/07/31 14:08:06 INFO Remoting: Starting remoting 17/07/31 14:08:06 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkExecutorActorSystem@...:36376] 17/07/31 14:08:06 INFO util.Utils: Successfully started service 'sparkExecutorActorSystem' on port 36376. 17/07/31 14:08:06 INFO storage.DiskBlockManager: Created local directory at /tmp/hadoop-mapr/nm-local-dir/usercache/cmphs/appcache/application_1501284102300_47651/blockmgr-244e0062-016e-4402-85c9-69f2ab9ef9d2 17/07/31 14:08:06 INFO storage.MemoryStore: MemoryStore started with capacity 2.7 GB 17/07/31 14:08:06 INFO executor.CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@...:43768 17/07/31 14:08:06 INFO executor.CoarseGrainedExecutorBackend: Successfully registered with driver 17/07/31 14:08:06 INFO executor.Executor: Starting executor ID 7 on host abcd.com 17/07/31 14:08:06 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44208. 17/07/31 14:08:06 INFO netty.NettyBlockTransferService: Server created on 44208 17/07/31 14:08:06 INFO storage.BlockManagerMaster: Trying to register BlockManager 17/07/31 14:08:06 INFO storage.BlockManagerMaster: Registered BlockManager 17/07/31 14:08:41 INFO executor.CoarseGrainedExecutorBackend: Driver commanded a shutdown 17/07/31 14:08:41 INFO storage.MemoryStore: MemoryStore cleared 17/07/31 14:08:41 INFO storage.BlockManager: BlockManager stopped 17/07/31 14:08:41 INFO util.ShutdownHookManager: Shutdown hook called End of LogType:stderr
LogType:stdout Log Upload Time:Mon Jul 31 14:08:42 -0700 2017 LogLength:0 Log Contents: End of LogType:stdout
