Date 1 - 2 of 2
Running OLAP on HBase with SparkGraphComputer fails on shuffle/Pregel message pass
Yevgeniy Ignatyev <yevgeniy...@...>
Recently we faced an issue with running PageRank on HBase: for comparison purposes we loaded our graph from Cassandra to the HBase deployment of the same size and unlike on Cassandra - all attempts to run page rank on that graph fail with initial cause pointing to the SparkExecutor line 165 in spark-gremlin:
It always happens with the message in logs that container requested more memory that allowed by its configuration, like:
According to logs error consistently seems to occur on the first message pass phase of vertex program - right after the initial iteration.
Here is one of configurations we tried to run OLAP on HBase, with the same Spark related properties we use to perform queries on Cassandra:
We tried to increase memory per executor as much as we can and tweaking gremlin.spark.graphStorageLevel without any success.
Did anybody experience similar issues with running on SparkGraphComputer with HBaseInputFormat/HBaseSnapshotInputFormat or probably on other backends?
Roy Yu <7604...@...>
I have the same promblem, have you ever solved it?toggle quoted message Show quoted text
On Wednesday, May 30, 2018 at 5:30:14 PM UTC+8 yevg...@... wrote:
|1 - 2 of 2|