Re: Edge details dropped from vertex object in sparkGraphComputer
"anj...@gmail.com" <anjani...@...>
Hi All,
toggle quoted messageShow quoted text
Thanks for all your inputs. After doing some more analysis found that in SparkGraphComputer (in tinker-pop library), vertex object has edge details in RDD till we add result to memory. mapReduce.addResultToMemory(finalMemory, outputRDD.writeMemoryRDD(graphComputerConfiguration, mapReduce.getMemoryKey(), reduceRDD)); writeMemoryRDD is using ouput format as "SequenceFileOutputFormat.class" which calls SequenceFile.class. I see vertex object has edge details till SequenceFile.class. Till here vertex object is of type ComputerVertex. But computerResult object does not have edge details in vertex object. I see in ComputerResult vertex object type is changed to DetachedVertex return new DefaultComputerResult(InputOutputHelper.getOutputGraph(graphComputerConfiguration, this.resultGraph, this.persist), finalMemory.asImmutable()); I think edges are getting dropped while de-serialising and converting object to DetachedVertex. But i was not able to figure out where its getting converted to DetachedVertex object. Below configs i am using: gremlin.graph: org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph spark.serializer: org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer Appreciate any suggestion/pointer to debug the issue. Thanks & Regards, Anjani
On Monday, 14 September 2020 at 12:04:31 UTC+5:30 anj...@... wrote: Thanks Marc for sharing detail.
|
|