Re: Edge details dropped from vertex object in sparkGraphComputer
Hi All,toggle quoted messageShow quoted text
Thanks for all your inputs. After doing some more analysis found that in SparkGraphComputer (in tinker-pop library), vertex object has edge details in RDD till we add result to memory. mapReduce.addResultToMemory(finalMemory, outputRDD.writeMemoryRDD(graphComputerConfiguration, mapReduce.getMemoryKey(), reduceRDD));
writeMemoryRDD is using ouput format as "SequenceFileOutputFormat.class" which calls SequenceFile.class. I see vertex object has edge details till SequenceFile.class. Till here vertex object is of type ComputerVertex.
But computerResult object does not have edge details in vertex object. I see in ComputerResult vertex object type is changed to DetachedVertex
return new DefaultComputerResult(InputOutputHelper.getOutputGraph(graphComputerConfiguration, this.resultGraph, this.persist), finalMemory.asImmutable());
I think edges are getting dropped while de-serialising and converting object to DetachedVertex. But i was not able to figure out where its getting converted to DetachedVertex object.
Below configs i am using:
Appreciate any suggestion/pointer to debug the issue.
Thanks & Regards,
On Monday, 14 September 2020 at 12:04:31 UTC+5:30 anj...@... wrote:
Thanks Marc for sharing detail.