Date
1 - 3 of 3
ShortestPathVertexProgram OOM issue
|
Abhay Pandit <abha...@...>
Hi Jan, I found my issue that there was some super node that was leading me to this issue. As per my finding ShortestPathVertexProgram not able to handle super nodes. But one thing I would like to say that this vertex program takes huge memory in normal case also. Say if your data size is 10G then it may take approx 10^3 memory or more but not less than that. Thanks, Abhay On Fri, 23 Aug 2019 at 02:13, 'Jan Jansen' via JanusGraph developers <janusgr...@...> wrote:
|
|
|
|
Jan Jansen <faro...@...>
Hello Thank you for writing this finding. If you like checkout try fix the OOM, we can support you. Our current focus is to update to a new version of the datastax driver and a new cassandra lib version. This can fix the problem. Greetings, Jan |
|
|
|
Abhay Pandit <abha...@...>
Hi, I am running SparkGraphComputer for ShortestPathVertexProgram and I am running into Out Of Memory issue with bigger clusters as well. Total Nodes: 10 Million Total Size Of Nodes: 10G approx Total Edge: 5M approx Cluster Config: Total Executors: 60 Each Executors Size: 128G I am using Janusgraph v0.4.0 backed by Cassandra and elasticseach Data Storage and Query Protocol: CQL Note: I am getting OOM where level of path is is more than 2. My Observations: I can see this warning in logs WARN Executor:66 - Managed memory leak detected; size = 2873804338 bytes, TID = 1276 Old Generation Heap taking maximum memory and GC is not happening as its show live references in memory. Any help is appreciated. GC Snapshot from an executor:  Note: I tried running ConnectedComponentVertexProgram found same issue with that as well. Thanks, Abhay |
|
|