|
ShortestPathVertexProgram OOM issue
Abhay Pandit <abha...@...>
Hi, I am running SparkGraphComputer for ShortestPathVertexProgram and I am running into Out Of Memory issue with bigger clusters as well. Total Nodes: 10 Million Total Size Of Nodes: 10G approx Total Edge: 5M approx Cluster Config: Total Executors: 60 Each Executors Size: 128G I am using Janusgraph v0.4.0 backed by Cassandra and elasticseach Data Storage and Query Protocol: CQL Note: I am getting OOM where level of path is is more than 2. My Observations: I can see this warning in logs WARN Executor:66 - Managed memory leak detected; size = 2873804338 bytes, TID = 1276 Old Generation Heap taking maximum memory and GC is not happening as its show live references in memory. Any help is appreciated. GC Snapshot from an executor:  Note: I tried running ConnectedComponentVertexProgram found same issue with that as well. Thanks, Abhay
|
|
|
|
Re: [ANNOUNCE] JanusGraph 0.4.0 Release
Misha Brukman <mbru...@...>
[ moving discussion from *-users@ to *-dev@ group ] FYI, since future users who are not already subscribed to the mailing list will most likely find this release via GitHub release page and not from this email, I took the liberty of adding the 3 bullet points under "Notable new features" and added them to the release page as well, ahead of the "Tested compatibility" section. This is something that users have been asking about on Twitter as well, so we should do this regularly in the future so folks know what to expect from each new release, right where they download the binaries.
On Tue, Jul 9, 2019 at 10:43 AM Florian Hockmann <f...@...> wrote:
|
|
|
|
[RESULT][VOTE] JanusGraph 0.4.0 release
Florian Hockmann <f...@...>
This vote is now closed with a total of 6 +1s, no +0s and no -1s. The results are: BINDING VOTES: +1 (3 -- Florian Hockmann, Ted Wilmes, Misha Brukman) 0 (0) -1 (0) NON-BINDING VOTES: +1 (3 -- Oleksandr Porunov, Abhay Pandit, Jan Jansen) 0 (0) -1 (0) Thank you very much,
|
|
|
|
Re: [VOTE] JanusGraph 0.4.0 release
Misha Brukman <mbru...@...>
Checked signature, skimmed the docs and sanity-checked binaries.
On Tue, Jul 2, 2019 at 3:34 PM Florian Hockmann <f...@...> wrote:
|
|
|
|
Re: [VOTE] JanusGraph 0.4.0 release
Jan Jansen <faro...@...>
Release looks good to me. JanusGraph 0.4 is running as readonly instance for some time without any problem. VOTE: +1 Thanks, Jan
|
|
|
|
Re: [DISCUSS] JanusGraph 0.5.0 release
Chris Hupman <chris...@...>
+1 I'd really like to see all of our current known github vulnerabilities get addressed in 0.5.0 as well. Also we should add a graph of the gods with schema constraints to improve testing and user adoption of the feature. Chris
|
|
|
|
Re: [VOTE] JanusGraph 0.4.0 release
Abhay Pandit <abha...@...>
VOTE: +1
On Fri, 5 Jul 2019 at 01:48, Ted Wilmes <twi...@...> wrote:
|
|
|
|
Re: [VOTE] JanusGraph 0.4.0 release
Ted Wilmes <twi...@...>
Release looks good. I verified signatures, skimmed through the docs, extracted the zip, tried out some of the new features listed in the change log (lots of good stuff in here!) and then ran a set of Gatling tests. VOTE: +1 Thanks, Ted
On Tuesday, July 2, 2019 at 3:08:38 PM UTC-5, Oleksandr Porunov wrote:
|
|
|
|
Re: [VOTE] JanusGraph 0.4.0 release
Oleksandr Porunov <alexand...@...>
Downloaded release artifacts. - Check the key exists in KEYS file: Done - Verify janusgraph-0.4.0-hadoop2.zip and janusgraph-0.4.0-hadoop2-doc.zip with asc keys: Done - Extract janusgraph-0.4.0.zip, build jars from it, include into personal project, check that all integration tests pass (ScyllaDB 3.0.3, ElasticSearch 6.6.0): Done - Check version related documentation changes: Done My VOTE is +1
On Tuesday, July 2, 2019 at 10:34:22 PM UTC+3, Florian Hockmann wrote:
|
|
|
|
[VOTE] JanusGraph 0.4.0 release
Florian Hockmann <f...@...>
Hello, We are happy to announce that JanusGraph 0.4.0 is ready for release. The release artifacts can be found at this location: https://github.com/JanusGraph/janusgraph/releases/tag/v0.4.0 A binary distribution is provided for user convenience: https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/janusgraph-0.4.0-hadoop2.zip The GPG key used to sign the release artifacts is available at: https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/KEYS The docs can be found here: https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/janusgraph-0.4.0-hadoop2-doc.zip (See also documentation preview site) The release tag in Git can be found here: https://github.com/JanusGraph/janusgraph/tree/v0.4.0 The release notes are available here: https://github.com/JanusGraph/janusgraph/blob/v0.4.0/docs/changelog.adoc This [VOTE] will open for the next 3 days --- closing Friday, July 5, 2019 at 10:00 PM CEST (UTC+2). All are welcome to review and vote on the release, but only votes from TSC members are binding. My vote is +1. Thank you, Florian Hockmann
|
|
|
|
Re: [DISCUSS] PR targeting for enhancement, feature, bugfix and breaking changes
Oleksandr Porunov <alexand...@...>
I also think that we need some policy but it is hard to say right now where should this "enhancement" go. Some improvements cannot go to older branches and some can. For example, it could be a storage version related enhancement in such case the enhancement must go to the branches where it would work (i.e. to the oldest branch where this storage version is enabled). Other improvements could be based on some work which is done in some branch. For example, we could refactor code in the latest branch and not in the old branches and the improvement is easily applied to the refactored code but is hard to be applied to old code in old branches. In such case I would say that the contributor should decide where it should be applied. I.e. in some cases the improvement can be applied to old branches but with a lot of additional work which the contributor don't won't to do. So it is hard for me to suggest a single policy which will work for everyone. I would say, it depends on the concrete case. P.S. I am using "enhancement" and "improvement" as synonims. By "enhancement" I mean "improvement" and not "new funcionality".
On Tuesday, July 2, 2019 at 4:40:42 PM UTC+3, Florian Hockmann wrote:
|
|
|
|
[DISCUSS] Accept money contributions
Oleksandr Porunov <alexand...@...>
Hello, I didn't research much, but I think we could try to enable some platform which will work to collect money contributions which we can spent on some expenses. I didn't research much how it works but I think we could use the contributed money for something like: - Buy commercial Travis plan to be able to run TinkerPop tests clearly https://travis-ci.com/plans - Hiring a full-time developer / developers to the project - Tickets to conferences to present JanusGraph It is just a simple ideas. I didn't research deep but the platforms we could look at are: - possibly some other platforms What the community think about this step? Do you think it could be useful? Best regards, Oleksandr Porunov
|
|
|
|
Re: [DISCUSS] JanusGraph 0.5.0 release
Jason Plurad <plu...@...>
+1 If there are folks in the community that want continued support for old/non-supported versions of backend dependencies, we would need those same folks to step up to help managing those maintenance branches. If you fall into that category, please speak up.
On Tuesday, July 2, 2019 at 4:46:13 AM UTC-4, Jan Jansen wrote:
|
|
|
|
Re: [DISCUSS] PR targeting for enhancement, feature, bugfix and breaking changes
Florian Hockmann <f...@...>
I'm fine with adding labels for features and breaking changes, but where I'm not sure about is this part:
Who decides here which branch to target? The contributor? Any reviewer? I think we should have a policy that specifies which kind of contribution should target which branch and it should ideally be easy to understand. Am Dienstag, 2. Juli 2019 15:38:29 UTC+2 schrieb Jason Plurad:
|
|
|
|
Re: [DISCUSS] PR targeting for enhancement, feature, bugfix and breaking changes
Jason Plurad <plu...@...>
Ah, fair enough, enhancement makes sense when explained more fully by example.
On Monday, July 1, 2019 at 9:19:00 AM UTC-4, Oleksandr Porunov wrote:
|
|
|
|
Re: [DISCUSS] JanusGraph 0.5.0 release
Jan Jansen <faro...@...>
I would also add dropping of Cassandra 2 support (including Thrift). These feature set sounds good for me to be integrate into JanusGraph 0.5. The update of TinkerPop 3.4.2 could be integrate also into 0.4.x. Greetings, Jan
|
|
|
|
[DISCUSS] JanusGraph 0.5.0 release
Oleksandr Porunov <alexand...@...>
Hi everyone, 0.2.3 and 0.3.2 are released and 0.4.0 is going to be released soon. I would like to add a new feature "Add support for ElasticSearch 7". The feature makes a small breaking changes, that is why it should be targeted to a new "feature branch". We discussed earlier that the community cannot support more than 2 branches right now. I see a small drawback in dropping support for "0.3" branch because all users who are running ES < 5 or Solr < 7 or Cassandra < 2.1 will not be supported anymore. But on the other side, we cannot support deprecated backends for too long. That is why, I would suggest creating "0.4" branch, making "master" branch as "0.5.0-SNAPSHOT" and support "0.3" branch till 0.3.3 release. After that we will support only "0.4" and "master" branches. In "0.5" release I would like to propose include: - Adding support for ES 7 (#1648) - Dropping mapping types for either ES 7 or ES 8 (need to discuss). We are required to drop support for mapping types in ES 8 but ES 7 mapping types are optional (can be dropped or saved). - Drop support for ES < 6 because ES 5.x is EOL - Do not retrieve unnecessary fields during index search (#1647 and #1652) - Update TinkerPop to 3.4.2 - Add support for raw lucene date range queries (#1615) - Extract JanusGraph Gremlin driver requirements (#1521) What the community members think about it? Are you OK with making "0.3.3" the last release in "0.3" branch and moving current master to "0.4" branch after the "0.4.0" release? I am proposing to make releases faster with smaller changes instead of waiting for a lot of changes to arrive. I don't see anything bad if we release one release every 1 or 2 months with several small changes. Best regards, Oleksandr Porunov
|
|
|
|
Re: [DISCUSS] PR targeting for enhancement, feature, bugfix and breaking changes
Oleksandr Porunov <alexand...@...>
If an issue suggests a performance improvement, can we label it as a "bugfix"?
On Monday, July 1, 2019 at 3:59:52 PM UTC+3, Jason Plurad wrote:
|
|
|
|
Re: [DISCUSS] PR targeting for enhancement, feature, bugfix and breaking changes
Jason Plurad <plu...@...>
I think "enhancement" can be easily confused with "feature". I'd suggest to stick with "bugfix" for items that would go into any branch.
On Monday, July 1, 2019 at 7:34:59 AM UTC-4, Oleksandr Porunov wrote:
|
|
|
|
[DISCUSS] PR targeting for enhancement, feature, bugfix and breaking changes
Oleksandr Porunov <alexand...@...>
Hello, Currently both new features and enhancements are labeled as "enhancement". I am proposing to introduce new labels: - "feature" - a new functionality - "breaking changes" - breaking changes After that I am proposing to use the next PR targeting scheme: - "breaking changes" and "feature" can go only to master branch - "enhancement" and "bugfix" can go to any branch I don't think that we need a special voting for this that is why I will wait for 1 week for other comments from contributors. If everyone is OK with it then we can stick to a new PR targeting scheme Best regards, Oleksandr
|
|
|