Date   

ShortestPathVertexProgram OOM issue

Abhay Pandit <abha...@...>
 

Hi,

I am running SparkGraphComputer for ShortestPathVertexProgram and I am running into Out Of Memory issue with bigger clusters as well.

Total Nodes: 10 Million
Total Size Of Nodes: 10G approx
Total Edge: 5M approx

Cluster Config:
Total Executors: 60
Each Executors Size: 128G

I am using Janusgraph v0.4.0 backed by Cassandra and elasticseach
Data Storage and Query Protocol: CQL

Note: I am getting OOM where level of path is is more than 2.
My Observations:
 I can see this warning in logs
WARN  Executor:66 - Managed memory leak detected; size = 2873804338 bytes, TID = 1276

Old Generation Heap taking maximum memory and GC is not happening as its show live references in memory.

Any help is appreciated.

GC Snapshot from an executor:
image.png
Note: I tried running ConnectedComponentVertexProgram found same issue with that as well.



Thanks,
Abhay


Re: [ANNOUNCE] JanusGraph 0.4.0 Release

Misha Brukman <mbru...@...>
 

[ moving discussion from *-users@ to *-dev@ group ]

FYI, since future users who are not already subscribed to the mailing list will most likely find this release via GitHub release page and not from this email, I took the liberty of adding the 3 bullet points under "Notable new features" and added them to the release page as well, ahead of the "Tested compatibility" section.

This is something that users have been asking about on Twitter as well, so we should do this regularly in the future so folks know what to expect from each new release, right where they download the binaries.

On Tue, Jul 9, 2019 at 10:43 AM Florian Hockmann <f...@...> wrote:
The JanusGraph Technical Steering Committee is excited to announce the release of JanusGraph 0.4.0.

JanusGraph is an Apache TinkerPop enabled property graph database with support for a variety of storage and indexing backends. Thank you to all of the contributors.

Notable new features in this release include:
  • Updates to key dependencies for Apache TinkerPop 3.4, Apache Cassandra 2.2, Apache HBase 2.1, Berkeley DB JE 7.5, Google Cloud Bigtable 1.11
  • Support CQL for OLAP which completes our CQL support
  • Performance improvements for pre-fetching of properties
The release artifacts can be found at this location:
    https://github.com/JanusGraph/janusgraph/releases/tag/v0.4.0

A binary distribution is provided for user convenience:
    https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/janusgraph-0.4.0-hadoop2.zip

The online docs can be found here:
    https://docs.janusgraph.org/0.4.0/index.html

To view the resolved issues and commits check the milestone here:
        https://github.com/JanusGraph/janusgraph/milestone/8?closed=1

Thank you very much,
Florian Hockmann

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgra...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/2911c180-78d6-428e-ba86-77190113063e%40googlegroups.com.


[RESULT][VOTE] JanusGraph 0.4.0 release

Florian Hockmann <f...@...>
 

This vote is now closed with a total of 6 +1s, no +0s and no -1s. The results are:

BINDING VOTES:

+1  (3 -- Florian Hockmann, Ted Wilmes,  Misha Brukman)
0   (0)
-1  (0)

NON-BINDING VOTES:

+1 (3 -- Oleksandr Porunov, Abhay Pandit, Jan Jansen)
0  (0)
-1 (0)

Thank you very much,

Florian Hockmann


Re: [VOTE] JanusGraph 0.4.0 release

Misha Brukman <mbru...@...>
 

Checked signature, skimmed the docs and sanity-checked binaries.

VOTE: +1


On Tue, Jul 2, 2019 at 3:34 PM Florian Hockmann <f...@...> wrote:
Hello,

We are happy to announce that JanusGraph 0.4.0 is ready for release.

The release artifacts can be found at this location:
        https://github.com/JanusGraph/janusgraph/releases/tag/v0.4.0

A binary distribution is provided for user convenience:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/janusgraph-0.4.0-hadoop2.zip

The GPG key used to sign the release artifacts is available at:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/KEYS

The docs can be found here:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/janusgraph-0.4.0-hadoop2-doc.zip
        (See also documentation preview site)

The release tag in Git can be found here:
        https://github.com/JanusGraph/janusgraph/tree/v0.4.0

The release notes are available here:
        https://github.com/JanusGraph/janusgraph/blob/v0.4.0/docs/changelog.adoc

This [VOTE] will open for the next 3 days --- closing Friday, July 5, 2019 at 10:00 PM CEST (UTC+2).
All are welcome to review and vote on the release, but only votes from TSC members are binding.
My vote is +1.


Thank you,
Florian Hockmann

--
You received this message because you are subscribed to the Google Groups "JanusGraph developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-dev/9a08d67e-1fab-462a-aa44-5468b2cdc0b3%40googlegroups.com.


Re: [VOTE] JanusGraph 0.4.0 release

Jan Jansen <faro...@...>
 

Release looks good to me. JanusGraph 0.4 is running as readonly instance for some time without any problem.

VOTE: +1

Thanks,
Jan



Re: [DISCUSS] JanusGraph 0.5.0 release

Chris Hupman <chris...@...>
 

+1

I'd really like to see all of our current known github vulnerabilities get addressed in 0.5.0 as well.

Also we should add a graph of the gods with schema constraints to improve testing and user adoption of the feature.

Chris


Re: [VOTE] JanusGraph 0.4.0 release

Abhay Pandit <abha...@...>
 

VOTE: +1

On Fri, 5 Jul 2019 at 01:48, Ted Wilmes <twi...@...> wrote:
Release looks good. I verified signatures, skimmed through the docs, extracted the zip, tried out some of the new features listed in the change log (lots of good stuff in here!) and then ran a set of Gatling tests.

VOTE: +1

Thanks,
Ted

On Tuesday, July 2, 2019 at 3:08:38 PM UTC-5, Oleksandr Porunov wrote:
Downloaded release artifacts. 
- Check the key exists in KEYS file: Done
- Verify janusgraph-0.4.0-hadoop2.zip and janusgraph-0.4.0-hadoop2-doc.zip with asc keys: Done
- Extract janusgraph-0.4.0.zip, build jars from it, include into personal project, check that all integration tests pass (ScyllaDB 3.0.3, ElasticSearch 6.6.0): Done
- Check version related documentation changes: Done

My VOTE is +1

On Tuesday, July 2, 2019 at 10:34:22 PM UTC+3, Florian Hockmann wrote:
Hello,

We are happy to announce that JanusGraph 0.4.0 is ready for release.

The release artifacts can be found at this location:
        https://github.com/JanusGraph/janusgraph/releases/tag/v0.4.0

A binary distribution is provided for user convenience:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/janusgraph-0.4.0-hadoop2.zip

The GPG key used to sign the release artifacts is available at:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/KEYS

The docs can be found here:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/janusgraph-0.4.0-hadoop2-doc.zip
        (See also documentation preview site)

The release tag in Git can be found here:
        https://github.com/JanusGraph/janusgraph/tree/v0.4.0

The release notes are available here:
        https://github.com/JanusGraph/janusgraph/blob/v0.4.0/docs/changelog.adoc

This [VOTE] will open for the next 3 days --- closing Friday, July 5, 2019 at 10:00 PM CEST (UTC+2).
All are welcome to review and vote on the release, but only votes from TSC members are binding.
My vote is +1.


Thank you,
Florian Hockmann

--
You received this message because you are subscribed to the Google Groups "JanusGraph developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-dev/fa7bb27b-b4e6-484a-8994-f62d133c3d9d%40googlegroups.com.


Re: [VOTE] JanusGraph 0.4.0 release

Ted Wilmes <twi...@...>
 

Release looks good. I verified signatures, skimmed through the docs, extracted the zip, tried out some of the new features listed in the change log (lots of good stuff in here!) and then ran a set of Gatling tests.

VOTE: +1

Thanks,
Ted


On Tuesday, July 2, 2019 at 3:08:38 PM UTC-5, Oleksandr Porunov wrote:
Downloaded release artifacts. 
- Check the key exists in KEYS file: Done
- Verify janusgraph-0.4.0-hadoop2.zip and janusgraph-0.4.0-hadoop2-doc.zip with asc keys: Done
- Extract janusgraph-0.4.0.zip, build jars from it, include into personal project, check that all integration tests pass (ScyllaDB 3.0.3, ElasticSearch 6.6.0): Done
- Check version related documentation changes: Done

My VOTE is +1

On Tuesday, July 2, 2019 at 10:34:22 PM UTC+3, Florian Hockmann wrote:
Hello,

We are happy to announce that JanusGraph 0.4.0 is ready for release.

The release artifacts can be found at this location:
        https://github.com/JanusGraph/janusgraph/releases/tag/v0.4.0

A binary distribution is provided for user convenience:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/janusgraph-0.4.0-hadoop2.zip

The GPG key used to sign the release artifacts is available at:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/KEYS

The docs can be found here:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/janusgraph-0.4.0-hadoop2-doc.zip
        (See also documentation preview site)

The release tag in Git can be found here:
        https://github.com/JanusGraph/janusgraph/tree/v0.4.0

The release notes are available here:
        https://github.com/JanusGraph/janusgraph/blob/v0.4.0/docs/changelog.adoc

This [VOTE] will open for the next 3 days --- closing Friday, July 5, 2019 at 10:00 PM CEST (UTC+2).
All are welcome to review and vote on the release, but only votes from TSC members are binding.
My vote is +1.


Thank you,
Florian Hockmann


Re: [VOTE] JanusGraph 0.4.0 release

Oleksandr Porunov <alexand...@...>
 

Downloaded release artifacts. 
- Check the key exists in KEYS file: Done
- Verify janusgraph-0.4.0-hadoop2.zip and janusgraph-0.4.0-hadoop2-doc.zip with asc keys: Done
- Extract janusgraph-0.4.0.zip, build jars from it, include into personal project, check that all integration tests pass (ScyllaDB 3.0.3, ElasticSearch 6.6.0): Done
- Check version related documentation changes: Done

My VOTE is +1


On Tuesday, July 2, 2019 at 10:34:22 PM UTC+3, Florian Hockmann wrote:
Hello,

We are happy to announce that JanusGraph 0.4.0 is ready for release.

The release artifacts can be found at this location:
        https://github.com/JanusGraph/janusgraph/releases/tag/v0.4.0

A binary distribution is provided for user convenience:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/janusgraph-0.4.0-hadoop2.zip

The GPG key used to sign the release artifacts is available at:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/KEYS

The docs can be found here:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/janusgraph-0.4.0-hadoop2-doc.zip
        (See also documentation preview site)

The release tag in Git can be found here:
        https://github.com/JanusGraph/janusgraph/tree/v0.4.0

The release notes are available here:
        https://github.com/JanusGraph/janusgraph/blob/v0.4.0/docs/changelog.adoc

This [VOTE] will open for the next 3 days --- closing Friday, July 5, 2019 at 10:00 PM CEST (UTC+2).
All are welcome to review and vote on the release, but only votes from TSC members are binding.
My vote is +1.


Thank you,
Florian Hockmann


[VOTE] JanusGraph 0.4.0 release

Florian Hockmann <f...@...>
 

Hello,

We are happy to announce that JanusGraph 0.4.0 is ready for release.

The release artifacts can be found at this location:
        https://github.com/JanusGraph/janusgraph/releases/tag/v0.4.0

A binary distribution is provided for user convenience:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/janusgraph-0.4.0-hadoop2.zip

The GPG key used to sign the release artifacts is available at:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/KEYS

The docs can be found here:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/janusgraph-0.4.0-hadoop2-doc.zip
        (See also documentation preview site)

The release tag in Git can be found here:
        https://github.com/JanusGraph/janusgraph/tree/v0.4.0

The release notes are available here:
        https://github.com/JanusGraph/janusgraph/blob/v0.4.0/docs/changelog.adoc

This [VOTE] will open for the next 3 days --- closing Friday, July 5, 2019 at 10:00 PM CEST (UTC+2).
All are welcome to review and vote on the release, but only votes from TSC members are binding.
My vote is +1.


Thank you,
Florian Hockmann


Re: [DISCUSS] PR targeting for enhancement, feature, bugfix and breaking changes

Oleksandr Porunov <alexand...@...>
 

I also think that we need some policy but it is hard to say right now where should this "enhancement" go. Some improvements cannot go to older branches and some can. For example, it could be a storage version related enhancement in such case the enhancement must go to the branches where it would work (i.e. to the oldest branch where this storage version is enabled). Other improvements could be based on some work which is done in some branch. For example, we could refactor code in the latest branch and not in the old branches and the improvement is easily applied to the refactored code but is hard to be applied to old code in old branches. In such case I would say that the contributor should decide where it should be applied. I.e. in some cases the improvement can be applied to old branches but with a lot of additional work which the contributor don't won't to do. So it is hard for me to suggest a single policy which will work for everyone. I would say, it depends on the concrete case. 

P.S. I am using "enhancement" and "improvement" as synonims. By "enhancement" I mean "improvement" and not "new funcionality".


On Tuesday, July 2, 2019 at 4:40:42 PM UTC+3, Florian Hockmann wrote:
I'm fine with adding labels for features and breaking changes, but where I'm not sure about is this part:

"enhancement" and "bugfix" can go to any branch

Who decides here which branch to target? The contributor? Any reviewer?

I think we should have a policy that specifies which kind of contribution should target which branch and it should ideally be easy to understand.

Am Dienstag, 2. Juli 2019 15:38:29 UTC+2 schrieb Jason Plurad:
Ah, fair enough, enhancement makes sense when explained more fully by example.

On Monday, July 1, 2019 at 9:19:00 AM UTC-4, Oleksandr Porunov wrote:
If an issue suggests a performance improvement, can we label it as a "bugfix"?

On Monday, July 1, 2019 at 3:59:52 PM UTC+3, Jason Plurad wrote:
I think "enhancement" can be easily confused with "feature". I'd suggest to stick with "bugfix" for items that would go into any branch.

On Monday, July 1, 2019 at 7:34:59 AM UTC-4, Oleksandr Porunov wrote:
Hello,

Currently both new features and enhancements are labeled as "enhancement".
I am proposing to introduce new labels: 
- "feature" - a new functionality
- "breaking changes" - breaking changes

After that I am proposing to use the next PR targeting scheme:
- "breaking changes" and "feature" can go only to master branch
- "enhancement" and "bugfix" can go to any branch

I don't think that we need a special voting for this that is why I will wait for 1 week for other comments from contributors. If everyone is OK with it then we can stick to a new PR targeting scheme

Best regards,
Oleksandr


[DISCUSS] Accept money contributions

Oleksandr Porunov <alexand...@...>
 

Hello,

I didn't research much, but I think we could try to enable some platform which will work to collect money contributions which we can spent on some expenses. I didn't research much how it works but I think we could use the contributed money for something like:
- Buy commercial Travis plan to be able to run TinkerPop tests clearly https://travis-ci.com/plans
- Hiring a full-time developer / developers to the project
- Tickets to conferences to present JanusGraph

It is just a simple ideas. I didn't research deep but the platforms we could look at are: 
- possibly some other platforms

What the community think about this step? Do you think it could be useful?

Best regards,
Oleksandr Porunov


Re: [DISCUSS] JanusGraph 0.5.0 release

Jason Plurad <plu...@...>
 

+1

If there are folks in the community that want continued support for old/non-supported versions of backend dependencies, we would need those same folks to step up to help managing those maintenance branches. If you fall into that category, please speak up.


On Tuesday, July 2, 2019 at 4:46:13 AM UTC-4, Jan Jansen wrote:
I would also add dropping of Cassandra 2 support (including Thrift).

These feature set sounds good for me to be integrate into JanusGraph 0.5.

The update of TinkerPop 3.4.2 could be integrate also into 0.4.x.

Greetings,
Jan


Re: [DISCUSS] PR targeting for enhancement, feature, bugfix and breaking changes

Florian Hockmann <f...@...>
 

I'm fine with adding labels for features and breaking changes, but where I'm not sure about is this part:

"enhancement" and "bugfix" can go to any branch

Who decides here which branch to target? The contributor? Any reviewer?

I think we should have a policy that specifies which kind of contribution should target which branch and it should ideally be easy to understand.

Am Dienstag, 2. Juli 2019 15:38:29 UTC+2 schrieb Jason Plurad:
Ah, fair enough, enhancement makes sense when explained more fully by example.

On Monday, July 1, 2019 at 9:19:00 AM UTC-4, Oleksandr Porunov wrote:
If an issue suggests a performance improvement, can we label it as a "bugfix"?

On Monday, July 1, 2019 at 3:59:52 PM UTC+3, Jason Plurad wrote:
I think "enhancement" can be easily confused with "feature". I'd suggest to stick with "bugfix" for items that would go into any branch.

On Monday, July 1, 2019 at 7:34:59 AM UTC-4, Oleksandr Porunov wrote:
Hello,

Currently both new features and enhancements are labeled as "enhancement".
I am proposing to introduce new labels: 
- "feature" - a new functionality
- "breaking changes" - breaking changes

After that I am proposing to use the next PR targeting scheme:
- "breaking changes" and "feature" can go only to master branch
- "enhancement" and "bugfix" can go to any branch

I don't think that we need a special voting for this that is why I will wait for 1 week for other comments from contributors. If everyone is OK with it then we can stick to a new PR targeting scheme

Best regards,
Oleksandr


Re: [DISCUSS] PR targeting for enhancement, feature, bugfix and breaking changes

Jason Plurad <plu...@...>
 

Ah, fair enough, enhancement makes sense when explained more fully by example.


On Monday, July 1, 2019 at 9:19:00 AM UTC-4, Oleksandr Porunov wrote:
If an issue suggests a performance improvement, can we label it as a "bugfix"?

On Monday, July 1, 2019 at 3:59:52 PM UTC+3, Jason Plurad wrote:
I think "enhancement" can be easily confused with "feature". I'd suggest to stick with "bugfix" for items that would go into any branch.

On Monday, July 1, 2019 at 7:34:59 AM UTC-4, Oleksandr Porunov wrote:
Hello,

Currently both new features and enhancements are labeled as "enhancement".
I am proposing to introduce new labels: 
- "feature" - a new functionality
- "breaking changes" - breaking changes

After that I am proposing to use the next PR targeting scheme:
- "breaking changes" and "feature" can go only to master branch
- "enhancement" and "bugfix" can go to any branch

I don't think that we need a special voting for this that is why I will wait for 1 week for other comments from contributors. If everyone is OK with it then we can stick to a new PR targeting scheme

Best regards,
Oleksandr


Re: [DISCUSS] JanusGraph 0.5.0 release

Jan Jansen <faro...@...>
 

I would also add dropping of Cassandra 2 support (including Thrift).

These feature set sounds good for me to be integrate into JanusGraph 0.5.

The update of TinkerPop 3.4.2 could be integrate also into 0.4.x.

Greetings,
Jan


[DISCUSS] JanusGraph 0.5.0 release

Oleksandr Porunov <alexand...@...>
 

Hi everyone,

0.2.3 and 0.3.2 are released and 0.4.0 is going to be released soon. I would like to add a new feature "Add support for ElasticSearch 7". The feature makes a small breaking changes, that is why it should be targeted to a new "feature branch". We discussed earlier that the community cannot support more than 2 branches right now.
I see a small drawback in dropping support for "0.3" branch because all users who are running ES < 5 or Solr < 7 or Cassandra < 2.1 will not be supported anymore. But on the other side, we cannot support deprecated backends for too long.
That is why, I would suggest creating "0.4" branch, making "master" branch as "0.5.0-SNAPSHOT" and support "0.3" branch till 0.3.3 release. After that we will support only "0.4" and "master" branches.

In "0.5" release I would like to propose include:
- Adding support for ES 7 (#1648)
- Dropping mapping types for either ES 7 or ES 8 (need to discuss). We are required to drop support for mapping types in ES 8 but ES 7 mapping types are optional (can be dropped or saved).
- Drop support for ES < 6 because ES 5.x is EOL
- Do not retrieve unnecessary fields during index search (#1647 and #1652)
- Update TinkerPop to 3.4.2
- Add support for raw lucene date range queries (#1615)
- Extract JanusGraph Gremlin driver requirements (#1521)

What the community members think about it? Are you OK with making "0.3.3" the last release in "0.3" branch and moving current master to "0.4" branch after the "0.4.0" release?

I am proposing to make releases faster with smaller changes instead of waiting for a lot of changes to arrive. I don't see anything bad if we release one release every 1 or 2 months with several small changes.

Best regards,
Oleksandr Porunov


Re: [DISCUSS] PR targeting for enhancement, feature, bugfix and breaking changes

Oleksandr Porunov <alexand...@...>
 

If an issue suggests a performance improvement, can we label it as a "bugfix"?


On Monday, July 1, 2019 at 3:59:52 PM UTC+3, Jason Plurad wrote:
I think "enhancement" can be easily confused with "feature". I'd suggest to stick with "bugfix" for items that would go into any branch.

On Monday, July 1, 2019 at 7:34:59 AM UTC-4, Oleksandr Porunov wrote:
Hello,

Currently both new features and enhancements are labeled as "enhancement".
I am proposing to introduce new labels: 
- "feature" - a new functionality
- "breaking changes" - breaking changes

After that I am proposing to use the next PR targeting scheme:
- "breaking changes" and "feature" can go only to master branch
- "enhancement" and "bugfix" can go to any branch

I don't think that we need a special voting for this that is why I will wait for 1 week for other comments from contributors. If everyone is OK with it then we can stick to a new PR targeting scheme

Best regards,
Oleksandr


Re: [DISCUSS] PR targeting for enhancement, feature, bugfix and breaking changes

Jason Plurad <plu...@...>
 

I think "enhancement" can be easily confused with "feature". I'd suggest to stick with "bugfix" for items that would go into any branch.


On Monday, July 1, 2019 at 7:34:59 AM UTC-4, Oleksandr Porunov wrote:
Hello,

Currently both new features and enhancements are labeled as "enhancement".
I am proposing to introduce new labels: 
- "feature" - a new functionality
- "breaking changes" - breaking changes

After that I am proposing to use the next PR targeting scheme:
- "breaking changes" and "feature" can go only to master branch
- "enhancement" and "bugfix" can go to any branch

I don't think that we need a special voting for this that is why I will wait for 1 week for other comments from contributors. If everyone is OK with it then we can stick to a new PR targeting scheme

Best regards,
Oleksandr


[DISCUSS] PR targeting for enhancement, feature, bugfix and breaking changes

Oleksandr Porunov <alexand...@...>
 

Hello,

Currently both new features and enhancements are labeled as "enhancement".
I am proposing to introduce new labels: 
- "feature" - a new functionality
- "breaking changes" - breaking changes

After that I am proposing to use the next PR targeting scheme:
- "breaking changes" and "feature" can go only to master branch
- "enhancement" and "bugfix" can go to any branch

I don't think that we need a special voting for this that is why I will wait for 1 week for other comments from contributors. If everyone is OK with it then we can stick to a new PR targeting scheme

Best regards,
Oleksandr

441 - 460 of 1601