Date   

[ANNOUNCEMENT] JanusGraph enabled donations on LFX Crowdfunding

Oleksandr Porunov
 

The JanusGraph Technical Steering Committee is excited to announce that JanusGraph is now accepting donations.


As you may know, most of JanusGraph contributors are not full-time JanusGraph employees, thus we came up with the idea to try to collect donations from the community to be able to hire full time employees to JanusGraph.


With your help JanusGraph will be able to produce releases much more often and we will be able to develop JanusGraph much faster.


JanusGraph Technical Steering Committee guarantees to be fully transparent with the community about any penny spent. We are accepting contributions via LFX Crowdfunding which has an open ledger where you can check all the transactions made and their descriptions.


LFX Crowdfunding link where JanusGraph accepts donations is : https://crowdfunding.lfx.linuxfoundation.org/projects/janusgraph


Best regards,

Oleksandr Porunov

on behalf of JanusGraph TSC


New TSC member: Boxuan Li

Oleksandr Porunov
 

On behalf of the JanusGraph Technical Steering Committee (TSC), I'm pleased to welcome a new Technical Steering Committee member on the project!

Boxuan Li has provided major contributions and has demonstrated an on-going commitment to the project. Being a TSC member enables assistance with the project management and to guide the direction of the project.

Congratulations, Boxuan Li!


Re: [DISCUSS] Dropping HBase 1 support

Boxuan Li
 

Hi Jan,

I think you could forward this to the user mailing list as well. Maybe janusgraph-hbase users could shed some light. I personally don’t have any opinion here.

Best,
Boxuan


「<Jan.jansen@...>」在 2021年7月14日 週三,下午4:13 寫道:

Hi

I looked into the Hbase 1 support after Porunov asked why I want to drop if the builds are passing: https://github.com/JanusGraph/janusgraph/pull/2213#issuecomment-861620348. It seems that we stop testing HBase 1 in our CI solution already in branch 0.3. The main issue was a wrong combination of maven flags for stop testing.

I tried to fix the flags and realized that isn't working against the Testcontainers solution, so revert internal back to commit before Testcontainers, see here https://github.com/GDATASoftwareAG/janusgraph/tree/test-hbase1. I had to fix some build issue than was able to execute tests which are failing https://github.com/GDATASoftwareAG/janusgraph/actions/runs/1024212663. (I didn't fix the HBase 2 build)

My Idea would be to drop HBase 1 support. (Currently, HBase 1 support already requires a custom build of JG.)

Any thoughts?

Greetings, Jan


[DISCUSS] Dropping HBase 1 support

Jansen, Jan
 

Hi

I looked into the Hbase 1 support after Porunov asked why I want to drop if the builds are passing: https://github.com/JanusGraph/janusgraph/pull/2213#issuecomment-861620348. It seems that we stop testing HBase 1 in our CI solution already in branch 0.3. The main issue was a wrong combination of maven flags for stop testing.

I tried to fix the flags and realized that isn't working against the Testcontainers solution, so revert internal back to commit before Testcontainers, see here https://github.com/GDATASoftwareAG/janusgraph/tree/test-hbase1. I had to fix some build issue than was able to execute tests which are failing https://github.com/GDATASoftwareAG/janusgraph/actions/runs/1024212663. (I didn't fix the HBase 2 build)

My Idea would be to drop HBase 1 support. (Currently, HBase 1 support already requires a custom build of JG.)

Any thoughts?

Greetings, Jan


Re: [DISCUSS] JanusGraph versioning

Boxuan Li
 

1.0.0 sounds good to me. Maybe we can target 1.0.0 after this 0.6.0 release. I think we'd better not rename the incoming release from 0.6.0 to 1.0.0, because it contains many new changes and may take some time (+ bug fixes, if any) to get stable. I would rather see a stable 1.0.0 release with few new features than an unstable one with many new features.

Best regards,
Boxuan Li


[DISCUSS] JanusGraph versioning

Oleksandr Porunov
 

Hi,

I would like to start a discussion about JanusGraph versioning.
Right now we have the next versioning:
<GA indicator?>.<features and breaking changes>.<patch>

So, for GA indicator we always have `0` and as far as I know we were waiting for JanusGraph to be stable to increment JanusGraph to 1.0.0 version.

That said, I'm not sure how exactly should it be considered as stable.
It was quite some time for JanusGraph to be used in production by many companies.

We could think about stability as:
- JanusGraph has been used in production for some time.
- There are no breaking changes for some time.

I think we meet the first case. I.e. JanusGraph is used in production but I'm not sure about the second option.

We do have breaking changes often but mostly they are related to drop of support for EOL versions or driver upgrades. I guess, we will have such breaking changes continiously because the support for old drivers will be dropped and new drivers will be supported. Thus, such breaking changes are kind of natural thing I guess.
I guess, bigger breaking changes are those, which influence storage layer (i.e. data). Last time we had such breaking changes was in 0.3.0 release (that said, they were very small and easy to be upgraded). So, if we count only such changes where you can't easily upgrade JanusGraph because you have data in old format - we meet second option as well in such situation.

In case we consider JanusGraph to be stable enough, should we upgrade it to 1.x.y version?

If we upgrade it, should we start following something like semantic versioning for all future versions (https://semver.org/) or should we think about different versioning / keep current versioning?

When should we upgrade JanusGraph to 1.x.y?

Should the first version `1.x.y` keep current `x.y` or reset it to `0.0`? I.e. should the first version be `1.7.0` or `1.0.0`?

My thoughts on the above questions are:
- We can consider JanusGraph to be stable enough
- After the upgrade it would make sense to start using the same version number as in semantic versioning (i.e. MAJOR.MINOR.PATCH).
- I guess we could do it on the next release after 0.6.0 but we potentially could rename 0.6.0 to 1.0.0 or 1.6.0 versions as well. I don't have good thoughts on that yet.
- Both 1.0.0 and 1.7.0 / 1.6.0 are good to me. I don't have good thoughts about it yet as well.

To be clear, I'm not insisting to bump JanusGraph to 1.x.y version immediately. What I wanted is to start a discussion about it to see other thoughts.

Best regards,
Oleksandr Porunov


JanusGraph Meetup #4 Recording

Ted Wilmes
 

Hello,
Thanks to all who attended the meetup yesterday. If you weren't able to make it, you can find the recording at: https://www.experoinc.com/online-seminar/janusgraph-community-meetup.

Thanks to our presenters: Marc, Saurabh, and Bruno, we had a really good set of material presented.

Thanks,
Ted


Re: [Meetup] JanusGraph Meetup May 18 covering JG OLAP approaches

Ted Wilmes
 

Hi Boxuan,
Yes, definitely. I'll post this under presentations on janusgraph.org. Also, I hadn't posted meetup 3 on there yet and finally tracked the link down, so that will also be up there shortly.

Thanks,
Ted


On Sun, May 16, 2021 at 10:22 AM Boxuan Li <liboxuan@...> wrote:
Hi Ted,

Thanks for organizing this! Do you have plans to record & release the video after the meetup? 10:30 ET is a bit late for some regions in APAC, so it would be great if there would be a video record.

Cheers,
Boxuan

On May 14, 2021, at 11:37 PM, Ted Wilmes <twilmes@...> wrote:

Hello,
We will be hosting a community meetup next week on Tuesday, May 18th at 9:30 central/10:30 eastern. We have a great set of speakers who will be discussing all things JanusGraph OLAP:

* Hadoop Marc who has helped many of us on the mailing list and in JG issues
* Saurabh Verma, principal engineer at Zeotap
* Bruno Berriso, engineer at Expero

If you're interested in signing up, here's the link: https://www.experoinc.com/get/janusgraph-user-group.

Thanks,
Ted


Re: [Meetup] JanusGraph Meetup May 18 covering JG OLAP approaches

Boxuan Li
 

Hi Ted,

Thanks for organizing this! Do you have plans to record & release the video after the meetup? 10:30 ET is a bit late for some regions in APAC, so it would be great if there would be a video record.

Cheers,
Boxuan

On May 14, 2021, at 11:37 PM, Ted Wilmes <twilmes@...> wrote:

Hello,
We will be hosting a community meetup next week on Tuesday, May 18th at 9:30 central/10:30 eastern. We have a great set of speakers who will be discussing all things JanusGraph OLAP:

* Hadoop Marc who has helped many of us on the mailing list and in JG issues
* Saurabh Verma, principal engineer at Zeotap
* Bruno Berriso, engineer at Expero

If you're interested in signing up, here's the link: https://www.experoinc.com/get/janusgraph-user-group.

Thanks,
Ted


[Meetup] JanusGraph Meetup May 18 covering JG OLAP approaches

Ted Wilmes
 

Hello,
We will be hosting a community meetup next week on Tuesday, May 18th at 9:30 central/10:30 eastern. We have a great set of speakers who will be discussing all things JanusGraph OLAP:

* Hadoop Marc who has helped many of us on the mailing list and in JG issues
* Saurabh Verma, principal engineer at Zeotap
* Bruno Berriso, engineer at Expero

If you're interested in signing up, here's the link: https://www.experoinc.com/get/janusgraph-user-group.

Thanks,
Ted


Re: Potential complexities of making secondary persistence atomic

rngcntr
 

Hi Florian!

That definitely sounds like a promising idea. You are most likely not the only one requiring consistency between storage and index data. In general, my advice is to start with a test case which explicitly addresses the situation where the primary persistence succeeds and the secondary persistence fails. From there, see what it needs to let the transaction fail. Once you have a solution for that, you can start the discussion by opening a PR. We will then see which issues could arise and try to solve them. The first prototype does not have to be perfect, a proof of concept is fine!

Best Regards,
Florian


Potential complexities of making secondary persistence atomic

florian.caesar <florian.caesar@...>
 

Hi!

Currently JanusGraph's philosophy is to fail transactions only if the primary persistence fails and allow failures in secondary persistence.
According to https://docs.janusgraph.org/advanced-topics/recovery/#transaction-failure, the recommended approach for dealing with secondary persistence failures is maintaining and continuously recovering from a write-ahead log.
This results in temporary inconsistencies between primary storage and mixed indices.

For my use case, potential inconsistencies are not acceptable. Currently there is no way of making JanusGraph fail a transaction if either primary or secondary persistence fails. Looking at the source code (starting at https://github.com/JanusGraph/janusgraph/blob/a73d39d8a071f8d5340aede4917653629ab595cb/janusgraph-core/src/main/java/org/janusgraph/graphdb/database/StandardJanusGraph.java#L743), it should be possible to re-order persistence to commit both secondary and primary storage at the same time. Naturally there is some cleanup to do in case of transaction failure, but it seems manageable to me.

What are the potential complexities of pursuing this? How does this mess with JanusGraph's logic & assumptions in other places?

If it seems fine, it should be an easy change. If nobody else wants to take it, I would also be happy to start a PR for this feature myself at some point.

Thanks for your input and time
Florian


Re: [DISCUSS] JanusGraph 0.6.0 release

Oleksandr Porunov
 

As Tinkerpop is very close for a release, I'm proposing to delay JanusGraph 0.6.0 release, so that we could ship 0.6.0 release of JanusGraph with either 3.4.11 or 3.5.0 version of Tinkerpop. I assume, if everything goes well, we should be able to release JanusGraph 0.6.0 version in the middle of May.


Re: [DISCUSS] JanusGraph 0.6.0 release

Jansen, Jan
 


Re: [Performance Optimization] Optimization around the `system_properties` table interaction

Boxuan Li
 

Yeah that makes sense. I saw you said “unreplicated” thus wondered. I am not familiar with how `system_properties` is handled, but just want to point out that it is very difficult if not impossible to change the data model while keeping backward compatibility at the same time.

On Apr 22, 2021, at 11:22 PM, sauverma <saurabhdec1988@...> wrote:

Hi Boxuan Li

We are using RF3.

What I meant is that essentially the data for system_properties is going to a single partition. Due to RF3, this partition is replicated 3 times.

Does this clarify?

Thanks
Saurabh



Re: [Performance Optimization] Optimization around the `system_properties` table interaction

sauverma
 

Hi Boxuan Li

We are using RF3.

What I meant is that essentially the data for system_properties is going to a single partition. Due to RF3, this partition is replicated 3 times.

Does this clarify?

Thanks
Saurabh


Re: [Performance Optimization] Optimization around the `system_properties` table interaction

Boxuan Li
 

Hi @sauverma,

I am just curious: I noticed you said "there is only 1 partition for system_properties unreplicated". Do you have storage.cql.replication-factor = 1?


Re: [Performance Optimization] Optimization around the `system_properties` table interaction

sauverma
 

Hi all

Updates on this issue

- We found that the periodic removal of system_properties (while the ingestion is running) leads to graph corruption (mentioned at high level at https://docs.janusgraph.org/advanced-topics/recovery/)
- The perf issue we saw were due to below reasons
     - improper handling on dataproc scaledown which lead to connections not getting closed to JG, and thus ever increasing system_properties table
     - unbounded access to the scylla caching layer, which is basically unthrottled access to scylla caching system, leading to other queries slowing down due to the system_properties single, hot partition
     - in addition to this, the data model for system_properties still needs to be fixed via usage of clustering keys, by design system_properties has only 1 SINGLE partition and all spark executors hit it while initialization leading to  query slow down -> query queuing -> query timeouts

Thanks


Re: [DISCUSS] JanusGraph 0.6.0 release

Jansen, Jan
 


What do you think? Sounds great
Would such a deadline be OK? I think it is good idea to have a deadline.


Von: janusgraph-dev@... <janusgraph-dev@...> im Auftrag von Oleksandr Porunov <alexandr.porunov@...>
Gesendet: Dienstag, 30. März 2021 18:49:50
An: janusgraph-dev@...
Betreff: Re: [janusgraph-dev] [DISCUSS] JanusGraph 0.6.0 release
 
I am good with waiting for TinkerPop 3.4.11 or 3.5.0 if we expect those releases soon, but I don't want to delay 0.6.0 release too much because the community sometimes asks for the release.
Also, if we release 0.6.0 without TinkerPop 3.5.0 version, I'm good with shipping 0.7.0 version even if that release is just 1 commit difference (TinkerPop upgrade).
As I said, I'm good with waiting for TinkerPop 3.4.11 or TinkerPop 3.5.0 but I would prefer to set a day till which we should start a releasing process (unless there are critical bugs).
I would propose to set a deadline on the 1st May. If there are no new TinkerPop releases till that day (due to some delays) then we start releasing process on the 1st May as is.
If there are new TinkerPop releases in April then we update JanusGraph to the latest TinkerPop release and start releasing process after that.

What do you think? Would such a deadline be OK?

Best regards,
Oleksandr


Re: [DISCUSS] JanusGraph 0.6.0 release

Oleksandr Porunov
 

I am good with waiting for TinkerPop 3.4.11 or 3.5.0 if we expect those releases soon, but I don't want to delay 0.6.0 release too much because the community sometimes asks for the release.
Also, if we release 0.6.0 without TinkerPop 3.5.0 version, I'm good with shipping 0.7.0 version even if that release is just 1 commit difference (TinkerPop upgrade).
As I said, I'm good with waiting for TinkerPop 3.4.11 or TinkerPop 3.5.0 but I would prefer to set a day till which we should start a releasing process (unless there are critical bugs).
I would propose to set a deadline on the 1st May. If there are no new TinkerPop releases till that day (due to some delays) then we start releasing process on the 1st May as is.
If there are new TinkerPop releases in April then we update JanusGraph to the latest TinkerPop release and start releasing process after that.

What do you think? Would such a deadline be OK?

Best regards,
Oleksandr

61 - 80 of 1582