Date   

Re: Potential complexities of making secondary persistence atomic

rngcntr
 

Hi Florian!

That definitely sounds like a promising idea. You are most likely not the only one requiring consistency between storage and index data. In general, my advice is to start with a test case which explicitly addresses the situation where the primary persistence succeeds and the secondary persistence fails. From there, see what it needs to let the transaction fail. Once you have a solution for that, you can start the discussion by opening a PR. We will then see which issues could arise and try to solve them. The first prototype does not have to be perfect, a proof of concept is fine!

Best Regards,
Florian


Potential complexities of making secondary persistence atomic

florian.caesar <florian.caesar@...>
 

Hi!

Currently JanusGraph's philosophy is to fail transactions only if the primary persistence fails and allow failures in secondary persistence.
According to https://docs.janusgraph.org/advanced-topics/recovery/#transaction-failure, the recommended approach for dealing with secondary persistence failures is maintaining and continuously recovering from a write-ahead log.
This results in temporary inconsistencies between primary storage and mixed indices.

For my use case, potential inconsistencies are not acceptable. Currently there is no way of making JanusGraph fail a transaction if either primary or secondary persistence fails. Looking at the source code (starting at https://github.com/JanusGraph/janusgraph/blob/a73d39d8a071f8d5340aede4917653629ab595cb/janusgraph-core/src/main/java/org/janusgraph/graphdb/database/StandardJanusGraph.java#L743), it should be possible to re-order persistence to commit both secondary and primary storage at the same time. Naturally there is some cleanup to do in case of transaction failure, but it seems manageable to me.

What are the potential complexities of pursuing this? How does this mess with JanusGraph's logic & assumptions in other places?

If it seems fine, it should be an easy change. If nobody else wants to take it, I would also be happy to start a PR for this feature myself at some point.

Thanks for your input and time
Florian


Re: [DISCUSS] JanusGraph 0.6.0 release

Oleksandr Porunov
 

As Tinkerpop is very close for a release, I'm proposing to delay JanusGraph 0.6.0 release, so that we could ship 0.6.0 release of JanusGraph with either 3.4.11 or 3.5.0 version of Tinkerpop. I assume, if everything goes well, we should be able to release JanusGraph 0.6.0 version in the middle of May.


Re: [DISCUSS] JanusGraph 0.6.0 release

Jansen, Jan
 


Re: [Performance Optimization] Optimization around the `system_properties` table interaction

Boxuan Li
 

Yeah that makes sense. I saw you said “unreplicated” thus wondered. I am not familiar with how `system_properties` is handled, but just want to point out that it is very difficult if not impossible to change the data model while keeping backward compatibility at the same time.

On Apr 22, 2021, at 11:22 PM, sauverma <saurabhdec1988@...> wrote:

Hi Boxuan Li

We are using RF3.

What I meant is that essentially the data for system_properties is going to a single partition. Due to RF3, this partition is replicated 3 times.

Does this clarify?

Thanks
Saurabh



Re: [Performance Optimization] Optimization around the `system_properties` table interaction

sauverma
 

Hi Boxuan Li

We are using RF3.

What I meant is that essentially the data for system_properties is going to a single partition. Due to RF3, this partition is replicated 3 times.

Does this clarify?

Thanks
Saurabh


Re: [Performance Optimization] Optimization around the `system_properties` table interaction

Boxuan Li
 

Hi @sauverma,

I am just curious: I noticed you said "there is only 1 partition for system_properties unreplicated". Do you have storage.cql.replication-factor = 1?


Re: [Performance Optimization] Optimization around the `system_properties` table interaction

sauverma
 

Hi all

Updates on this issue

- We found that the periodic removal of system_properties (while the ingestion is running) leads to graph corruption (mentioned at high level at https://docs.janusgraph.org/advanced-topics/recovery/)
- The perf issue we saw were due to below reasons
     - improper handling on dataproc scaledown which lead to connections not getting closed to JG, and thus ever increasing system_properties table
     - unbounded access to the scylla caching layer, which is basically unthrottled access to scylla caching system, leading to other queries slowing down due to the system_properties single, hot partition
     - in addition to this, the data model for system_properties still needs to be fixed via usage of clustering keys, by design system_properties has only 1 SINGLE partition and all spark executors hit it while initialization leading to  query slow down -> query queuing -> query timeouts

Thanks


Re: [DISCUSS] JanusGraph 0.6.0 release

Jansen, Jan
 


What do you think? Sounds great
Would such a deadline be OK? I think it is good idea to have a deadline.


Von: janusgraph-dev@... <janusgraph-dev@...> im Auftrag von Oleksandr Porunov <alexandr.porunov@...>
Gesendet: Dienstag, 30. März 2021 18:49:50
An: janusgraph-dev@...
Betreff: Re: [janusgraph-dev] [DISCUSS] JanusGraph 0.6.0 release
 
I am good with waiting for TinkerPop 3.4.11 or 3.5.0 if we expect those releases soon, but I don't want to delay 0.6.0 release too much because the community sometimes asks for the release.
Also, if we release 0.6.0 without TinkerPop 3.5.0 version, I'm good with shipping 0.7.0 version even if that release is just 1 commit difference (TinkerPop upgrade).
As I said, I'm good with waiting for TinkerPop 3.4.11 or TinkerPop 3.5.0 but I would prefer to set a day till which we should start a releasing process (unless there are critical bugs).
I would propose to set a deadline on the 1st May. If there are no new TinkerPop releases till that day (due to some delays) then we start releasing process on the 1st May as is.
If there are new TinkerPop releases in April then we update JanusGraph to the latest TinkerPop release and start releasing process after that.

What do you think? Would such a deadline be OK?

Best regards,
Oleksandr


Re: [DISCUSS] JanusGraph 0.6.0 release

Oleksandr Porunov
 

I am good with waiting for TinkerPop 3.4.11 or 3.5.0 if we expect those releases soon, but I don't want to delay 0.6.0 release too much because the community sometimes asks for the release.
Also, if we release 0.6.0 without TinkerPop 3.5.0 version, I'm good with shipping 0.7.0 version even if that release is just 1 commit difference (TinkerPop upgrade).
As I said, I'm good with waiting for TinkerPop 3.4.11 or TinkerPop 3.5.0 but I would prefer to set a day till which we should start a releasing process (unless there are critical bugs).
I would propose to set a deadline on the 1st May. If there are no new TinkerPop releases till that day (due to some delays) then we start releasing process on the 1st May as is.
If there are new TinkerPop releases in April then we update JanusGraph to the latest TinkerPop release and start releasing process after that.

What do you think? Would such a deadline be OK?

Best regards,
Oleksandr


Re: [DISCUSS] JanusGraph 0.6.0 release

rngcntr
 

If we wait for TinkerPop 3.5.0 (or 3.4.11), we can probably also release the configurable batch sizes in 0.6.0 which should bring a noticable performance gain if used properly.

Kind regards, Florian


Re: [DISCUSS] JanusGraph 0.6.0 release

Jansen, Jan
 

Hi

I would be great to release JanusGraph 0.6.0.

I just want to mention that TinkerPop 3.5.0 is planed to be released in April.

We don't release new version in frequent time span, so should we wait to release or not?

Best regards, Jan


Re: [DISCUSS] JanusGraph 0.6.0 release

Boxuan Li
 

Thanks for organizing this, Oleksandr! I would like to have Fix potential ThreadLocal transaction leak released in 0.6.0. If there is no further comment I plan to merge it this weekend.

Graph Reindexing Issue Fix: this PR is actually WIP. If I have some time at the weekend, I'll try to see if I can help fix it.

Best regards,
Boxuan Li


[DISCUSS] JanusGraph 0.6.0 release

Oleksandr Porunov
 

Hi everyone,

I would like to start a discussion about JanusGraph 0.6.0 release. There were multiple requests from the community about ETA of 0.6.0 release.
I wanted to check if there are any features / bugfixes which we want to include into 0.6.0 release?
Right now we have the next PRs / issues targeted 0.6.0 release:
Reduce configuration options for CFG - will check soon.
Bump bigtable-hbase-2.x-shaded from 1.16.0 to 1.19.0 - this is just a version bump, but I didn't test JanusGraph compatibility with Bigtable.
Add exprimental support for java 11 - This PR adds experimental support for Java 11, but looks like many tests won't work with Java 11. So it's hard to say if JanusGraph will be compatible with Java 11 when used with Hbase for example or different settings. Also, it's hard to say about relation between TinkerPop and JanusGraph regarding this matter. TinkerPop added Java 11 support to 3.5.0 version, but this version isn't released yet. I guess this might influence some TinkerPop tests and functionality, but I didn't check that yet.
Graph Reindexing Issue Fix - this PR is still under review.
Add job to run all TinkerPop tests on schedule - I think this issue isn't hard to be resolved, so I think we can resolve it. Nevertheless it shouldn't block 0.6.0 release anyhow because we run TinkerPop tests before the release.

Are there any other work we need to add to 0.6.0 version?
From my point of view, I think we can release 0.6.0 version after the above issues / PRs are resolved or re-targeted. That said, if I miss something, please, add relevant issues / PRs to the milestone or discuss it in this thread.

Best regards,
Oleksandr Porunov


Re: Multiple vertices generated for the same index value and vertex properties missing with RF3

sauverma
 

Hi Marc

I've submitted this as a bug https://github.com/JanusGraph/janusgraph/issues/2515

Thanks


Re: Multiple vertices generated for the same index value and vertex properties missing with RF3

hadoopmarc@...
 


Re: [Performance Issue] Large partitions formed on janusgraph_ids table leading to read perf issues (throughput reduces to 1/3rd of original)

hadoopmarc@...
 


Multiple vertices generated for the same index value and vertex properties missing with RF3

sauverma
 

Hi JG community

We are facing a weird issue with Replication factor 3 with scylla DB as backend.  

- we are using index for looking up V in the graph
- observation is that for many vertices with the same index value, multiple vertex IDs are generated as seen below


Has anyone else encountered the same issue with RF3, with RF1 the issue is not encountered

Thanks


Re: [Performance Optimization] Optimization around the `system_properties` table interaction

sauverma
 

On Mon, Feb 22, 2021 at 01:58 AM, <simone3.cattani@...> wrote:
columns
Hi @simone

We were planning to segregate the static configurations from the runtime dynamic configurations (last update TS from client, etc.).
AFAIK only the static configurations are required by the janusgraph clients while initializing.

Thanks
Saurabh


Re: [Performance Optimization] Optimization around the `system_properties` table interaction

simone3.cattani@...
 

HI @sauverma,
nice, we are truncating the table, too. It's good to have your confirmation that it could safely workaround the issue.
We will analyze the `gc_grace_seconds` setting now.

Just for curiosity: are you fixing it changing the `KCVSConfiguration` in order to store properties as rows instead of columns? 

81 - 100 of 1592