Hi Oleksandr,

I can share some high-level information here.

The transactional graph store backend consists two main parts:

• Key-Column-Value (KCV) store: match with JanusGraph’s data model and getSlice/mutations needs. Based on our LSM-tree-based KV store Jungle, which uses append-only B+tree for SSTables and has lower overhead compared to RocksDB (see this paper for more details:https://www.usenix.org/conference/hotstorage19/presentation/ahn ). We have chunking to optimize for supernodes with millions of edges in adjacency lists.
• Distributed transactions: we developed a simple and efficient distributed transaction protocol, called GRIT. A demo was published in ICDE 2019 conference (attached). If you are interested in a short description of its principles, see this one page write-up: https://drive.google.com/file/d/1yxT0KffldCoV5Dz55376jnpBucKpvIdp/view and a few slides linked from this one pager. For transaction support for the graph store backend, we don’t need GTM in the architecture. We only need DBTM for a single scale-out DB. We have DBTM OCC implementation, LogStore (see paper for details: https://www.usenix.org/conference/hotcloud19/presentation/ahn ), and LogPlayer. LogStore replication uses Raft protocol, which is based on NuRaft that we open-sourced a couple of weeks ago (https://tech.ebayinc.com/engineering/nuraft-a-lightweight-c-raft-core/ )

The replication is through LogStores, which the LogPlayers play to the backend servers within the same DCs.

-Gene

On Wednesday, August 28, 2019 at 1:49:11 AM UTC-7, Oleksandr Porunov wrote:

Hi Gene,

It is always interesting to see new store backends.

It would be interesting to see the main concepts behind this backend.

Personally I am interested in transactions management mechanism. Is it works something like InnoDB transactions in MySQL? Is the row with the data is locked during the "update" / "read for update"? Is the data is locked in all replicas? If no, how do you choose what replicas to lock? etc.

Please, share some details when this backend is open source.

Best regards,

Oleksandr

On Tuesday, August 27, 2019 at 8:59:56 PM UTC+3, Gene Z wrote:

Hi JanusGraph developers,

We at eBay have been developing a transactional scale-out graph store backend specifically for JanusGraph to address data inconsistency problems with non-transactional backends.

The system consists of various components written in C++, at different maturity levels. But overall system works functionally. We intend to move the further development to the open source with Apache 2.0 license and try to see the interests and potential contributions from the open source community.

If you are interested, and can contribute to develop and or use/test the system, please let me know, and I can share more details, in the meantime we are going through the open source process.

Best regards,

Gene

Hi Oleksandr,

I can share some high-level information here.

The transactional graph store backend consists two main parts:

• Key-Column-Value (KCV) store: match with JanusGraph’s data model and getSlice/mutations needs. Based on our LSM-tree-based KV store Jungle, which uses append-only B+tree for SSTables and has lower overhead compared to RocksDB (see this paper for more details:https://www.usenix.org/conference/hotstorage19/presentation/ahn ). We have chunking to optimize for supernodes with millions of edges in adjacency lists.
• Distributed transactions: we developed a simple and efficient distributed transaction protocol, called GRIT. A demo was published in ICDE 2019 conference (attached). If you are interested in a short description of its principles, see this one page write-up: https://drive.google.com/file/d/1yxT0KffldCoV5Dz55376jnpBucKpvIdp/view and a few slides linked from this one pager. For transaction support for the graph store backend, we don’t need GTM in the architecture. We only need DBTM for a single scale-out DB. We have DBTM OCC implementation, LogStore (see paper for details: https://www.usenix.org/conference/hotcloud19/presentation/ahn ), and LogPlayer. LogStore replication uses Raft protocol, which is based on NuRaft that we open-sourced a couple of weeks ago (https://tech.ebayinc.com/engineering/nuraft-a-lightweight-c-raft-core/ )

The replication is through LogStores, which the LogPlayers play to the backend servers within the same DCs.

-Gene

On Wednesday, August 28, 2019 at 1:49:11 AM UTC-7, Oleksandr Porunov wrote:

Hi Gene,

It is always interesting to see new store backends.

It would be interesting to see the main concepts behind this backend.

Personally I am interested in transactions management mechanism. Is it works something like InnoDB transactions in MySQL? Is the row with the data is locked during the "update" / "read for update"? Is the data is locked in all replicas? If no, how do you choose what replicas to lock? etc.

Please, share some details when this backend is open source.

Best regards,

Oleksandr

On Tuesday, August 27, 2019 at 8:59:56 PM UTC+3, Gene Z wrote:

Hi JanusGraph developers,

We at eBay have been developing a transactional scale-out graph store backend specifically for JanusGraph to address data inconsistency problems with non-transactional backends.

The system consists of various components written in C++, at different maturity levels. But overall system works functionally. We intend to move the further development to the open source with Apache 2.0 license and try to see the interests and potential contributions from the open source community.

If you are interested, and can contribute to develop and or use/test the system, please let me know, and I can share more details, in the meantime we are going through the open source process.

Best regards,

Gene

Hi Gene,

It is always interesting to see new store backends.

It would be interesting to see the main concepts behind this backend.

Personally I am interested in transactions management mechanism. Is it works something like InnoDB transactions in MySQL? Is the row with the data is locked during the "update" / "read for update"? Is the data is locked in all replicas? If no, how do you choose what replicas to lock? etc.

Please, share some details when this backend is open source.

Best regards,

Oleksandr

On Tuesday, August 27, 2019 at 8:59:56 PM UTC+3, Gene Z wrote:

Hi JanusGraph developers,

We at eBay have been developing a transactional scale-out graph store backend specifically for JanusGraph to address data inconsistency problems with non-transactional backends.

The system consists of various components written in C++, at different maturity levels. But overall system works functionally. We intend to move the further development to the open source with Apache 2.0 license and try to see the interests and potential contributions from the open source community.

If you are interested, and can contribute to develop and or use/test the system, please let me know, and I can share more details, in the meantime we are going through the open source process.

Best regards,

Gene

Hi JanusGraph developers,

We at eBay have been developing a transactional scale-out graph store backend specifically for JanusGraph to address data inconsistency problems with non-transactional backends.

The system consists of various components written in C++, at different maturity levels. But overall system works functionally. We intend to move the further development to the open source with Apache 2.0 license and try to see the interests and potential contributions from the open source community.

If you are interested, and can contribute to develop and or use/test the system, please let me know, and I can share more details, in the meantime we are going through the open source process.

Best regards,

Gene