Hello,

In eBay, we now have JanusGraph deployed in production, with FoundationDB being the backend store. The details on the design, deployment and the eBay use cases can be found in last year’s FoundationDB Summit (https://www.youtube.com/watch?v=EtB1BPG00PE). We took the earlier version of JanusGraph-FoundationDB plugin developed by Ted Wilmes in February 2019, and have made bug fixing and improvement over the last two years.  Now we would like to contribute our code back to the open source community, with the following improved features:

(1)Some bug fixing for the current JanusGraph FoundationDB plugin, and support of Async. Iterator. With Asynchronous Iterator, we can greatly leverage FoundationDB’s Asynchronous Iterator in terms of on-demand data stream pulling, better memory efficiency, better parallelism for MultiQuery support. We introduce “iterator” mode, along with the current mode that we call “list” mode.

(2)Enable Prefetching on Global Read Version of FoundationDB. This allows read-only query to use a slightly delayed GRV for snap-shot read (configurable, say 100 ms delayed), but save the GRV fetching time from the Primary DC. In the cross-datacenter production environment, the saving can be as large as > 50 ms due to this GRV pre-fetching.

(3) Support Read-Only GraphDB from the specified environment variables at GraphDB engine start-up.

(4)Context propagation that allows the Client Application to express “read-only” optimization support from Feature 2 mentioned above, to the FoundationDB storage plugin. So that write-involved queries will not be optimized. Furthermore, we can allow transaction reset with “read-commit-write” mode to be per-transaction, and only for the “read-only” queries, further optimization for “read-only” queries.The write-involved queries will still follow "serializable" mode for strong consistency.

(5)Enable TLS that is supported by FoundationDB. Security is important in the production environment.

(6)Enable Prometheus Metrics Monitoring. So that we can have good visibility across multiple datacenter deployment in the production environment. From the metrics collected, dashboards and alerts can be further developed easily. We can share our Grafana dashboards JSON configuration files and Prometheus alert rules used in our production environment  to the open-source community also. 

The first PR on Feature 1 above can be found at: https://github.com/JanusGraph/janusgraph-foundationdb/pull/50

Regards,

Jun Li and Hieu Nguyen

--
You received this message because you are subscribed to the Google Groups "JanusGraph developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-dev/43fd48ac-9211-48ee-a1ea-240354010dc2n%40googlegroups.com.

Hello,

In eBay, we now have JanusGraph deployed in production, with FoundationDB being the backend store. The details on the design, deployment and the eBay use cases can be found in last year’s FoundationDB Summit (https://www.youtube.com/watch?v=EtB1BPG00PE). We took the earlier version of JanusGraph-FoundationDB plugin developed by Ted Wilmes in February 2019, and have made bug fixing and improvement over the last two years.  Now we would like to contribute our code back to the open source community, with the following improved features:

(1)Some bug fixing for the current JanusGraph FoundationDB plugin, and support of Async. Iterator. With Asynchronous Iterator, we can greatly leverage FoundationDB’s Asynchronous Iterator in terms of on-demand data stream pulling, better memory efficiency, better parallelism for MultiQuery support. We introduce “iterator” mode, along with the current mode that we call “list” mode.

(2)Enable Prefetching on Global Read Version of FoundationDB. This allows read-only query to use a slightly delayed GRV for snap-shot read (configurable, say 100 ms delayed), but save the GRV fetching time from the Primary DC. In the cross-datacenter production environment, the saving can be as large as > 50 ms due to this GRV pre-fetching.

(3) Support Read-Only GraphDB from the specified environment variables at GraphDB engine start-up.

(4)Context propagation that allows the Client Application to express “read-only” optimization support from Feature 2 mentioned above, to the FoundationDB storage plugin. So that write-involved queries will not be optimized. Furthermore, we can allow transaction reset with “read-commit-write” mode to be per-transaction, and only for the “read-only” queries, further optimization for “read-only” queries.The write-involved queries will still follow "serializable" mode for strong consistency.

(5)Enable TLS that is supported by FoundationDB. Security is important in the production environment.

(6)Enable Prometheus Metrics Monitoring. So that we can have good visibility across multiple datacenter deployment in the production environment. From the metrics collected, dashboards and alerts can be further developed easily. We can share our Grafana dashboards JSON configuration files and Prometheus alert rules used in our production environment  to the open-source community also. 

The first PR on Feature 1 above can be found at: https://github.com/JanusGraph/janusgraph-foundationdb/pull/50

Regards,

Jun Li and Hieu Nguyen

Hello,

In eBay, we now have JanusGraph deployed in production, with FoundationDB being the backend store. The details on the design, deployment and the eBay use cases can be found in last year’s FoundationDB Summit (https://www.youtube.com/watch?v=EtB1BPG00PE). We took the earlier version of JanusGraph-FoundationDB plugin developed by Ted Wilmes in February 2019, and have made bug fixing and improvement over the last two years.  Now we would like to contribute our code back to the open source community, with the following improved features:

(1)Some bug fixing for the current JanusGraph FoundationDB plugin, and support of Async. Iterator. With Asynchronous Iterator, we can greatly leverage FoundationDB’s Asynchronous Iterator in terms of on-demand data stream pulling, better memory efficiency, better parallelism for MultiQuery support. We introduce “iterator” mode, along with the current mode that we call “list” mode.

(2)Enable Prefetching on Global Read Version of FoundationDB. This allows read-only query to use a slightly delayed GRV for snap-shot read (configurable, say 100 ms delayed), but save the GRV fetching time from the Primary DC. In the cross-datacenter production environment, the saving can be as large as > 50 ms due to this GRV pre-fetching.

(3) Support Read-Only GraphDB from the specified environment variables at GraphDB engine start-up.

(4)Context propagation that allows the Client Application to express “read-only” optimization support from Feature 2 mentioned above, to the FoundationDB storage plugin. So that write-involved queries will not be optimized. Furthermore, we can allow transaction reset with “read-commit-write” mode to be per-transaction, and only for the “read-only” queries, further optimization for “read-only” queries.The write-involved queries will still follow "serializable" mode for strong consistency.

(5)Enable TLS that is supported by FoundationDB. Security is important in the production environment.

(6)Enable Prometheus Metrics Monitoring. So that we can have good visibility across multiple datacenter deployment in the production environment. From the metrics collected, dashboards and alerts can be further developed easily. We can share our Grafana dashboards JSON configuration files and Prometheus alert rules used in our production environment  to the open-source community also. 

The first PR on Feature 1 above can be found at: https://github.com/JanusGraph/janusgraph-foundationdb/pull/50

Regards,

Jun Li and Hieu Nguyen