Note: lists.lfaidata.foundation will be down for maintenance on Monday, September 26th, starting at 9AM Pacific Time (4PM Monday September 26, 2022 UTC), for approximately one hour.
I tried rerunning the scaling test on a fresh graph with ids.block-size=10000000 , unfortunately I haven't seen any performance gain.
I also tried ids.block-size=10000000 and ids.authority.conflict-avoidance-mode=GLOBAL_AUTO, but there also there was no performance gain.
I tried something else which turned out to be very successful:
instead of inserting all the properties in the graph, I tried only inserting the ones necessary to feed the composite indexes and vertex-centric indexes. The indexes are used to execute efficiently the "get element or create it" logic. This test scaled quite nicely up to 64 indexers (instead of 4 before)!
My best guess for why it is the case: they reduced the amount of work the ScyllaDB coordinators had to do by:
I would happily continue digging into this, unfortunately we have other priorities that turned up. We're putting the testing on the side for the moment.
I thought I would post my complete findings/guess anyway in case they are useful to someone.
Thank you so much for your help!