Re: new contribution - production-worthy in-memory backend with no external dependencies

Chris Hupman <chris...@...>

I haven't really looked threw the PR much yet, but from your comments I would lean towards updating the existing backend, assuming the test suite doesn't see a performance hit. 

Also, thank you for the contribution. You probably already know this, but given the size of the PR I anticipate it'll take a good chunk of time to properly review. As a result I would like to pre-emptively ask you to be patient with the process. 



On Thursday, March 7, 2019 at 10:53:00 AM UTC-8, Dmitry Kovalev wrote:
Hi JanusGraph team,

We would like to contribute an improved implementation of in-memory backend for JanusGraph.

There are many possible applications for both embedded and standalone JanusGraph with a 100% in-memory backend – i.e. cases where the graph can potentially fit within the boundaries of one JVM, is built dynamically and in parallel transactions, and the performance during build-out of the graph and querying it is critical. A quick search in the issues and mailing list seems to confirm that there is general interest in this kind of use case, both for JanusGraph specifically and for an open-source in-memory graph in general (e.g. TinkerGraph may not fit the bill in all cases as it is not transactional etc).
However there seems to be a bit of a gap in JanusGraph offering in this space. Current implementation of JanusGraph’s in-memory backend has not changed since Titan times, and is still declared as “for testing only”, “not ready for production use”.


We have done some analysis a while back which shows that the main obstacle for production use of current in-memory backend is enormous overhead when storing key values (likely a trade-off vs simplicity of implementation for something that was only intended for testing purposes).
Quite often the total overhead of wrapping data structures and references is bigger than the actual data being stored. 

After a series of improvements this overhead was significantly reduced (2x-5x depending on the “shape” of the data stored and the size of individual data entries). 

The modified version of in-memory backend is successfully used in production, handling up to 70+ concurrent read/write transactions at any one moment, 10+ millions of vertices (of which quite a few have up to 60 properties) and about 3x more edges, within one JVM.

More detailed analysis and memory profiling of current vs improved implementation is attached to the corresponding github issue

The initial PR is here:

NOTE: the PR currently suggests adding the new backend rather than modifying the existing one, as it seemed cleaner and easier to compare performance within one codebase branch.

However, having read recent discussions about adding more backends into main repository vs maintenance costs, I am starting to think that actually modifying the existing in-memory backend could be a better idea.

This is because:

a)      the modifications are fairly straightforward and do not change the general structure of existing backend

b)      don’t add too much extra code or any external dependencies

c)       don’t require any additional documentation or setup instructions, at least initially

d)      all existing (and a few extra) integrity tests pass and using it in place of current version is unlikely to create any issues for current (test-only) uses

e)      the same backend will simply be fit for production use, no new backends to maintain

Any thoughts on the contribution as a whole and new backend vs updating the existing one are much appreciated.

Many thanks,


Join { to automatically receive all group messages.