Date
1 - 3 of 3
Any advice on performance concern of JanusGraph with Cassandra&Elastic Search?
hazalkecoglu@...
Hi everyone,
We are working on a project that we would like to use JanusGraph. Our system will consist of 100MM nodes and 1B edges between those nodes.
We are going to work with last 90 days data.System needs to work like uploading 1B edges and deleting last 90 days of everyday.
So far we tried to experience JanusGraph with Cassandra and ElasticSearch.
We want to learn about your experiences and also contribute with ours during the project.
Is there anyone who worked with that huge volume of data? What should be our concerns when to work with that kind of big data?
Also what will be the best and fast approach of uploading 1B edges everyday?
Thanks a lot
hadoopmarc@...
Many organizations use JanusGraph on this scale. Insertion of data is slow so you need massive parallel operations to do an entire bulk load overnight. Most people use tools like Apache Spark for this. A useful blog series on this, can be found at (you see, there is a wide community!):
https://www.experoinc.com/post/janusgraph-nuts-and-bolts-part-1-write-performance
https://www.experoinc.com/post/have-you-had-your-janusgraph-tuneup
https://www.scylladb.com/2020/05/14/zeotap-a-graph-of-twenty-billion-ids-built-on-scylla-and-janusgraph/
Of course, you have to monitor the Cassandra and Elasticsearch clusters to check whether they are well balanced and not overloaded. Although JanusGraph can handle some overloading ("TemporaryBackendException") there are limits to this.
If you do not have any Cassandra legacy or experience and with the knowledge that JanusGraph fully supports ScyllaDb, you probably can believe the pitch made by Scylla itself:
https://www.scylladb.com/scylla-vs-cassandra/
Best wishes, Marc
https://www.experoinc.com/post/janusgraph-nuts-and-bolts-part-1-write-performance
https://www.experoinc.com/post/have-you-had-your-janusgraph-tuneup
https://www.scylladb.com/2020/05/14/zeotap-a-graph-of-twenty-billion-ids-built-on-scylla-and-janusgraph/
Of course, you have to monitor the Cassandra and Elasticsearch clusters to check whether they are well balanced and not overloaded. Although JanusGraph can handle some overloading ("TemporaryBackendException") there are limits to this.
If you do not have any Cassandra legacy or experience and with the knowledge that JanusGraph fully supports ScyllaDb, you probably can believe the pitch made by Scylla itself:
https://www.scylladb.com/scylla-vs-cassandra/
Best wishes, Marc