1. Messages
  2. Backup & Restore of Janusgraph Data with Mixed Index Backend (Elastisearch)

Re: Backup & Restore of Janusgraph Data with Mixed Index Backend (Elastisearch)

Boxuan Li
 

Yeah, reindexing can be slow in that case. You could try the transaction recovery mechanism as described in https://docs.janusgraph.org/advanced-topics/recovery/#transaction-failure which makes use of write-ahead log and requires a process dedicated to run transaction recovery continuously.

> Can I somehow make Janusgraph respect that and fail the transaction if it can't persist to the indexing backend

Unfortunately no. That seems to be a legitimate requirement and even I don’t know why this is not allowed at the moment. You may want to raise a feature request on GitHub, and/or fork JanusGraph and apply that change.

On May 1, 2021, at 10:16 PM, florian.caesar via lists.lfaidata.foundation <florian.caesar=protonmail.com@...> wrote:

Hi Boxuan, 

thank you for the detailed response.

What if inconsistency between my primary storage and my indexing backend is not tolerable? Can I somehow make Janusgraph respect that and fail the transaction if it can't persist to the indexing backend? 

As for the recovery options, reindexing seems reasonable. Though I'm worried that reindexing all mixed indices that way will be very slow for large graphs with many mixed indices. 

Florian




-------- Original Message --------
On 1 May 2021, 13:46, Boxuan Li < liboxuan@...> wrote:

Hi Florian,

JanusGraph's philosophy is that your primary storage (ScyllaDB in your case) is the primary and authoritative source of truth, and inconsistency between your mixed index backend and storage layer is tolerable. For example, your transaction would succeed if data is persisted successfully in your primary storage but not the mixed index backend. To fix the inconsistency, you could periodically run the reindex OLAP job, and you could set up the transaction recovery process as described in https://docs.janusgraph.org/advanced-topics/recovery/#transaction-failure.

For your use case, I would suggest running reindex job after you restore data.

Cheers,
Boxuan
Join {janusgraph-users@lists.lfaidata.foundation to automatically receive all group messages.