1. Janusgraph-Users
  2. Topics

Backup & Restore of Janusgraph Data with Mixed Index Backend (Elastisearch)

florian.caesar
 

Hi,

what is the recommended approach for backing up the Janusgraph storage layer (ScyllaDB in my case) together with a mixed index backend (Elasticsearch)?
I know I can back up & restore them separately, but that seems like it might lead to inconsistencies since it's not coordinated.
I would appreciate input from anyone who has ever run Janusgraph in production before - thanks!

Regards,

Florian
Boxuan Li
 

Hi Florian,

JanusGraph's philosophy is that your primary storage (ScyllaDB in your case) is the primary and authoritative source of truth, and inconsistency between your mixed index backend and storage layer is tolerable. For example, your transaction would succeed if data is persisted successfully in your primary storage but not the mixed index backend. To fix the inconsistency, you could periodically run the reindex OLAP job, and you could set up the transaction recovery process as described in https://docs.janusgraph.org/advanced-topics/recovery/#transaction-failure.

For your use case, I would suggest running reindex job after you restore data.

Cheers,
Boxuan
florian.caesar
 

Hi Boxuan,

thank you for the detailed response.

What if inconsistency between my primary storage and my indexing backend is not tolerable? Can I somehow make Janusgraph respect that and fail the transaction if it can't persist to the indexing backend?

As for the recovery options, reindexing seems reasonable. Though I'm worried that reindexing all mixed indices that way will be very slow for large graphs with many mixed indices.

Florian




-------- Original Message --------
On 1 May 2021, 13:46, Boxuan Li < liboxuan@...> wrote:

Hi Florian,

JanusGraph's philosophy is that your primary storage (ScyllaDB in your case) is the primary and authoritative source of truth, and inconsistency between your mixed index backend and storage layer is tolerable. For example, your transaction would succeed if data is persisted successfully in your primary storage but not the mixed index backend. To fix the inconsistency, you could periodically run the reindex OLAP job, and you could set up the transaction recovery process as described in https://docs.janusgraph.org/advanced-topics/recovery/#transaction-failure.

For your use case, I would suggest running reindex job after you restore data.

Cheers,
Boxuan

Boxuan Li
 

Yeah, reindexing can be slow in that case. You could try the transaction recovery mechanism as described in https://docs.janusgraph.org/advanced-topics/recovery/#transaction-failure which makes use of write-ahead log and requires a process dedicated to run transaction recovery continuously.

> Can I somehow make Janusgraph respect that and fail the transaction if it can't persist to the indexing backend

Unfortunately no. That seems to be a legitimate requirement and even I don’t know why this is not allowed at the moment. You may want to raise a feature request on GitHub, and/or fork JanusGraph and apply that change.

On May 1, 2021, at 10:16 PM, florian.caesar via lists.lfaidata.foundation <florian.caesar=protonmail.com@...> wrote:

Hi Boxuan, 

thank you for the detailed response.

What if inconsistency between my primary storage and my indexing backend is not tolerable? Can I somehow make Janusgraph respect that and fail the transaction if it can't persist to the indexing backend? 

As for the recovery options, reindexing seems reasonable. Though I'm worried that reindexing all mixed indices that way will be very slow for large graphs with many mixed indices. 

Florian




-------- Original Message --------
On 1 May 2021, 13:46, Boxuan Li < liboxuan@...> wrote:

Hi Florian,

JanusGraph's philosophy is that your primary storage (ScyllaDB in your case) is the primary and authoritative source of truth, and inconsistency between your mixed index backend and storage layer is tolerable. For example, your transaction would succeed if data is persisted successfully in your primary storage but not the mixed index backend. To fix the inconsistency, you could periodically run the reindex OLAP job, and you could set up the transaction recovery process as described in https://docs.janusgraph.org/advanced-topics/recovery/#transaction-failure.

For your use case, I would suggest running reindex job after you restore data.

Cheers,
Boxuan
florian.caesar
 

Thanks again. Yeah, might end up doing that, but it seems like a complicated solution.. hmm.

Regarding the feature request, I'll dig into the code and ask around the janusgraph-dev group :)
hadoopmarc@...
 

In theory (not used in practice) the following should be possible:
  1. make a snapshot of the ScyllaDB keyspace
  2. after the ScyllaDB snapshot is written, make a snapshot of corresponding ES mixed indices
  3. restore all snapshots on separate temporary clusters (doing this manually on a production cluster is a no-go)
  4. find the latest writetime in the ScyllaDB snapshot
  5. try all ES index items later than this timestamp and remove them if the corresponding vertices cannot be retrieved from ScyllaDB
  6. make a new snapshot of the ES mixed indices
This is rather cumbersome, of course, but it would allow for a fast restore of consistent indices (this does not deal with the other issue, the partially succeeded transactions).

Best wishes,   Marc
florian.caesar
 

Awesome, yes, that's very similar to what I was planning!
It's not perfect and definitely needs to tested thoroughly, but it should be much faster and reasonably scriptable.
I'll let you all know how it goes when I get to setting this up.. hopefully won't be long, a decade or so at most.

Thanks!

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Monday, May 3rd, 2021 at 9:06 AM, <hadoopmarc@...> wrote:
In theory (not used in practice) the following should be possible:

  1. make a snapshot of the ScyllaDB keyspace
  2. after the ScyllaDB snapshot is written, make a snapshot of corresponding ES mixed indices
  3. restore all snapshots on separate temporary clusters (doing this manually on a production cluster is a no-go)
  4. find the latest writetime in the ScyllaDB snapshot
  5. try all ES index items later than this timestamp and remove them if the corresponding vertices cannot be retrieved from ScyllaDB
  6. make a new snapshot of the ES mixed indices
This is rather cumbersome, of course, but it would allow for a fast restore of consistent indices (this does not deal with the other issue, the partially succeeded transactions).

Best wishes,   Marc

rngcntr
 

Although the solution presented by Marc is also the closest to a consistent backup that I can think of, there are obviously caveats to it. Updates of values which were written after the time of the Scylla snapshot could be present in ES, corrupting the state of the index. Therefore, checking the pure existence of a vertex in Scylla may not be sophisticated enough to guarantee a consistent state. Verifying the property values explicitly can be helpful here, but that still leaves us with the question how to handle mismatches of this kind.
Just keep that in mind when using such a backup strategy in your environment.

Best regards,
Florian
florian.caesar
 

Yeah, good point, it's a bit hairy. Having potentially inconsistent index backups makes them much less attractive. Though I guess I could run a reindex job on just the delta since last Scylla write time and last ES write time.
As a simpler alternative, how about pausing write transactions for say ~1s and initiating simultaneous backups of my Scylla and ES clusters during that time?
From what I can tell, both backup mechanisms guarantee snapshot isolation. A short write pause should ensure that all writes have propagated.
What caveats do you see with this approach?

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Monday, May 3rd, 2021 at 9:49 AM, rngcntr <florian.grieskamp@...> wrote:

Although the solution presented by Marc is also the closest to a consistent backup that I can think of, there are obviously caveats to it. Updates of values which were written after the time of the Scylla snapshot could be present in ES, corrupting the state of the index. Therefore, checking the pure existence of a vertex in Scylla may not be sophisticated enough to guarantee a consistent state. Verifying the property values explicitly can be helpful here, but that still leaves us with the question how to handle mismatches of this kind.
Just keep that in mind when using such a backup strategy in your environment.

Best regards,
Florian

rngcntr
 

If your use case can handle the downtime, stopping writes and waiting until all changes are propagated to both the storage and the index backend sounds like a viable solution. However, I have no idea about the order of magnitude of the necessary downtime.

1 - 10 of 10