Date   

How to circumvent transaction cache?

timon.schneider@...
 

Our application has transactions editing many vertices representing elements of a branch. This branch is also represented by a vertex that has boolean property isPublished. Before committing such a transaction, we need to know whether another user set the isPublished property on the branch vertex to true, in which case the transaction should be rolled back.

Here’s the problem:
* User A reads the branch vertex but doesn’t close transaction
* User B changes the isPublished property to true and commits (while A is still making changes)
* User A read locks the vertex with an external locking API
* User A queries the branch vertex again (to make sure isPublished is still false) in the same thread but gets the old values because of the transaction cache.
Now user A can commit data even though the branch isPublished is true.

I know it’s possible to use createThreadedTx() to circumvent the ThreadLocal transaction cache. However, such refreshes will be very common in our application and ideally we would be able to execute a refresh within the main transaction to minimise complexity and workarounds. Is this possible? And if not, are there any possibilities to turn off transaction cache entirely?

Thanks in advance,
Timon


Re: Not able to reindex with bigtable as backend

hadoopmarc@...
 

The vertex centric index is written to the storage backend, so I guess the section on write performance configs should be relevant:
https://docs.janusgraph.org/advanced-topics/bulk-loading/#optimizing-writes-and-reads

If have no idea whether row locking plays a role in writing the vertex centric index. If so, the config properties you mention are relevant and maybe also the config for batch loading, which disables locking:
https://docs.janusgraph.org/advanced-topics/bulk-loading/#batch-loading

Id allocation does not seem relevant (it has its own error messages so you would notice).

Marc


Re: Not able to reindex with bigtable as backend

liqingtaobkd@...
 

Thanks a lot for your reply Marc. I browsed through the older threads and didn't find a good solution for this. 

"BigTable cannot keep up with your index repair workers" - could you provide a little bit insights for what an index repair job does, or any documentation?
I was trying a few storage settings and didn't get any luck yet: storage.write-time/storage.lock.wait-time/storage.lock.expiry-time/etc. Do you think it will make a difference? 

As you suggested, I'll try delete the index and retry from start.
For our application, we do need to have the option of reindexing current data, so I'll need to make sure it works. Do you see similar issue for Cassandra? We deploy it on GCP so we try Bigtable first.
Do you have any recommendation on backend storage for GCP please?


Re: Not able to reindex with bigtable as backend

hadoopmarc@...
 

I checked on the existing issues and the following one looks similar to your issue:
https://github.com/JanusGraph/janusgraph/issues/1803

There are also some older questions in the janusgraph users list. Only workaround seems to be to define the index before adding the data.

Best wishes,     Marc


Re: Not able to reindex with bigtable as backend

hadoopmarc@...
 

The stacktraces you sent are not from reindexing but from an index repair job. TemporaryBackendException is usually an indication of unbalanced distributed system components; apparently BigTable cannot keep up with your index repair workers. Is it still possible to delete the index and retry from the start?

Otherwise, you could try if reindexing works with just a small graph. There is little to go on right now.

Best wishes,    Marc


Re: ConfiguredGraphFactory and Authentication not working

Jansen, Jan
 


Re: ConfiguredGraphFactory and Authentication not working

hadoopmarc@...
 

Hi Vinayak,

The information you provide is still a puzzle of little pieces which is hard to make sense of. Do you mean either of the following:

  1. there is some behaviour of janusgraph that is undocumented. Please provide steps to reproduce the issue.
  2. you created a graph with v.0.4.0 and now you have trouble reading it with janusgraph v0.5.2 . According to the upgrade instuctions in the changelog, https://docs.janusgraph.org/changelog/, there are no specific instructions to read v0.4.x graphs with janusgraph v0.5.x, in other words this is not expected to be an issue.
Or is it something else? Please be very specific.

You state "The difference which I saw between the two was when I start 0.4.0 automatically configuredgraphfactory schema was created in Cassandra, but in 0.5.2 janusgraph schema is created by default. This may be the reason for it."  Can you elaborate on that. How do the different schema's look like. What are the differences in the yaml and properties config files?

Best wishes,    Marc


Delete label is very difficult

vamsi.lingala@...
 
Edited

Deleting all edges in a label/all vertices in a label from gremlin is almost impossible..
g.E().hasLabel('MAID-BRAND-012021').drop()
g.V().hasLabel('BRAND').drop()


it gets timeout even thought you increase timeout to larger limits.
I think it scans all vertices/edges and then finds which are belong to that label..which is very inefficient


Re: Not able to reindex with bigtable as backend

liqingtaobkd@...
 

Sorry for multiple emails. My found the final error from Janusgraph. Not sure it's a bigtable issue or a janusgraph/bigtable compatibility issue. Can anybody help to take a look?

3035627 [Thread-8] ERROR org.janusgraph.graphdb.olap.job.IndexRepairJob - Transaction commit threw runtime exception:

org.janusgraph.core.JanusGraphException: Could not commit transaction due to exception during persistence at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1449) at org.janusgraph.graphdb.olap.job.IndexUpdateJob.workerIterationEnd(IndexUpdateJob.java:136) at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:208) at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118) at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:285) Caused by: org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:56) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:91) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:133) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:196) at org.janusgraph.diskstorage.BackendTransaction.commit(BackendTransaction.java:150) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1440) ... 4 more Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT1M40S at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:100) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) ... 9 more Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at org.janusgraph.diskstorage.hbase.HBaseStoreManager.mutateMany(HBaseStoreManager.java:460) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:79) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:94) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:91) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) ... 10 more Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: IllegalStateException: 1 time, servers with issues: bigtable.googleapis.com at com.google.cloud.bigtable.hbase.BatchExecutor.batchCallback(BatchExecutor.java:288) at com.google.cloud.bigtable.hbase.BatchExecutor.batch(BatchExecutor.java:207) at com.google.cloud.bigtable.hbase.AbstractBigtableTable.batch(AbstractBigtableTable.java:185) at org.janusgraph.diskstorage.hbase.HTable1_0.batch(HTable1_0.java:51) at org.janusgraph.diskstorage.hbase.HBaseStoreManager.mutateMany(HBaseStoreManager.java:455) ... 14 more


Re: Not able to reindex with bigtable as backend

liqingtaobkd@...
 

Found follow error in the log. janusgraph version 0.5.3 with bigtable as backend. Any suggestions pls? I have been stuck with it for a few days...

"org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at org.janusgraph.diskstorage.hbase.HBaseStoreManager.mutateMany(HBaseStoreManager.java:460) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:79) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:94) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:91) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:91) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:133) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:196) at org.janusgraph.diskstorage.BackendTransaction.commit(BackendTransaction.java:150) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1440) at org.janusgraph.graphdb.olap.job.IndexUpdateJob.workerIterationEnd(IndexUpdateJob.java:136) at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:208) at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118) at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:285) Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: IllegalStateException: 1 time, servers with issues: bigtable.googleapis.com at com.google.cloud.bigtable.hbase.BatchExecutor.batchCallback(BatchExecutor.java:288) at com.google.cloud.bigtable.hbase.BatchExecutor.batch(BatchExecutor.java:207) at com.google.cloud.bigtable.hbase.AbstractBigtableTable.batch(AbstractBigtableTable.java:185) at org.janusgraph.diskstorage.hbase.HTable1_0.batch(HTable1_0.java:51) at org.janusgraph.diskstorage.hbase.HBaseStoreManager.mutateMany(HBaseStoreManager.java:455) ... 14 more "


Re: ConfiguredGraphFactory and Authentication not working

Vinayak Bali
 

Hi Marc,

It was working with 0.4.0. After the update to 0.5.2, it is not working. The difference which I saw between the two was when I start 0.4.0 automatically configuredgraphfactory schema was created in Cassandra, but in 0.5.2 janusgraph schema is created by default. This may be the reason for it. Please let me know how to solve the issue.

Thanks & Regards,
Vinayak

On Sun, Feb 28, 2021 at 9:26 PM <hadoopmarc@...> wrote:
Hi Vinayak,

I played around a bit with the ConfigurationManagementGraph myself. The client error you describe occurs when you run ":remote console" twice. It works as a toggle switch, in other words, the command you run was probably interpreted as for a locally embedded janusgraph instance (which was not present).

Best wishes,    Marc


Re: JanusGraphIndex how to retrieve constraint (indexOnly) specified for the global index?

cmilowka
 

Works like a charm, thank you Bo Xuan Li.


Re: Not able to reindex with bigtable as backend

liqingtaobkd@...
 

Thanks for the reply. I carefully followed each step described in the doc. Before the reindex, I closed all the open transactions and management instance. I sent the reindex command from the console and it never returns (at least for 10h+):
mgmt.updateIndex(mgmt.getRelationIndex(flowsTo, "flowsToByTimestamp"), SchemaAction.REINDEX).get()

So I don't have a chance to commit.

But from my monitoring of janusgraph and bigtable, there is no activity after 30min.

Do you have any further suggestion?


Re: ConfiguredGraphFactory and Authentication not working

hadoopmarc@...
 

Hi Vinayak,

I played around a bit with the ConfigurationManagementGraph myself. The client error you describe occurs when you run ":remote console" twice. It works as a toggle switch, in other words, the command you run was probably interpreted as for a locally embedded janusgraph instance (which was not present).

Best wishes,    Marc


Re: Not able to reindex with bigtable as backend

hadoopmarc@...
 

Please be sure to run all the steps (including a graph.tx().rollback() before index creation and a mgmt.commit() after update of the index) from the example in:

https://docs.janusgraph.org/index-management/index-performance/#vertex-centric-indexes

Best wishes,   Marc


Not able to reindex with bigtable as backend

liqingtaobkd@...
 

Hi community,

I am new to Janusgraph and am trying to build a POC. I ran into an issue that I really need some help. I am running Janusgraph on GCP with bigtable as storage backend. I am trying to create. a new vertex centric index. Since I already have some data in the database, I will need to do a re-index. The index is in "REGISTERED" status right now. 
Here are the steps that I took:

mgmt = graph.openManagement()
flowsTo = mgmt.getEdgeLabel('flowsTo')
mgmt.updateIndex(mgmt.getRelationIndex(flowsTo, "flowsToByTimestamp"), SchemaAction.REINDEX).get()
 
I am not sure if there is any way to check the status of the reindex process. I just check my janusgraph/bigtable's cpu/memory, etc, and it shows that the reindex should have finished in half an hour. However the reindex command never returned for hours, and the status is still "registered". If I try to reindex again, it shows
"Another job with the same id is already running: flowsToByTimestamp[flowsTo]"

I don't see any error from server side so I have no clue right now. Any suggestion will be greatly appreciated. Thanks!


Re: JanusGraphIndex how to retrieve constraint (indexOnly) specified for the global index?

Boxuan Li
 

You can do this:
JanusGraphSchemaType constraint = ManagementSystem.getGraphIndexDirect("indexName", ((ManagementSystem) mgmt).getWrappedTx()).getSchemaTypeConstraint();
If (constraint == null) then there is no constraint. Otherwise constraint.name() returns your indexOnly label.

Best regards,
Boxuan

On Feb 25, 2021, at 2:43 PM, hadoopmarc@... wrote:

Thanks. Still, my earlier answer remains unsatisfactory. Somehow, a new JanusGraph instance must be able to load an existing schema from the system table and have the information about label constraints available. Otherwise, the instance would not be able to index newly added vertices and edges correctly.

Best wishes,   Marc


Re: ConfiguredGraphFactory and Authentication not working

hadoopmarc@...
 

Hi Vinayak,

You showed the error message from the remote console. Does gremlin server itself log anything particular about the ConfigurationManagementGraph after a restart?

Marc


Re: ConfiguredGraphFactory and Authentication not working

Vinayak Bali
 

Hi Marc,

Tried the approach you mentioned, still facing the same issue. In one of the blogs, I read that it is due to the Cassandra cluster is first configured with janusgraphfactory and it works for the new Cassandra cluster. But this is not an acceptable solution. 

Thanks & Regards,
Vinayak

On Fri, Feb 26, 2021 at 1:04 PM <hadoopmarc@...> wrote:
Hi Vinayak,

Could you try in the gremlin-server.yaml with just:
graphs: {
  ConfigurationManagementGraph: conf/janusgraph-cql-configurationgraph.properties
}
So, without the graph, graph1 and graph2 keys.

This is a speculative answer, but this is the only thing that seems to be different from the ref docs, so we need to test the assumption that these additional keys are not harmful.

Best wishes,    Marc


Re: ConfiguredGraphFactory and Authentication not working

hadoopmarc@...
 

Hi Vinayak,

Could you try in the gremlin-server.yaml with just:
graphs: {
  ConfigurationManagementGraph: conf/janusgraph-cql-configurationgraph.properties
}
So, without the graph, graph1 and graph2 keys.

This is a speculative answer, but this is the only thing that seems to be different from the ref docs, so we need to test the assumption that these additional keys are not harmful.

Best wishes,    Marc

1001 - 1020 of 6656