Could this be related to delays in indexing? I don't know how to figure out of such exists, but assuming this happens - 

how does a traversal behave when looking up based on an index key when the key is not yet indexed?

On Thursday, December 17, 2020 at 10:54:32 AM UTC+2 Assaf Schwartz wrote:

Hi All,

I'm experiencing an issues with running JanusGraph (on top of GKE) against BigTable.

This is the general setup description:

• We are using a single node BigTable cluster (for development / integration purposes) with the vanilla 0.5.2 docker.
• Indexing is configured to be done with ES (also running on GKE)
• JanusGraph is configured through environment variables:
  
  • janusgraph.storage.backend=hbase
    
  • janusgraph.storage.hbase.ext.hbase.client.connection.impl=com.google.cloud.bigtable.hbase2_x.BigtableConnection
    
  • janusgraph.storage.hbase.ext.google.bigtable.project.id=<project_id>
    
  • janusgraph.storage.hbase.ext.google.bigtable.instance.id=<bt>
    
• Interaction with JanusGraph are done only through a single gRPC server that is running gremlin-python, let's call it DB-SERVER.
  
• Last time we've done testing against BT was with version 0.4.1 of JanusGraph, precompiled to support HBase1.
• All of our components communicate via gRPC.

Description of the problem:

1. The DB-SERVER creates a Vertex i, generate some XML to represent work to be done, and sends it to another service for processing, let's call in ORCHESTRATOR.
2. The ORCHESTRATOR generates two properties, w and r (local identifiers) and sends them back to the DB-SERVER, so they will be set as properties on Vertex i. These two properties are also mixed String indexes.
3. After setting the properties, DB-SERVER will ack ORCHESTRATOR, which will start processing. As part of the processing, ORCHESTRATOR will send updates back to the DB-SERVER using w and r.
4. On getting these updates DB-SERVER, it will try looking up Vertex i based on w and r, like so:
   g.V().has("r", <some_r>).has("w", <some_w>).next()
5. At that point, a null / None is returned as the traversal fails to find Vertex i.
6. Trying the same traversal in a separate console (python and gremlin) does fetch the vertex. Since it's a single instance cluster, I ruled out any eventual consistency issues.

I'm not sure if it's a regression introduced after 0.4.1.

I've also validated that db-caching is turned off.

Help! :)

Many thanks in advance,

Assaf