Date
1 - 4 of 4
Diagnosing slow write speeds to BigTable
|
AC
Hey there, folks. Firstly I want to say thanks for your help with the previous bug we uncovered. I'm evaluating JanusGraph performance on BigTable and observing very slow write speeds when writing even a single vertex and committing a transaction. Starting a new transaction, writing a single vertex, and committing the transaction takes at minimum 5-6 seconds. BigTable metrics indicate that the backend is never taking more than 100ms (max) to perform a write. It's hard to imagine that any amount of overhead on the BigTable side would bring this up to 5-6 seconds. The basic BigTable stats inside our application also look reasonable. Here is the current configuration: "storage.backend": "hbase" "metrics.enabled": true "cache.db-cache": false "query.batch": true "storage.page-size": 1000 "storage.hbase.ext.hbase.client.connection.impl": "com.google.cloud.bigtable.hbase2_x.BigtableConnection" "storage.hbase.ext.google.bigtable.grpc.retry.deadlineexceeded.enable": true "storage.hbase.ext.google.bigtable.grpc.channel.count": 50 "storage.lock.retries": 5 "storage.lock.wait-time": 50.millis This is running in a GCP container that is rather beefy and not doing anything else, and is located in the same region as the BigTable cluster. Other traffic to/from the container seems fine. I'm currently using hbase-shaded-client rev 2.1.5 since that's aligned to JanusGraph 0.5.3 which we are currently using. I experimented with up to 2.4.8 and saw no difference. I'm also using bigtable-hbase-2.x-shaded 1.25.1, the latest stable revision. I'm at a loss how to progress further with my diagnosis, as all evidence indicates that the latency is originating with JanusGraph's operation. How can I better find and eliminate the source of this latency? Thanks! |
|
|
|
Boxuan Li
I am not an expert on this and I've never used BigTable or GCP before, but here are my two cents:
1) Did you test the read speed? Is it also very slow compared to writing? 2) Did you try using an HBase/Bigtable client (in the same GCP container as your JanusGraph instance) to write to your BigTable cluster? If it's also very slow then the problem might be with your network or other setups. Best, Boxuan |
|
|
|
AC
Hey again Boxuan, thanks for your help in this thread! 2) That is a good idea, I will try making some writes to BigTable outside of JanusGraph in this container. However, considering that the BigTable client stats and BigTable server stats both report low latencies from within the JanusGraph application, this is looking like a JanusGraph-related issue. I will report back with results today. On Tue, Nov 16, 2021 at 11:48 AM Boxuan Li <liboxuan@...> wrote: I am not an expert on this and I've never used BigTable or GCP before, but here are my two cents: |
|
|
|
AC
I have a follow-up question in addition to my reply above: Is there any guide for understanding the JanusGraph metrics available? I have written a basic metrics integration but I'm finding it quite hard to interpret the metrics that are being produced. On Tue, Nov 16, 2021 at 12:35 PM AC via lists.lfaidata.foundation <acrane=twitter.com@...> wrote:
|
|
|