1. Messages
  2. Reindex job

Re: Reindex job

Jerry He <jerr...@...>
 

Looked at the code.   The timeout is 180 seconds.
private static final int TIMEOUT_MS = 180000;

It is not customizable.  There is a configuration property 'storage.read-time', which does not seem to apply or be used here.

It is trying to fetch the next batch of rows from the backend. I am still curious why it could timeout with 180 seconds.

Thanks.

Jerry

On Wed, May 31, 2017 at 10:26 PM, Joe Obernberger <joseph.o...@...> wrote:

Hi Jerry - HBase appears to be healthy.  The graph is about 65 million nodes and 74 million edges.

-Joe


On 6/1/2017 1:05 AM, Jerry He wrote:
How big is your graph?  Is your HBase healthy? 
I wonder what the default timeout duration is. 
It is like a JanusGraph timeout, not a HBase timeout.

Thanks.

Jerry

On Wed, May 31, 2017 at 2:36 PM, Joe Obernberger <joseph.o...@...> wrote:
Hi All - I have a graph that I'd like to add an index to.  I've tried it through gremlin with a smaller graph and the procedure works fine.  With a larger graph, however, I get a timeout error:

gremlin> mgmt.updateIndex(mgmt.getGraphIndex("fullNameIndex"),SchemaAction.REINDEX).get()
17:18:34 ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor - Exception occured during job execution: {}
org.janusgraph.diskstorage.TemporaryBackendException: Timed out waiting for next row data - storage error likely
        at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor.run(StandardScannerExecutor.java:150)
        at java.lang.Thread.run(Thread.java:745)
17:18:34 ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor - Processing thread interrupted while waiting on queue or processing data
java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:272)
17:18:34 ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor - Could not load data from storage: {}
java.lang.RuntimeException: java.io.InterruptedIOException
        at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97)
        at com.google.common.collect.Iterators$7.computeNext(Iterators.java:650)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
        at org.janusgraph.diskstorage.hbase.HBaseKeyColumnValueStore$RowIterator.hasNext(HBaseKeyColumnValueStore.java:295)
        at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$DataPuller.run(StandardScannerExecutor.java:325)
Caused by: java.io.InterruptedIOException
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
        at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
        at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:410)
        at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371)
        at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:94)
        ... 5 more
17:18:34 ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor - Could not load data from storage: {}
java.lang.RuntimeException: java.io.InterruptedIOException
        at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97)
        at com.google.common.collect.Iterators$7.computeNext(Iterators.java:650)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
        at org.janusgraph.diskstorage.hbase.HBaseKeyColumnValueStore$RowIterator.hasNext(HBaseKeyColumnValueStore.java:295)
        at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$DataPuller.run(StandardScannerExecutor.java:325)
Caused by: java.io.InterruptedIOException
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:214)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
        at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
        at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:410)
        at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371)
        at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:94)
        ... 5 more
17:18:34 ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor - Could not load data from storage: {}
java.lang.RuntimeException: java.io.InterruptedIOException
        at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97)
        at com.google.common.collect.Iterators$7.computeNext(Iterators.java:650)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
        at org.janusgraph.diskstorage.hbase.HBaseKeyColumnValueStore$RowIterator.hasNext(HBaseKeyColumnValueStore.java:295)
        at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$DataPuller.run(StandardScannerExecutor.java:325)
Caused by: java.io.InterruptedIOException
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:214)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
        at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
        at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:410)
        at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371)
        at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:94)
        ... 5 more
org.janusgraph.diskstorage.TemporaryBackendException: Timed out waiting for next row data - storage error likely
Type ':help' or ':h' for help.
Display stack trace? [yN]n


Any ideas?  I've also tried this through Java code, but I get a different error:

17/05/31 17:14:37 INFO job.IndexRepairJob: Found index fullNameIndex
17/05/31 17:14:37 ERROR scan.StandardScannerExecutor: Exception trying to setup the job:
java.lang.IllegalStateException: Operation cannot be executed because the enclosing transaction is closed
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.verifyOpen(StandardJanusGraphTx.java:299)
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.getRelationType(StandardJanusGraphTx.java:891)
        at org.janusgraph.graphdb.query.QueryUtil.getType(QueryUtil.java:61)
        at org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryBuilder.constructQueryWithoutProfile(BasicVertexCentricQueryBuilder.java:456)
        at org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryBuilder.constructQuery(BasicVertexCentricQueryBuilder.java:399)
        at org.janusgraph.graphdb.olap.QueryContainer$QueryBuilder.relations(QueryContainer.java:129)
        at org.janusgraph.graphdb.olap.QueryContainer$QueryBuilder.edges(QueryContainer.java:165)
        at org.janusgraph.graphdb.olap.job.IndexRepairJob.getQueries(IndexRepairJob.java:216)
        at org.janusgraph.graphdb.olap.VertexJobConverter.getQueries(VertexJobConverter.java:157)
        at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor.run(StandardScannerExecutor.java:103)
        at java.lang.Thread.run(Thread.java:745)
17/05/31 17:14:37 ERROR job.IndexRepairJob: Transaction commit threw runtime exception:
java.lang.IllegalArgumentException: The transaction has already been closed
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1356)
        at org.janusgraph.graphdb.database.management.ManagementSystem.commit(ManagementSystem.java:235)
        at org.janusgraph.graphdb.olap.job.IndexUpdateJob.workerIterationEnd(IndexUpdateJob.java:133)
        at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:76)
        at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118)
        at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor.run(StandardScannerExecutor.java:125)
        at java.lang.Thread.run(Thread.java:745)
Exception in thread "Thread-14" java.lang.IllegalArgumentException: The transaction has already been closed
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1356)
        at org.janusgraph.graphdb.database.management.ManagementSystem.commit(ManagementSystem.java:235)
        at org.janusgraph.graphdb.olap.job.IndexUpdateJob.workerIterationEnd(IndexUpdateJob.java:133)
        at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:76)
        at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118)
        at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor.run(StandardScannerExecutor.java:125)
        at java.lang.Thread.run(Thread.java:745)
17/05/31 17:14:37 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
17/05/31 17:14:37 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x35b92203553bdf6
17/05/31 17:14:37 INFO zookeeper.ZooKeeper: Session: 0x35b92203553bdf6 closed
17/05/31 17:14:37 INFO zookeeper.ClientCnxn: EventThread shut down

Thanks for any ideas!

-Joe

--
You received this message because you are subscribed to the Google Groups "JanusGraph users list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Virus-free. www.avg.com


Join {janusgraph-users@lists.lfaidata.foundation to automatically receive all group messages.