Date
1 - 4 of 4
Reindex job
|
Jerry He <jerr...@...>
Looked at the code. The timeout is 180 seconds. private static final int TIMEOUT_MS = 180000; It is not customizable. There is a configuration property 'storage.read-time', which does not seem to apply or be used here. It is trying to fetch the next batch of rows from the backend. I am still curious why it could timeout with 180 seconds. Thanks. Jerry On Wed, May 31, 2017 at 10:26 PM, Joe Obernberger <joseph.o...@...> wrote:
|
|
|
|
Joe Obernberger <joseph.o...@...>
Hi Jerry - HBase appears to be healthy. The graph is about 65 million nodes and 74 million edges. -Joe On 6/1/2017 1:05 AM, Jerry He wrote:
|
|
|
|
Jerry He <jerr...@...>
How big is your graph? Is your HBase healthy? I wonder what the default timeout duration is. It is like a JanusGraph timeout, not a HBase timeout. Thanks. Jerry On Wed, May 31, 2017 at 2:36 PM, Joe Obernberger <joseph.o...@...> wrote: Hi All - I have a graph that I'd like to add an index to. I've tried it through gremlin with a smaller graph and the procedure works fine. With a larger graph, however, I get a timeout error: |
|
|
|
Joe Obernberger <joseph.o...@...>
Hi All - I have a graph that I'd like to add an index to. I've tried it through gremlin with a smaller graph and the procedure works fine. With a larger graph, however, I get a timeout error:
gremlin> mgmt.updateIndex(mgmt.getGraphIndex("fullNameIndex"),SchemaAction.REINDEX).get() 17:18:34 ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor - Exception occured during job execution: {} org.janusgraph.diskstorage.TemporaryBackendException: Timed out waiting for next row data - storage error likely at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor.run(StandardScannerExecutor.java:150) at java.lang.Thread.run(Thread.java:745) 17:18:34 ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor - Processing thread interrupted while waiting on queue or processing data java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:272) 17:18:34 ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor - Could not load data from storage: {} java.lang.RuntimeException: java.io.InterruptedIOException at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97) at com.google.common.collect.Iterators$7.computeNext(Iterators.java:650) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.janusgraph.diskstorage.hbase.HBaseKeyColumnValueStore$RowIterator.hasNext(HBaseKeyColumnValueStore.java:295) at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$DataPuller.run(StandardScannerExecutor.java:325) Caused by: java.io.InterruptedIOException at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:410) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371) at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:94) ... 5 more 17:18:34 ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor - Could not load data from storage: {} java.lang.RuntimeException: java.io.InterruptedIOException at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97) at com.google.common.collect.Iterators$7.computeNext(Iterators.java:650) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.janusgraph.diskstorage.hbase.HBaseKeyColumnValueStore$RowIterator.hasNext(HBaseKeyColumnValueStore.java:295) at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$DataPuller.run(StandardScannerExecutor.java:325) Caused by: java.io.InterruptedIOException at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:214) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:410) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371) at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:94) ... 5 more 17:18:34 ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor - Could not load data from storage: {} java.lang.RuntimeException: java.io.InterruptedIOException at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97) at com.google.common.collect.Iterators$7.computeNext(Iterators.java:650) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.janusgraph.diskstorage.hbase.HBaseKeyColumnValueStore$RowIterator.hasNext(HBaseKeyColumnValueStore.java:295) at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$DataPuller.run(StandardScannerExecutor.java:325) Caused by: java.io.InterruptedIOException at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:214) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:410) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371) at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:94) ... 5 more org.janusgraph.diskstorage.TemporaryBackendException: Timed out waiting for next row data - storage error likely Type ':help' or ':h' for help. Display stack trace? [yN]n Any ideas? I've also tried this through Java code, but I get a different error: 17/05/31 17:14:37 INFO job.IndexRepairJob: Found index fullNameIndex 17/05/31 17:14:37 ERROR scan.StandardScannerExecutor: Exception trying to setup the job: java.lang.IllegalStateException: Operation cannot be executed because the enclosing transaction is closed at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.verifyOpen(StandardJanusGraphTx.java:299) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.getRelationType(StandardJanusGraphTx.java:891) at org.janusgraph.graphdb.query.QueryUtil.getType(QueryUtil.java:61) at org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryBuilder.constructQueryWithoutProfile(BasicVertexCentricQueryBuilder.java:456) at org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryBuilder.constructQuery(BasicVertexCentricQueryBuilder.java:399) at org.janusgraph.graphdb.olap.QueryContainer$QueryBuilder.relations(QueryContainer.java:129) at org.janusgraph.graphdb.olap.QueryContainer$QueryBuilder.edges(QueryContainer.java:165) at org.janusgraph.graphdb.olap.job.IndexRepairJob.getQueries(IndexRepairJob.java:216) at org.janusgraph.graphdb.olap.VertexJobConverter.getQueries(VertexJobConverter.java:157) at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor.run(StandardScannerExecutor.java:103) at java.lang.Thread.run(Thread.java:745) 17/05/31 17:14:37 ERROR job.IndexRepairJob: Transaction commit threw runtime exception: java.lang.IllegalArgumentException: The transaction has already been closed at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1356) at org.janusgraph.graphdb.database.management.ManagementSystem.commit(ManagementSystem.java:235) at org.janusgraph.graphdb.olap.job.IndexUpdateJob.workerIterationEnd(IndexUpdateJob.java:133) at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:76) at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118) at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor.run(StandardScannerExecutor.java:125) at java.lang.Thread.run(Thread.java:745) Exception in thread "Thread-14" java.lang.IllegalArgumentException: The transaction has already been closed at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1356) at org.janusgraph.graphdb.database.management.ManagementSystem.commit(ManagementSystem.java:235) at org.janusgraph.graphdb.olap.job.IndexUpdateJob.workerIterationEnd(IndexUpdateJob.java:133) at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:76) at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118) at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor.run(StandardScannerExecutor.java:125) at java.lang.Thread.run(Thread.java:745) 17/05/31 17:14:37 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService 17/05/31 17:14:37 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x35b92203553bdf6 17/05/31 17:14:37 INFO zookeeper.ZooKeeper: Session: 0x35b92203553bdf6 closed 17/05/31 17:14:37 INFO zookeeper.ClientCnxn: EventThread shut down Thanks for any ideas! -Joe |
|
|
