Hi All - I have a graph that I'd like to add an index to. I've tried it through gremlin with a smaller graph and the procedure works fine. With a larger graph, however, I get a timeout error:
gremlin> mgmt.updateIndex(mgmt.getGraphIndex("fullNameIndex"),SchemaAction.REINDEX).get()
17:18:34 ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor - Exception occured during job execution: {}
org.janusgraph.diskstorage.TemporaryBackendException: Timed out waiting for next row data - storage error likely
at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor.run(StandardScannerExecutor.java:150)
at java.lang.Thread.run(Thread.java:745)
17:18:34 ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor - Processing thread interrupted while waiting on queue or processing data
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:272)
17:18:34 ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor - Could not load data from storage: {}
java.lang.RuntimeException: java.io.InterruptedIOException
at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:650)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.janusgraph.diskstorage.hbase.HBaseKeyColumnValueStore$RowIterator.hasNext(HBaseKeyColumnValueStore.java:295)
at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$DataPuller.run(StandardScannerExecutor.java:325)
Caused by: java.io.InterruptedIOException
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:410)
at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371)
at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:94)
... 5 more
17:18:34 ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor - Could not load data from storage: {}
java.lang.RuntimeException: java.io.InterruptedIOException
at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:650)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.janusgraph.diskstorage.hbase.HBaseKeyColumnValueStore$RowIterator.hasNext(HBaseKeyColumnValueStore.java:295)
at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$DataPuller.run(StandardScannerExecutor.java:325)
Caused by: java.io.InterruptedIOException
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:214)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:410)
at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371)
at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:94)
... 5 more
17:18:34 ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor - Could not load data from storage: {}
java.lang.RuntimeException: java.io.InterruptedIOException
at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:650)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.janusgraph.diskstorage.hbase.HBaseKeyColumnValueStore$RowIterator.hasNext(HBaseKeyColumnValueStore.java:295)
at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$DataPuller.run(StandardScannerExecutor.java:325)
Caused by: java.io.InterruptedIOException
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:214)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:410)
at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371)
at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:94)
... 5 more
org.janusgraph.diskstorage.TemporaryBackendException: Timed out waiting for next row data - storage error likely
Type ':help' or ':h' for help.
Display stack trace? [yN]n
Any ideas? I've also tried this through Java code, but I get a different error:
17/05/31 17:14:37 INFO job.IndexRepairJob: Found index fullNameIndex
17/05/31 17:14:37 ERROR scan.StandardScannerExecutor: Exception trying to setup the job:
java.lang.IllegalStateException: Operation cannot be executed because the enclosing transaction is closed
at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.verifyOpen(StandardJanusGraphTx.java:299)
at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.getRelationType(StandardJanusGraphTx.java:891)
at org.janusgraph.graphdb.query.QueryUtil.getType(QueryUtil.java:61)
at org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryBuilder.constructQueryWithoutProfile(BasicVertexCentricQueryBuilder.java:456)
at org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryBuilder.constructQuery(BasicVertexCentricQueryBuilder.java:399)
at org.janusgraph.graphdb.olap.QueryContainer$QueryBuilder.relations(QueryContainer.java:129)
at org.janusgraph.graphdb.olap.QueryContainer$QueryBuilder.edges(QueryContainer.java:165)
at org.janusgraph.graphdb.olap.job.IndexRepairJob.getQueries(IndexRepairJob.java:216)
at org.janusgraph.graphdb.olap.VertexJobConverter.getQueries(VertexJobConverter.java:157)
at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor.run(StandardScannerExecutor.java:103)
at java.lang.Thread.run(Thread.java:745)
17/05/31 17:14:37 ERROR job.IndexRepairJob: Transaction commit threw runtime exception:
java.lang.IllegalArgumentException: The transaction has already been closed
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1356)
at org.janusgraph.graphdb.database.management.ManagementSystem.commit(ManagementSystem.java:235)
at org.janusgraph.graphdb.olap.job.IndexUpdateJob.workerIterationEnd(IndexUpdateJob.java:133)
at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:76)
at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118)
at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor.run(StandardScannerExecutor.java:125)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "Thread-14" java.lang.IllegalArgumentException: The transaction has already been closed
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1356)
at org.janusgraph.graphdb.database.management.ManagementSystem.commit(ManagementSystem.java:235)
at org.janusgraph.graphdb.olap.job.IndexUpdateJob.workerIterationEnd(IndexUpdateJob.java:133)
at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:76)
at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118)
at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor.run(StandardScannerExecutor.java:125)
at java.lang.Thread.run(Thread.java:745)
17/05/31 17:14:37 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
17/05/31 17:14:37 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x35b92203553bdf6
17/05/31 17:14:37 INFO zookeeper.ZooKeeper: Session: 0x35b92203553bdf6 closed
17/05/31 17:14:37 INFO zookeeper.ClientCnxn: EventThread shut down
Thanks for any ideas!
-Joe
