MapReduceIndexManagement reindex not completing successfully


nigel...@...
 

I am using a snapshot build, janusgraph-0.2.0-SNAPSHOT-hadoop2, and I am trying to reindex a mixed index using a map reduce job.

graph = JanusGraphFactory.open('target.properties')
mgmt = graph.openManagement()
mr = new MapReduceIndexManagement(graph)
mr.updateIndex(mgmt.getGraphIndex("mixedV"), SchemaAction.REINDEX).get()

This starts up, and tries to do some stuff (I have tried stepping through the code). There is a warning at start
15:07:16 WARN  org.apache.hadoop.mapreduce.JobResourceUploader  - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.

I don't think that is relevant.

After a short time I get multiple warnings like this


15:08:09 WARN  org.apache.thrift.transport.TIOStreamTransport  - Error closing output stream.

java.net.SocketException: Socket closed

at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:116)

at java.net.SocketOutputStream.write(SocketOutputStream.java:153)

at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)

at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)

at java.io.FilterOutputStream.close(FilterOutputStream.java:158)

at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)

at org.apache.thrift.transport.TSocket.close(TSocket.java:194)

at org.apache.thrift.transport.TFramedTransport.close(TFramedTransport.java:89)

at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.closeClient(ThriftSyncConnectionFactoryImpl.java:272)

at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.access$800(ThriftSyncConnectionFactoryImpl.java:92)

at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection$2.run(ThriftSyncConnectionFactoryImpl.java:254)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)


Eventually the job fails

15:08:37 WARN  org.apache.hadoop.mapred.LocalJobRunner  - job_local311804379_0001

java.lang.Exception: org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception

at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)

at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)

Caused by: org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception

at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)

at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)

at org.janusgraph.diskstorage.log.kcvs.KCVSLog.readSetting(KCVSLog.java:818)

at org.janusgraph.diskstorage.log.kcvs.KCVSLog.<init>(KCVSLog.java:270)

at org.janusgraph.diskstorage.log.kcvs.KCVSLogManager.openLog(KCVSLogManager.java:225)

at org.janusgraph.diskstorage.Backend.initialize(Backend.java:275)

at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1841)

at org.janusgraph.graphdb.database.StandardJanusGraph.<init>(StandardJanusGraph.java:134)

at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:107)

at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:97)

at org.janusgraph.hadoop.scan.HadoopVertexScanMapper.setup(HadoopVertexScanMapper.java:37)

at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)

at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S

at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)

at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)

... 19 more

Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend

at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getNamesSlice(AstyanaxKeyColumnValueStore.java:128)

at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getNamesSlice(AstyanaxKeyColumnValueStore.java:92)

at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getSlice(AstyanaxKeyColumnValueStore.java:81)

at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77)

at org.janusgraph.diskstorage.keycolumnvalue.KCVSUtil.get(KCVSUtil.java:52)

at org.janusgraph.diskstorage.log.kcvs.KCVSLog$3.call(KCVSLog.java:821)

at org.janusgraph.diskstorage.log.kcvs.KCVSLog$3.call(KCVSLog.java:818)

at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)

at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)

at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)

... 20 more

Caused by: com.netflix.astyanax.connectionpool.exceptions.TransportException: TransportException: [host=127.0.0.1(127.0.0.1):9160, latency=0(0), attempts=1]org.apache.thrift.transport.TTransportException: java.net.SocketException: Bad file descriptor

at com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:197)

at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:139)

at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:119)

at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:352)

at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$4.execute(ThriftColumnFamilyQueryImpl.java:538)

at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getNamesSlice(AstyanaxKeyColumnValueStore.java:126)

... 29 more

Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Bad file descriptor

at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)

at org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:158)

at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)

at org.apache.cassandra.thrift.Cassandra$Client.send_set_keyspace(Cassandra.java:602)

at org.apache.cassandra.thrift.Cassandra$Client.set_keyspace(Cassandra.java:594)

at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:127)

... 33 more

Caused by: java.net.SocketException: Bad file descriptor

at java.net.SocketOutputStream.socketWrite0(Native Method)

at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)

at java.net.SocketOutputStream.write(SocketOutputStream.java:153)

at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)

at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)

at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)

... 38 more

15:08:37 WARN  org.apache.thrift.transport.TIOStreamTransport  - Error closing output stream.

java.net.SocketException: Socket closed

at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:116)

at java.net.SocketOutputStream.write(SocketOutputStream.java:153)

at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)

at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)

at java.io.FilterOutputStream.close(FilterOutputStream.java:158)

at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)

at org.apache.thrift.transport.TSocket.close(TSocket.java:194)

at org.apache.thrift.transport.TFramedTransport.close(TFramedTransport.java:89)

at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.closeClient(ThriftSyncConnectionFactoryImpl.java:272)

at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.access$800(ThriftSyncConnectionFactoryImpl.java:92)

at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection$2.run(ThriftSyncConnectionFactoryImpl.java:254)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

java.io.IOException: MapReduce JobID job_local311804379_0001 terminated abnormally: state=FAILED, failureinfo=NA

Type ':help' or ':h' for help.

Display stack trace? [yN]y

java.util.concurrent.ExecutionException: java.io.IOException: MapReduce JobID job_local311804379_0001 terminated abnormally: state=FAILED, failureinfo=NA

at org.janusgraph.hadoop.MapReduceIndexManagement$FailedJobFuture.get(MapReduceIndexManagement.java:297)

at org.janusgraph.hadoop.MapReduceIndexManagement$FailedJobFuture.get(MapReduceIndexManagement.java:267)

at java_util_concurrent_Future$get.call(Unknown Source)

at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)

at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)

at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)

at groovysh_evaluate.run(groovysh_evaluate:3)

at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)

at org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:70)

at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:191)

at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)

at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)

at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)

at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1218)

at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)

at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:72)

at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)

at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:95)

at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)

at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)

at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1218)

at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)

at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)

at org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:124)

at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)

at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)

at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)

at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1218)

at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)

at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)

at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:83)

at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)

at org.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:169)

at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)

at org.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:478)

Caused by: java.io.IOException: MapReduce JobID job_local311804379_0001 terminated abnormally: state=FAILED, failureinfo=NA

at org.janusgraph.hadoop.scan.HadoopScanRunner.runJob(HadoopScanRunner.java:147)

at org.janusgraph.hadoop.MapReduceIndexManagement.updateIndex(MapReduceIndexManagement.java:186)

at org.janusgraph.hadoop.MapReduceIndexManagement$updateIndex.call(Unknown Source)

at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)

at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)

at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)

... 42 more


Other map-reduce jobs seem to run (e.g. page rank and some of the other demos).

I can reindex with the Management API. I am assuming our graphs will get too big for that. This is a small graph with a few thousand nodes. Cassandra running locally on one machine.

Any comments or hints on getting this running would be most welcome.




Join {janusgraph-users@lists.lfaidata.foundation to automatically receive all group messages.