How big is your graph?  Is your HBase healthy? 

I wonder what the default timeout duration is. 

It is like a JanusGraph timeout, not a HBase timeout.

Thanks.

Jerry

On Wed, May 31, 2017 at 2:36 PM, Joe Obernberger <joseph.o...@...> wrote:

Hi All - I have a graph that I'd like to add an index to.  I've tried it through gremlin with a smaller graph and the procedure works fine.  With a larger graph, however, I get a timeout error:

gremlin> mgmt.updateIndex(mgmt.getGraphIndex("fullNameIndex"),SchemaAction.REINDEX).get()
17:18:34 ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor - Exception occured during job execution: {}
org.janusgraph.diskstorage.TemporaryBackendException: Timed out waiting for next row data - storage error likely
        at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor.run(StandardScannerExecutor.java:150)
        at java.lang.Thread.run(Thread.java:745)
17:18:34 ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor - Processing thread interrupted while waiting on queue or processing data
java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$Processor.run(StandardScannerExecutor.java:272)
17:18:34 ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor - Could not load data from storage: {}
java.lang.RuntimeException: java.io.InterruptedIOException
        at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97)
        at com.google.common.collect.Iterators$7.computeNext(Iterators.java:650)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
        at org.janusgraph.diskstorage.hbase.HBaseKeyColumnValueStore$RowIterator.hasNext(HBaseKeyColumnValueStore.java:295)
        at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$DataPuller.run(StandardScannerExecutor.java:325)
Caused by: java.io.InterruptedIOException
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
        at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
        at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:410)
        at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371)
        at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:94)
        ... 5 more
17:18:34 ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor - Could not load data from storage: {}
java.lang.RuntimeException: java.io.InterruptedIOException
        at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97)
        at com.google.common.collect.Iterators$7.computeNext(Iterators.java:650)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
        at org.janusgraph.diskstorage.hbase.HBaseKeyColumnValueStore$RowIterator.hasNext(HBaseKeyColumnValueStore.java:295)
        at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$DataPuller.run(StandardScannerExecutor.java:325)
Caused by: java.io.InterruptedIOException
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:214)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
        at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
        at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:410)
        at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371)
        at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:94)
        ... 5 more
17:18:34 ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor - Could not load data from storage: {}
java.lang.RuntimeException: java.io.InterruptedIOException
        at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97)
        at com.google.common.collect.Iterators$7.computeNext(Iterators.java:650)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
        at org.janusgraph.diskstorage.hbase.HBaseKeyColumnValueStore$RowIterator.hasNext(HBaseKeyColumnValueStore.java:295)
        at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor$DataPuller.run(StandardScannerExecutor.java:325)
Caused by: java.io.InterruptedIOException
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:214)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
        at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
        at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:410)
        at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371)
        at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:94)
        ... 5 more
org.janusgraph.diskstorage.TemporaryBackendException: Timed out waiting for next row data - storage error likely
Type ':help' or ':h' for help.
Display stack trace? [yN]n


Any ideas?  I've also tried this through Java code, but I get a different error:

17/05/31 17:14:37 INFO job.IndexRepairJob: Found index fullNameIndex
17/05/31 17:14:37 ERROR scan.StandardScannerExecutor: Exception trying to setup the job:
java.lang.IllegalStateException: Operation cannot be executed because the enclosing transaction is closed
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.verifyOpen(StandardJanusGraphTx.java:299)
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.getRelationType(StandardJanusGraphTx.java:891)
        at org.janusgraph.graphdb.query.QueryUtil.getType(QueryUtil.java:61)
        at org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryBuilder.constructQueryWithoutProfile(BasicVertexCentricQueryBuilder.java:456)
        at org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryBuilder.constructQuery(BasicVertexCentricQueryBuilder.java:399)
        at org.janusgraph.graphdb.olap.QueryContainer$QueryBuilder.relations(QueryContainer.java:129)
        at org.janusgraph.graphdb.olap.QueryContainer$QueryBuilder.edges(QueryContainer.java:165)
        at org.janusgraph.graphdb.olap.job.IndexRepairJob.getQueries(IndexRepairJob.java:216)
        at org.janusgraph.graphdb.olap.VertexJobConverter.getQueries(VertexJobConverter.java:157)
        at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor.run(StandardScannerExecutor.java:103)
        at java.lang.Thread.run(Thread.java:745)
17/05/31 17:14:37 ERROR job.IndexRepairJob: Transaction commit threw runtime exception:
java.lang.IllegalArgumentException: The transaction has already been closed
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1356)
        at org.janusgraph.graphdb.database.management.ManagementSystem.commit(ManagementSystem.java:235)
        at org.janusgraph.graphdb.olap.job.IndexUpdateJob.workerIterationEnd(IndexUpdateJob.java:133)
        at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:76)
        at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118)
        at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor.run(StandardScannerExecutor.java:125)
        at java.lang.Thread.run(Thread.java:745)
Exception in thread "Thread-14" java.lang.IllegalArgumentException: The transaction has already been closed
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1356)
        at org.janusgraph.graphdb.database.management.ManagementSystem.commit(ManagementSystem.java:235)
        at org.janusgraph.graphdb.olap.job.IndexUpdateJob.workerIterationEnd(IndexUpdateJob.java:133)
        at org.janusgraph.graphdb.olap.job.IndexRepairJob.workerIterationEnd(IndexRepairJob.java:76)
        at org.janusgraph.graphdb.olap.VertexJobConverter.workerIterationEnd(VertexJobConverter.java:118)
        at org.janusgraph.diskstorage.keycolumnvalue.scan.StandardScannerExecutor.run(StandardScannerExecutor.java:125)
        at java.lang.Thread.run(Thread.java:745)
17/05/31 17:14:37 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
17/05/31 17:14:37 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x35b92203553bdf6
17/05/31 17:14:37 INFO zookeeper.ZooKeeper: Session: 0x35b92203553bdf6 closed
17/05/31 17:14:37 INFO zookeeper.ClientCnxn: EventThread shut down

Thanks for any ideas!

-Joe

--
You received this message because you are subscribed to the Google Groups "JanusGraph users list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Virus-free. www.avg.com

Hi John - I'm also very interested in how to do this.  We recently built a graph stored in HBase, and when we run g.E().count(), it took some 5+ hours to complete from the gremlin shell (79 million edges).  Is there any 'how to' or getting started guide on how to use Spark+YARN with this?

Thank you!

-Joe

On 5/31/2017 1:06 PM, 'John Helmsen' via JanusGraph users list wrote:

Gentlemen and Ladies,

Currently our group is trying to stand up an instance of JanusGraph/Titan that performs OLAP operations using SparkGraphComputer in TinkerPop.  To do OLAP,.we wish to use Spark with Yarn.  So far, however, we have not been able to successfully launch any distributed queries, such as count(), using this approach.  While we can post stack traces, etc, I'd like to ask a different question first.

Has anyone gotten the system to perform Spark operations using YARN?

If so, how?

--
You received this message because you are subscribed to the Google Groups "JanusGraph users list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Virus-free. www.avg.com

Gentlemen and Ladies,

Currently our group is trying to stand up an instance of JanusGraph/Titan that performs OLAP operations using SparkGraphComputer in TinkerPop.  To do OLAP,.we wish to use Spark with Yarn.  So far, however, we have not been able to successfully launch any distributed queries, such as count(), using this approach.  While we can post stack traces, etc, I'd like to ask a different question first.

Has anyone gotten the system to perform Spark operations using YARN?

If so, how?

--
You received this message because you are subscribed to the Google Groups "JanusGraph users list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgra...@....
For more options, visit https://groups.google.com/d/optout.

Virus-free. www.avg.com

Would it make sense to add Scala and Spark versions to the compatibility chart for clarity, even though they're tied to TinkerPop version?

On Wed, May 31, 2017 at 11:15 AM, Jason Plurad <p...@...> wrote:

I'll note that those product versions are a bit ahead of what is listed in the version compatibility matrix.

Also, as discussed in a previous thread, the supported Scala version is 2.10.5.

On Tuesday, May 30, 2017 at 8:18:20 AM UTC-4, Prabin acharya wrote:

I am trying to configure JanusGraph with Apache Cassandra 3.1 and Solr 6.1.0 .

I cannot connect to the Storage backend from the code , though all of my configuration are correct .. 

--
You received this message because you are subscribed to the Google Groups "JanusGraph users list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I am trying to configure JanusGraph with Apache Cassandra 3.1 and Solr 6.1.0 .

I cannot connect to the Storage backend from the code , though all of my configuration are correct .. 

Hi Jimmy,

I started building a list of companies using JanusGraph in production; you can see the current list here: https://github.com/JanusGraph/janusgraph#users (and the logos at the bottom of http://janusgraph.org) and more additions are on the way.

They appear to be happy with JanusGraph, but I'll let them chime in if they want to provide any additional details.

BTW, if anyone else is a production user of JanusGraph, please get in touch with me and let's get you added on the list as well!

Misha

On Fri, Apr 7, 2017 at 4:03 AM, Jimmy <xuliuc...@...> wrote:

Lovely and promising project! I want to know if anyone is using JanusGraph in production at present？Thanks!

--
You received this message because you are subscribed to the Google Groups "JanusGraph users list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgra...@....
For more options, visit https://groups.google.com/d/optout.

Hello!

I need to load CSV files into JanusGraph with a size of 1.7 gigabytes.

How can I increase the load speed? 

I added to the config:

storage.batch-loading=true

schema.default=none

I have almost no access to the disk, but the CPU is always busy.

In the profiler, the hot spot is: org.janusgraph.graphdb.internal.AbstractElement equals (8%).

Ah, you used indexOnly() to restrict the index by vertex label, so in order to utilize the index, your query must include the vertex label. Either of these would work since ultimately they are optimized to the same traversal:

g.V().hasLabel("ip4address").has("ip4addressString", "192.168.155.212")
g.V().has("ip4address", "ip4addressString", "192.168.155.212")

On Friday, May 19, 2017 at 1:57:25 PM UTC-4, CGen wrote:

PropertyKey ip4addressStringPropertyKey = mgmt.makePropertyKey("ip4addressString").dataType(String.class).make();

VertexLabel ip4addressVertexLabel = mgmt.makeVertexLabel("ip4address").make();

JanusGraphIndex ip4addressStringIndex = mgmt.buildIndex("compositeIp4addressString", Vertex.class).addKey(ip4addressStringPropertyKey).indexOnly(ip4addressVertexLabel).unique().buildCompositeIndex();

        mgmt.setConsistency(ip4addressStringIndex, ConsistencyModifier.LOCK);

пятница, 19 мая 2017 г., 17:36:46 UTC+3 пользователь Jason Plurad написал:

Can you share your property and index definition?

On Friday, May 19, 2017 at 6:37:09 AM UTC-4, CGen wrote:

And one more thing. My index exists and works because 

Adding this property for key [ip4addressString] and value [192.168.155.212] violates a uniqueness constraint [compositeIp4addressString]

пятница, 19 мая 2017 г., 12:53:16 UTC+3 пользователь CGen написал:

Thanks for the answer. I do not understand a little.

In my case I'm looking for a Vertex: traversalSource.V().has("ip4addressString", addressString);

And I get a warning: Query requires iterating over all vertices [(ip4addressString = 192.168.155.212)]. For better performance, use indexes

ip4addressString is a unique CompositeIndex. Why is my index not used?

четверг, 18 мая 2017 г., 21:11:03 UTC+3 пользователь Jason Plurad написал:

No, it is not a bug. It is a warning letting you know that there is a query that is doing a global scan of all vertices. In this case, the specific call is `g.V().count().next()` which is trying to determine whether there is any data in the graph. Something like `g.V().hasNext()` would be a less expensive check, although it would still issue the warning.

This small example can get away a call like this because there are only 12 vertices in the loaded graph, but it would be an expensive operation if there were 1 billion vertices.

Thanks for the feedback.
-- Jason

On Thursday, May 18, 2017 at 12:36:05 PM UTC-4, Ярослав Станко wrote:

Hello!

I get the example : https://github.com/pluradj/janusgraph-java-example

Why is there a warning in the log?

"WARN org.janusgraph.graphdb.transaction.StandardJanusGraphTx - Query requires iterating over all vertices [()]. For better performance, use indexes"

In my project the same thing.

Is this a bug? How to fix it?

PropertyKey ip4addressStringPropertyKey = mgmt.makePropertyKey("ip4addressString").dataType(String.class).make();

VertexLabel ip4addressVertexLabel = mgmt.makeVertexLabel("ip4address").make();

JanusGraphIndex ip4addressStringIndex = mgmt.buildIndex("compositeIp4addressString", Vertex.class).addKey(ip4addressStringPropertyKey).indexOnly(ip4addressVertexLabel).unique().buildCompositeIndex();

        mgmt.setConsistency(ip4addressStringIndex, ConsistencyModifier.LOCK);

пятница, 19 мая 2017 г., 17:36:46 UTC+3 пользователь Jason Plurad написал:

Can you share your property and index definition?

On Friday, May 19, 2017 at 6:37:09 AM UTC-4, CGen wrote:

And one more thing. My index exists and works because 

Adding this property for key [ip4addressString] and value [192.168.155.212] violates a uniqueness constraint [compositeIp4addressString]

пятница, 19 мая 2017 г., 12:53:16 UTC+3 пользователь CGen написал:

Thanks for the answer. I do not understand a little.

In my case I'm looking for a Vertex: traversalSource.V().has("ip4addressString", addressString);

And I get a warning: Query requires iterating over all vertices [(ip4addressString = 192.168.155.212)]. For better performance, use indexes

ip4addressString is a unique CompositeIndex. Why is my index not used?

четверг, 18 мая 2017 г., 21:11:03 UTC+3 пользователь Jason Plurad написал:

No, it is not a bug. It is a warning letting you know that there is a query that is doing a global scan of all vertices. In this case, the specific call is `g.V().count().next()` which is trying to determine whether there is any data in the graph. Something like `g.V().hasNext()` would be a less expensive check, although it would still issue the warning.

This small example can get away a call like this because there are only 12 vertices in the loaded graph, but it would be an expensive operation if there were 1 billion vertices.

Thanks for the feedback.
-- Jason

On Thursday, May 18, 2017 at 12:36:05 PM UTC-4, Ярослав Станко wrote:

Hello!

I get the example : https://github.com/pluradj/janusgraph-java-example

Why is there a warning in the log?

"WARN org.janusgraph.graphdb.transaction.StandardJanusGraphTx - Query requires iterating over all vertices [()]. For better performance, use indexes"

In my project the same thing.

Is this a bug? How to fix it?