1. janusgraph-users@lists.lfaidata.foundation
  2. Messages
   Date   
[Performance Issue] Large partitions formed on janusgraph_ids table leading to read perf issues (throughput reduces to 1/3rd of original)

sauverma
 

Hi all

We are using janusgraph at zeotap at humongous scale (~70B V and 50B E) backed by scylla.

Right now I am facing an issue in janusgraph_ids table, wherein there are large partitions created in the scylla DB, and this is leading to huge read performance issues. The queries hitting janusgraph_ids table are range queries and with large partitions, the reads are becoming super slow.  

I would like to know if anyone else has observed similar issue, is there a set of configurations that need to be checked or something else you would suggest.

In scylla grafana board, this issue is seen with high number of foreground read tasks.



Thanks
Saurabh
Multiple vertices generated for the same index value and vertex properties missing with RF3

sauverma
 

We are facing a weird issue with Replication factor 3 with scylla DB as backend.  

- we are using index for looking up V in the graph
- observation is that for many vertices with the same index value, multiple vertex IDs are generated as seen below



Has anyone else encountered the same issue with RF3, with RF1 the issue is not encountered

Thanks
Saurabh Verma
Re: Script16.groovy: 2: unable to resolve class StandardJanusGraph

Vinayak Bali
 

Hi Marc,

If possible please share the code used, let me check if it can be used to modify my code.

Thanks & Regards,
Vinayak

On Tue, 9 Mar 2021, 12:54 pm , <hadoopmarc@...> wrote:
Hi Vinayak,

When I replayed your imports I got the same error, but when I tried again it worked. I guess the delay for downloading dependencies was too high and second time the cache could be used.

Best wishes,    Marc

Re: Script16.groovy: 2: unable to resolve class StandardJanusGraph

hadoopmarc@...
 

Hi Vinayak,

When I replayed your imports I got the same error, but when I tried again it worked. I guess the delay for downloading dependencies was too high and second time the cache could be used.

Best wishes,    Marc
Re: Script16.groovy: 2: unable to resolve class StandardJanusGraph

Vinayak Bali
 

Hi Marc,

Explicitly, imported StandardJanusGraph using the following import command which is not working:
org.janusgraph.graphdb.database.StandardJanusGraph

Tried Graph approach too, it throws the following error:

Script158.groovy: 1: unable to resolve class LinkedBlockingQueue
 @ line 1, column 1.
   class WaitingQueue<E> extends LinkedBlockingQueue<E>
   ^

1 error
Type ':help' or ':h' for help.
Display stack trace? [yN]y
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
Script158.groovy: 1: unable to resolve class LinkedBlockingQueue
 @ line 1, column 1.
   class WaitingQueue<E> extends LinkedBlockingQueue<E>
   ^

1 error

at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:311)
at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:980)
at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:647)
at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:596)
at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:390)
at groovy.lang.GroovyClassLoader.access$300(GroovyClassLoader.java:89)
at groovy.lang.GroovyClassLoader$5.provide(GroovyClassLoader.java:330)
at groovy.lang.GroovyClassLoader$5.provide(GroovyClassLoader.java:327)
at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:147)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:325)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:309)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:251)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine$GroovyCacheLoader.lambda$load$0(GremlinGroovyScriptEngine.java:819)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.CompletableFuture.asyncSupplyStage(CompletableFuture.java:1618)
at java.util.concurrent.CompletableFuture.supplyAsync(CompletableFuture.java:1843)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine$GroovyCacheLoader.load(GremlinGroovyScriptEngine.java:817)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine$GroovyCacheLoader.load(GremlinGroovyScriptEngine.java:812)
at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:3117)
at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:144)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$16(BoundedLocalCache.java:1968)
at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1966)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1949)
at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:113)
at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:67)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.getScriptClass(GremlinGroovyScriptEngine.java:567)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:374)
at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$0(GremlinExecutor.java:267)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

All the import statements are:
import org.janusgraph.core.ConfiguredGraphFactory
//import org.janusgraph.core.Cardinality
//import org.apache.tinkerpop.gremlin.structure.Vertex

import java.util.concurrent.Executors
import java.util.concurrent.LinkedBlockingQueue
import java.util.concurrent.ThreadPoolExecutor
import java.util.concurrent.TimeUnit

import org.janusgraph.graphdb.database.StandardJanusGraph
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource
import com.opencsv.CSVReaderBuilder
import com.opencsv.CSVReader
import com.opencsv.enums.CSVReaderNullFieldIndicator

Based on your experience, please suggest an approach using which I can solve the issue.

Thanks & Regards,
Vinayak

On Mon, Mar 8, 2021 at 5:50 PM <hadoopmarc@...> wrote:
Hi Vinayak,

What may confuse you is that the gremlin console does a lot of under the hood imports, but it doesn't import all janusgraph classes. So, you can solve this in two ways:
  1. Preferred: rather use the underlying interface for specifiying the type, so line 3 private Graph graph;
  2. Optional: do an explicit import of StandardJanusGraph
gremlin> StandardJanusGraph
No such property: StandardJanusGraph for class: groovysh_evaluate
Type ':help' or ':h' for help.
Display stack trace? [yN]n
gremlin> Graph
==>interface org.apache.tinkerpop.gremlin.structure.Graph


Best wishes,    Marc

Re: Driver's for java connectivity

Vinayak Bali
 

Hi,

The goal is to execute a query which retrieves all the data from database and store it to a file to analyse the results 

Thanks 
Vinayak 

On Mon, 8 Mar 2021, 8:49 pm Kevin Schmidt, <ktschmidt@...> wrote:
Vinayak,

If your goal is simply to get your entire graph into a JSON-based format, you may want to consider GraphSON (http://tinkerpop.apache.org/docs/3.4.1/dev/io/#graphson).

Kevin

On Mon, Mar 8, 2021 at 7:13 AM Vinayak Bali <vinayakbali16@...> wrote:
Hi Kevin,

To verify whether the issue is from the server or client-side, we are trying to execute the same query on the server and output the data in the file. But still it's not possible. Janusgraph is using Cassandra Dameon, hence both are on the same jvm. We are testing for further use-cases which can contain more data than now.

Thanks & Regards,
Vinayak

On Mon, Mar 8, 2021 at 8:37 PM Kevin Schmidt <ktschmidt@...> wrote:
Vinayak,

Is the out of memory error in the server or the client?  And are Janusgraph and Cassandra running in the same JVM or are you running Cassandra separately?

And what is your use case for your client needing to retrieve your entire graph?

Kevin

On Mon, Mar 8, 2021 at 7:01 AM Vinayak Bali <vinayakbali16@...> wrote:
Hi Kevin,

JVM size at the server where janusgraph and Cassandra (same machine) is set to 24GB. On the API side, we tried setting it to 8GB. Is there any other way using which we can write the query result  to a file in json format ??

Thanks & Regards,
Vinayak

On Mon, Mar 8, 2021 at 8:20 PM Kevin Schmidt <ktschmidt@...> wrote:
Vinayak,

What about the memory settings, specifically the max heap, for the JVM Janusgraph is running in?

Your traversal is effectively retrieving your entire graph.  Is that really what you need to do?  Or is this just a test and your real scenarios wouldn't be doing that?  Doing a retrieval like this is going to require a lot of memory and your JVM simply isn't configured with enough to complete it, thus the out of memory error.

Kevin

On Mon, Mar 8, 2021 at 6:35 AM Vinayak Bali <vinayakbali16@...> wrote:
Hi Kevin,

Query: g.V().as('a').outE().as('b').inV().as('c').select('a','b','c').by(valueMap().by(unfold()))

Memory Settings:
maxInitialLineLength: 4096
maxHeaderSize: 8192
maxChunkSize: 8192
maxContentLength: 2000000000
maxAccumulationBufferComponents: 1024
resultIterationBatchSize: 64
writeBufferLowWaterMark: 32768
writeBufferHighWaterMark: 65536

Thanks & Regards,
Vinayak

On Mon, Mar 8, 2021 at 8:01 PM Kevin Schmidt <ktschmidt@...> wrote:
Vinayak,

What is the query?  It sounds like whatever it is requires instantiating a lot of objects in order to process the results and construct the JSON.

What are your memory settings for the Janusgraph server?

Kevin

On Mon, Mar 8, 2021 at 6:11 AM Vinayak Bali <vinayakbali16@...> wrote:
Hi Marc,

While trying to execute a query using HTTP throws an out memory error. The number of records returned is more hence can't get the entire output. I want an entire output of the query in the file JSON format. Size of graph nodes: 7 lakh and edges: 3 lakh.

Thanks & Regards,
Vinayak 

On Mon, Mar 8, 2021 at 5:52 PM <hadoopmarc@...> wrote:
Hi Vinayak,

What actually is the "desired operation", it is not clear to me? In other words, what is the result after you have run the desired operation?

Best wishes,    Marc

Re: Driver's for java connectivity

Kevin Schmidt
 

Vinayak,

If your goal is simply to get your entire graph into a JSON-based format, you may want to consider GraphSON (http://tinkerpop.apache.org/docs/3.4.1/dev/io/#graphson).

Kevin

On Mon, Mar 8, 2021 at 7:13 AM Vinayak Bali <vinayakbali16@...> wrote:
Hi Kevin,

To verify whether the issue is from the server or client-side, we are trying to execute the same query on the server and output the data in the file. But still it's not possible. Janusgraph is using Cassandra Dameon, hence both are on the same jvm. We are testing for further use-cases which can contain more data than now.

Thanks & Regards,
Vinayak

On Mon, Mar 8, 2021 at 8:37 PM Kevin Schmidt <ktschmidt@...> wrote:
Vinayak,

Is the out of memory error in the server or the client?  And are Janusgraph and Cassandra running in the same JVM or are you running Cassandra separately?

And what is your use case for your client needing to retrieve your entire graph?

Kevin

On Mon, Mar 8, 2021 at 7:01 AM Vinayak Bali <vinayakbali16@...> wrote:
Hi Kevin,

JVM size at the server where janusgraph and Cassandra (same machine) is set to 24GB. On the API side, we tried setting it to 8GB. Is there any other way using which we can write the query result  to a file in json format ??

Thanks & Regards,
Vinayak

On Mon, Mar 8, 2021 at 8:20 PM Kevin Schmidt <ktschmidt@...> wrote:
Vinayak,

What about the memory settings, specifically the max heap, for the JVM Janusgraph is running in?

Your traversal is effectively retrieving your entire graph.  Is that really what you need to do?  Or is this just a test and your real scenarios wouldn't be doing that?  Doing a retrieval like this is going to require a lot of memory and your JVM simply isn't configured with enough to complete it, thus the out of memory error.

Kevin

On Mon, Mar 8, 2021 at 6:35 AM Vinayak Bali <vinayakbali16@...> wrote:
Hi Kevin,

Query: g.V().as('a').outE().as('b').inV().as('c').select('a','b','c').by(valueMap().by(unfold()))

Memory Settings:
maxInitialLineLength: 4096
maxHeaderSize: 8192
maxChunkSize: 8192
maxContentLength: 2000000000
maxAccumulationBufferComponents: 1024
resultIterationBatchSize: 64
writeBufferLowWaterMark: 32768
writeBufferHighWaterMark: 65536

Thanks & Regards,
Vinayak

On Mon, Mar 8, 2021 at 8:01 PM Kevin Schmidt <ktschmidt@...> wrote:
Vinayak,

What is the query?  It sounds like whatever it is requires instantiating a lot of objects in order to process the results and construct the JSON.

What are your memory settings for the Janusgraph server?

Kevin

On Mon, Mar 8, 2021 at 6:11 AM Vinayak Bali <vinayakbali16@...> wrote:
Hi Marc,

While trying to execute a query using HTTP throws an out memory error. The number of records returned is more hence can't get the entire output. I want an entire output of the query in the file JSON format. Size of graph nodes: 7 lakh and edges: 3 lakh.

Thanks & Regards,
Vinayak 

On Mon, Mar 8, 2021 at 5:52 PM <hadoopmarc@...> wrote:
Hi Vinayak,

What actually is the "desired operation", it is not clear to me? In other words, what is the result after you have run the desired operation?

Best wishes,    Marc

Re: Driver's for java connectivity

Vinayak Bali
 

Hi Kevin,

To verify whether the issue is from the server or client-side, we are trying to execute the same query on the server and output the data in the file. But still it's not possible. Janusgraph is using Cassandra Dameon, hence both are on the same jvm. We are testing for further use-cases which can contain more data than now.

Thanks & Regards,
Vinayak

On Mon, Mar 8, 2021 at 8:37 PM Kevin Schmidt <ktschmidt@...> wrote:
Vinayak,

Is the out of memory error in the server or the client?  And are Janusgraph and Cassandra running in the same JVM or are you running Cassandra separately?

And what is your use case for your client needing to retrieve your entire graph?

Kevin

On Mon, Mar 8, 2021 at 7:01 AM Vinayak Bali <vinayakbali16@...> wrote:
Hi Kevin,

JVM size at the server where janusgraph and Cassandra (same machine) is set to 24GB. On the API side, we tried setting it to 8GB. Is there any other way using which we can write the query result  to a file in json format ??

Thanks & Regards,
Vinayak

On Mon, Mar 8, 2021 at 8:20 PM Kevin Schmidt <ktschmidt@...> wrote:
Vinayak,

What about the memory settings, specifically the max heap, for the JVM Janusgraph is running in?

Your traversal is effectively retrieving your entire graph.  Is that really what you need to do?  Or is this just a test and your real scenarios wouldn't be doing that?  Doing a retrieval like this is going to require a lot of memory and your JVM simply isn't configured with enough to complete it, thus the out of memory error.

Kevin

On Mon, Mar 8, 2021 at 6:35 AM Vinayak Bali <vinayakbali16@...> wrote:
Hi Kevin,

Query: g.V().as('a').outE().as('b').inV().as('c').select('a','b','c').by(valueMap().by(unfold()))

Memory Settings:
maxInitialLineLength: 4096
maxHeaderSize: 8192
maxChunkSize: 8192
maxContentLength: 2000000000
maxAccumulationBufferComponents: 1024
resultIterationBatchSize: 64
writeBufferLowWaterMark: 32768
writeBufferHighWaterMark: 65536

Thanks & Regards,
Vinayak

On Mon, Mar 8, 2021 at 8:01 PM Kevin Schmidt <ktschmidt@...> wrote:
Vinayak,

What is the query?  It sounds like whatever it is requires instantiating a lot of objects in order to process the results and construct the JSON.

What are your memory settings for the Janusgraph server?

Kevin

On Mon, Mar 8, 2021 at 6:11 AM Vinayak Bali <vinayakbali16@...> wrote:
Hi Marc,

While trying to execute a query using HTTP throws an out memory error. The number of records returned is more hence can't get the entire output. I want an entire output of the query in the file JSON format. Size of graph nodes: 7 lakh and edges: 3 lakh.

Thanks & Regards,
Vinayak 

On Mon, Mar 8, 2021 at 5:52 PM <hadoopmarc@...> wrote:
Hi Vinayak,

What actually is the "desired operation", it is not clear to me? In other words, what is the result after you have run the desired operation?

Best wishes,    Marc

Re: Driver's for java connectivity

Kevin Schmidt
 

Vinayak,

Is the out of memory error in the server or the client?  And are Janusgraph and Cassandra running in the same JVM or are you running Cassandra separately?

And what is your use case for your client needing to retrieve your entire graph?

Kevin

On Mon, Mar 8, 2021 at 7:01 AM Vinayak Bali <vinayakbali16@...> wrote:
Hi Kevin,

JVM size at the server where janusgraph and Cassandra (same machine) is set to 24GB. On the API side, we tried setting it to 8GB. Is there any other way using which we can write the query result  to a file in json format ??

Thanks & Regards,
Vinayak

On Mon, Mar 8, 2021 at 8:20 PM Kevin Schmidt <ktschmidt@...> wrote:
Vinayak,

What about the memory settings, specifically the max heap, for the JVM Janusgraph is running in?

Your traversal is effectively retrieving your entire graph.  Is that really what you need to do?  Or is this just a test and your real scenarios wouldn't be doing that?  Doing a retrieval like this is going to require a lot of memory and your JVM simply isn't configured with enough to complete it, thus the out of memory error.

Kevin

On Mon, Mar 8, 2021 at 6:35 AM Vinayak Bali <vinayakbali16@...> wrote:
Hi Kevin,

Query: g.V().as('a').outE().as('b').inV().as('c').select('a','b','c').by(valueMap().by(unfold()))

Memory Settings:
maxInitialLineLength: 4096
maxHeaderSize: 8192
maxChunkSize: 8192
maxContentLength: 2000000000
maxAccumulationBufferComponents: 1024
resultIterationBatchSize: 64
writeBufferLowWaterMark: 32768
writeBufferHighWaterMark: 65536

Thanks & Regards,
Vinayak

On Mon, Mar 8, 2021 at 8:01 PM Kevin Schmidt <ktschmidt@...> wrote:
Vinayak,

What is the query?  It sounds like whatever it is requires instantiating a lot of objects in order to process the results and construct the JSON.

What are your memory settings for the Janusgraph server?

Kevin

On Mon, Mar 8, 2021 at 6:11 AM Vinayak Bali <vinayakbali16@...> wrote:
Hi Marc,

While trying to execute a query using HTTP throws an out memory error. The number of records returned is more hence can't get the entire output. I want an entire output of the query in the file JSON format. Size of graph nodes: 7 lakh and edges: 3 lakh.

Thanks & Regards,
Vinayak 

On Mon, Mar 8, 2021 at 5:52 PM <hadoopmarc@...> wrote:
Hi Vinayak,

What actually is the "desired operation", it is not clear to me? In other words, what is the result after you have run the desired operation?

Best wishes,    Marc

Re: Driver's for java connectivity

Vinayak Bali
 

Hi Kevin,

JVM size at the server where janusgraph and Cassandra (same machine) is set to 24GB. On the API side, we tried setting it to 8GB. Is there any other way using which we can write the query result  to a file in json format ??

Thanks & Regards,
Vinayak

On Mon, Mar 8, 2021 at 8:20 PM Kevin Schmidt <ktschmidt@...> wrote:
Vinayak,

What about the memory settings, specifically the max heap, for the JVM Janusgraph is running in?

Your traversal is effectively retrieving your entire graph.  Is that really what you need to do?  Or is this just a test and your real scenarios wouldn't be doing that?  Doing a retrieval like this is going to require a lot of memory and your JVM simply isn't configured with enough to complete it, thus the out of memory error.

Kevin

On Mon, Mar 8, 2021 at 6:35 AM Vinayak Bali <vinayakbali16@...> wrote:
Hi Kevin,

Query: g.V().as('a').outE().as('b').inV().as('c').select('a','b','c').by(valueMap().by(unfold()))

Memory Settings:
maxInitialLineLength: 4096
maxHeaderSize: 8192
maxChunkSize: 8192
maxContentLength: 2000000000
maxAccumulationBufferComponents: 1024
resultIterationBatchSize: 64
writeBufferLowWaterMark: 32768
writeBufferHighWaterMark: 65536

Thanks & Regards,
Vinayak

On Mon, Mar 8, 2021 at 8:01 PM Kevin Schmidt <ktschmidt@...> wrote:
Vinayak,

What is the query?  It sounds like whatever it is requires instantiating a lot of objects in order to process the results and construct the JSON.

What are your memory settings for the Janusgraph server?

Kevin

On Mon, Mar 8, 2021 at 6:11 AM Vinayak Bali <vinayakbali16@...> wrote:
Hi Marc,

While trying to execute a query using HTTP throws an out memory error. The number of records returned is more hence can't get the entire output. I want an entire output of the query in the file JSON format. Size of graph nodes: 7 lakh and edges: 3 lakh.

Thanks & Regards,
Vinayak 

On Mon, Mar 8, 2021 at 5:52 PM <hadoopmarc@...> wrote:
Hi Vinayak,

What actually is the "desired operation", it is not clear to me? In other words, what is the result after you have run the desired operation?

Best wishes,    Marc

Re: Driver's for java connectivity

Kevin Schmidt
 

Vinayak,

What about the memory settings, specifically the max heap, for the JVM Janusgraph is running in?

Your traversal is effectively retrieving your entire graph.  Is that really what you need to do?  Or is this just a test and your real scenarios wouldn't be doing that?  Doing a retrieval like this is going to require a lot of memory and your JVM simply isn't configured with enough to complete it, thus the out of memory error.

Kevin

On Mon, Mar 8, 2021 at 6:35 AM Vinayak Bali <vinayakbali16@...> wrote:
Hi Kevin,

Query: g.V().as('a').outE().as('b').inV().as('c').select('a','b','c').by(valueMap().by(unfold()))

Memory Settings:
maxInitialLineLength: 4096
maxHeaderSize: 8192
maxChunkSize: 8192
maxContentLength: 2000000000
maxAccumulationBufferComponents: 1024
resultIterationBatchSize: 64
writeBufferLowWaterMark: 32768
writeBufferHighWaterMark: 65536

Thanks & Regards,
Vinayak

On Mon, Mar 8, 2021 at 8:01 PM Kevin Schmidt <ktschmidt@...> wrote:
Vinayak,

What is the query?  It sounds like whatever it is requires instantiating a lot of objects in order to process the results and construct the JSON.

What are your memory settings for the Janusgraph server?

Kevin

On Mon, Mar 8, 2021 at 6:11 AM Vinayak Bali <vinayakbali16@...> wrote:
Hi Marc,

While trying to execute a query using HTTP throws an out memory error. The number of records returned is more hence can't get the entire output. I want an entire output of the query in the file JSON format. Size of graph nodes: 7 lakh and edges: 3 lakh.

Thanks & Regards,
Vinayak 

On Mon, Mar 8, 2021 at 5:52 PM <hadoopmarc@...> wrote:
Hi Vinayak,

What actually is the "desired operation", it is not clear to me? In other words, what is the result after you have run the desired operation?

Best wishes,    Marc

Re: Driver's for java connectivity

Vinayak Bali
 

Hi Kevin,

Query: g.V().as('a').outE().as('b').inV().as('c').select('a','b','c').by(valueMap().by(unfold()))

Memory Settings:
maxInitialLineLength: 4096
maxHeaderSize: 8192
maxChunkSize: 8192
maxContentLength: 2000000000
maxAccumulationBufferComponents: 1024
resultIterationBatchSize: 64
writeBufferLowWaterMark: 32768
writeBufferHighWaterMark: 65536

Thanks & Regards,
Vinayak

On Mon, Mar 8, 2021 at 8:01 PM Kevin Schmidt <ktschmidt@...> wrote:
Vinayak,

What is the query?  It sounds like whatever it is requires instantiating a lot of objects in order to process the results and construct the JSON.

What are your memory settings for the Janusgraph server?

Kevin

On Mon, Mar 8, 2021 at 6:11 AM Vinayak Bali <vinayakbali16@...> wrote:
Hi Marc,

While trying to execute a query using HTTP throws an out memory error. The number of records returned is more hence can't get the entire output. I want an entire output of the query in the file JSON format. Size of graph nodes: 7 lakh and edges: 3 lakh.

Thanks & Regards,
Vinayak 

On Mon, Mar 8, 2021 at 5:52 PM <hadoopmarc@...> wrote:
Hi Vinayak,

What actually is the "desired operation", it is not clear to me? In other words, what is the result after you have run the desired operation?

Best wishes,    Marc

Re: Driver's for java connectivity

Kevin Schmidt
 

Vinayak,

What is the query?  It sounds like whatever it is requires instantiating a lot of objects in order to process the results and construct the JSON.

What are your memory settings for the Janusgraph server?

Kevin

On Mon, Mar 8, 2021 at 6:11 AM Vinayak Bali <vinayakbali16@...> wrote:
Hi Marc,

While trying to execute a query using HTTP throws an out memory error. The number of records returned is more hence can't get the entire output. I want an entire output of the query in the file JSON format. Size of graph nodes: 7 lakh and edges: 3 lakh.

Thanks & Regards,
Vinayak 

On Mon, Mar 8, 2021 at 5:52 PM <hadoopmarc@...> wrote:
Hi Vinayak,

What actually is the "desired operation", it is not clear to me? In other words, what is the result after you have run the desired operation?

Best wishes,    Marc

Re: Driver's for java connectivity

Vinayak Bali
 

Hi Marc,

While trying to execute a query using HTTP throws an out memory error. The number of records returned is more hence can't get the entire output. I want an entire output of the query in the file JSON format. Size of graph nodes: 7 lakh and edges: 3 lakh.

Thanks & Regards,
Vinayak 

On Mon, Mar 8, 2021 at 5:52 PM <hadoopmarc@...> wrote:
Hi Vinayak,

What actually is the "desired operation", it is not clear to me? In other words, what is the result after you have run the desired operation?

Best wishes,    Marc

Re: How to circumvent transaction cache?

timon.schneider@...
 

Currently using HBase.

Consider the following:
User A decides to set isPublished of Vertex X from false to true, does not commit yet.
User B changes isPublished of Vertex X from false to true and commits immediately.
User A commits and will get an error because the property value is not the same anymore as at the start of the transaction.

Why wouldn't it be possible for JG to provide user A with select for update functionality that allows user A to select vertex X for update, do edits to other elements, commit and get the same message as in the example above if the property on vertex X is changed?
JMX authentication for cassandra

Vinayak Bali
 

Hi All,
There is a requirement to enable the JMX authentication of Cassandra. The steps followed are as follows:

1. cd /home/user1/janusgraph-full-0.5.2/conf/cassandra
2. vi cassandra-env.sh
3. original : 
LOCAL_JMX=yes
if [ "$LOCAL_JMX" = "yes" ]; then
  JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT -XX:+DisableExplicitGC"
else
  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"
#  JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStore=/path/to/keystore"
#  JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStorePassword=<keystore-password>"
#  JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStore=/path/to/truststore"
#  JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStorePassword=<truststore-password>"
#  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.need.client.auth=true"
#  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.registry.ssl=true"
#  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.protocols=<enabled-protocols>"
#  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.cipher.suites=<enabled-cipher-suites>"
fi

Updated:
LOCAL_JMX=yes
if [ "$LOCAL_JMX" = "no" ]; then
  JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT -XX:+DisableExplicitGC"
else
  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"
#  JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStore=/path/to/keystore"
#  JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStorePassword=<keystore-password>"
#  JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStore=/path/to/truststore"
#  JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStorePassword=<truststore-password>"
#  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.need.client.auth=true"
#  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.registry.ssl=true"
#  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.protocols=<enabled-protocols>"
#  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.cipher.suites=<enabled-cipher-suites>"
fi

When I try to restart the service using command getting an error.
/home/user1/janusgraph-full-0.5.2/bin/janusgraph start

Forking Cassandra...
Running `nodetool statusthrift`........................ timeout exceeded (60 seconds)
See /home/fusionops/janusgraph-full-0.5.2/bin/../logs/cassandra.log for Cassandra log output.

bin/nodetool status

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/fusionops/janusgraph-full-0.5.2/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/fusionops/janusgraph-full-0.5.2/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
error: Authentication failed! Credentials required
-- StackTrace --
java.lang.SecurityException: Authentication failed! Credentials required
at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticationFailure(JMXPluggableAuthenticator.java:211)
at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticate(JMXPluggableAuthenticator.java:163)
at org.apache.cassandra.utils.JMXServerUtils$JMXPluggableAuthenticatorWrapper.authenticate(JMXServerUtils.java:279)
at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:232)
at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
at javax.management.remote.rmi.RMIJRMPServerImpl$ExportedWrapper.newClient(RMIJRMPServerImpl.java:253)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:164)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:235)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:180)
at com.sun.proxy.$Proxy7.newClient(Unknown Source)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:183)
at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:150)
at org.apache.cassandra.tools.NodeTool$NodeToolCmd.connect(NodeTool.java:302)
at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:242)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158)

Are there any other changes required ??

Thanks & Regards,
Vinayak
Re: How to circumvent transaction cache?

Boxuan Li
 

Hi Timon, what exactly is your data storage setup? For example, if you are using Cassandra (with replication) then there is no guarantee your current transaction can read latest value right after another transaction commits (even if your transaction does not cache anything).

JanusGraph is a layer built on top of your storage backend. Unfortunately, it cannot provide any guarantee that is not provided by the underlying storage backend at the first place. If you are concerned about ACID maybe you should use BerkeleyDB or FoundationDB.
 

On Mar 8, 2021, at 7:44 PM, timon.schneider@... wrote:

Hi all,

On Fri, Mar 5, 2021 at 05:32 PM, Ted Wilmes wrote:
Since you do not have ACID guarantees in most cases, I think you could still have a case where another transaction commits prior to your commit even though you saw isPublished = false when you check it. One possible way around this without ACID would be to process all mutations for a branch on one thread, effectively single threading access to it so that you could know that no other user was writing to the branch while you were reading.
I actually aim to keep the system ACID compliant. The only thing is (which I struggle to implement in JG) is that the edits only can be committed after a lock and read is done on the branch vertex' isPublished property. The problem is JG doesn't offer select for update functionality. I need to read the branch vertex to get the Id and lock it but while I'm getting it the isPublished property can be set to true by another user. Getting the vertex, locking it, and refreshing the data could be an option however it's not supported by JG.

Isn't this a shortcoming of JG that many users have issues with?

I think the single thread solution you suggest is not an option as our application is a meta data editor where multiple users should be able to edit elements of a branch simultaneously.

@Bo Xuan Li
I'm very much concerned with the consistency of the data. The check on the branch vertex is just a read operation necessary to guarantee that the branch is not published at the point of persisting the edits.

Re: Driver's for java connectivity

hadoopmarc@...
 

Hi Vinayak,

What actually is the "desired operation", it is not clear to me? In other words, what is the result after you have run the desired operation?

Best wishes,    Marc
Re: Script16.groovy: 2: unable to resolve class StandardJanusGraph

hadoopmarc@...
 

Hi Vinayak,

What may confuse you is that the gremlin console does a lot of under the hood imports, but it doesn't import all janusgraph classes. So, you can solve this in two ways:
  1. Preferred: rather use the underlying interface for specifiying the type, so line 3 private Graph graph;
  2. Optional: do an explicit import of StandardJanusGraph
gremlin> StandardJanusGraph
No such property: StandardJanusGraph for class: groovysh_evaluate
Type ':help' or ':h' for help.
Display stack trace? [yN]n
gremlin> Graph
==>interface org.apache.tinkerpop.gremlin.structure.Graph


Best wishes,    Marc
Re: How to circumvent transaction cache?

timon.schneider@...
 

Hi all,


On Fri, Mar 5, 2021 at 05:32 PM, Ted Wilmes wrote:
Since you do not have ACID guarantees in most cases, I think you could still have a case where another transaction commits prior to your commit even though you saw isPublished = false when you check it. One possible way around this without ACID would be to process all mutations for a branch on one thread, effectively single threading access to it so that you could know that no other user was writing to the branch while you were reading.
I actually aim to keep the system ACID compliant. The only thing is (which I struggle to implement in JG) is that the edits only can be committed after a lock and read is done on the branch vertex' isPublished property. The problem is JG doesn't offer select for update functionality. I need to read the branch vertex to get the Id and lock it but while I'm getting it the isPublished property can be set to true by another user. Getting the vertex, locking it, and refreshing the data could be an option however it's not supported by JG.

Isn't this a shortcoming of JG that many users have issues with?

I think the single thread solution you suggest is not an option as our application is a meta data editor where multiple users should be able to edit elements of a branch simultaneously.

@Bo Xuan Li
I'm very much concerned with the consistency of the data. The check on the branch vertex is just a read operation necessary to guarantee that the branch is not published at the point of persisting the edits.
981 - 1000 of 6678