Date   

Re: Need info regarding transaction recovery completion

Boxuan Li
 

Hi Radhika,

Unfortunately, there is no such API. If you are willing to dive into JanusGraph source code, you can modify StandardTransactionLogProcessor::fixSecondaryFailure method and build JanusGraph by yourself.

You are also welcome to create a feature request on GitHub issues. Probably we should allow users to register a callback method when recovery is done.

Best,
Boxuan



Need info regarding transaction recovery completion

Radhika Kundam
 
Edited

Hi Team,

I am using JanusGraphFactory.startTransactionRecovery to recover secondary failure entries.
We need to perform some other action once recovery is completed, I couldn't find any API to know the status of recovery.
Is there any way to know if the recovery of all the failure entries is completed.

I would appreciate you help.

Thanks,
Radhika


org.janusgraph.diskstorage.PermanentBackendException: Read 1 locks with our rid but mismatched timestamps

Ronnie
 

Hi,

Environment
- JanusGraph 0.5.3 on JDK: 1.8
- Backend: Cassandra 3.11.3 running on JDK 1.8

Warning and error during first time server startup
2021-08-17T22:26:20,861 - WARN  [main:o.j.d.l.c.ConsistentKeyLocker@510] - Skipping outdated lock on KeyColumn [k=0x 16-165-160-103-105- 30- 71-114- 97-112-104- 95- 78- 97-109-101- 95- 73-110-100-101-248, c=0x  0] with our rid ( 48- 97- 55- 50- 97- 97- 57- 57- 49- 56- 57- 56- 57- 45-115-104- 97-114-101-100-106- 97-110-117-115-103-114- 97-112-104- 48- 49- 45-112- 50- 55- 45-101-110-103- 45-105-110- 48- 51- 45-113-117- 97-108-121-115- 45- 99-111-109- 49) but mismatched timestamp (actual ts 2021-08-17T22:26:20.755981Z, expected ts 2021-08-17T22:26:20.755981926Z)
2021-08-17T22:26:20,863 - ERROR [main:o.j.g.d.StandardJanusGraph@724] - Could not commit transaction [1] due to storage exception in system-commit
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Read 1 locks with our rid  48- 97- 55- 50- 97- 97- 57- 57- 49- 56- 57- 56- 57- 45-115-104- 97-114-101-100-106- 97-110-117-115-103-114- 97-112-104- 48- 49- 45-112- 50- 55- 45-101-110-103- 45-105-110- 48- 51- 45-113-117- 97-108-121-115- 45- 99-111-109- 49 but mismatched timestamps; no lock column contained our timestamp (2021-08-17T22:26:20.755981926Z)
at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSeniority(ConsistentKeyLocker.java:542)
at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:468)
at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:118)
at org.janusgraph.diskstorage.locking.AbstractLocker.checkLocks(AbstractLocker.java:351)
... 27 more
2021-08-17T22:26:20,864 - ERROR [main:o.a.t.g.s.u.ServerGremlinExecutor@87] - Could not invoke constructor on class org.janusgraph.graphdb.management.JanusGraphManager (defined by the 'graphManager' setting) with one argument of class Settings
Graph configuration:
gremlin.graph=org.janusgraph.core.ConfiguredGraphFactory
graph.graphname=ConfigurationManagementGraph
graph.timestamps=MICRO
storage.backend=cql
storage.hostname=10.114.171.91,10.114.171.92,10.114.171.93
storage.cql.keyspace= sharedjanusgraph
storage.read-time=50000
cache.db-cache = true
cache.db-cache-clean-wait = 20
cache.db-cache-time = 180000
cache.db-cache-size = 0.4
tx.log-tx=true
tx.max-commit-time=15000
metrics.enabled=False
metrics.jmx.enabled=False
cluster.max-partitions=32
Note: Explicitly set graph.timestamps=MICRO ; when setting this to NANO as suggested here https://stackoverflow.com/questions/58916854/janusgraph-janusgraphexception-could-not-commit-transaction-due-to-exception-dur i get the following error:
java.lang.IllegalArgumentException: Timestamp overflow detected: 2021-08-17T23:20:11.611614212Z
at org.janusgraph.diskstorage.log.kcvs.KCVSLog.getTimeSlice(KCVSLog.java:330)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog.add(KCVSLog.java:418)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog.add(KCVSLog.java:394)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog.add(KCVSLog.java:377)
at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:690)
at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1438)
... 14 more
Any pointers why this time resolution mismatch is happening?

Thanks,
Ronnie


Re: Janusgraph 0.6.0

Oleksandr Porunov
 

Thanks. I opened the PR here: https://github.com/JanusGraph/janusgraph/pull/2760


Re: Janusgraph 0.6.0

toom@...
 

I confirm, adding lucene-backward-codecs-8.9.0.jar in lib folder solves my problem.

Toom.


Re: Janusgraph 0.6.0

Oleksandr Porunov
 

Hi, we upgraded Lucene to 8.9.0 version. Do you think the problem will be resolved if we include `lucene-backward-codecs.jar` in the classpath?


Janusgraph 0.6.0

toom@...
 

Hi,

I'm testing the new pre-release of Janusgraph (from https://github.com/JanusGraph/janusgraph/releases/download/v0.6.0/janusgraph-0.6.0.zip) with the Berkeley/Lucene database created from JG 0.5.3 and all calls of index fail (below the full stacktrace).
Is there a migration process, maybe a reindex ? The error message contains "Could not load codec 'Lucene70'.  Did you forget to add lucene-backward-codecs.jar?". Do you plan to include this jar in your JanusGraph distribution ?

Best regards,

Toom.

-- 
org.janusgraph.core.JanusGraphException: Could not call index
        at org.janusgraph.graphdb.util.SubqueryIterator.<init>(SubqueryIterator.java:67)
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1430)
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1322)
        at org.janusgraph.graphdb.query.QueryProcessor$LimitAdjustingIterator.getNewIterator(QueryProcessor.java:206)
        at org.janusgraph.graphdb.query.LimitAdjustingIterator.hasNext(LimitAdjustingIterator.java:69)
        at org.janusgraph.graphdb.query.ResultSetIterator.nextInternal(ResultSetIterator.java:55)
        at org.janusgraph.graphdb.query.ResultSetIterator.<init>(ResultSetIterator.java:45)
        at org.janusgraph.graphdb.query.QueryProcessor.iterator(QueryProcessor.java:68)
        at org.janusgraph.graphdb.query.graph.GraphCentricQueryBuilder.lambda$iterables$1(GraphCentricQueryBuilder.java:239)
        at org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphStep.lambda$executeGraphCentricQuery$2(JanusGraphStep.java:202)
        at org.janusgraph.graphdb.util.ProfiledIterator.<init>(ProfiledIterator.java:36)
        at org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphStep.executeGraphCentricQuery(JanusGraphStep.java:202)
        at org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphStep.lambda$null$0(JanusGraphStep.java:105)
        at java.lang.Iterable.forEach(Iterable.java:75)
        at org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphStep.lambda$new$1(JanusGraphStep.java:105)
        at org.apache.tinkerpop.gremlin.process.traversal.step.map.GraphStep.processNextStart(GraphStep.java:157)
        at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:150)
        at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:222)
        at java_util_Iterator$hasNext.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:119)
        at org.apache.tinkerpop.gremlin.console.Console$_closure3.doCall(Console.groovy:257)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:37)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:52)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
        at org.codehaus.groovy.tools.shell.Groovysh.setLastResult(Groovysh.groovy:463)
        at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:63)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:168)
        at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:201)
        at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
        at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1217)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:144)
        at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:83)
        at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:120)
        at org.codehaus.groovy.tools.shell.Shell$leftShift$2.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
        at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:93)
        at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1217)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:144)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:164)
        at org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:138)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:156)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:160)
        at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:57)
        at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1217)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:144)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:164)
        at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:97)
        at java_lang_Runnable$run.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:119)
        at org.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:170)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:80)
        at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:59)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:237)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:265)
        at org.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:524)
Caused by: org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
        at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54)
        at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:488)
        at org.janusgraph.diskstorage.BackendTransaction.indexQuery(BackendTransaction.java:416)
        at org.janusgraph.graphdb.database.IndexSerializer.query(IndexSerializer.java:596)
        at org.janusgraph.graphdb.util.SubqueryIterator.<init>(SubqueryIterator.java:65)
        ... 108 more
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Permanent exception while executing backend operation IndexQuery
        at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:79)
        at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:52)
        ... 112 more
Caused by: java.lang.IllegalArgumentException: Could not load codec 'Lucene70'.  Did you forget to add lucene-backward-codecs.jar?
        at org.apache.lucene.index.SegmentInfos.readCodec(SegmentInfos.java:449)
        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:356)
        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291)
        at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:64)
        at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:61)
        at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:720)
        at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:84)
        at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:64)
        at org.janusgraph.diskstorage.lucene.LuceneIndex$Transaction.getSearcher(LuceneIndex.java:1105)
        at org.janusgraph.diskstorage.lucene.LuceneIndex$Transaction.access$000(LuceneIndex.java:1090)
        at org.janusgraph.diskstorage.lucene.LuceneIndex.query(LuceneIndex.java:596)
        at org.janusgraph.diskstorage.indexing.IndexTransaction.queryStream(IndexTransaction.java:110)
        at org.janusgraph.diskstorage.BackendTransaction$6.call(BackendTransaction.java:419)
        at org.janusgraph.diskstorage.BackendTransaction$6.call(BackendTransaction.java:416)
        at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:66)
        ... 113 more
        Suppressed: org.apache.lucene.index.CorruptIndexException: checksum passed (47229baa). possibly transient resource issue, or a Lucene or JVM bug (resource=BufferedChecksumIndexInput(MMapIndexInput(path=".../segments_w")))
                at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:466)
                at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:434)
                ... 126 more
Caused by: java.lang.IllegalArgumentException: An SPI class of type org.apache.lucene.codecs.Codec with name 'Lucene70' does not exist.  You need to add the corresponding JAR file supporting this SPI to your classpath.  The current classpath supports the following names: [Lucene87]
        at org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:116)
        at org.apache.lucene.codecs.Codec.forName(Codec.java:116)
        at org.apache.lucene.index.SegmentInfos.readCodec(SegmentInfos.java:445)
        ... 127 more
 


Re: What are the implications of using Object.class property type?

hadoopmarc@...
 

Hi Laura,

One code example says more than 1000 words:

gremlin> graph = TinkerFactory.createModern()
==>tinkergraph[vertices:6 edges:6]
gremlin> g=graph.traversal(
traversal(    traversal()   
gremlin> g=graph.traversal()
==>graphtraversalsource[tinkergraph[vertices:6 edges:6], standard]
gremlin> g.addV().property("lang", 45)
==>v[13]
gremlin> g.V().elementMap()
==>[id:1,label:person,name:marko,age:29]
==>[id:2,label:person,name:vadas,age:27]
==>[id:3,label:software,name:lop,lang:java]
==>[id:4,label:person,name:josh,age:32]
==>[id:5,label:software,name:ripple,lang:java]
==>[id:6,label:person,name:peter,age:35]
==>[id:13,label:vertex,lang:45]
gremlin> g.V().values("lang")
==>java
==>java
==>45
gremlin> g.V().values("lang").group().by(map{it->it.get().getClass()}).by(count())
==>[class java.lang.String:2,class java.lang.Integer:1]
gremlin>

So, this query shows you all occurring data types of a specific property in the graph.
Strictly speaking, gremlin OLAP queries are queries using withComputer(). I tend to use the term a bit looser including analytical queries requiring a full table scan.

Best wishes,

Marc


Re: What are the implications of using Object.class property type?

Laura Morales <lauretas@...>
 

your graph will be more difficult to use if you do not know what data type is in a property. Users would have to explore for themselves what object types are in the graph. This means a lot of OLAP queries with lots of waiting time and a large load on the graph system.
What would be an example of a OLAP query? There is a way to get a property's type in a gremlin query?


Re: What are the implications of using Object.class property type?

hadoopmarc@...
 

Hi Laura,

Some remarks:
  • primitive types can be stored more efficiently than general objects (an integer is exactly 32 bits, an object an be any size)
  • for the CompositeIndex the objects are fine as long as they implement the equals() method
  • your graph will be more difficult to use if you do not know what data type is in a property. Users would have to explore for themselves what object types are in the graph. This means a lot of OLAP queries with lots of waiting time and a large load on the graph system. If users only use their own data, you can give each user its own graph.
  • if you use Gremlin Server, the supported protocols will not know how to serialize arbitrary objects, so you have to instruct the client to request the string values of objects only. Then, you could have asked your users to enter object strings into janusgraph in the first place.

Best wishes,    Marc


Fw: Re: [janusgraph-users] What are the implications of using Object.class property type?

Laura Morales <lauretas@...>
 

ERRATA

The composite index does not work when searching by comparison, ie. .has('name', lt(50)) (I get the usual warning "Query requires iterating over all vertices" and it returns zero vertexes)
I get the warning but the vertex with name=42 *is* returned correctly


Re: What are the implications of using Object.class property type?

Laura Morales <lauretas@...>
 

I'd like to understand a little bit more about what's going on under the hood when creating a new property with .dataType(Object.class) vs any other specific type eg. .dataType(String.class) or .dataType(Integer.class)
I'm able to create a "name" property like this:

mgmt.makePropertyKey('name').dataType(Object.class).make()

and this is what I've noticed:

- it allows me to create these vertexes
g.addV('alice').property('name', 'Alice')
g.addV('terminator').property('name', 42)
- it allows me to create a composite index, but not a mixed index (confirming what was said in the other thread)
- the composite index works when searching for an exact match, ie. .has('name', 'Alice') and .has('name', 42). The composite index does not work when searching by comparison, ie. .has('name', lt(50)) (I get the usual warning "Query requires iterating over all vertices" and it returns zero vertexes)

I'm only interested into this because I have a graph where multiple people contribute, it would be very nice to not having to deal with explicit property types, if Object.class is an option. For my particular use case I could live without mixed indexes, and I wouldn't mind a small performance deficit (size and/or speed) introduced by the usage of Object as a general type. But I really struggle to understand what's going on. What's the difference between Object and specific types from Janus' point of view? Are types only useful for enforcing a particular schema when inserting data, or there's more to it?



Sent: Wednesday, July 21, 2021 at 10:34 AM
From: hadoopmarc@...
To: janusgraph-users@...
Subject: Re: [janusgraph-users] What are the implications of using Object.class property type?
Hi Laura,

A similar question was posed recently:
https://lists.lfaidata.foundation/g/janusgraph-users/message/5986[https://lists.lfaidata.foundation/g/janusgraph-users/message/5986?p=,,,20,0,0,0::recentpostdate%252Fsticky,,mixedindex,20,2,0,83929827]

So,
1. Only for the CompositeIndex
2. In your specific example, you could use the java Integer class ( https://docs.oracle.com/javase/8/docs/api/java/lang/Integer.html[https://docs.oracle.com/javase/8/docs/api/java/lang/Integer.html] ), because its constructor takes either integer type or string type and it has the equals() method implemented.

Best wishes,    Marc


Re: JanusGraph-GRPC

rngcntr
 

Hi!

As far as I know, your assumption is correct. However, Janusgraph-GRPC is still in the early stages of development and I guess it will still take a few versions before it will become available. That's also the reason, why the feature is still undocumented. If you have any specific questsions, I would recommend to reach out to Jan Jansen (@farodin91 on GitHub) because he drives the development behind Janusgraph-GRPC.

Best regards,
Florian


JanusGraph-GRPC

schwartz@...
 

Hello!
I've seen some references for JanusGraph-GRPC as something that might replace the ManagementSystem, but can't quite find any more documentation.
Can anyone shed some light?

Many thanks!


Re: Not able to enable Write-ahead logs using tx.log-tx for existing JanusGraph setup

Radhika Kundam
 

Boxuan,

Thank you for the response. Will try again as mentioned in the configuration.

Regards,
Radhika


On Fri, Aug 6, 2021 at 11:53 PM Boxuan Li <liboxuan@...> wrote:
Is it expected that tx.log-tx works only for fresh JanusGraph setup?

No. It should work well for your existing JanusGraph setup too. Note that it is a GLOBAL option so it must be changed for the entire cluster. See https://docs.janusgraph.org/basics/configuration/#global-configuration

Best,
Boxuan


Re: Data Loading Script Optimization

hadoopmarc@...
 

Hi Vinayak,

Good to see some progress!

Some suggestions:
  • Is 40% relative to a single core or to all cores (e.g. CPU usage for a java process in top can be 800% if 8 cores are present)?
  • Ncore * 100% is not necessarily the maximum CPU load of the groovy process + storage backend if the loading becomes IO limited. Can you find out what IO usage is?
  • Do you use CompositeIndices on the properties "name" and "e-mail" for the has() filters?
  • Regarding the idea from Nicolas, I would rather use a ConcurrentMap that maps ORG id's to vertex id's, but only fill it as you go for the ORG's that you add or lookup. The JanusGraph transaction and database caches should be large enough to hold the vertices to be referenced two or more times, thus accommodating g.V(id) lookups.
  • On a single system Apache Spark will not help you.

Best wishes,    Marc


Re: Data Loading Script Optimization

Nicolas Trangosi
 

Hi,
You could first create a local cache for ORG by retrying first all ORG:

Map<String, Long> orgCache = g.V().has('vertexLabel', 'ORG').project("name", "id").by("orgName").by(T.id)...

Then replace __.V().has('vertexLabel', 'ORG').has('orgName', orgName) by __V(orgCache.get(orgName))

Same trick, may be used for persons to remove the coalesce if you know that you import more users than already exist in db.

Nicolas


Le lun. 9 août 2021 à 10:07, Vinayak Bali <vinayakbali16@...> a écrit :
Hi Marc, 

To avoid confusion, including a new transaction at line number 39, as well as at line no 121.  
Line 39: GraphTraversalSource g = graph.newTransaction().traversal();
Line 121: g = ctx.g = graph.newTransaction().traversal();
The total time took was 11 mins. The maximum amount of cpu utilization was 40%. As the hardware configuration of the instance is at higher side, still we have enough RAM to increase the performance. 
Request you to share if it's possible to increase the performance further following some process(Hadoop/spark) etc based on your experience. 

Thanks & Regards,
Vinayak

On Sun, Aug 8, 2021 at 1:53 PM <hadoopmarc@...> wrote:
Hi Vinayak,

Yes, it should be possible to improve on the 3% CPU usage.

The newTransaction() should be added to line 39 (GraphTraversalSource g = graph.traversal();) as the global g from line 121 is not used.

Marc



--

  

Nicolas Trangosi

Lead back

+33 (0)6 77 86 66 44      

   




Ce message et ses pièces jointes peuvent contenir des informations confidentielles ou privilégiées et ne doivent donc pas être diffusés, exploités ou copiés sans autorisation. 
Si vous avez reçu ce message par erreur, veuillez le signaler a l'expéditeur et le détruire ainsi que les pièces jointes. 
Les messages électroniques étant susceptibles d'altération, DCbrain décline toute responsabilité si ce message a été altéré, déformé ou falsifié. Merci. 

This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, DCbrain is not liable for messages that have been modified, changed or falsified. Thank you.


Re: Data Loading Script Optimization

Vinayak Bali
 

Hi Marc, 

To avoid confusion, including a new transaction at line number 39, as well as at line no 121.  
Line 39: GraphTraversalSource g = graph.newTransaction().traversal();
Line 121: g = ctx.g = graph.newTransaction().traversal();
The total time took was 11 mins. The maximum amount of cpu utilization was 40%. As the hardware configuration of the instance is at higher side, still we have enough RAM to increase the performance. 
Request you to share if it's possible to increase the performance further following some process(Hadoop/spark) etc based on your experience. 

Thanks & Regards,
Vinayak

On Sun, Aug 8, 2021 at 1:53 PM <hadoopmarc@...> wrote:
Hi Vinayak,

Yes, it should be possible to improve on the 3% CPU usage.

The newTransaction() should be added to line 39 (GraphTraversalSource g = graph.traversal();) as the global g from line 121 is not used.

Marc


Re: Data Loading Script Optimization

hadoopmarc@...
 

Hi Vinayak,

Yes, it should be possible to improve on the 3% CPU usage.

The newTransaction() should be added to line 39 (GraphTraversalSource g = graph.traversal();) as the global g from line 121 is not used.

Marc


Re: Data Loading Script Optimization

Vinayak Bali
 

Hi Marc, 

The storage backend used is Cassandra. 
Yes, storage backend janusgraph and load scripts are on the same server.
specified storage.batch-loading=true 
CPU usage is very low not more than 3 percent. The machine has higher hardware configurations. So, I need suggestions on how we can make full use of the hardware.
I will use graph.newTransaction().traversal() replacing line 121 in the code and share the results. 
Current line: g = ctx.g = graph.traversal();
Modified : g = ctx.g = graph.newTransaction().traversal();
Please validate and confirm the changes. 
As data increases, we should use global GraphTraversalSource g at the bottom of the script for the bulk loading.

Thanks & Regards,
Vinayak


On Sat, Aug 7, 2021 at 6:21 PM <hadoopmarc@...> wrote:
Hi Vinayak,

What storage backend do you use? Do I understand right that the storage backend and the load script all run on the same server? If, so, are all available CPU resources actively used during batch loading? What is CPU usage of the groovy process and what of the storage backend?

Specific details in the script:
  • did you specify storage.batch-loading=true
  • I am not sure whether each traversal() call on the graph gets its own thread-independent transaction (that is why ask for the groovy CPU usage). Maybe you need g = graph.newTransaction().traversal() in CsvImporter
  • I assume that the global GraphTraversalSource g at the bottom of the script is not used for the bulk loading.
Best wishes,    Marc

561 - 580 of 6651