You already specified the vertex-centrex index on the amount key to be ordered while creating the index. By explicitly reordering the results in the traversal, the index cannot take effect because the reordering needs alls vertices to be retrieved instead of just the first 500.

HTH,    Marc

Op zaterdag 25 juli 2020 om 20:52:37 UTC+2 schreef 18...@...:

Hi JanusGraph team,

I have created a vertex-centric indexes for vertices. As follows, now I want to use the index to get the information of the top 500 edges in descending sort. However, I find that the execution time is the same as that without vertex index. How can I use the index to sort faster and extract the information of the first 500 edges more quickly?

Here's the graph I've built：

graph=JanusGraphFactory.open(‘janusgraph-cql-es-server-test2.properties’) mgmt = graph.openManagement() mgmt.makeVertexLabel('VirtualAddress').make() addr = mgmt.makePropertyKey('address').dataType(String.class).cardinality(SINGLE).make() token_addr = mgmt.makePropertyKey('token_addr').dataType(String.class).cardinality(SINGLE).make() transfer_to=mgmt.makeEdgeLabel('TRANSFER_TO').multiplicity(MULTI).make() amount = mgmt.makePropertyKey('amount').dataType(Double.class).cardinality(SINGLE).make() tx_hash = mgmt.makePropertyKey('tx_hash').dataType(String.class).cardinality(SINGLE).make() tx_index = mgmt.makePropertyKey('tx_index').dataType(Integer.class).cardinality(SINGLE).make() created_time = mgmt.makePropertyKey('created_time').dataType(Date.class).cardinality(SINGLE).make() updated_time = mgmt.makePropertyKey('updated_time').dataType(Date.class).cardinality(SINGLE).make() mgmt.buildIndex('addressComposite', Vertex.class).addKey(addr).buildCompositeIndex() mgmt.buildIndex('addressTokenUniqComposite', Vertex.class).addKey(addr).addKey(token_addr).unique().buildCompositeIndex() mgmt.buildEdgeIndex(transfer_to,"transferOutAmountTs", Direction.OUT, Order.desc,amount,created_time) mgmt.buildEdgeIndex(transfer_to,"transferOutTs", Direction.OUT, Order.desc,created_time) mgmt.commit()

Here's the data I inserted, building a starting point, a million edges associated with it, and 100 endpoints,

graph_conf = 'janusgraph-cql-es-server-test2.properties'

graph = JanusGraphFactory.open(graph_conf)

g = graph.traversal()

String line = "5244613,tx_hash_00,token_addr_00,from_addr_00,to_addr_00,,,6000,19305.57174337591,72,1520896044"

int start_value = 1

int end_value = 1000000

line = "tx_hash_00,token_addr_00,from_addr_00,to_addr_00,6000,72,1520896044"

cloumns = line.split(',', -1)

(tx_hash, token_addr, from_addr, to_addr, amount, log_index, timestamp) = cloumns

from_addr_node = g.addV('VirtualAddress').property('address', from_addr).property('token_addr', token_addr).next()

from_id = from_addr_node.id()

amount = amount.toBigDecimal()

tx_index = log_index.toInteger()

for (int i = start_value; i <= end_value; i++) {

    to_addr_node = g.addV('VirtualAddress').property('address', to_addr + String.valueOf(i)).property('token_addr', token_addr).next()

    to_id = to_addr_node.id()

    Date ts = new Date((timestamp.toLong() - i) * 1000)

    g.addE('TRANSFER_TO').from(g.V(from_id)).to(g.V(to_id))

            .property('amount', amount + i)

            .property('tx_hash', tx_hash)

            .property('tx_index', tx_index + i)

            .property('created_time', ts)

            .next()

    if (i % 20000 == 0) {

        println("[total:${i}]")

        System.sleep(500)

        g.tx().commit()

        graph.close()

        System.sleep(5000)

        graph = JanusGraphFactory.open(graph_conf)

        g = graph.traversal()

        System.sleep(5000)

    }

    g.tx().commit()

}

graph.close()

 

Here are my query criteria:

g.V().has('address', ‘from_addr_00').outE('TRANSFER_TO').order().by(‘amount’,desc).limit(500).valueMap().toList()

Also are there any other ways of creating users??

On Monday, July 20, 2020 at 3:29:54 PM UTC+5:30, sparshneel chanchlani wrote:

Hi, 

I am actually trying to add authentication to Janusgraph. I am actually referring the link below

https://docs.janusgraph.org/basics/server/#authentication-over-websocket

below is may credentials DB config:

gremlin.graph=org.janusgraph.core.JanusGraphFactory

storage.backend=cql

storage.hostname= 10.xx.xx.xx

storage.port= 9042

storage.username= cassandrauser

storage.password= cassandrapwd

storage.cql.keyspace=creds_db

storage.read-time=50000

storage.cql.write-consistency-level=LOCAL_QUORUM

storage.cql.read-consistency-level=LOCAL_QUORUM

cluster.max-partitions=32

storage.lock.wait-time=5000

storage.lock.retries=5

ids.block-size=100000

Actually when i start the gremlin-server the creds_db and the graphDB creates successfully. The issue, i am not able to create the credentials using Credentials(graph) groovy script, i am trying through gremlin-consle see below.

 g1 = JanusGraphFactory.open('conf/gremlin-server/janusgraph-server-credentials.properties')

==>standardjanusgraph[cql:[]]

gremlin> creds = Credentials(g1)

No signature of method: groovysh_evaluate.Credentials() is applicable for argument types: (org.janusgraph.graphdb.database.StandardJanusGraph) values: [standardjanusgraph[cql:[]

Actaully the groovysh_evaluate script does not support standard graph as parameters. What should be my credentials.properties for cassandra??

Thanks,

Sparshneel

Hello,

I've got the same issue with the latest version of JanusGraph and Atlas from master branch. Did you manage somehow appropriate type/serializer registration to produce GraphSON output? I'd like to visualise graph via Cytoscape or Graphexp. Thanks for any advice!

I've tried already - gremlin config (using Scylla and ES):

attributes.custom.attribute1.attribute-class=org.apache.atlas.typesystem.types.DataTypes.TypeCategory

attributes.custom.attribute1.serializer-class=org.apache.atlas.repository.graphdb.janus.serializer.TypeCategorySerializer

attributes.custom.attribute2.attribute-class=java.util.ArrayList

attributes.custom.attribute2.serializer-class=org.janusgraph.graphdb.database.serialize.attribute.SerializableSerializer

attributes.custom.attribute3.attribute-class=java.math.BigInteger

attributes.custom.attribute3.serializer-class=org.apache.atlas.repository.graphdb.janus.serializer.BigIntegerSerializer

attributes.custom.attribute4.attribute-class=java.math.BigDecimal

attributes.custom.attribute4.serializer-class=org.apache.atlas.repository.graphdb.janus.serializer.BigDecimalSerializer

then from gremlin cli:

graph.io(IoCore.graphson()).writeGraph("/atlas.json")

resulting into:

org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException: Could not find a type identifier for the class : class org.apache.atlas.typesystem.types.DataTypes$TypeCategory. Make sure the value to serialize has a type identifier registered for its class.

Dne středa 15. ledna 2020 v 14:31:09 UTC+1 uživatel mar...@... napsal:

Hi,

See a similar question on:

https://gitter.im/janusgraph/janusgraph?at=5c7d868a212d0c1e1ad19eef

HTH,   Marc

Op woensdag 15 januari 2020 14:11:14 UTC+1 schreef qi...@...:

hello i am new in JanusGraph. When i use Janusgraph in Apache Atlas, i found a question, the error is :

Could not find a type identifier for the class : class org.apache.atlas.typesystem.types.DataTypes$TypeCategory. Make sure the value to serialize has a type identifier registered for its class.
org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException: Could not find a type identifier for the class : class org.apache.atlas.typesystem.types.DataTypes$TypeCategory. Make sure the value to serialize has a type identifier registered for its class.
at org.apache.tinkerpop.shaded.jackson.databind.ser.DefaultSerializerProvider._wrapAsIOE(DefaultSerializerProvider.java:509)
at org.apache.tinkerpop.shaded.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:482)
at org.apache.tinkerpop.shaded.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3893)
at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:3164)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONWriter.writeVertex(GraphSONWriter.java:82)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONWriter.writeVertices(GraphSONWriter.java:110)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONWriter.writeGraph(GraphSONWriter.java:71)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONIo.writeGraph(GraphSONIo.java:83)
at org.apache.tinkerpop.gremlin.structure.io.Io$writeGraph.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
at groovysh_evaluate.run(groovysh_evaluate:3)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:236)
at org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:71)
at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:196)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1225)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:145)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:72)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:95)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1225)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:145)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:165)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:130)
at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.

how can i solve it? thank you very much

Hi JanusGraph team,

We are exploring the possibility of janusgraph in our company. We are planning to use dynamic graph creation/deletion using ConfiguredGraphFactory. We have deployed 2 janusgraph instances. Our plan is to write java library to creating, deletion and other management activities like Index creation etc . I haven't yet found a way to create/delete a graph using java apis by using ConfiguredGraphFactory.

From my investigation, only way to do so using by connecting to gremlin server and sending string commands like 

 - ConfiguredGraphFactory.create(graphName)

 - ConfiguredGraphFactory.drop(graphName)

Is this the only way to create/delete graph? Please adivce.

Thanks,

anumodh 

Hi JanusGraph team,

I have created a vertex-centric indexes for vertices. As follows, now I want to use the index to get the information of the top 500 edges in descending sort. However, I find that the execution time is the same as that without vertex index. How can I use the index to sort faster and extract the information of the first 500 edges more quickly?

Here's the graph I've built：

graph=JanusGraphFactory.open(‘janusgraph-cql-es-server-test2.properties’) mgmt = graph.openManagement() mgmt.makeVertexLabel('VirtualAddress').make() addr = mgmt.makePropertyKey('address').dataType(String.class).cardinality(SINGLE).make() token_addr = mgmt.makePropertyKey('token_addr').dataType(String.class).cardinality(SINGLE).make() transfer_to=mgmt.makeEdgeLabel('TRANSFER_TO').multiplicity(MULTI).make() amount = mgmt.makePropertyKey('amount').dataType(Double.class).cardinality(SINGLE).make() tx_hash = mgmt.makePropertyKey('tx_hash').dataType(String.class).cardinality(SINGLE).make() tx_index = mgmt.makePropertyKey('tx_index').dataType(Integer.class).cardinality(SINGLE).make() created_time = mgmt.makePropertyKey('created_time').dataType(Date.class).cardinality(SINGLE).make() updated_time = mgmt.makePropertyKey('updated_time').dataType(Date.class).cardinality(SINGLE).make() mgmt.buildIndex('addressComposite', Vertex.class).addKey(addr).buildCompositeIndex() mgmt.buildIndex('addressTokenUniqComposite', Vertex.class).addKey(addr).addKey(token_addr).unique().buildCompositeIndex() mgmt.buildEdgeIndex(transfer_to,"transferOutAmountTs", Direction.OUT, Order.desc,amount,created_time) mgmt.buildEdgeIndex(transfer_to,"transferOutTs", Direction.OUT, Order.desc,created_time) mgmt.commit()

Here's the data I inserted, building a starting point, a million edges associated with it, and 100 endpoints,

graph_conf = 'janusgraph-cql-es-server-test2.properties'

graph = JanusGraphFactory.open(graph_conf)

g = graph.traversal()

String line = "5244613,tx_hash_00,token_addr_00,from_addr_00,to_addr_00,,,6000,19305.57174337591,72,1520896044"

int start_value = 1

int end_value = 1000000

line = "tx_hash_00,token_addr_00,from_addr_00,to_addr_00,6000,72,1520896044"

cloumns = line.split(',', -1)

(tx_hash, token_addr, from_addr, to_addr, amount, log_index, timestamp) = cloumns

from_addr_node = g.addV('VirtualAddress').property('address', from_addr).property('token_addr', token_addr).next()

from_id = from_addr_node.id()

amount = amount.toBigDecimal()

tx_index = log_index.toInteger()

for (int i = start_value; i <= end_value; i++) {

    to_addr_node = g.addV('VirtualAddress').property('address', to_addr + String.valueOf(i)).property('token_addr', token_addr).next()

    to_id = to_addr_node.id()

    Date ts = new Date((timestamp.toLong() - i) * 1000)

    g.addE('TRANSFER_TO').from(g.V(from_id)).to(g.V(to_id))

            .property('amount', amount + i)

            .property('tx_hash', tx_hash)

            .property('tx_index', tx_index + i)

            .property('created_time', ts)

            .next()

    if (i % 20000 == 0) {

        println("[total:${i}]")

        System.sleep(500)

        g.tx().commit()

        graph.close()

        System.sleep(5000)

        graph = JanusGraphFactory.open(graph_conf)

        g = graph.traversal()

        System.sleep(5000)

    }

    g.tx().commit()

}

graph.close()

 

Here are my query criteria:

g.V().has('address', ‘from_addr_00').outE('TRANSFER_TO').order().by(‘amount’,desc).limit(500).valueMap().toList()

I use TinkerPop’s Hadoop-Gremlin to import data.

Take grateful-dead.txt as an example, I want to import vertex without edges.

But I get error like this, "java.util.NoSuchElementException".

Hi!

Try this 

spark.io.compression.codec=snappy

четверг, 23 июля 2020 г., 1:57:38 UTC+3 пользователь Fábio Dapper написал:

Hello, we have a Cluster with CLOUDERA CDH 6.3.2 and I'm trying to run Janusgraph on the Cluster with YARN and HBASE, but without success.

(it's OK with SPARK Local)

Version SPARK 2.4.2

HBASE: 2.1.0-cdh6.3.2

Janusgraph (v 0.5.2 and v0.4.1)

I did a lot of searching, but I didn't find any recent references, and they all use older versions of SPARK and Janusgraph.

Some examples:

1) https://docs.janusgraph.org/advanced-topics/hadoop/  

2) http://tinkerpop.apache.org/docs/current/recipes/#olap-spark-yarn

3) http://yaaics.blogspot.com/2017/07/configuring-janusgraph-for-spark-yarn.html

According to these references, I followed the following steps:

1. Copy the following files to the Janusgraph "lib" directory:
1. spark-yarn-2.11-2.4.0.jar
2. scala-reflect-2.10.5.jar
3. hadoop-yarn-server-web-proxy-2.7.2.jar
4. guice-servlet-3.0.jar
2. Generate a "/tmp/spark-gremlin-0.5.2.zip" file containing all the .jar files from "janusgraph / lib /".
3. Create a configuration file called 'test.properties' from “conf/hadoop-graph/read-hbase-standalone-cluster.properties” by adding (or modifying) the properties below:
   
   

        janusgraphmr.ioformat.conf.storage.hostname=XXX.XXX.XXX.XXX 

spark.master= yarn

#spark.deploy-mode=client

spark.submit.deployMode=client

spark.executor.memory=1g

spark.yarn.dist.jars=/tmp/spark-gremlin-0-5-2.zip

spark.yarn.archive=/tmp/spark-gremlin-0-5-2.zip

spark.yarn.appMasterEnv.CLASSPATH=./__spark_libs__/*:[hadoop_conf_dir]

spark.executor.extraClassPath=./__spark_libs__/*:/[hadoop_conf_dir]

spark.driver.extraLibraryPath=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native 

Then I ran the following commands:

graph = GraphFactory.open(conf/hadoop-graph/test.properties)
g = graph.traversal().withComputer(SparkGraphComputer)
g.V().count()

Can someone help me?

a) Are these problems related to version incompatibility?

b) Has anyone successfully used similar infrastructure?

c) Would anyone know how to determine a correct version of the necessary libraries?

d) Any suggestion?

Thank you all !!!

 Below is a copy of the Yarn Log from my last attempt.

ERROR org.apache.spark.scheduler.TaskSetManager  - Task 0 in stage 0.0 failed 4 times; aborting job
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, [SERVER_NAME], executor 1): java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$6.apply(TorrentBroadcast.scala:304)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$6.apply(TorrentBroadcast.scala:304)
at scala.Option.map(Option.scala:146)
at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:304)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$apply$2.apply(TorrentBroadcast.scala:235)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:211)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1326)
at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:207)
at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:66)
at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:66)
at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:96)
at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:89)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Thank you!!

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgra...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/467a21c7-b103-4c1a-9404-a514e4335671o%40googlegroups.com.

Hello, we have a Cluster with CLOUDERA CDH 6.3.2 and I'm trying to run Janusgraph on the Cluster with YARN and HBASE, but without success.

(it's OK with SPARK Local)

Version SPARK 2.4.2

HBASE: 2.1.0-cdh6.3.2

Janusgraph (v 0.5.2 and v0.4.1)

I did a lot of searching, but I didn't find any recent references, and they all use older versions of SPARK and Janusgraph.

Some examples:

1) https://docs.janusgraph.org/advanced-topics/hadoop/  

2) http://tinkerpop.apache.org/docs/current/recipes/#olap-spark-yarn

3) http://yaaics.blogspot.com/2017/07/configuring-janusgraph-for-spark-yarn.html

According to these references, I followed the following steps:

1. Copy the following files to the Janusgraph "lib" directory:
1. spark-yarn-2.11-2.4.0.jar
2. scala-reflect-2.10.5.jar
3. hadoop-yarn-server-web-proxy-2.7.2.jar
4. guice-servlet-3.0.jar
2. Generate a "/tmp/spark-gremlin-0.5.2.zip" file containing all the .jar files from "janusgraph / lib /".
3. Create a configuration file called 'test.properties' from “conf/hadoop-graph/read-hbase-standalone-cluster.properties” by adding (or modifying) the properties below:
   
   

        janusgraphmr.ioformat.conf.storage.hostname=XXX.XXX.XXX.XXX 

spark.master= yarn

#spark.deploy-mode=client

spark.submit.deployMode=client

spark.executor.memory=1g

spark.yarn.dist.jars=/tmp/spark-gremlin-0-5-2.zip

spark.yarn.archive=/tmp/spark-gremlin-0-5-2.zip

spark.yarn.appMasterEnv.CLASSPATH=./__spark_libs__/*:[hadoop_conf_dir]

spark.executor.extraClassPath=./__spark_libs__/*:/[hadoop_conf_dir]

spark.driver.extraLibraryPath=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native 

Then I ran the following commands:

graph = GraphFactory.open(conf/hadoop-graph/test.properties)
g = graph.traversal().withComputer(SparkGraphComputer)
g.V().count()

Can someone help me?

a) Are these problems related to version incompatibility?

b) Has anyone successfully used similar infrastructure?

c) Would anyone know how to determine a correct version of the necessary libraries?

d) Any suggestion?

Thank you all !!!

 Below is a copy of the Yarn Log from my last attempt.

ERROR org.apache.spark.scheduler.TaskSetManager  - Task 0 in stage 0.0 failed 4 times; aborting job
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, [SERVER_NAME], executor 1): java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$6.apply(TorrentBroadcast.scala:304)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$6.apply(TorrentBroadcast.scala:304)
at scala.Option.map(Option.scala:146)
at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:304)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$apply$2.apply(TorrentBroadcast.scala:235)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:211)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1326)
at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:207)
at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:66)
at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:66)
at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:96)
at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:89)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Thank you!!

Hi,

See a similar question on:

https://gitter.im/janusgraph/janusgraph?at=5c7d868a212d0c1e1ad19eef

HTH,   Marc

Op woensdag 15 januari 2020 14:11:14 UTC+1 schreef qi...@...:

hello i am new in JanusGraph. When i use Janusgraph in Apache Atlas, i found a question, the error is :

Could not find a type identifier for the class : class org.apache.atlas.typesystem.types.DataTypes$TypeCategory. Make sure the value to serialize has a type identifier registered for its class.
org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException: Could not find a type identifier for the class : class org.apache.atlas.typesystem.types.DataTypes$TypeCategory. Make sure the value to serialize has a type identifier registered for its class.
at org.apache.tinkerpop.shaded.jackson.databind.ser.DefaultSerializerProvider._wrapAsIOE(DefaultSerializerProvider.java:509)
at org.apache.tinkerpop.shaded.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:482)
at org.apache.tinkerpop.shaded.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3893)
at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:3164)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONWriter.writeVertex(GraphSONWriter.java:82)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONWriter.writeVertices(GraphSONWriter.java:110)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONWriter.writeGraph(GraphSONWriter.java:71)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONIo.writeGraph(GraphSONIo.java:83)
at org.apache.tinkerpop.gremlin.structure.io.Io$writeGraph.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
at groovysh_evaluate.run(groovysh_evaluate:3)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:236)
at org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:71)
at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:196)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1225)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:145)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:72)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:95)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1225)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:145)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:165)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:130)
at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.

how can i solve it? thank you very much

Hi,

I have a question regarding the performance of basic operations in JanusGraph. We use obsolete Titan in our project (with Persistit storage backend, so the Titan version is 0.4.4) and we wanted to compare it with JanusGraph in order to get rid of the deprecated database but to maintain (or better improve) our current performance. 

A brief report about our results is provided here: https://github.com/svitaluc/graphbench/tree/master/results. 

The source code of the benchmark can be found on the upper level here https://github.com/svitaluc/graphbench.

Based on the results I have a few questions:

- How is it possible that the import with JanusGraph is so slow in comparison with Titan? Is there any additional setting that we perhaps missed?

- Getting vertices with their neighbors/edges have tremendously worse performance than in the case of Titan. How is this possible? It seems to me like the model was changed and the edges are no longer stored in the same row because I do not think that there would be such a difference between Persistit and Cassandra. On the other hand, there would be no reason for the model change of course.

- Is the reasoning about cache behavior correct? Is there a part of cache that cannot be influenced? Maybe in the case of JanusGraph, this issue is negligible.

Thank you for any hint and response!

Cheers,

Lucie

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgra...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/27693e05-285f-4445-a715-59604ff5f995o%40googlegroups.com.

Hi,

> How do we know if the graph has been dropped?   

As I am using Cassandra and ES as backend, I can see that data as been dropped by doing a direct request to these components.

>   Is there any way of clearing all the data, the schema without restarting the server. 

I have found any yet but I do not look into the gremlin server code.  I do not have this requirement, so I do search for it. May be using a proxy design pattern on graph object may do the trick.

Nicolas

Le jeudi 16 juillet 2020 à 14:53:03 UTC+2, d...@... a écrit :

Hi Nicolas,

How do we know if the graph has been dropped? You said you have to restart the server after doing that. I am doing the same, but I don't want to. Is there any way of clearing all the data, the schema without restarting the server.

Also, JanusGraphFactory.drop(getJanusGraph()) is the same as JanusGraphFactory.drop(graph) - because that function passes the same object and it still doesn't work. I don't want to restart the server as it's a remote server. How do I achieve that?

On Wednesday, July 15, 2020 at 3:43:13 PM UTC-4, Nicolas Trangosi wrote:

Hi,

I am able to drop remotely  the graph using the script:

JanusGraphFactory.drop(graph);[]

After the script, I need to restart janusGraph in order to re-create the graph.

Could you sent the janusgraph logs with the stack trace ?

Also, instead of  JanusGraphFactory.drop(getJanusGraph());  , could you try with JanusGraphFactory.drop(graph);

Kind regards,

Nicolas

Le mercredi 15 juillet 2020 à 00:44:36 UTC+2, d...@... a écrit :

Hi,

I'm looking to drop a graph, but the JanusgraphFactory.drop(graph) doesn't work for me. I'm using JAVA to connect to the remote server and load/remove data.

Using the below properties file  for remote-properties gremlin.remote.remoteConnectionClass=org.apache.tinkerpop.gremlin.driver.remote.DriverRemoteConnection
gremlin.remote.driver.clusterFile=conf/remote-objects.yaml
gremlin.remote.driver.sourceName=g

Below file for remote-objects - 

hosts: [hostname]
port: 8182
serializer: {
className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0,
config: {
ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry]
}
}
Using this to open the graph - 
conf = new PropertiesConfiguration(propFileName);

// using the remote graph for queries
graph = JanusGraphFactory.open("inmemory");
g = graph.traversal().withRemote(conf);
return g;

Using this to drop the graph:
if (graph != null) {
LOGGER.info("Dropping janusgraph function");
JanusGraphFactory.drop(getJanusGraph());
}

Any help would be appreciated.
Thanks!

Ce message et ses pièces jointes peuvent contenir des informations confidentielles ou privilégiées et ne doivent donc pas être diffusés, exploités ou copiés sans autorisation. 

Si vous avez reçu ce message par erreur, veuillez le signaler a l'expéditeur et le détruire ainsi que les pièces jointes. 

Les messages électroniques étant susceptibles d'altération, DCbrain décline toute responsabilité si ce message a été altéré, déformé ou falsifié. Merci. 

This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, DCbrain is not liable for messages that have been modified, changed or falsified. Thank you.

Ce message et ses pièces jointes peuvent contenir des informations confidentielles ou privilégiées et ne doivent donc pas être diffusés, exploités ou copiés sans autorisation. 

Si vous avez reçu ce message par erreur, veuillez le signaler a l'expéditeur et le détruire ainsi que les pièces jointes. 

Les messages électroniques étant susceptibles d'altération, DCbrain décline toute responsabilité si ce message a été altéré, déformé ou falsifié. Merci. 

This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, DCbrain is not liable for messages that have been modified, changed or falsified. Thank you.

Try ./janusgraph.sh clean

On Tue, Jul 21, 2020, 12:20 PM Nicolas Trangosi <nic...@...> wrote:

Hi,

> How do we know if the graph has been dropped?   

As I am using Cassandra and ES as backend, I can see that data as been dropped by doing a direct request to these components.

>   Is there any way of clearing all the data, the schema without restarting the server. 

I have found any yet but I do not look into the gremlin server code.  I do not have this requirement, so I do search for it. May be using a proxy design pattern on graph object may do the trick.

Nicolas

Le jeudi 16 juillet 2020 à 14:53:03 UTC+2, d...@... a écrit :

Hi Nicolas,

How do we know if the graph has been dropped? You said you have to restart the server after doing that. I am doing the same, but I don't want to. Is there any way of clearing all the data, the schema without restarting the server.

Also, JanusGraphFactory.drop(getJanusGraph()) is the same as JanusGraphFactory.drop(graph) - because that function passes the same object and it still doesn't work. I don't want to restart the server as it's a remote server. How do I achieve that?

On Wednesday, July 15, 2020 at 3:43:13 PM UTC-4, Nicolas Trangosi wrote:

Hi,

I am able to drop remotely  the graph using the script:

JanusGraphFactory.drop(graph);[]

After the script, I need to restart janusGraph in order to re-create the graph.

Could you sent the janusgraph logs with the stack trace ?

Also, instead of  JanusGraphFactory.drop(getJanusGraph());  , could you try with JanusGraphFactory.drop(graph);

Kind regards,

Nicolas

Le mercredi 15 juillet 2020 à 00:44:36 UTC+2, d...@... a écrit :

Hi,

I'm looking to drop a graph, but the JanusgraphFactory.drop(graph) doesn't work for me. I'm using JAVA to connect to the remote server and load/remove data.

Using the below properties file  for remote-properties gremlin.remote.remoteConnectionClass=org.apache.tinkerpop.gremlin.driver.remote.DriverRemoteConnection
gremlin.remote.driver.clusterFile=conf/remote-objects.yaml
gremlin.remote.driver.sourceName=g

Below file for remote-objects - 

hosts: [hostname]
port: 8182
serializer: {
className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0,
config: {
ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry]
}
}
Using this to open the graph - 
conf = new PropertiesConfiguration(propFileName);

// using the remote graph for queries
graph = JanusGraphFactory.open("inmemory");
g = graph.traversal().withRemote(conf);
return g;

Using this to drop the graph:
if (graph != null) {
LOGGER.info("Dropping janusgraph function");
JanusGraphFactory.drop(getJanusGraph());
}

Any help would be appreciated.
Thanks!

Ce message et ses pièces jointes peuvent contenir des informations confidentielles ou privilégiées et ne doivent donc pas être diffusés, exploités ou copiés sans autorisation. 

Si vous avez reçu ce message par erreur, veuillez le signaler a l'expéditeur et le détruire ainsi que les pièces jointes. 

Les messages électroniques étant susceptibles d'altération, DCbrain décline toute responsabilité si ce message a été altéré, déformé ou falsifié. Merci. 

This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, DCbrain is not liable for messages that have been modified, changed or falsified. Thank you.

Ce message et ses pièces jointes peuvent contenir des informations confidentielles ou privilégiées et ne doivent donc pas être diffusés, exploités ou copiés sans autorisation. 

Si vous avez reçu ce message par erreur, veuillez le signaler a l'expéditeur et le détruire ainsi que les pièces jointes. 

Les messages électroniques étant susceptibles d'altération, DCbrain décline toute responsabilité si ce message a été altéré, déformé ou falsifié. Merci. 

This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, DCbrain is not liable for messages that have been modified, changed or falsified. Thank you.

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgra...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/ab1d0d52-ce84-4eeb-ad24-e3e189eb4ed5n%40googlegroups.com.