Date   

Not able to connect when 1 of 3 nodes is down in the Cassandra cluster

Bharat Dighe <bdi...@...>
 

I am using titan 1.0 and planning to move to Janus very soon.

I have following keyspace

CREATE KEYSPACE my_ks WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': '2', 'DC2': '1'}  AND durable_writes = true;

Current status of the cluster nodes is as follows, one of the node in DC1 is down.

|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns    Host ID                               Rack
DN  IP1  
2.8 MB     256     ?       2a5abdad-af65-48e7-a74c-d40f1f759460  rac2
UN  IP2  
4.33 MB    256     ?       4897d661-24d3-4d30-b07a-00a8103635f6  rac1
Datacenter: Sunnyside_DC
========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns    Host ID                               Rack
UN  IP3  
5.24 MB    256     ?       f830e2a9-6eea-4617-88dd-d63e44beb115  rac1


Titan is able to connect to node in DC2 but fails to join the UP node in DC1.

TitanGraph graph = TitanFactory.build().
 
set("storage.backend", "cassandra").
 
set("storage.hostname", "IP2").
 
set("storage.port", 9160).
 
set("storage.cassandra.keyspace", "my_ks").
 
set("storage.read-only", false).
 
set("query.force-index", false).
 
set("storage.cassandra.astyanax.connection-pool-type","ROUND_ROBIN").
 
set("storage.cassandra.astyanax.node-discovery-type","NONE").
 
set("storage.cassandra.read-consistency-level","LOCAL_QUORUM").
 
set("storage.cassandra.write-consistency-level","LOCAL_QUORUM").
 
set("storage.cassandra.atomic-batch-mutate",false).
 open
();

It gives following exception:

22:11:19,655 ERROR CountingConnectionPoolMonitor:94 - com.netflix.astyanax.connectionpool.exceptions.TokenRangeOfflineException: TokenRangeOfflineException: [host=IP2:9160, latency=100(100), attempts=1]UnavailableException()
com
.netflix.astyanax.connectionpool.exceptions.TokenRangeOfflineException: TokenRangeOfflineException: [host=IP2:9160, latency=100(100), attempts=1]UnavailableException()
 at com
.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:165)
 at com
.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:65)
 at com
.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:28)
 at com
.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:153)
 at com
.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:119)
 at com
.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:352)
 at com
.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$4.execute(ThriftColumnFamilyQueryImpl.java:538)
 at com
.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getNamesSlice(AstyanaxKeyColumnValueStore.java:112)
 at com
.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getNamesSlice(AstyanaxKeyColumnValueStore.java:78)
 at com
.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getSlice(AstyanaxKeyColumnValueStore.java:67)
 at com
.thinkaurelius.titan.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:91)
 at com
.thinkaurelius.titan.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:1)
 at com
.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:133)
 at com
.thinkaurelius.titan.diskstorage.util.BackendOperation$1.call(BackendOperation.java:147)
 at com
.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56)
 at com
.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42)
 at com
.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:144)
 at com
.thinkaurelius.titan.diskstorage.configuration.backend.KCVSConfiguration.get(KCVSConfiguration.java:88)
 at com
.thinkaurelius.titan.diskstorage.configuration.BasicConfiguration.isFrozen(BasicConfiguration.java:93)
 at com
.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1338)
 at com
.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:94)
 at com
.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:84)
 at com
.thinkaurelius.titan.core.TitanFactory$Builder.open(TitanFactory.java:139)
 at 
TestGraph.main(TestGraph.java:20)
Caused by: UnavailableException()
 at org
.apache.cassandra.thrift.Cassandra$multiget_slice_result$multiget_slice_resultStandardScheme.read(Cassandra.java:14687)
 at org
.apache.cassandra.thrift.Cassandra$multiget_slice_result$multiget_slice_resultStandardScheme.read(Cassandra.java:14633)
 at org
.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:14559)
 at org
.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
 at org
.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:741)
 at org
.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:725)
 at com
.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$4$1.internalExecute(ThriftColumnFamilyQueryImpl.java:544)
 at com
.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$4$1.internalExecute(ThriftColumnFamilyQueryImpl.java:541)
 at com
.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:60)
 
... 22 more


Please help me to resolve this.

Thanks
Bharat


Re: Merging two graphs.

Jason Plurad <plu...@...>
 

You could use some of the methods from TinkerPop's ElementHelper to help copy over properties.
http://tinkerpop.apache.org/javadocs/3.2.3/full/org/apache/tinkerpop/gremlin/structure/util/ElementHelper.html

The element ids will be different between your 2 JanusGraph instances, so as you've noted, you'll either have to track the ids or have sufficient indexes configured to do lookups.


On Wednesday, July 19, 2017 at 10:37:28 AM UTC-4, Gwiz wrote:
Hello,

I have an in-memory Janusgraph instance and I would like to merge that with my Cassandra/ES based graph. What is the best (efficient) way to do this?

When I merge, I don't need to insert the vertices that are already in my Cassandra/ES based graph. I know which vertices in the in-memory graph exists in the Cassandra/ES based graph. 

I don't see any Java API to take a Vertex (or list of Vertices) from one Janusgraph instance and add them to another. All I see is the following:
  • addVertex(String)
  • addVertex(Object... objects)
My original plan is to insert the new vertices from in-memory graph into my Cassandra/ES based graph, get the IDs back and use them to insert Edges.

I appreciate any help.

Thanks.


Re: when release 0.2.0?

Jason Plurad <plu...@...>
 

I think it's getting pretty close. I'd guess July is a stretch, but August should be possible.

Several great things are already in place: TinkerPop 3.2.5 support, Cassandra CQL support, ES 5.4.2 support, Solr 6.6.0 support.
* There are a few more PRs in the queue that need to get merged, including this one on OLAP compatibility with Cassandra 3.0+.
* There is also some discussion going on regarding the Cassandra source code tree organization, which also needs to be completed.
* A recently identified migration issue from Titan must be fixed.

I could be missing others. Here's how you and anybody else in the community can help:
* Help triage the issues. Try to reproduce them. Add comments. If there's something critical that's needed for 0.2, let it be known.
* Help review code in the pull requests.
* Test the master branch in your test environments. It already has cassandra-cql and ES 5.x support in place, so you can help us make sure it works especially for your use cases.


On Thursday, July 20, 2017 at 10:27:59 PM UTC-4, Ranger Tsao wrote:
I want use JanusGraph in my production,but I need two feature: cassandra-cql and es 5.x


Re: how can i remove the index

Chin Huang <chinhu...@...>
 

We were in the same situation. The problem is the index cannot be manually moved out the "INSTALLED" state. We have to recreate in the index and make sure no open transactions during mgmt.commit()

Please also check this thread, with similar issue of having index stuck in the "INSTALLED" state.

https://mail.google.com/mail/u/0/#search/installed/15c7ee77373b6ea8

On Fri, Jul 21, 2017 at 9:15 AM, David Pitera <piter...@...> wrote:
Your propertyKey `phone` on your index `phoneIndex` is in the `INSTALLED` state, so you need to `REGISTER` it and wait for it to become registered before you can attempt to `enable` it. After it is enabled, you can disable and then remove it.

Some code that should help cause it goes through the whole installed/registered/enabled phase is:




On Thu, Jul 20, 2017 at 11:43 PM, 李平 <lipin...@...> wrote:
public class GraphTest {

public static void main(String[] args) throws Exception {

BaseConfiguration baseConfiguration = new BaseConfiguration();
baseConfiguration.setProperty("storage.backend", "hbase");
baseConfiguration.setProperty("storage.hostname", "192.168.1.108");
baseConfiguration.setProperty("gremlin.graph", "org.janusgraph.core.JanusGraphFactory");
JanusGraph janusGraph = JanusGraphFactory.open(baseConfiguration);
GraphTraversalSource g = janusGraph.traversal();
g.tx().rollback();
JanusGraphManagement janusGraphManagement = janusGraph.openManagement();
JanusGraphIndex phoneIndex = janusGraphManagement.getGraphIndex("phoneIndex");
janusGraphManagement.updateIndex(phoneIndex, SchemaAction.DISABLE_INDEX).get();
janusGraphManagement.commit();
g.tx().commit();
ManagementSystem.awaitGraphIndexStatus(janusGraph,"phoneIndex")
.status(SchemaStatus.DISABLED)
.timeout(10, ChronoUnit.MINUTES)
.call();
janusGraphManagement.updateIndex(phoneIndex, SchemaAction.REMOVE_INDEX).get();
janusGraphManagement.commit();
janusGraph.tx().commit();
System.out.println("---------------------remove index sucess...");

the log is :Some key(s) on index phoneIndex do not currently have status DISABLED: phone=INSTALLED


--
You received this message because you are subscribed to the Google Groups "JanusGraph users list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "JanusGraph users list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: how can i remove the index

David Pitera <piter...@...>
 

Your propertyKey `phone` on your index `phoneIndex` is in the `INSTALLED` state, so you need to `REGISTER` it and wait for it to become registered before you can attempt to `enable` it. After it is enabled, you can disable and then remove it.

Some code that should help cause it goes through the whole installed/registered/enabled phase is:




On Thu, Jul 20, 2017 at 11:43 PM, 李平 <lipin...@...> wrote:
public class GraphTest {

public static void main(String[] args) throws Exception {

BaseConfiguration baseConfiguration = new BaseConfiguration();
baseConfiguration.setProperty("storage.backend", "hbase");
baseConfiguration.setProperty("storage.hostname", "192.168.1.108");
baseConfiguration.setProperty("gremlin.graph", "org.janusgraph.core.JanusGraphFactory");
JanusGraph janusGraph = JanusGraphFactory.open(baseConfiguration);
GraphTraversalSource g = janusGraph.traversal();
g.tx().rollback();
JanusGraphManagement janusGraphManagement = janusGraph.openManagement();
JanusGraphIndex phoneIndex = janusGraphManagement.getGraphIndex("phoneIndex");
janusGraphManagement.updateIndex(phoneIndex, SchemaAction.DISABLE_INDEX).get();
janusGraphManagement.commit();
g.tx().commit();
ManagementSystem.awaitGraphIndexStatus(janusGraph,"phoneIndex")
.status(SchemaStatus.DISABLED)
.timeout(10, ChronoUnit.MINUTES)
.call();
janusGraphManagement.updateIndex(phoneIndex, SchemaAction.REMOVE_INDEX).get();
janusGraphManagement.commit();
janusGraph.tx().commit();
System.out.println("---------------------remove index sucess...");

the log is :Some key(s) on index phoneIndex do not currently have status DISABLED: phone=INSTALLED


--
You received this message because you are subscribed to the Google Groups "JanusGraph users list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


how can i remove the index

李平 <lipin...@...>
 

public class GraphTest {

public static void main(String[] args) throws Exception {

BaseConfiguration baseConfiguration = new BaseConfiguration();
baseConfiguration.setProperty("storage.backend", "hbase");
baseConfiguration.setProperty("storage.hostname", "192.168.1.108");
baseConfiguration.setProperty("gremlin.graph", "org.janusgraph.core.JanusGraphFactory");
JanusGraph janusGraph = JanusGraphFactory.open(baseConfiguration);
GraphTraversalSource g = janusGraph.traversal();
g.tx().rollback();
JanusGraphManagement janusGraphManagement = janusGraph.openManagement();
JanusGraphIndex phoneIndex = janusGraphManagement.getGraphIndex("phoneIndex");
janusGraphManagement.updateIndex(phoneIndex, SchemaAction.DISABLE_INDEX).get();
janusGraphManagement.commit();
g.tx().commit();
ManagementSystem.awaitGraphIndexStatus(janusGraph,"phoneIndex")
.status(SchemaStatus.DISABLED)
.timeout(10, ChronoUnit.MINUTES)
.call();
janusGraphManagement.updateIndex(phoneIndex, SchemaAction.REMOVE_INDEX).get();
janusGraphManagement.commit();
janusGraph.tx().commit();
System.out.println("---------------------remove index sucess...");

the log is :Some key(s) on index phoneIndex do not currently have status DISABLED: phone=INSTALLED



when release 0.2.0?

Ranger Tsao <cao....@...>
 

I want use JanusGraph in my production,but I need two feature: cassandra-cql and es 5.x


Re: Geoshape property in remote gremlin query, GraphSON

Robert Dale <rob...@...>
 

Using cluster/client will work with the v1 serializer.

gremlin-server.yaml:

serializers:
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV1d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}


conf/remote.yaml:
hosts: [localhost]
port
: 8182
serializer
: { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV1d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] } }


connect:
gremlin> cluster = Cluster.open('conf/remote.yaml')
==>localhost/127.0.0.1:8182
gremlin
> client = cluster.connect();
==>org.apache.tinkerpop.gremlin.driver.Client$ClusteredClient@434514d8
gremlin> client.submit("g.addV().property('geo', Geoshape.point(0.0,1.1))")
==>result{object={id=4152, label=vertex, type=vertex, properties={geo=[{id=sn-37c-sl, value=POINT (1.1 0)}]}} class=java.util.HashMap}
gremlin> client.submit("g.V().values('geo')").all().get().get(0).getObject()
==>POINT (1.1 0)
gremlin> client.submit("g.V().values('geo')").all().get().get(0).getObject().getClass()
==>class org.janusgraph.core.attribute.Geoshape



You won't be able to use withRemote() with the V1 serializer.



On Wednesday, July 19, 2017 at 3:16:37 PM UTC-4, Robert Dale wrote:
It seems Geoshape GraphSON support is hardcoded to v1 although I couldn't get it to work with that either.  If you have to use GraphSON instead of Gryo, then you could checkout master, apply this patch, and rebuild. I created an  issue to support multiple versions of serializers  https://github.com/JanusGraph/janusgraph/issues/420

diff --git a/janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/io/graphson/JanusGraphSONModule.java b/janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/io/graphson/JanusGraphSONModule.java
index
6ef907b..8168309 100644
--- a/janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/io/graphson/JanusGraphSONModule.java
+++ b/janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/io/graphson/JanusGraphSONModule.java
@@ -50,10 +50,10 @@ public class JanusGraphSONModule extends TinkerPopJacksonModule {
     
private JanusGraphSONModule() {
         
super("janusgraph");
         addSerializer
(RelationIdentifier.class, new RelationIdentifierSerializer());
-        addSerializer(Geoshape.class, new Geoshape.GeoshapeGsonSerializerV1d0());
+        addSerializer(Geoshape.class, new Geoshape.GeoshapeGsonSerializerV2d0());
 
         addDeserializer
(RelationIdentifier.class, new RelationIdentifierDeserializer());
-        addDeserializer(Geoshape.class, new Geoshape.GeoshapeGsonDeserializerV1d0());
+        addDeserializer(Geoshape.class, new Geoshape.GeoshapeGsonDeserializerV2d0());
     
}
 
     
private static final JanusGraphSONModule INSTANCE = new JanusGraphSONModule();



On Tuesday, July 18, 2017 at 5:47:50 PM UTC-4, Conrad Rosenbrock wrote:
I am trying to assign a value to a property with the native Geoshape type. I have it serialized into JSON as follows (where g is aliased to the traversal on gremlin server):

{"@value": {"type": "point", "coordinates": [{"@value": 1.1, "@type": "g:Double"}, {"@value": 2.2, "@type": "g:Double"}]}, "@type": "g:Geoshape"}

In the gremlin console, I can easily type 

Geoshape.point(1.1, 2.2)

and it works perfectly. I am sure that it is something quite simple. Here is the error:

Request [PooledUnsafeDirectByteBuf(ridx: 653, widx: 653, cap: 687)] could not be deserialized by org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV2d0.

For reference, I do have the following serializer in the gremlin server config:

{ className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV2d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}

which should direct gremlin server to the relevant deserializer in Janus.

Thanks!


Re: How Can I make a statistics,i.e:how many vertexes or edges?

spirit...@...
 

I run into a new problem when I config the gremlin and spark-onYarn according the post.
Am I lacking the dependencies or version conflicts?

10:47:47,199  INFO KryoShimServiceLoader:117 - Set KryoShimService provider to org.apache.tinkerpop.gremlin.hadoop.structure.io.HadoopPoolShimService@4b31a708 (class org.apache.tinkerpop.gremlin.hadoop.structure.io.HadoopPoolShimService) because its priority value (0) is the highest available
10:47:47,199  INFO KryoShimServiceLoader:123 - Configuring KryoShimService provider org.apache.tinkerpop.gremlin.hadoop.structure.io.HadoopPoolShimService@4b31a708 with user-provided configuration
10:47:51,447  INFO SparkContext:58 - Running Spark version 1.6.1
10:47:51,495  INFO SecurityManager:58 - Changing view acls to: rc
10:47:51,496  INFO SecurityManager:58 - Changing modify acls to: rc
10:47:51,496  INFO SecurityManager:58 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(rc); users with modify permissions: Set(rc)
10:47:51,855  INFO Utils:58 - Successfully started service 'sparkDriver' on port 41967.
10:47:52,450  INFO Slf4jLogger:80 - Slf4jLogger started
10:47:52,504  INFO Remoting:74 - Starting remoting
10:47:52,666  INFO Remoting:74 - Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@...:50605]
10:47:52,673  INFO Utils:58 - Successfully started service 'sparkDriverActorSystem' on port 50605.
10:47:53,428  INFO SparkEnv:58 - Registering MapOutputTracker
10:47:53,448  INFO SparkEnv:58 - Registering BlockManagerMaster
10:47:53,460  INFO DiskBlockManager:58 - Created local directory at /tmp/blockmgr-94bbe487-7cf4-4cf5-bcc2-fc538487f31a
10:47:53,473  INFO MemoryStore:58 - MemoryStore started with capacity 2.4 GB
10:47:53,591  INFO SparkEnv:58 - Registering OutputCommitCoordinator
10:47:53,755  INFO Server:272 - jetty-8.y.z-SNAPSHOT
10:47:53,809  INFO AbstractConnector:338 - Started SelectChannelConnector@0.0.0.0:4040
10:47:53,810  INFO Utils:58 - Successfully started service 'SparkUI' on port 4040.
10:47:53,813  INFO SparkUI:58 - Started SparkUI at http://10.200.48.112:4040
spark.yarn.driver.memoryOverhead is set but does not apply in client mode.
10:47:54,996  INFO TimelineClientImpl:296 - Timeline service address: http://dl-rc-optd-ambari-master-v-test-1.host.dataengine.com:8188/ws/v1/timeline/
10:47:55,307  INFO ConfiguredRMFailoverProxyProvider:100 - Failing over to rm2
10:47:55,333  INFO Client:58 - Requesting a new application from cluster with 8 NodeManagers
10:47:55,351  INFO Client:58 - Verifying our application has not requested more than the maximum memory capability of the cluster (10240 MB per container)
10:47:55,351  INFO Client:58 - Will allocate AM container, with 896 MB memory including 384 MB overhead
10:47:55,352  INFO Client:58 - Setting up container launch context for our AM
10:47:55,355  INFO Client:58 - Setting up the launch environment for our AM container
10:47:55,367  INFO Client:58 - Preparing resources for our AM container
10:47:56,298  INFO Client:58 - Uploading resource file:/rc/lib/spark_lib/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar -> hdfs://chorustest/user/rc/.sparkStaging/application_1499824261147_0015/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar
10:47:59,369  INFO Client:58 - Uploading resource file:/tmp/spark-ea70c397-fad0-44bc-ae1f-7248ed3f3003/__spark_conf__1134932846047586070.zip -> hdfs://chorustest/user/rc/.sparkStaging/application_1499824261147_0015/__spark_conf__1134932846047586070.zip
10:47:59,442  WARN Client:70 -
hdp.version is not found,
Please set HDP_VERSION=xxx in spark-env.sh,
or set -Dhdp.version=xxx in spark.{driver|yarn.am}.extraJavaOptions
or set SPARK_JAVA_OPTS="-Dhdp.verion=xxx" in spark-env.sh
If you're running Spark under HDP.

10:47:59,456  INFO SecurityManager:58 - Changing view acls to: rc
10:47:59,456  INFO SecurityManager:58 - Changing modify acls to: rc
10:47:59,456  INFO SecurityManager:58 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(rc); users with modify permissions: Set(rc)
10:47:59,463  INFO Client:58 - Submitting application 15 to ResourceManager
10:47:59,694  INFO YarnClientImpl:273 - Submitted application application_1499824261147_0015
java.lang.NoSuchMethodError: org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.bindToYarn(Lorg/apache/hadoop/yarn/api/records/ApplicationId;Lscala/Option;)V


在 2017年7月17日星期一 UTC+8下午5:53:32,spi...@...写道:

My graph has about 100 million vertexes and 200 million edges. But if use the following code, it is too slow.
GraphTraversal<Vertex, Long> countV = traversal.V().count();
while (countV.hasNext()){
System.out.println("countV:" + countV.next());
}

GraphTraversal<Edge, Long> countE = traversal.E().count();
while (countE.hasNext()){
System.out.println("countE:" + countE.next());
}

I want to computer the count of vertex or edge directly through Hbase. The following code is:
 SnapshotCounter.HBaseGetter entryGetter = new SnapshotCounter.HBaseGetter();
       
EntryList entryList = StaticArrayEntryList.ofBytes(
                result
.getMap().get(Bytes.toBytes("e")).entrySet(),
                entryGetter
);
       
StandardTitanTx tx = (StandardTitanTx) graph.newTransaction();
       
System.out.println("Entry list size: " + entryList.size());
       
int cnt = 0;
//        IDInspector inspector = graph.getIDInspector();
        for (Entry entry : entryList) {
           
RelationCache relation = graph.getEdgeSerializer().readRelation(entry, false, tx);
//            Direction direction = graph.getEdgeSerializer().parseDirection(entry);
//            System.out.println("Direction is:" + direction.name());
//            System.out.println("relation is:" + relation);

//            System.out.println("numProperties: " + relation.numProperties());
//            Iterator<LongObjectCursor<Object>> longObjectCursorIterator = relation.propertyIterator();
//            LongObjectCursor<Object> next = longObjectCursorIterator.next();
//            System.out.println("key is:" + next.key);
//            System.out.println("value is:" + next.value);
//            System.out.println("next.toString is:" + next.toString());



            RelationType type = tx.getExistingRelationType(relation.typeId);

           
Iterator<Edge> edgeIterator1 = type.edges(Direction.BOTH);
           
while (edgeIterator1.hasNext()){
               
Edge next11 = edgeIterator1.next();
               
System.out.println("relType is :" + next11.property("relType"));
           
}

//             if (type.isEdgeLabel() && !tx.getIdInspector().isEdgeLabelId(relation.relationId)){
//            if (type.isEdgeLabel() &&  !graph.getIDManager().isEdgeLabelId(relation.relationId) &&
//                    !tx.getIdInspector().isRelationTypeId(type.longId())) {
            if (type.isEdgeLabel() ) {
                cnt
++;
               
System.out.print("isSystemRelationTypeId: ");
               
System.out.println(graph.getIDManager().isSystemRelationTypeId(relation.typeId));

               
System.out.print("isEdgeLabelId: ");
               
System.out.println(graph.getIDManager().isEdgeLabelId(relation.typeId));

               
System.out.print("type isEdgeLabel: ");
               
System.out.println(type.isEdgeLabel());

               
System.out.print("relationId isSystemRelationTypeId: ");
               
System.out.println(graph.getIDManager().isSystemRelationTypeId(relation.relationId));
               
System.out.println(entry.getValue().toString());
           
}
       
}
       
System.out.println("Edge count: " + cnt);

I made a test by making a small graph-- two vertexes and two edges.


But I just get the count of the edge is one, expecting two. Is there any problem? Please help....This problem bugs me. Thanks~~~


Re: How Can I make a statistics,i.e:how many vertexes or edges?

spirit...@...
 

I config the gremlin and spark-on-yarn according to the post.
But I ran into the problem
10:47:47,199  INFO KryoShimServiceLoader:117 - Set KryoShimService provider to org.apache.tinkerpop.gremlin.hadoop.structure.io.HadoopPoolShimService@4b31a708 (class org.apache.tinkerpop.gremlin.hadoop.structure.io.HadoopPoolShimService) because its priority value (0) is the highest available
10:47:47,199  INFO KryoShimServiceLoader:123 - Configuring KryoShimService provider org.apache.tinkerpop.gremlin.hadoop.structure.io.HadoopPoolShimService@4b31a708 with user-provided configuration
10:47:51,447  INFO SparkContext:58 - Running Spark version 1.6.1
10:47:51,495  INFO SecurityManager:58 - Changing view acls to: rc
10:47:51,496  INFO SecurityManager:58 - Changing modify acls to: rc
10:47:51,496  INFO SecurityManager:58 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(rc); users with modify permissions: Set(rc)
10:47:51,855  INFO Utils:58 - Successfully started service 'sparkDriver' on port 41967.
10:47:52,450  INFO Slf4jLogger:80 - Slf4jLogger started
10:47:52,504  INFO Remoting:74 - Starting remoting
10:47:52,666  INFO Remoting:74 - Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@...:50605]
10:47:52,673  INFO Utils:58 - Successfully started service 'sparkDriverActorSystem' on port 50605.
10:47:53,428  INFO SparkEnv:58 - Registering MapOutputTracker
10:47:53,448  INFO SparkEnv:58 - Registering BlockManagerMaster
10:47:53,460  INFO DiskBlockManager:58 - Created local directory at /tmp/blockmgr-94bbe487-7cf4-4cf5-bcc2-fc538487f31a
10:47:53,473  INFO MemoryStore:58 - MemoryStore started with capacity 2.4 GB
10:47:53,591  INFO SparkEnv:58 - Registering OutputCommitCoordinator
10:47:53,755  INFO Server:272 - jetty-8.y.z-SNAPSHOT
10:47:53,809  INFO AbstractConnector:338 - Started SelectChannelConnector@0.0.0.0:4040
10:47:53,810  INFO Utils:58 - Successfully started service 'SparkUI' on port 4040.
10:47:53,813  INFO SparkUI:58 - Started SparkUI at http://10.200.48.112:4040
spark.yarn.driver.memoryOverhead is set but does not apply in client mode.
10:47:54,996  INFO TimelineClientImpl:296 - Timeline service address: http://dl-rc-optd-ambari-master-v-test-1.host.dataengine.com:8188/ws/v1/timeline/
10:47:55,307  INFO ConfiguredRMFailoverProxyProvider:100 - Failing over to rm2
10:47:55,333  INFO Client:58 - Requesting a new application from cluster with 8 NodeManagers
10:47:55,351  INFO Client:58 - Verifying our application has not requested more than the maximum memory capability of the cluster (10240 MB per container)
10:47:55,351  INFO Client:58 - Will allocate AM container, with 896 MB memory including 384 MB overhead
10:47:55,352  INFO Client:58 - Setting up container launch context for our AM
10:47:55,355  INFO Client:58 - Setting up the launch environment for our AM container
10:47:55,367  INFO Client:58 - Preparing resources for our AM container
10:47:56,298  INFO Client:58 - Uploading resource file:/rc/lib/spark_lib/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar -> hdfs://chorustest/user/rc/.sparkStaging/application_1499824261147_0015/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar
10:47:59,369  INFO Client:58 - Uploading resource file:/tmp/spark-ea70c397-fad0-44bc-ae1f-7248ed3f3003/__spark_conf__1134932846047586070.zip -> hdfs://chorustest/user/rc/.sparkStaging/application_1499824261147_0015/__spark_conf__1134932846047586070.zip
10:47:59,442  WARN Client:70 -
hdp.version is not found,
Please set HDP_VERSION=xxx in spark-env.sh,
or set -Dhdp.version=xxx in spark.{driver|yarn.am}.extraJavaOptions
or set SPARK_JAVA_OPTS="-Dhdp.verion=xxx" in spark-env.sh
If you're running Spark under HDP.

10:47:59,456  INFO SecurityManager:58 - Changing view acls to: rc
10:47:59,456  INFO SecurityManager:58 - Changing modify acls to: rc
10:47:59,456  INFO SecurityManager:58 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(rc); users with modify permissions: Set(rc)
10:47:59,463  INFO Client:58 - Submitting application 15 to ResourceManager
10:47:59,694  INFO YarnClientImpl:273 - Submitted application application_1499824261147_0015
java.lang.NoSuchMethodError: org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.bindToYarn(Lorg/apache/hadoop/yarn/api/records/ApplicationId;Lscala/Option;)V
 I lost the dependencies? or version conflicts?


在 2017年7月18日星期二 UTC+8下午10:13:53,Jason Plurad写道:

If you scroll back just a few days in the message history of this group, you'll find a link to this nice blog post: "Configuring JanusGraph for spark-yarn" https://groups.google.com/d/msg/janusgraph-users/9e82gcUTB4M/evKFnB3cAgAJ

HadoopMarc covers doing an OLAP vertex count with JanusGraph + HBase, which it sounds like what you're trying to do, and it has an example properties file.

I can't really tell what you're trying to do in that code snippet. It would be best if you would share the code publicly on GitHub or BitBucket or something similar so if somebody wanted to try it out, it would be easy to do.

> anybody online?  please help me~

JanusGraph is run by volunteers contributing to the open source project. Immediate responses may not happen. Using the mailing list and searching its archive is your best bet for learning from the community because there are several hundred folks that are subscribed to this list.


On Monday, July 17, 2017 at 11:35:47 PM UTC-4, spirit888hill wrote:
anybody online?  please help me~

在 2017年7月17日星期一 UTC+8下午5:53:32,写道:
My graph has about 100 million vertexes and 200 million edges. But if use the following code, it is too slow.
GraphTraversal<Vertex, Long> countV = traversal.V().count();
while (countV.hasNext()){
System.out.println("countV:" + countV.next());
}

GraphTraversal<Edge, Long> countE = traversal.E().count();
while (countE.hasNext()){
System.out.println("countE:" + countE.next());
}

I want to computer the count of vertex or edge directly through Hbase. The following code is:
 SnapshotCounter.HBaseGetter entryGetter = new SnapshotCounter.HBaseGetter();
       
EntryList entryList = StaticArrayEntryList.ofBytes(
                result
.getMap().get(Bytes.toBytes("e")).entrySet(),
                entryGetter
);
       
StandardTitanTx tx = (StandardTitanTx) graph.newTransaction();
       
System.out.println("Entry list size: " + entryList.size());
       
int cnt = 0;
//        IDInspector inspector = graph.getIDInspector();
        for (Entry entry : entryList) {
           
RelationCache relation = graph.getEdgeSerializer().readRelation(entry, false, tx);
//            Direction direction = graph.getEdgeSerializer().parseDirection(entry);
//            System.out.println("Direction is:" + direction.name());
//            System.out.println("relation is:" + relation);

//            System.out.println("numProperties: " + relation.numProperties());
//            Iterator<LongObjectCursor<Object>> longObjectCursorIterator = relation.propertyIterator();
//            LongObjectCursor<Object> next = longObjectCursorIterator.next();
//            System.out.println("key is:" + next.key);
//            System.out.println("value is:" + next.value);
//            System.out.println("next.toString is:" + next.toString());



            RelationType type = tx.getExistingRelationType(relation.typeId);

           
Iterator<Edge> edgeIterator1 = type.edges(Direction.BOTH);
           
while (edgeIterator1.hasNext()){
               
Edge next11 = edgeIterator1.next();
               
System.out.println("relType is :" + next11.property("relType"));
           
}

//             if (type.isEdgeLabel() && !tx.getIdInspector().isEdgeLabelId(relation.relationId)){
//            if (type.isEdgeLabel() &&  !graph.getIDManager().isEdgeLabelId(relation.relationId) &&
//                    !tx.getIdInspector().isRelationTypeId(type.longId())) {
            if (type.isEdgeLabel() ) {
                cnt
++;
               
System.out.print("isSystemRelationTypeId: ");
               
System.out.println(graph.getIDManager().isSystemRelationTypeId(relation.typeId));

               
System.out.print("isEdgeLabelId: ");
               
System.out.println(graph.getIDManager().isEdgeLabelId(relation.typeId));

               
System.out.print("type isEdgeLabel: ");
               
System.out.println(type.isEdgeLabel());

               
System.out.print("relationId isSystemRelationTypeId: ");
               
System.out.println(graph.getIDManager().isSystemRelationTypeId(relation.relationId));
               
System.out.println(entry.getValue().toString());
           
}
       
}
       
System.out.println("Edge count: " + cnt);

I made a test by making a small graph-- two vertexes and two edges.


But I just get the count of the edge is one, expecting two. Is there any problem? Please help....This problem bugs me. Thanks~~~


Re: Geoshape property in remote gremlin query, GraphSON

Robert Dale <rob...@...>
 

It seems Geoshape GraphSON support is hardcoded to v1 although I couldn't get it to work with that either.  If you have to use GraphSON instead of Gryo, then you could checkout master, apply this patch, and rebuild. I created an  issue to support multiple versions of serializers  https://github.com/JanusGraph/janusgraph/issues/420

diff --git a/janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/io/graphson/JanusGraphSONModule.java b/janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/io/graphson/JanusGraphSONModule.java
index
6ef907b..8168309 100644
--- a/janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/io/graphson/JanusGraphSONModule.java
+++ b/janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/io/graphson/JanusGraphSONModule.java
@@ -50,10 +50,10 @@ public class JanusGraphSONModule extends TinkerPopJacksonModule {
     
private JanusGraphSONModule() {
         
super("janusgraph");
         addSerializer
(RelationIdentifier.class, new RelationIdentifierSerializer());
-        addSerializer(Geoshape.class, new Geoshape.GeoshapeGsonSerializerV1d0());
+        addSerializer(Geoshape.class, new Geoshape.GeoshapeGsonSerializerV2d0());
 
         addDeserializer
(RelationIdentifier.class, new RelationIdentifierDeserializer());
-        addDeserializer(Geoshape.class, new Geoshape.GeoshapeGsonDeserializerV1d0());
+        addDeserializer(Geoshape.class, new Geoshape.GeoshapeGsonDeserializerV2d0());
     
}
 
     
private static final JanusGraphSONModule INSTANCE = new JanusGraphSONModule();



On Tuesday, July 18, 2017 at 5:47:50 PM UTC-4, Conrad Rosenbrock wrote:
I am trying to assign a value to a property with the native Geoshape type. I have it serialized into JSON as follows (where g is aliased to the traversal on gremlin server):

{"@value": {"type": "point", "coordinates": [{"@value": 1.1, "@type": "g:Double"}, {"@value": 2.2, "@type": "g:Double"}]}, "@type": "g:Geoshape"}

In the gremlin console, I can easily type 

Geoshape.point(1.1, 2.2)

and it works perfectly. I am sure that it is something quite simple. Here is the error:

Request [PooledUnsafeDirectByteBuf(ridx: 653, widx: 653, cap: 687)] could not be deserialized by org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV2d0.

For reference, I do have the following serializer in the gremlin server config:

{ className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV2d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}

which should direct gremlin server to the relevant deserializer in Janus.

Thanks!


Merging two graphs.

Gwiz <feed...@...>
 

Hello,

I have an in-memory Janusgraph instance and I would like to merge that with my Cassandra/ES based graph. What is the best (efficient) way to do this?

When I merge, I don't need to insert the vertices that are already in my Cassandra/ES based graph. I know which vertices in the in-memory graph exists in the Cassandra/ES based graph. 

I don't see any Java API to take a Vertex (or list of Vertices) from one Janusgraph instance and add them to another. All I see is the following:
  • addVertex(String)
  • addVertex(Object... objects)
My original plan is to insert the new vertices from in-memory graph into my Cassandra/ES based graph, get the IDs back and use them to insert Edges.

I appreciate any help.

Thanks.


Re: How Can I make a statistics,i.e:how many vertexes or edges?

spirit...@...
 

Oooops, I'm so sorry. HadoopMarc is a person.:)

在 2017年7月18日星期二 UTC+8下午10:13:53,Jason Plurad写道:

If you scroll back just a few days in the message history of this group, you'll find a link to this nice blog post: "Configuring JanusGraph for spark-yarn" https://groups.google.com/d/msg/janusgraph-users/9e82gcUTB4M/evKFnB3cAgAJ

HadoopMarc covers doing an OLAP vertex count with JanusGraph + HBase, which it sounds like what you're trying to do, and it has an example properties file.

I can't really tell what you're trying to do in that code snippet. It would be best if you would share the code publicly on GitHub or BitBucket or something similar so if somebody wanted to try it out, it would be easy to do.

> anybody online?  please help me~

JanusGraph is run by volunteers contributing to the open source project. Immediate responses may not happen. Using the mailing list and searching its archive is your best bet for learning from the community because there are several hundred folks that are subscribed to this list.


On Monday, July 17, 2017 at 11:35:47 PM UTC-4, spirit888hill wrote:
anybody online?  please help me~

在 2017年7月17日星期一 UTC+8下午5:53:32,写道:
My graph has about 100 million vertexes and 200 million edges. But if use the following code, it is too slow.
GraphTraversal<Vertex, Long> countV = traversal.V().count();
while (countV.hasNext()){
System.out.println("countV:" + countV.next());
}

GraphTraversal<Edge, Long> countE = traversal.E().count();
while (countE.hasNext()){
System.out.println("countE:" + countE.next());
}

I want to computer the count of vertex or edge directly through Hbase. The following code is:
 SnapshotCounter.HBaseGetter entryGetter = new SnapshotCounter.HBaseGetter();
       
EntryList entryList = StaticArrayEntryList.ofBytes(
                result
.getMap().get(Bytes.toBytes("e")).entrySet(),
                entryGetter
);
       
StandardTitanTx tx = (StandardTitanTx) graph.newTransaction();
       
System.out.println("Entry list size: " + entryList.size());
       
int cnt = 0;
//        IDInspector inspector = graph.getIDInspector();
        for (Entry entry : entryList) {
           
RelationCache relation = graph.getEdgeSerializer().readRelation(entry, false, tx);
//            Direction direction = graph.getEdgeSerializer().parseDirection(entry);
//            System.out.println("Direction is:" + direction.name());
//            System.out.println("relation is:" + relation);

//            System.out.println("numProperties: " + relation.numProperties());
//            Iterator<LongObjectCursor<Object>> longObjectCursorIterator = relation.propertyIterator();
//            LongObjectCursor<Object> next = longObjectCursorIterator.next();
//            System.out.println("key is:" + next.key);
//            System.out.println("value is:" + next.value);
//            System.out.println("next.toString is:" + next.toString());



            RelationType type = tx.getExistingRelationType(relation.typeId);

           
Iterator<Edge> edgeIterator1 = type.edges(Direction.BOTH);
           
while (edgeIterator1.hasNext()){
               
Edge next11 = edgeIterator1.next();
               
System.out.println("relType is :" + next11.property("relType"));
           
}

//             if (type.isEdgeLabel() && !tx.getIdInspector().isEdgeLabelId(relation.relationId)){
//            if (type.isEdgeLabel() &&  !graph.getIDManager().isEdgeLabelId(relation.relationId) &&
//                    !tx.getIdInspector().isRelationTypeId(type.longId())) {
            if (type.isEdgeLabel() ) {
                cnt
++;
               
System.out.print("isSystemRelationTypeId: ");
               
System.out.println(graph.getIDManager().isSystemRelationTypeId(relation.typeId));

               
System.out.print("isEdgeLabelId: ");
               
System.out.println(graph.getIDManager().isEdgeLabelId(relation.typeId));

               
System.out.print("type isEdgeLabel: ");
               
System.out.println(type.isEdgeLabel());

               
System.out.print("relationId isSystemRelationTypeId: ");
               
System.out.println(graph.getIDManager().isSystemRelationTypeId(relation.relationId));
               
System.out.println(entry.getValue().toString());
           
}
       
}
       
System.out.println("Edge count: " + cnt);

I made a test by making a small graph-- two vertexes and two edges.


But I just get the count of the edge is one, expecting two. Is there any problem? Please help....This problem bugs me. Thanks~~~


Re: How Can I make a statistics,i.e:how many vertexes or edges?

spirit...@...
 

Thanks Jason for your reply.
I'll show my code on GitHub later.
And I want to know what the HadoopMarc is? You mean Hadoop MapReduce??

在 2017年7月18日星期二 UTC+8下午10:13:53,Jason Plurad写道:

If you scroll back just a few days in the message history of this group, you'll find a link to this nice blog post: "Configuring JanusGraph for spark-yarn" https://groups.google.com/d/msg/janusgraph-users/9e82gcUTB4M/evKFnB3cAgAJ

HadoopMarc covers doing an OLAP vertex count with JanusGraph + HBase, which it sounds like what you're trying to do, and it has an example properties file.

I can't really tell what you're trying to do in that code snippet. It would be best if you would share the code publicly on GitHub or BitBucket or something similar so if somebody wanted to try it out, it would be easy to do.

> anybody online?  please help me~

JanusGraph is run by volunteers contributing to the open source project. Immediate responses may not happen. Using the mailing list and searching its archive is your best bet for learning from the community because there are several hundred folks that are subscribed to this list.


On Monday, July 17, 2017 at 11:35:47 PM UTC-4, spirit888hill wrote:
anybody online?  please help me~

在 2017年7月17日星期一 UTC+8下午5:53:32,写道:
My graph has about 100 million vertexes and 200 million edges. But if use the following code, it is too slow.
GraphTraversal<Vertex, Long> countV = traversal.V().count();
while (countV.hasNext()){
System.out.println("countV:" + countV.next());
}

GraphTraversal<Edge, Long> countE = traversal.E().count();
while (countE.hasNext()){
System.out.println("countE:" + countE.next());
}

I want to computer the count of vertex or edge directly through Hbase. The following code is:
 SnapshotCounter.HBaseGetter entryGetter = new SnapshotCounter.HBaseGetter();
       
EntryList entryList = StaticArrayEntryList.ofBytes(
                result
.getMap().get(Bytes.toBytes("e")).entrySet(),
                entryGetter
);
       
StandardTitanTx tx = (StandardTitanTx) graph.newTransaction();
       
System.out.println("Entry list size: " + entryList.size());
       
int cnt = 0;
//        IDInspector inspector = graph.getIDInspector();
        for (Entry entry : entryList) {
           
RelationCache relation = graph.getEdgeSerializer().readRelation(entry, false, tx);
//            Direction direction = graph.getEdgeSerializer().parseDirection(entry);
//            System.out.println("Direction is:" + direction.name());
//            System.out.println("relation is:" + relation);

//            System.out.println("numProperties: " + relation.numProperties());
//            Iterator<LongObjectCursor<Object>> longObjectCursorIterator = relation.propertyIterator();
//            LongObjectCursor<Object> next = longObjectCursorIterator.next();
//            System.out.println("key is:" + next.key);
//            System.out.println("value is:" + next.value);
//            System.out.println("next.toString is:" + next.toString());



            RelationType type = tx.getExistingRelationType(relation.typeId);

           
Iterator<Edge> edgeIterator1 = type.edges(Direction.BOTH);
           
while (edgeIterator1.hasNext()){
               
Edge next11 = edgeIterator1.next();
               
System.out.println("relType is :" + next11.property("relType"));
           
}

//             if (type.isEdgeLabel() && !tx.getIdInspector().isEdgeLabelId(relation.relationId)){
//            if (type.isEdgeLabel() &&  !graph.getIDManager().isEdgeLabelId(relation.relationId) &&
//                    !tx.getIdInspector().isRelationTypeId(type.longId())) {
            if (type.isEdgeLabel() ) {
                cnt
++;
               
System.out.print("isSystemRelationTypeId: ");
               
System.out.println(graph.getIDManager().isSystemRelationTypeId(relation.typeId));

               
System.out.print("isEdgeLabelId: ");
               
System.out.println(graph.getIDManager().isEdgeLabelId(relation.typeId));

               
System.out.print("type isEdgeLabel: ");
               
System.out.println(type.isEdgeLabel());

               
System.out.print("relationId isSystemRelationTypeId: ");
               
System.out.println(graph.getIDManager().isSystemRelationTypeId(relation.relationId));
               
System.out.println(entry.getValue().toString());
           
}
       
}
       
System.out.println("Edge count: " + cnt);

I made a test by making a small graph-- two vertexes and two edges.


But I just get the count of the edge is one, expecting two. Is there any problem? Please help....This problem bugs me. Thanks~~~


Geoshape property in remote gremlin query, GraphSON

rosen...@...
 

I am trying to assign a value to a property with the native Geoshape type. I have it serialized into JSON as follows (where g is aliased to the traversal on gremlin server):

{"@value": {"type": "point", "coordinates": [{"@value": 1.1, "@type": "g:Double"}, {"@value": 2.2, "@type": "g:Double"}]}, "@type": "g:Geoshape"}

In the gremlin console, I can easily type 

Geoshape.point(1.1, 2.2)

and it works perfectly. I am sure that it is something quite simple. Here is the error:

Request [PooledUnsafeDirectByteBuf(ridx: 653, widx: 653, cap: 687)] could not be deserialized by org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV2d0.

For reference, I do have the following serializer in the gremlin server config:

{ className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV2d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}

which should direct gremlin server to the relevant deserializer in Janus.

Thanks!


Re: How Can I make a statistics,i.e:how many vertexes or edges?

Jason Plurad <plu...@...>
 

If you scroll back just a few days in the message history of this group, you'll find a link to this nice blog post: "Configuring JanusGraph for spark-yarn" https://groups.google.com/d/msg/janusgraph-users/9e82gcUTB4M/evKFnB3cAgAJ

HadoopMarc covers doing an OLAP vertex count with JanusGraph + HBase, which it sounds like what you're trying to do, and it has an example properties file.

I can't really tell what you're trying to do in that code snippet. It would be best if you would share the code publicly on GitHub or BitBucket or something similar so if somebody wanted to try it out, it would be easy to do.

> anybody online?  please help me~

JanusGraph is run by volunteers contributing to the open source project. Immediate responses may not happen. Using the mailing list and searching its archive is your best bet for learning from the community because there are several hundred folks that are subscribed to this list.


On Monday, July 17, 2017 at 11:35:47 PM UTC-4, spirit888hill wrote:
anybody online?  please help me~

在 2017年7月17日星期一 UTC+8下午5:53:32,写道:
My graph has about 100 million vertexes and 200 million edges. But if use the following code, it is too slow.
GraphTraversal<Vertex, Long> countV = traversal.V().count();
while (countV.hasNext()){
System.out.println("countV:" + countV.next());
}

GraphTraversal<Edge, Long> countE = traversal.E().count();
while (countE.hasNext()){
System.out.println("countE:" + countE.next());
}

I want to computer the count of vertex or edge directly through Hbase. The following code is:
 SnapshotCounter.HBaseGetter entryGetter = new SnapshotCounter.HBaseGetter();
       
EntryList entryList = StaticArrayEntryList.ofBytes(
                result
.getMap().get(Bytes.toBytes("e")).entrySet(),
                entryGetter
);
       
StandardTitanTx tx = (StandardTitanTx) graph.newTransaction();
       
System.out.println("Entry list size: " + entryList.size());
       
int cnt = 0;
//        IDInspector inspector = graph.getIDInspector();
        for (Entry entry : entryList) {
           
RelationCache relation = graph.getEdgeSerializer().readRelation(entry, false, tx);
//            Direction direction = graph.getEdgeSerializer().parseDirection(entry);
//            System.out.println("Direction is:" + direction.name());
//            System.out.println("relation is:" + relation);

//            System.out.println("numProperties: " + relation.numProperties());
//            Iterator<LongObjectCursor<Object>> longObjectCursorIterator = relation.propertyIterator();
//            LongObjectCursor<Object> next = longObjectCursorIterator.next();
//            System.out.println("key is:" + next.key);
//            System.out.println("value is:" + next.value);
//            System.out.println("next.toString is:" + next.toString());



            RelationType type = tx.getExistingRelationType(relation.typeId);

           
Iterator<Edge> edgeIterator1 = type.edges(Direction.BOTH);
           
while (edgeIterator1.hasNext()){
               
Edge next11 = edgeIterator1.next();
               
System.out.println("relType is :" + next11.property("relType"));
           
}

//             if (type.isEdgeLabel() && !tx.getIdInspector().isEdgeLabelId(relation.relationId)){
//            if (type.isEdgeLabel() &&  !graph.getIDManager().isEdgeLabelId(relation.relationId) &&
//                    !tx.getIdInspector().isRelationTypeId(type.longId())) {
            if (type.isEdgeLabel() ) {
                cnt
++;
               
System.out.print("isSystemRelationTypeId: ");
               
System.out.println(graph.getIDManager().isSystemRelationTypeId(relation.typeId));

               
System.out.print("isEdgeLabelId: ");
               
System.out.println(graph.getIDManager().isEdgeLabelId(relation.typeId));

               
System.out.print("type isEdgeLabel: ");
               
System.out.println(type.isEdgeLabel());

               
System.out.print("relationId isSystemRelationTypeId: ");
               
System.out.println(graph.getIDManager().isSystemRelationTypeId(relation.relationId));
               
System.out.println(entry.getValue().toString());
           
}
       
}
       
System.out.println("Edge count: " + cnt);

I made a test by making a small graph-- two vertexes and two edges.


But I just get the count of the edge is one, expecting two. Is there any problem? Please help....This problem bugs me. Thanks~~~


Read partitioned vertex (ID=8202), but partitioned vertex filtering is disabled

spirit...@...
 

I make graph statistics by using hadoop-gremlin and SparkGraphComputer.

config file is:
gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph
gremlin.hadoop.graphInputFormat=org.janusgraph.hadoop.formats.hbase.HBaseInputFormat
#gremlin.hadoop.graphOutputFormat=org.apache.hadoop.mapreduce.lib.output.NullOutputFormat
gremlin.hadoop.graphOutputFormat=org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoOutputFormat

gremlin.hadoop.jarsInDistributedCache=true
gremlin.hadoop.inputLocation=none
gremlin.hadoop.outputLocation=output
#####
# hbase config
####
janusgraphmr.ioformat.conf.storage.backend=hbase
janusgraphmr.ioformat.conf.storage.hostname=dl-rc-optd-ambari-slave-v-test-1.host.dataengine.com,dl-rc-optd-ambari-slave-v-test-2.host.dataengine.com,dl-rc-optd-ambari-slave-v-test-3.host.dataengine.com
janusgraphmr.ioformat.conf.storage.port=2181
janusgraphmr.ioformat.conf.storage.hbase.table=bluesharp_hanxi1

#####################################
# GiraphGraphComputer Configuration #
#####################################
giraph.minWorkers=1
giraph.maxWorkers=1
giraph.SplitMasterWorker=false
giraph.useOutOfCoreGraph=true
giraph.useOutOfCoreMessages=true
mapred.map.child.java.opts=-Xmx1024m
mapred.reduce.child.java.opts=-Xmx1024m
giraph.numInputThreads=4
giraph.numComputeThreads=4
giraph.maxMessagesInMemory=100000
####################################
# SparkGraphComputer Configuration #
####################################
spark.master=local[4]
spark.executor.memory=1g
spark.serializer=org.apache.spark.serializer.KryoSerializer
#spark.kryo.registrator=org.janusgraph.hadoop.serialize.JanusGraphKryoRegistrator

But, I run into the problem, the following is:
19:20:58 ERROR org.apache.spark.scheduler.TaskSetManager  - Task 0 in stage 0.0 failed 1 times; aborting job
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.IllegalStateException: Read partitioned vertex (ID=8202), but partitioned vertex filtering is disabled.
at com.google.common.base.Preconditions.checkState(Preconditions.java:197)
at org.janusgraph.hadoop.formats.util.JanusGraphVertexDeserializer.readHadoopVertex(JanusGraphVertexDeserializer.java:84)
at org.janusgraph.hadoop.formats.util.GiraphRecordReader.nextKeyValue(GiraphRecordReader.java:60)
at org.apache.spark.rdd.NewHadoopRDD$$anon$1.hasNext(NewHadoopRDD.scala:168)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:29)
at org.apache.tinkerpop.gremlin.util.iterator.IteratorUtils$4.advance(IteratorUtils.java:298)
at org.apache.tinkerpop.gremlin.util.iterator.IteratorUtils$4.hasNext(IteratorUtils.java:269)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:388)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:144)
at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.fold(TraversableOnce.scala:199)
at scala.collection.AbstractIterator.fold(Iterator.scala:1157)
at org.apache.spark.rdd.RDD$$anonfun$fold$1$$anonfun$19.apply(RDD.scala:1086)
at org.apache.spark.rdd.RDD$$anonfun$fold$1$$anonfun$19.apply(RDD.scala:1086)
at org.apache.spark.SparkContext$$anonfun$36.apply(SparkContext.scala:1951)
at org.apache.spark.SparkContext$$anonfun$36.apply(SparkContext.scala:1951)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

Is my config file is wrong? `Cause I didn't find the config-template. AnyBody can help me ? Thank you very much


Re: How Can I make a statistics,i.e:how many vertexes or edges?

spirit...@...
 

anybody online?  please help me~

在 2017年7月17日星期一 UTC+8下午5:53:32,spi...@...写道:

My graph has about 100 million vertexes and 200 million edges. But if use the following code, it is too slow.
GraphTraversal<Vertex, Long> countV = traversal.V().count();
while (countV.hasNext()){
System.out.println("countV:" + countV.next());
}

GraphTraversal<Edge, Long> countE = traversal.E().count();
while (countE.hasNext()){
System.out.println("countE:" + countE.next());
}

I want to computer the count of vertex or edge directly through Hbase. The following code is:
 SnapshotCounter.HBaseGetter entryGetter = new SnapshotCounter.HBaseGetter();
       
EntryList entryList = StaticArrayEntryList.ofBytes(
                result
.getMap().get(Bytes.toBytes("e")).entrySet(),
                entryGetter
);
       
StandardTitanTx tx = (StandardTitanTx) graph.newTransaction();
       
System.out.println("Entry list size: " + entryList.size());
       
int cnt = 0;
//        IDInspector inspector = graph.getIDInspector();
        for (Entry entry : entryList) {
           
RelationCache relation = graph.getEdgeSerializer().readRelation(entry, false, tx);
//            Direction direction = graph.getEdgeSerializer().parseDirection(entry);
//            System.out.println("Direction is:" + direction.name());
//            System.out.println("relation is:" + relation);

//            System.out.println("numProperties: " + relation.numProperties());
//            Iterator<LongObjectCursor<Object>> longObjectCursorIterator = relation.propertyIterator();
//            LongObjectCursor<Object> next = longObjectCursorIterator.next();
//            System.out.println("key is:" + next.key);
//            System.out.println("value is:" + next.value);
//            System.out.println("next.toString is:" + next.toString());



            RelationType type = tx.getExistingRelationType(relation.typeId);

           
Iterator<Edge> edgeIterator1 = type.edges(Direction.BOTH);
           
while (edgeIterator1.hasNext()){
               
Edge next11 = edgeIterator1.next();
               
System.out.println("relType is :" + next11.property("relType"));
           
}

//             if (type.isEdgeLabel() && !tx.getIdInspector().isEdgeLabelId(relation.relationId)){
//            if (type.isEdgeLabel() &&  !graph.getIDManager().isEdgeLabelId(relation.relationId) &&
//                    !tx.getIdInspector().isRelationTypeId(type.longId())) {
            if (type.isEdgeLabel() ) {
                cnt
++;
               
System.out.print("isSystemRelationTypeId: ");
               
System.out.println(graph.getIDManager().isSystemRelationTypeId(relation.typeId));

               
System.out.print("isEdgeLabelId: ");
               
System.out.println(graph.getIDManager().isEdgeLabelId(relation.typeId));

               
System.out.print("type isEdgeLabel: ");
               
System.out.println(type.isEdgeLabel());

               
System.out.print("relationId isSystemRelationTypeId: ");
               
System.out.println(graph.getIDManager().isSystemRelationTypeId(relation.relationId));
               
System.out.println(entry.getValue().toString());
           
}
       
}
       
System.out.println("Edge count: " + cnt);

I made a test by making a small graph-- two vertexes and two edges.


But I just get the count of the edge is one, expecting two. Is there any problem? Please help....This problem bugs me. Thanks~~~


Re: Exception: IllegalStateException: The vertex or type is not associated with this transaction

arunab...@...
 

Thank you Robert! This helps :)

Arunabh


On Monday, July 17, 2017 at 4:32:23 PM UTC-7, Robert Dale wrote:
Don't mix the Graph API and the Traversal API.  Prefer the Traversal API.

// in two traversals
newNode = g.addV("B").property("x",x).property("y",y).property("z",z).property("w", w).next()
g.V().hasLabel("A").has("x",x).has("y").addE("rt").to(V(newNode)).iterate()

// In one traversal
g.addV("B").property("x",x).property("y",y).property("z",z).property("w", w).as("newNode").V().hasLabel("A").has("x",x).has("y").addE("rt").to("newNode").iterate()

g.tx().commit()



Robert Dale

On Mon, Jul 17, 2017 at 6:53 PM, aa5186 <aru...@...> wrote:
Hi all,

I am trying to add a new vertex and associate a new edge from it to an existing vertex. It doesn't allow me to add the edge and throws the exception:
java.lang.IllegalStateException: The vertex or type is not associated with this transaction [v[16552]]

```
List<Vertex> nodes = g.V().hasLabel("A").
has("x",x).
has("y",y).toList();
Vertex newNode = tx.addVertex(T.label,"B",
"x",x,
"y",y,
"z",z,
"w",w);
for(Vertex node: nodes){

node.addEdge("rt",newNode);
}
graph.tx().commit();
```
From what I understand, the "nodes" are not part of the tx transaction (...?) and therefore, the edge between "node" and "newNode" is not possible.
Can someone please point me in the right direction as I am new to Graph databases?

Thanks!

--
You received this message because you are subscribed to the Google Groups "JanusGraph users list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Exception: IllegalStateException: The vertex or type is not associated with this transaction

Robert Dale <rob...@...>
 

Don't mix the Graph API and the Traversal API.  Prefer the Traversal API.

// in two traversals
newNode = g.addV("B").property("x",x).property("y",y).property("z",z).property("w", w).next()
g.V().hasLabel("A").has("x",x).has("y").addE("rt").to(V(newNode)).iterate()

// In one traversal
g.addV("B").property("x",x).property("y",y).property("z",z).property("w", w).as("newNode").V().hasLabel("A").has("x",x).has("y").addE("rt").to("newNode").iterate()

g.tx().commit()



Robert Dale

On Mon, Jul 17, 2017 at 6:53 PM, aa5186 <arunab...@...> wrote:
Hi all,

I am trying to add a new vertex and associate a new edge from it to an existing vertex. It doesn't allow me to add the edge and throws the exception:
java.lang.IllegalStateException: The vertex or type is not associated with this transaction [v[16552]]

```
List<Vertex> nodes = g.V().hasLabel("A").
has("x",x).
has("y",y).toList();
Vertex newNode = tx.addVertex(T.label,"B",
"x",x,
"y",y,
"z",z,
"w",w);
for(Vertex node: nodes){

node.addEdge("rt",newNode);
}
graph.tx().commit();
```
From what I understand, the "nodes" are not part of the tx transaction (...?) and therefore, the edge between "node" and "newNode" is not possible.
Can someone please point me in the right direction as I am new to Graph databases?

Thanks!

--
You received this message because you are subscribed to the Google Groups "JanusGraph users list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6221 - 6240 of 6678