Date   

Re: JG Schema - addConnection seem to create duplicate connections

Peter Molnar
 

Hi Marc,

Thanks a lot for looking into this. As requested, I filled an issue about this on Github: https://github.com/JanusGraph/janusgraph/issues/2950

Thanks,
Peter


Re: Janusgraph embedded multi instance(JVM) data sync issue

Pawan Shriwas
 

Hi Marc,

I don't think data cache was created due to elastic search/mixed index only. I have seen this on basic property/node without index as well. I am thinking let's work on basic node/property then we can plan for mixed index cases.

Any suggestions for  basic case without an index backend?

Thanks,
Pawan


On Sat, Jan 15, 2022 at 5:16 PM <hadoopmarc@...> wrote:
Hi Pawan,

OK, let's investigate further. You say that the issue occurs for both vertex creation and modification. Let's take the clearest case first: vertex creation with an indexed property. So, in your system setup, if you have added a new vertex with embedded intance1, sometimes it takes a minute or more before a query for this vertex (based on its property value) on instance2 returns the vertex. This can only mean that the elasticserch index sometimes does not return the new property value. This on its turn means that an elasticsearch replica has not yet been synced with the data about the new vertex.

Indeed, the janusgraph-elastic configs have a key index.[X].elasticsearch.bulk-refresh (default: false) which can be set to any of the values in:
https://www.elastic.co/guide/en/elasticsearch/reference/7.16/docs-refresh.html

One can check the correspondence between this janusgraph config item and the elasticsearch API parameter in:
https://github.com/JanusGraph/janusgraph/blob/v0.6.0/janusgraph-es/src/main/java/org/janusgraph/diskstorage/es/rest/RestElasticSearchClient.java

So, can you see what happens with the other possible values for index.[X].elasticsearch.bulk-refresh?

Best wishes,    Marc



--
Thanks & Regard

PAWAN SHRIWAS


Re: JG Schema - addConnection seem to create duplicate connections

hadoopmarc@...
 

Hi Peter,

Thanks for reporting. I think it is a bug. I checked with the standalone gremlin REPL of janusgraph-0.6.0, using:
graph = JanusGraphFactory.open('conf/janusgraph-inmemory.properties')

This gives the same results and if you add the from toEdge connections first, the FromEdge gets 4 connections.

You can check that two of the four connections are redundant, that is, they refer to the same edge in the schema:

gremlin> edges[1].mappedConnections()
==>org.janusgraph.core.Connection@1fecfaea
==>org.janusgraph.core.Connection@4872669f
==>org.janusgraph.core.Connection@483f286e
==>org.janusgraph.core.Connection@4bb147ec
gremlin> edges[1].mappedConnections()[0].getConnectionEdge()
==>e[hs0-el-1th-st][525-~T$SchemaRelated->1037]
gremlin> edges[1].mappedConnections()[1].getConnectionEdge()
==>e[ikg-el-1th-171][525-~T$SchemaRelated->1549]
gremlin> edges[1].mappedConnections()[2].getConnectionEdge()
==>e[hs0-el-1th-st][525-~T$SchemaRelated->1037]
gremlin> edges[1].mappedConnections()[3].getConnectionEdge()
==>e[ikg-el-1th-171][525-~T$SchemaRelated->1549]

Finally, I checked that the schema results remain the same if you add the following config properties to the graph (as suggested by the ref docs):
schema.default=none
schema.constraints=true

Can you please report this as an issue on: https://github.com/JanusGraph/janusgraph/issues

Best wishes,   Marc


On Tue, Jan 11, 2022 at 01:06 PM, Peter Molnar wrote:
mgmt = graph.openManagement();


Re: Fastest way to check if a property key is mixed indexed or not

hadoopmarc@...
 

Hi Harshit,

The performance impact for JanusGraph when including a property key in multiple mixed indices, is negligable (the selection of the index for a specific query will be a tat slower). Additional mixed indices imply a heavier load on the indexing backend (in particular memory and storage, CPU during inserts) but with little impact on response times if the cluster is dimensioned properly.

Marc


Re: Janusgraph embedded multi instance(JVM) data sync issue

hadoopmarc@...
 

Hi Pawan,

OK, let's investigate further. You say that the issue occurs for both vertex creation and modification. Let's take the clearest case first: vertex creation with an indexed property. So, in your system setup, if you have added a new vertex with embedded intance1, sometimes it takes a minute or more before a query for this vertex (based on its property value) on instance2 returns the vertex. This can only mean that the elasticserch index sometimes does not return the new property value. This on its turn means that an elasticsearch replica has not yet been synced with the data about the new vertex.

Indeed, the janusgraph-elastic configs have a key index.[X].elasticsearch.bulk-refresh (default: false) which can be set to any of the values in:
https://www.elastic.co/guide/en/elasticsearch/reference/7.16/docs-refresh.html

One can check the correspondence between this janusgraph config item and the elasticsearch API parameter in:
https://github.com/JanusGraph/janusgraph/blob/v0.6.0/janusgraph-es/src/main/java/org/janusgraph/diskstorage/es/rest/RestElasticSearchClient.java

So, can you see what happens with the other possible values for index.[X].elasticsearch.bulk-refresh?

Best wishes,    Marc


Re: Fastest way to check if a property key is mixed indexed or not

Harshit Sharma
 

Will there be any performance impact if i will index a property key in multiple indices (mixed index)?


On Sat, 15 Jan, 2022, 3:55 pm , <hadoopmarc@...> wrote:
Hi Harshit,

The concept "property is indexed or not" is ambiguous because an index can have multiple property keys. If you want to know if there is an index with a specific property key as the only key, indeed you would have to do something like in your example code (but modified).

Best wishes,   Marc


Re: Fastest way to check if a property key is mixed indexed or not

hadoopmarc@...
 

Hi Harshit,

The concept "property is indexed or not" is ambiguous because an index can have multiple property keys. If you want to know if there is an index with a specific property key as the only key, indeed you would have to do something like in your example code (but modified).

Best wishes,   Marc


Re: Janusgraph embedded multi instance(JVM) data sync issue

Pawan Shriwas
 

Hi Marc,

I have removed cache properties from instances and we already have new transactions for each api operation but still facing stale data issues in other instances for some time.

Below is the code which is used for the new transaction for each operation.

In my embedded janusgraph service, We always create new translations for each api operation using below code and do commit or rollback at the end of api operation.  but sometimes it works and sometimes not. Is it a sync kind of issue which varies between graph instances in multiple services(JVM).

// Create graph instance code(once service start) 
  String filePath = ConfigUtils.getString(GraphConstants.GRAPH_FILE_PATH);
  JanusGraph graphinstance = embeddedConnection.open(filePath);

// create transaction code for each api operation
  JanusgraphTransaction threadedTransaction=  graphinstance.getGraphInstance().newTransaction();

// we do commit or rollback at end of each api operation
        threadedTransaction.commit();
                 //or 
        threadedTransaction.rollback();

Let me know if anything related to configuration or any code needs to tried for the same.

Thanks,
Pawan

On Fri, Jan 7, 2022 at 1:45 PM <hadoopmarc@...> wrote:
Hi Pawan,

Your requirement for instant synchronization cannot work with JanusGraph caches enabled, because JanusGraph will get data from the cache if available, instead of getting the latest data from the backend. So,

  • cache.db-cache = false
  • be sure to start a new transaction before querying for the latest data (e.g. by executing a g.tx().commit())
Best wishes,    Marc



--
Thanks & Regard

PAWAN SHRIWAS


Fastest way to check if a property key is mixed indexed or not

Harshit Sharma
 

Is there a way I can check if a particular property is indexed or not?

I know the following method but there I will have to traverse all indexes

List<JanusgraphIndex> indexList = mgmt.getIndexes(Vertex.class)
For(index : indexList){
  propertyKeys = index.getFieldKeys()
  if (propertyKeys.contains("KEY1")
      return true;
}
return false;

is there a better way to do the same?
--
Regards,

Harshit Sharma
+91-9901459920


Re: New Property keys in existing index getting stuck in registered state

Harshit Sharma
 

Is it allowed to index the same property key in two different indexes.
For example I created a property key graphId and created two indexes vertexIndex, edgeIndex.
Index graphId in both indexes.

The problem I'm facing is this graphId index is getting enabled in vertexIndex because I'm creating it first but it is getting stuck in REGISTERED state for edgeIndex

On Wed, Jan 12, 2022 at 6:38 AM Boxuan Li <liboxuan@...> wrote:
Can you post the stacktrace (or the place where NPE is thrown)?

On Wed, Jan 12, 2022 at 2:50 AM Harshit Sharma <harshit.sharma1080@...> wrote:
That is not working. According to Documentation https://docs.janusgraph.org/schema/index-management/index-performance/
After build index i'm calling ManagementSystem.awaitGraphIndexStatus(graph, INDEX_NAME).call()
but this call is throwing nullPointerException.

On Tue, Jan 11, 2022 at 8:18 PM Boxuan Li <liboxuan@...> wrote:
Hi Harshit,


On Tue, Jan 11, 2022 at 10:43 PM Harshit Sharma <harshit.sharma1080@...> wrote:
Sorry because of type posting again -

If I'm adding new keys to an existing index, many keys are getting stuck in the registered or installed state
for example, I indexed a key "graphId" for the vertex in an index "graphVertexIndex"
and I indexed the same key for edges in index names "graphEdgeIndex".
Now it is getting enabled in "graphVertexIndex" but getting stuck in the Registered state in "graphEdgeIndex"

On Tue, Jan 11, 2022 at 8:10 PM Harshit Sharma via lists.lfaidata.foundation <harshit.sharma1080=gmail.com@...> wrote:
If I'm adding new keys to an existing index, many keys are getting stuck in the registered or installed state
for example, I indexed a key "graphId" for the vertex in an index graphVertexIndex
while I indexed the same key for an edge in graphEdgeIndex
it is getting enabled in graphVertexIndex but stuck in the Registered state

--
Regards,

Harshit Sharma
+91-9901459920



--
Regards,

Harshit Sharma
+91-9901459920



--
Regards,

Harshit Sharma
+91-9901459920



--
Regards,

Harshit Sharma
+91-9901459920


Re: New Property keys in existing index getting stuck in registered state

Boxuan Li
 

Can you post the stacktrace (or the place where NPE is thrown)?

On Wed, Jan 12, 2022 at 2:50 AM Harshit Sharma <harshit.sharma1080@...> wrote:
That is not working. According to Documentation https://docs.janusgraph.org/schema/index-management/index-performance/
After build index i'm calling ManagementSystem.awaitGraphIndexStatus(graph, INDEX_NAME).call()
but this call is throwing nullPointerException.

On Tue, Jan 11, 2022 at 8:18 PM Boxuan Li <liboxuan@...> wrote:
Hi Harshit,


On Tue, Jan 11, 2022 at 10:43 PM Harshit Sharma <harshit.sharma1080@...> wrote:
Sorry because of type posting again -

If I'm adding new keys to an existing index, many keys are getting stuck in the registered or installed state
for example, I indexed a key "graphId" for the vertex in an index "graphVertexIndex"
and I indexed the same key for edges in index names "graphEdgeIndex".
Now it is getting enabled in "graphVertexIndex" but getting stuck in the Registered state in "graphEdgeIndex"

On Tue, Jan 11, 2022 at 8:10 PM Harshit Sharma via lists.lfaidata.foundation <harshit.sharma1080=gmail.com@...> wrote:
If I'm adding new keys to an existing index, many keys are getting stuck in the registered or installed state
for example, I indexed a key "graphId" for the vertex in an index graphVertexIndex
while I indexed the same key for an edge in graphEdgeIndex
it is getting enabled in graphVertexIndex but stuck in the Registered state

--
Regards,

Harshit Sharma
+91-9901459920



--
Regards,

Harshit Sharma
+91-9901459920



--
Regards,

Harshit Sharma
+91-9901459920


Re: New Property keys in existing index getting stuck in registered state

Harshit Sharma
 

That is not working. According to Documentation https://docs.janusgraph.org/schema/index-management/index-performance/
After build index i'm calling ManagementSystem.awaitGraphIndexStatus(graph, INDEX_NAME).call()
but this call is throwing nullPointerException.

On Tue, Jan 11, 2022 at 8:18 PM Boxuan Li <liboxuan@...> wrote:
Hi Harshit,


On Tue, Jan 11, 2022 at 10:43 PM Harshit Sharma <harshit.sharma1080@...> wrote:
Sorry because of type posting again -

If I'm adding new keys to an existing index, many keys are getting stuck in the registered or installed state
for example, I indexed a key "graphId" for the vertex in an index "graphVertexIndex"
and I indexed the same key for edges in index names "graphEdgeIndex".
Now it is getting enabled in "graphVertexIndex" but getting stuck in the Registered state in "graphEdgeIndex"

On Tue, Jan 11, 2022 at 8:10 PM Harshit Sharma via lists.lfaidata.foundation <harshit.sharma1080=gmail.com@...> wrote:
If I'm adding new keys to an existing index, many keys are getting stuck in the registered or installed state
for example, I indexed a key "graphId" for the vertex in an index graphVertexIndex
while I indexed the same key for an edge in graphEdgeIndex
it is getting enabled in graphVertexIndex but stuck in the Registered state

--
Regards,

Harshit Sharma
+91-9901459920



--
Regards,

Harshit Sharma
+91-9901459920



--
Regards,

Harshit Sharma
+91-9901459920


Re: Concurrent Transactions on JG Edges

Aman <amandeep.srivastava1996@...>
 

Hi,

Can someone please advise if previously mentioned way is incorrect for using lock consistencies on edge property keys (version in my case)?

Regards,
Aman

On Mon, 10 Jan, 2022, 10:26 pm Aman via lists.lfaidata.foundation, <amandeep.srivastava1996=gmail.com@...> wrote:
Hi Marc,

Thank you for your response. This is how I'm using it:

1. I created a property key called version, which is incremented at the service layer every time an update is made on an element (vertex or edge). 

mgmt = graph.openManagement()
version = mgmt.makePropertyKey('version').dataType(Integer.class).make()
mgmt.setConsistency(name, ConsistencyModifier.LOCK)
mgmt.buildIndex("vertexIndex", Vertex.class).addKey(version, Mapping.DEFAULT.asParameter()).buildMixedIndex("esIndex")
mgmt.buildIndex("edgeIndex", Edge.class).addKey(version, Mapping.DEFAULT.asParameter()).buildMixedIndex("esIndex")
mgmt.commit()

In case mixed index already exists, I simply add the property in both vertex and edge mixed indices:
index = mgmt.getGraphIndex("edgeIndex or vertexIndex")
mgmt.addIndexKey(index, version)
In case

2. For every new edge label I create, I'm attaching this property to the edge.

mgmt = graph.openManagement() relatedTo = mgmt.makeEdgeLabel('relatedTo').make() mgmt.addProperties(relatedTo, version) mgmt.commit()

How I tested: When I'm making multiple concurrent calls via Jmeter to update an existing vertex, only one request passes, and rest throw a permanent locking exception. However, when I run same test for edges, multiple updates are successful. Few that fail are because of version check logic at service layer which kicks in a few ms after first few updates are successful.

I'm using Janus with Cassandra and Elastic Search.

Regards,

Aman


Re: New Property keys in existing index getting stuck in registered state

Boxuan Li
 

Hi Harshit,


On Tue, Jan 11, 2022 at 10:43 PM Harshit Sharma <harshit.sharma1080@...> wrote:
Sorry because of type posting again -

If I'm adding new keys to an existing index, many keys are getting stuck in the registered or installed state
for example, I indexed a key "graphId" for the vertex in an index "graphVertexIndex"
and I indexed the same key for edges in index names "graphEdgeIndex".
Now it is getting enabled in "graphVertexIndex" but getting stuck in the Registered state in "graphEdgeIndex"

On Tue, Jan 11, 2022 at 8:10 PM Harshit Sharma via lists.lfaidata.foundation <harshit.sharma1080=gmail.com@...> wrote:
If I'm adding new keys to an existing index, many keys are getting stuck in the registered or installed state
for example, I indexed a key "graphId" for the vertex in an index graphVertexIndex
while I indexed the same key for an edge in graphEdgeIndex
it is getting enabled in graphVertexIndex but stuck in the Registered state

--
Regards,

Harshit Sharma
+91-9901459920



--
Regards,

Harshit Sharma
+91-9901459920


Re: New Property keys in existing index getting stuck in registered state

Harshit Sharma
 

Sorry because of type posting again -

If I'm adding new keys to an existing index, many keys are getting stuck in the registered or installed state
for example, I indexed a key "graphId" for the vertex in an index "graphVertexIndex"
and I indexed the same key for edges in index names "graphEdgeIndex".
Now it is getting enabled in "graphVertexIndex" but getting stuck in the Registered state in "graphEdgeIndex"


On Tue, Jan 11, 2022 at 8:10 PM Harshit Sharma via lists.lfaidata.foundation <harshit.sharma1080=gmail.com@...> wrote:
If I'm adding new keys to an existing index, many keys are getting stuck in the registered or installed state
for example, I indexed a key "graphId" for the vertex in an index graphVertexIndex
while I indexed the same key for an edge in graphEdgeIndex
it is getting enabled in graphVertexIndex but stuck in the Registered state

--
Regards,

Harshit Sharma
+91-9901459920



--
Regards,

Harshit Sharma
+91-9901459920


New Property keys in existing index getting stuck in registered state

Harshit Sharma
 

If I'm adding new keys to an existing index, many keys are getting stuck in the registered or installed state
for example, I indexed a key "graphId" for the vertex in an index graphVertexIndex
while I indexed the same key for an edge in graphEdgeIndex
it is getting enabled in graphVertexIndex but stuck in the Registered state

--
Regards,

Harshit Sharma
+91-9901459920


JG Schema - addConnection seem to create duplicate connections

Peter Molnar
 

Hi All,

I have a strange behaviour while using the addConnection method for creating JG schema constraints. It seems it creates duplicated connections in some cases. Please see below how to reproduce this with JG v0.6 and Cassandra v3.11.11 backend. I just used the below snippets in Gremlin Console 3.5.1 to connect to JG remotely.

The graph is supposed model transactions and parties involved in the transaction. It has transaction, person and entity nodes. A transaction is supposed to have one "from party" (either person or entity) connected with FROM edge and one "to party" (either person or entity) connected with TO edge. For example (Person A) --FROM--> (Transaction #1) --TO--> (Entity A) for expressing that "Transaction #1" was performed between "Person A" and "Entity A" and the source of the transaction was "Person A".

Creating dynamic graph and schema
-------
map = new HashMap();
map.put("storage.backend","cql");
map.put("storage.hostname","cassandra");
map.put("query.force-index", "false");
map.put("schema.default", "default");
map.put("schema.constraints", "false");
map.put("graph.graphname", "transactionGraph")
ConfiguredGraphFactory.createConfiguration(new MapConfiguration(map));
 
graph = ConfiguredGraphFactory.open("transactionGraph");
mgmt = graph.openManagement();
 
transaction = mgmt.makeVertexLabel('transaction').make();
person = mgmt.makeVertexLabel('person').make();
entity = mgmt.makeVertexLabel('entity').make();
 
fromEdge = mgmt.makeEdgeLabel('FROM').multiplicity(ONE2ONE).make()
mgmt.addConnection(fromEdge, person, transaction)
mgmt.addConnection(fromEdge, entity, transaction)
 
toEdge = mgmt.makeEdgeLabel('TO').multiplicity(ONE2ONE).make()
mgmt.addConnection(toEdge, transaction, person)
mgmt.addConnection(toEdge, transaction, entity)
 
mgmt.commit()

Checking connections of the schema
----------------
mgmt = graph.openManagement()
edges = mgmt.getRelationTypes(EdgeLabel.class)
fromEdge = edges[0]
toEdge = edges[1]
 
fromEdge.mappedConnections().size() // as I would expect, it has two connections
toEdge.mappedConnections().size() // why 4 connections are here? I would expect only two connections similarly to the FROM edge
 
mgmt.close()

--------------

Could you please have a look and let me know if this is a feature or a bug?

Thanks, Peter


Re: Janusgraph not able to find suitable index for a index enabled property key

Harshit Sharma
 

I think I found the issue, I was using default mapping for all properties which are created in separate transactions.

I have a few questions -

1. Does JG support TEXTSTRING mapping SET cardinality?
2. Just want to be sure, if Default Mapping is equal to TEXT Mapping?

On Mon, Jan 10, 2022 at 10:08 PM Harshit Sharma via lists.lfaidata.foundation <harshit.sharma1080=gmail.com@...> wrote:
It looks something like this ->


graph_search_graph-index-edge: {

mappings: {

- properties: {
   - all: { 
        type: "text"
      }, 
- graphId: {
     type: "text",
     -copy_to: [ "all"]
   }
- graphId__STRING: { type: "keyword" }

- svc_edge_key_s: {
      type: "text", 
      - copy_to: [ "all"]
}
}
}

On Mon, Jan 10, 2022 at 9:46 PM <sergeymetallic@...> wrote:
Could you please provide a ES mapping that was generated for this property?



--
Regards,

Harshit Sharma
+91-9901459920



--
Regards,

Harshit Sharma
+91-9901459920


Re: Concurrent Transactions on JG Edges

Aman <amandeep.srivastava1996@...>
 

Hi Marc,

Thank you for your response. This is how I'm using it:

1. I created a property key called version, which is incremented at the service layer every time an update is made on an element (vertex or edge). 

mgmt = graph.openManagement()
version = mgmt.makePropertyKey('version').dataType(Integer.class).make()
mgmt.setConsistency(name, ConsistencyModifier.LOCK)
mgmt.buildIndex("vertexIndex", Vertex.class).addKey(version, Mapping.DEFAULT.asParameter()).buildMixedIndex("esIndex")
mgmt.buildIndex("edgeIndex", Edge.class).addKey(version, Mapping.DEFAULT.asParameter()).buildMixedIndex("esIndex")
mgmt.commit()

In case mixed index already exists, I simply add the property in both vertex and edge mixed indices:
index = mgmt.getGraphIndex("edgeIndex or vertexIndex")
mgmt.addIndexKey(index, version)
In case

2. For every new edge label I create, I'm attaching this property to the edge.

mgmt = graph.openManagement() relatedTo = mgmt.makeEdgeLabel('relatedTo').make() mgmt.addProperties(relatedTo, version) mgmt.commit()

How I tested: When I'm making multiple concurrent calls via Jmeter to update an existing vertex, only one request passes, and rest throw a permanent locking exception. However, when I run same test for edges, multiple updates are successful. Few that fail are because of version check logic at service layer which kicks in a few ms after first few updates are successful.

I'm using Janus with Cassandra and Elastic Search.

Regards,

Aman


Re: Janusgraph not able to find suitable index for a index enabled property key

Harshit Sharma
 

It looks something like this ->


graph_search_graph-index-edge: {

mappings: {

- properties: {
   - all: { 
        type: "text"
      }, 
- graphId: {
     type: "text",
     -copy_to: [ "all"]
   }
- graphId__STRING: { type: "keyword" }

- svc_edge_key_s: {
      type: "text", 
      - copy_to: [ "all"]
}
}
}

On Mon, Jan 10, 2022 at 9:46 PM <sergeymetallic@...> wrote:
Could you please provide a ES mapping that was generated for this property?



--
Regards,

Harshit Sharma
+91-9901459920

241 - 260 of 6588