Date   

Re: Janusgraph not able to find suitable index for a index enabled property key

Harshit Sharma
 

sorry for the typo,  it is addIndexKey() only what I'm using. 

Example-

Created a vertex label -> vertexA
created a property key -> vertexA_key_s
create mixedIndex if not exist -> vertex_index 
add property key "vertexA_key_s" in vertex_index

Created a vertex label -> vertexB
created a proproperty key -> vertexB_key_s
create mixedIndex if not exist -> vertex_index 
add property key "vertexB_key_s" in vertex_index

Created a edge label -> edge1
created a proproperty key -> edge1_key_s
create mixedIndex if not exist -> edge_index 

*Note -> We are crating index (vertex_index, edge_index) on the start of the service with some system properties like graphID, name, etc)

PrintIndexes in janusgraph -

Vertex Index Name. |    Type.  | Unique.  | Backing.   | key:   Status
--------------------------------------------------------------------------------------------
vertex_index. |    Mixed | false.  | search.   | vertexA_key_s: ENABLED
| vertexB_key_s: ENABLED


Edge Index (VCI) Name. |    Type.  | Unique.  | Backing.   | key  | Status
--------------------------------------------------------------------------------------------
edge_index. |    Mixed | false.  | search.   | vedge1_key_s: ENABLED


In elastic i'm able to search on that property using endpoint /_search?q=oltp_vertexA_key_s




On Mon, Jan 10, 2022 at 12:51 PM Boxuan Li <liboxuan@...> wrote:
Hi Harshit,

Thanks for the explanation. I noticed your example code contains the following snippet:

if(existingIndex) 
                janusgraphIndex = mgmt.getGraphIndex(VERTEX_INDEX_NAME)
                mgmt.addIndex(janusgraphIndex, propertyKey)

I don’t have a computer with me right now, but IIRC there is no “addIndex” API. Do you mean by “addIndexKey”?

Another question: let’s say you create a new mixed index with property A. Later, you add a new property B to it using “mgmt.addIndexKey” API. Then you create a vertex with both property A and property B. Now, are you only able to leverage the mixed index to query property A only, but not property B? Also, what new document(s) do you observe on ES? A concrete example would be greatly appreciated if possible.

Best,
Boxuan

On Mon, Jan 10, 2022 at 3:04 PM Harshit Sharma <harshit.sharma1080@...> wrote:
Hi Boxuan,

Thanks for the quick response. Actually, I'm following a strict schema in my service. 
i.e we first register a schema and then use it for vertex/edge creation.

So I'm not creating schema and vertex/edge in the same transaction. Only property key and index creation are happening in one transaction that is also at starting of the service only. From next time is index already exists then my service adds new properties to it.

These new properties are created after the creation of the index in a separate transaction are visible in management API printIndexes and printSchema. Also, they are visible in elastic mapping.



On Mon, Jan 10, 2022 at 12:26 PM Boxuan Li <liboxuan@...> wrote:
Sorry I misread your email. I thought you were using TEXT mapping only.

You said you were able to create index, add vertices and edges, and query them utilizing index in the same transaction, but not in a new transaction. Can you give complete code examples that demonstrate how you do so in the same transaction AND in a new transaction, as Marc asked? Specifically, do include how you open and commit/rollback transactions.

In addition, I wonder if this always happens no matter how long you wait, or only happens in a short period of time. Schema propagation takes some time, and you may see a sync delay across JanusGraph instances. If you create the index on one JanusGraph instance, and immediately insert & query data in another JanusGraph instance, it might be the case that schema propagation is not done yet and your other JanusGraph instance is not aware of the recently created index.

Another thing you could check is the Elasticsearch. After creating the index, do you see the index on your ES cluster? After inserting vertices/edges, do you see documents created at ES side?

Best,
Boxuan

On Mon, Jan 10, 2022 at 1:41 PM Harshit Sharma <harshit.sharma1080@...> wrote:
Only For SET cardinality, I'm using TEXT mapping because looks like SET does not work with TEXTSTRING.

On Mon, Jan 10, 2022 at 11:07 AM Harshit Sharma via lists.lfaidata.foundation <harshit.sharma1080=gmail.com@...> wrote:
but is that the reason I'm not able to use property keys created in a new transaction?
Also, can we mix index property keys of cardinality SET?

On Mon, Jan 10, 2022 at 10:56 AM Boxuan Li <liboxuan@...> wrote:
Hi Harshit,

You should use STRING mapping if you want exact string match. TEXT mapping is for full text search. See 

Best,
Boxuan

On Mon, Jan 10, 2022 at 1:15 PM Harshit Sharma <harshit.sharma1080@...> wrote:
Hi Marc,

1. Creation of property keys and indices in the schema + commit
      graph = (Janusgraph) GraphDBConfigurator.getGraph()
      mgmt = graph.openManagement()
      
     For Vertex ->
           propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SET).make()
          if(existingIndex) 
                janusgraphIndex = mgmt.getGraphIndex(VERTEX_INDEX_NAME)
                mgmt.addIndex(janusgraphIndex, propertyKey)
          else
               indexBuilder = mgmt.buildIndex(VERTEX_INDEX_NAME,  Vertex.class)
               if(cardinality is SET)
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXT.asParameter() : Mapping.DEFAULT.asParameter())
               else
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXTSTRING.asParameter() : Mapping.DEFAULT.asParameter())
  
    For Edge ->
            propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SINGLE).make()
             propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SINGLE).make()
          if(existingIndex) 
                janusgraphIndex = mgmt.getGraphIndex(EDGE_INDEX_NAME)
                mgmt.addIndex(janusgraphIndex, propertyKey)
          else
               indexBuilder = mgmt.buildIndex(EDGE_INDEX_NAME,  Vertex.class)
               if(cardinality is SET)
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXT.asParameter() : Mapping.DEFAULT.asParameter())
               else
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXTSTRING.asParameter() : Mapping.DEFAULT.asParameter())
         
            mgmt.commit();
    


       3.   Query of a vertex based on a property value in another transaction
             Let say i created a property key "domain_vertex_key1_s" for vertex
             Query = g.V().has("domain_vertex_key1_s", "val1);
             
             Let say I created a property key "domain_edge_key1_s" for edge
            
             Query = g.V().has("domain_vertex_key1_s","val2");
          

On Mon, 10 Jan, 2022, 12:46 am , <hadoopmarc@...> wrote:
Hi Harshit,

Can you please describe the steps you have taken in more detail:
  • creation of property keys and indices in the schema + commit,
  • creation of a vertex with a property in one committed transaction,
  • query of a vertex based on a property value in another transaction.

Marc



--
Regards,

Harshit Sharma
+91-9901459920



--
Regards,

Harshit Sharma
+91-9901459920



--
Regards,

Harshit Sharma
+91-9901459920



--
Regards,

Harshit Sharma
+91-9901459920


Re: Janusgraph not able to find suitable index for a index enabled property key

Boxuan Li
 

Hi Harshit,

Thanks for the explanation. I noticed your example code contains the following snippet:

if(existingIndex) 
                janusgraphIndex = mgmt.getGraphIndex(VERTEX_INDEX_NAME)
                mgmt.addIndex(janusgraphIndex, propertyKey)

I don’t have a computer with me right now, but IIRC there is no “addIndex” API. Do you mean by “addIndexKey”?

Another question: let’s say you create a new mixed index with property A. Later, you add a new property B to it using “mgmt.addIndexKey” API. Then you create a vertex with both property A and property B. Now, are you only able to leverage the mixed index to query property A only, but not property B? Also, what new document(s) do you observe on ES? A concrete example would be greatly appreciated if possible.

Best,
Boxuan

On Mon, Jan 10, 2022 at 3:04 PM Harshit Sharma <harshit.sharma1080@...> wrote:
Hi Boxuan,

Thanks for the quick response. Actually, I'm following a strict schema in my service. 
i.e we first register a schema and then use it for vertex/edge creation.

So I'm not creating schema and vertex/edge in the same transaction. Only property key and index creation are happening in one transaction that is also at starting of the service only. From next time is index already exists then my service adds new properties to it.

These new properties are created after the creation of the index in a separate transaction are visible in management API printIndexes and printSchema. Also, they are visible in elastic mapping.



On Mon, Jan 10, 2022 at 12:26 PM Boxuan Li <liboxuan@...> wrote:
Sorry I misread your email. I thought you were using TEXT mapping only.

You said you were able to create index, add vertices and edges, and query them utilizing index in the same transaction, but not in a new transaction. Can you give complete code examples that demonstrate how you do so in the same transaction AND in a new transaction, as Marc asked? Specifically, do include how you open and commit/rollback transactions.

In addition, I wonder if this always happens no matter how long you wait, or only happens in a short period of time. Schema propagation takes some time, and you may see a sync delay across JanusGraph instances. If you create the index on one JanusGraph instance, and immediately insert & query data in another JanusGraph instance, it might be the case that schema propagation is not done yet and your other JanusGraph instance is not aware of the recently created index.

Another thing you could check is the Elasticsearch. After creating the index, do you see the index on your ES cluster? After inserting vertices/edges, do you see documents created at ES side?

Best,
Boxuan

On Mon, Jan 10, 2022 at 1:41 PM Harshit Sharma <harshit.sharma1080@...> wrote:
Only For SET cardinality, I'm using TEXT mapping because looks like SET does not work with TEXTSTRING.

On Mon, Jan 10, 2022 at 11:07 AM Harshit Sharma via lists.lfaidata.foundation <harshit.sharma1080=gmail.com@...> wrote:
but is that the reason I'm not able to use property keys created in a new transaction?
Also, can we mix index property keys of cardinality SET?

On Mon, Jan 10, 2022 at 10:56 AM Boxuan Li <liboxuan@...> wrote:
Hi Harshit,

You should use STRING mapping if you want exact string match. TEXT mapping is for full text search. See 

Best,
Boxuan

On Mon, Jan 10, 2022 at 1:15 PM Harshit Sharma <harshit.sharma1080@...> wrote:
Hi Marc,

1. Creation of property keys and indices in the schema + commit
      graph = (Janusgraph) GraphDBConfigurator.getGraph()
      mgmt = graph.openManagement()
      
     For Vertex ->
           propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SET).make()
          if(existingIndex) 
                janusgraphIndex = mgmt.getGraphIndex(VERTEX_INDEX_NAME)
                mgmt.addIndex(janusgraphIndex, propertyKey)
          else
               indexBuilder = mgmt.buildIndex(VERTEX_INDEX_NAME,  Vertex.class)
               if(cardinality is SET)
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXT.asParameter() : Mapping.DEFAULT.asParameter())
               else
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXTSTRING.asParameter() : Mapping.DEFAULT.asParameter())
  
    For Edge ->
            propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SINGLE).make()
             propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SINGLE).make()
          if(existingIndex) 
                janusgraphIndex = mgmt.getGraphIndex(EDGE_INDEX_NAME)
                mgmt.addIndex(janusgraphIndex, propertyKey)
          else
               indexBuilder = mgmt.buildIndex(EDGE_INDEX_NAME,  Vertex.class)
               if(cardinality is SET)
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXT.asParameter() : Mapping.DEFAULT.asParameter())
               else
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXTSTRING.asParameter() : Mapping.DEFAULT.asParameter())
         
            mgmt.commit();
    


       3.   Query of a vertex based on a property value in another transaction
             Let say i created a property key "domain_vertex_key1_s" for vertex
             Query = g.V().has("domain_vertex_key1_s", "val1);
             
             Let say I created a property key "domain_edge_key1_s" for edge
            
             Query = g.V().has("domain_vertex_key1_s","val2");
          

On Mon, 10 Jan, 2022, 12:46 am , <hadoopmarc@...> wrote:
Hi Harshit,

Can you please describe the steps you have taken in more detail:
  • creation of property keys and indices in the schema + commit,
  • creation of a vertex with a property in one committed transaction,
  • query of a vertex based on a property value in another transaction.

Marc



--
Regards,

Harshit Sharma
+91-9901459920



--
Regards,

Harshit Sharma
+91-9901459920



--
Regards,

Harshit Sharma
+91-9901459920


Re: Janusgraph not able to find suitable index for a index enabled property key

Harshit Sharma
 

Hi Boxuan,

Thanks for the quick response. Actually, I'm following a strict schema in my service. 
i.e we first register a schema and then use it for vertex/edge creation.

So I'm not creating schema and vertex/edge in the same transaction. Only property key and index creation are happening in one transaction that is also at starting of the service only. From next time is index already exists then my service adds new properties to it.

These new properties are created after the creation of the index in a separate transaction are visible in management API printIndexes and printSchema. Also, they are visible in elastic mapping.



On Mon, Jan 10, 2022 at 12:26 PM Boxuan Li <liboxuan@...> wrote:
Sorry I misread your email. I thought you were using TEXT mapping only.

You said you were able to create index, add vertices and edges, and query them utilizing index in the same transaction, but not in a new transaction. Can you give complete code examples that demonstrate how you do so in the same transaction AND in a new transaction, as Marc asked? Specifically, do include how you open and commit/rollback transactions.

In addition, I wonder if this always happens no matter how long you wait, or only happens in a short period of time. Schema propagation takes some time, and you may see a sync delay across JanusGraph instances. If you create the index on one JanusGraph instance, and immediately insert & query data in another JanusGraph instance, it might be the case that schema propagation is not done yet and your other JanusGraph instance is not aware of the recently created index.

Another thing you could check is the Elasticsearch. After creating the index, do you see the index on your ES cluster? After inserting vertices/edges, do you see documents created at ES side?

Best,
Boxuan

On Mon, Jan 10, 2022 at 1:41 PM Harshit Sharma <harshit.sharma1080@...> wrote:
Only For SET cardinality, I'm using TEXT mapping because looks like SET does not work with TEXTSTRING.

On Mon, Jan 10, 2022 at 11:07 AM Harshit Sharma via lists.lfaidata.foundation <harshit.sharma1080=gmail.com@...> wrote:
but is that the reason I'm not able to use property keys created in a new transaction?
Also, can we mix index property keys of cardinality SET?

On Mon, Jan 10, 2022 at 10:56 AM Boxuan Li <liboxuan@...> wrote:
Hi Harshit,

You should use STRING mapping if you want exact string match. TEXT mapping is for full text search. See 

Best,
Boxuan

On Mon, Jan 10, 2022 at 1:15 PM Harshit Sharma <harshit.sharma1080@...> wrote:
Hi Marc,

1. Creation of property keys and indices in the schema + commit
      graph = (Janusgraph) GraphDBConfigurator.getGraph()
      mgmt = graph.openManagement()
      
     For Vertex ->
           propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SET).make()
          if(existingIndex) 
                janusgraphIndex = mgmt.getGraphIndex(VERTEX_INDEX_NAME)
                mgmt.addIndex(janusgraphIndex, propertyKey)
          else
               indexBuilder = mgmt.buildIndex(VERTEX_INDEX_NAME,  Vertex.class)
               if(cardinality is SET)
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXT.asParameter() : Mapping.DEFAULT.asParameter())
               else
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXTSTRING.asParameter() : Mapping.DEFAULT.asParameter())
  
    For Edge ->
            propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SINGLE).make()
             propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SINGLE).make()
          if(existingIndex) 
                janusgraphIndex = mgmt.getGraphIndex(EDGE_INDEX_NAME)
                mgmt.addIndex(janusgraphIndex, propertyKey)
          else
               indexBuilder = mgmt.buildIndex(EDGE_INDEX_NAME,  Vertex.class)
               if(cardinality is SET)
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXT.asParameter() : Mapping.DEFAULT.asParameter())
               else
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXTSTRING.asParameter() : Mapping.DEFAULT.asParameter())
         
            mgmt.commit();
    


       3.   Query of a vertex based on a property value in another transaction
             Let say i created a property key "domain_vertex_key1_s" for vertex
             Query = g.V().has("domain_vertex_key1_s", "val1);
             
             Let say I created a property key "domain_edge_key1_s" for edge
            
             Query = g.V().has("domain_vertex_key1_s","val2");
          

On Mon, 10 Jan, 2022, 12:46 am , <hadoopmarc@...> wrote:
Hi Harshit,

Can you please describe the steps you have taken in more detail:
  • creation of property keys and indices in the schema + commit,
  • creation of a vertex with a property in one committed transaction,
  • query of a vertex based on a property value in another transaction.

Marc



--
Regards,

Harshit Sharma
+91-9901459920



--
Regards,

Harshit Sharma
+91-9901459920



--
Regards,

Harshit Sharma
+91-9901459920


Re: Janusgraph not able to find suitable index for a index enabled property key

Boxuan Li
 

Sorry I misread your email. I thought you were using TEXT mapping only.

You said you were able to create index, add vertices and edges, and query them utilizing index in the same transaction, but not in a new transaction. Can you give complete code examples that demonstrate how you do so in the same transaction AND in a new transaction, as Marc asked? Specifically, do include how you open and commit/rollback transactions.

In addition, I wonder if this always happens no matter how long you wait, or only happens in a short period of time. Schema propagation takes some time, and you may see a sync delay across JanusGraph instances. If you create the index on one JanusGraph instance, and immediately insert & query data in another JanusGraph instance, it might be the case that schema propagation is not done yet and your other JanusGraph instance is not aware of the recently created index.

Another thing you could check is the Elasticsearch. After creating the index, do you see the index on your ES cluster? After inserting vertices/edges, do you see documents created at ES side?

Best,
Boxuan

On Mon, Jan 10, 2022 at 1:41 PM Harshit Sharma <harshit.sharma1080@...> wrote:
Only For SET cardinality, I'm using TEXT mapping because looks like SET does not work with TEXTSTRING.

On Mon, Jan 10, 2022 at 11:07 AM Harshit Sharma via lists.lfaidata.foundation <harshit.sharma1080=gmail.com@...> wrote:
but is that the reason I'm not able to use property keys created in a new transaction?
Also, can we mix index property keys of cardinality SET?

On Mon, Jan 10, 2022 at 10:56 AM Boxuan Li <liboxuan@...> wrote:
Hi Harshit,

You should use STRING mapping if you want exact string match. TEXT mapping is for full text search. See 

Best,
Boxuan

On Mon, Jan 10, 2022 at 1:15 PM Harshit Sharma <harshit.sharma1080@...> wrote:
Hi Marc,

1. Creation of property keys and indices in the schema + commit
      graph = (Janusgraph) GraphDBConfigurator.getGraph()
      mgmt = graph.openManagement()
      
     For Vertex ->
           propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SET).make()
          if(existingIndex) 
                janusgraphIndex = mgmt.getGraphIndex(VERTEX_INDEX_NAME)
                mgmt.addIndex(janusgraphIndex, propertyKey)
          else
               indexBuilder = mgmt.buildIndex(VERTEX_INDEX_NAME,  Vertex.class)
               if(cardinality is SET)
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXT.asParameter() : Mapping.DEFAULT.asParameter())
               else
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXTSTRING.asParameter() : Mapping.DEFAULT.asParameter())
  
    For Edge ->
            propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SINGLE).make()
             propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SINGLE).make()
          if(existingIndex) 
                janusgraphIndex = mgmt.getGraphIndex(EDGE_INDEX_NAME)
                mgmt.addIndex(janusgraphIndex, propertyKey)
          else
               indexBuilder = mgmt.buildIndex(EDGE_INDEX_NAME,  Vertex.class)
               if(cardinality is SET)
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXT.asParameter() : Mapping.DEFAULT.asParameter())
               else
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXTSTRING.asParameter() : Mapping.DEFAULT.asParameter())
         
            mgmt.commit();
    


       3.   Query of a vertex based on a property value in another transaction
             Let say i created a property key "domain_vertex_key1_s" for vertex
             Query = g.V().has("domain_vertex_key1_s", "val1);
             
             Let say I created a property key "domain_edge_key1_s" for edge
            
             Query = g.V().has("domain_vertex_key1_s","val2");
          

On Mon, 10 Jan, 2022, 12:46 am , <hadoopmarc@...> wrote:
Hi Harshit,

Can you please describe the steps you have taken in more detail:
  • creation of property keys and indices in the schema + commit,
  • creation of a vertex with a property in one committed transaction,
  • query of a vertex based on a property value in another transaction.

Marc



--
Regards,

Harshit Sharma
+91-9901459920



--
Regards,

Harshit Sharma
+91-9901459920


Re: Janusgraph not able to find suitable index for a index enabled property key

Harshit Sharma
 

Only For SET cardinality, I'm using TEXT mapping because looks like SET does not work with TEXTSTRING.


On Mon, Jan 10, 2022 at 11:07 AM Harshit Sharma via lists.lfaidata.foundation <harshit.sharma1080=gmail.com@...> wrote:
but is that the reason I'm not able to use property keys created in a new transaction?
Also, can we mix index property keys of cardinality SET?

On Mon, Jan 10, 2022 at 10:56 AM Boxuan Li <liboxuan@...> wrote:
Hi Harshit,

You should use STRING mapping if you want exact string match. TEXT mapping is for full text search. See 

Best,
Boxuan

On Mon, Jan 10, 2022 at 1:15 PM Harshit Sharma <harshit.sharma1080@...> wrote:
Hi Marc,

1. Creation of property keys and indices in the schema + commit
      graph = (Janusgraph) GraphDBConfigurator.getGraph()
      mgmt = graph.openManagement()
      
     For Vertex ->
           propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SET).make()
          if(existingIndex) 
                janusgraphIndex = mgmt.getGraphIndex(VERTEX_INDEX_NAME)
                mgmt.addIndex(janusgraphIndex, propertyKey)
          else
               indexBuilder = mgmt.buildIndex(VERTEX_INDEX_NAME,  Vertex.class)
               if(cardinality is SET)
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXT.asParameter() : Mapping.DEFAULT.asParameter())
               else
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXTSTRING.asParameter() : Mapping.DEFAULT.asParameter())
  
    For Edge ->
            propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SINGLE).make()
             propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SINGLE).make()
          if(existingIndex) 
                janusgraphIndex = mgmt.getGraphIndex(EDGE_INDEX_NAME)
                mgmt.addIndex(janusgraphIndex, propertyKey)
          else
               indexBuilder = mgmt.buildIndex(EDGE_INDEX_NAME,  Vertex.class)
               if(cardinality is SET)
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXT.asParameter() : Mapping.DEFAULT.asParameter())
               else
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXTSTRING.asParameter() : Mapping.DEFAULT.asParameter())
         
            mgmt.commit();
    


       3.   Query of a vertex based on a property value in another transaction
             Let say i created a property key "domain_vertex_key1_s" for vertex
             Query = g.V().has("domain_vertex_key1_s", "val1);
             
             Let say I created a property key "domain_edge_key1_s" for edge
            
             Query = g.V().has("domain_vertex_key1_s","val2");
          

On Mon, 10 Jan, 2022, 12:46 am , <hadoopmarc@...> wrote:
Hi Harshit,

Can you please describe the steps you have taken in more detail:
  • creation of property keys and indices in the schema + commit,
  • creation of a vertex with a property in one committed transaction,
  • query of a vertex based on a property value in another transaction.

Marc



--
Regards,

Harshit Sharma
+91-9901459920



--
Regards,

Harshit Sharma
+91-9901459920


Re: Janusgraph not able to find suitable index for a index enabled property key

Harshit Sharma
 

but is that the reason I'm not able to use property keys created in a new transaction?
Also, can we mix index property keys of cardinality SET?

On Mon, Jan 10, 2022 at 10:56 AM Boxuan Li <liboxuan@...> wrote:
Hi Harshit,

You should use STRING mapping if you want exact string match. TEXT mapping is for full text search. See 

Best,
Boxuan

On Mon, Jan 10, 2022 at 1:15 PM Harshit Sharma <harshit.sharma1080@...> wrote:
Hi Marc,

1. Creation of property keys and indices in the schema + commit
      graph = (Janusgraph) GraphDBConfigurator.getGraph()
      mgmt = graph.openManagement()
      
     For Vertex ->
           propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SET).make()
          if(existingIndex) 
                janusgraphIndex = mgmt.getGraphIndex(VERTEX_INDEX_NAME)
                mgmt.addIndex(janusgraphIndex, propertyKey)
          else
               indexBuilder = mgmt.buildIndex(VERTEX_INDEX_NAME,  Vertex.class)
               if(cardinality is SET)
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXT.asParameter() : Mapping.DEFAULT.asParameter())
               else
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXTSTRING.asParameter() : Mapping.DEFAULT.asParameter())
  
    For Edge ->
            propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SINGLE).make()
             propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SINGLE).make()
          if(existingIndex) 
                janusgraphIndex = mgmt.getGraphIndex(EDGE_INDEX_NAME)
                mgmt.addIndex(janusgraphIndex, propertyKey)
          else
               indexBuilder = mgmt.buildIndex(EDGE_INDEX_NAME,  Vertex.class)
               if(cardinality is SET)
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXT.asParameter() : Mapping.DEFAULT.asParameter())
               else
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXTSTRING.asParameter() : Mapping.DEFAULT.asParameter())
         
            mgmt.commit();
    


       3.   Query of a vertex based on a property value in another transaction
             Let say i created a property key "domain_vertex_key1_s" for vertex
             Query = g.V().has("domain_vertex_key1_s", "val1);
             
             Let say I created a property key "domain_edge_key1_s" for edge
            
             Query = g.V().has("domain_vertex_key1_s","val2");
          

On Mon, 10 Jan, 2022, 12:46 am , <hadoopmarc@...> wrote:
Hi Harshit,

Can you please describe the steps you have taken in more detail:
  • creation of property keys and indices in the schema + commit,
  • creation of a vertex with a property in one committed transaction,
  • query of a vertex based on a property value in another transaction.

Marc



--
Regards,

Harshit Sharma
+91-9901459920


Re: Janusgraph not able to find suitable index for a index enabled property key

Boxuan Li
 

Hi Harshit,

You should use STRING mapping if you want exact string match. TEXT mapping is for full text search. See 

Best,
Boxuan

On Mon, Jan 10, 2022 at 1:15 PM Harshit Sharma <harshit.sharma1080@...> wrote:
Hi Marc,

1. Creation of property keys and indices in the schema + commit
      graph = (Janusgraph) GraphDBConfigurator.getGraph()
      mgmt = graph.openManagement()
      
     For Vertex ->
           propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SET).make()
          if(existingIndex) 
                janusgraphIndex = mgmt.getGraphIndex(VERTEX_INDEX_NAME)
                mgmt.addIndex(janusgraphIndex, propertyKey)
          else
               indexBuilder = mgmt.buildIndex(VERTEX_INDEX_NAME,  Vertex.class)
               if(cardinality is SET)
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXT.asParameter() : Mapping.DEFAULT.asParameter())
               else
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXTSTRING.asParameter() : Mapping.DEFAULT.asParameter())
  
    For Edge ->
            propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SINGLE).make()
             propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SINGLE).make()
          if(existingIndex) 
                janusgraphIndex = mgmt.getGraphIndex(EDGE_INDEX_NAME)
                mgmt.addIndex(janusgraphIndex, propertyKey)
          else
               indexBuilder = mgmt.buildIndex(EDGE_INDEX_NAME,  Vertex.class)
               if(cardinality is SET)
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXT.asParameter() : Mapping.DEFAULT.asParameter())
               else
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXTSTRING.asParameter() : Mapping.DEFAULT.asParameter())
         
            mgmt.commit();
    


       3.   Query of a vertex based on a property value in another transaction
             Let say i created a property key "domain_vertex_key1_s" for vertex
             Query = g.V().has("domain_vertex_key1_s", "val1);
             
             Let say I created a property key "domain_edge_key1_s" for edge
            
             Query = g.V().has("domain_vertex_key1_s","val2");
          

On Mon, 10 Jan, 2022, 12:46 am , <hadoopmarc@...> wrote:
Hi Harshit,

Can you please describe the steps you have taken in more detail:
  • creation of property keys and indices in the schema + commit,
  • creation of a vertex with a property in one committed transaction,
  • query of a vertex based on a property value in another transaction.

Marc


Re: Janusgraph not able to find suitable index for a index enabled property key

Harshit Sharma
 

Hi Marc,

1. Creation of property keys and indices in the schema + commit
      graph = (Janusgraph) GraphDBConfigurator.getGraph()
      mgmt = graph.openManagement()
      
     For Vertex ->
           propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SET).make()
          if(existingIndex) 
                janusgraphIndex = mgmt.getGraphIndex(VERTEX_INDEX_NAME)
                mgmt.addIndex(janusgraphIndex, propertyKey)
          else
               indexBuilder = mgmt.buildIndex(VERTEX_INDEX_NAME,  Vertex.class)
               if(cardinality is SET)
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXT.asParameter() : Mapping.DEFAULT.asParameter())
               else
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXTSTRING.asParameter() : Mapping.DEFAULT.asParameter())
  
    For Edge ->
            propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SINGLE).make()
             propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType (String.class).cardinality(Cardinality.SINGLE).make()
          if(existingIndex) 
                janusgraphIndex = mgmt.getGraphIndex(EDGE_INDEX_NAME)
                mgmt.addIndex(janusgraphIndex, propertyKey)
          else
               indexBuilder = mgmt.buildIndex(EDGE_INDEX_NAME,  Vertex.class)
               if(cardinality is SET)
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXT.asParameter() : Mapping.DEFAULT.asParameter())
               else
                        indexBuilder.addKey(propertyKey, String.class.equals(propertyKey.dataType) ? Mapping.TEXTSTRING.asParameter() : Mapping.DEFAULT.asParameter())
         
            mgmt.commit();
    


       3.   Query of a vertex based on a property value in another transaction
             Let say i created a property key "domain_vertex_key1_s" for vertex
             Query = g.V().has("domain_vertex_key1_s", "val1);
             
             Let say I created a property key "domain_edge_key1_s" for edge
            
             Query = g.V().has("domain_vertex_key1_s","val2");
          


On Mon, 10 Jan, 2022, 12:46 am , <hadoopmarc@...> wrote:
Hi Harshit,

Can you please describe the steps you have taken in more detail:
  • creation of property keys and indices in the schema + commit,
  • creation of a vertex with a property in one committed transaction,
  • query of a vertex based on a property value in another transaction.

Marc


Re: Janusgraph not able to find suitable index for a index enabled property key

hadoopmarc@...
 

Hi Harshit,

Can you please describe the steps you have taken in more detail:
  • creation of property keys and indices in the schema + commit,
  • creation of a vertex with a property in one committed transaction,
  • query of a vertex based on a property value in another transaction.

Marc


Janusgraph not able to find suitable index for a index enabled property key

Harshit Sharma
 

I'm working on a Janusgraph application. To improve gremlin query performance we are creating two mixed indexes, one for vertices and one for edges.
Now Janusgraph can query indexes for property keys that are created and indexed at the time of index creation i.e in the same transaction. If I'm creating and indexing a new property key in a new transaction then Janusgraph is not able to query them using indexing, instead, it does a complete graph scan.
Using Janusgeaph management API I checked that all property keys are indexed and enabled, even then Janusgraph is scanning a complete graph for querying on an indexed property key.
Backend index engine -> ElasticSearch
Backend Storage -> Cassandra

Is there anything I'm missing? Any help would be greatly appreciated.


Re: Janusgraph embedded multi instance(JVM) data sync issue

hadoopmarc@...
 

Hi Pawan,

Your requirement for instant synchronization cannot work with JanusGraph caches enabled, because JanusGraph will get data from the cache if available, instead of getting the latest data from the backend. So,

  • cache.db-cache = false
  • be sure to start a new transaction before querying for the latest data (e.g. by executing a g.tx().commit())
Best wishes,    Marc


Re: Janusgraph embedded multi instance(JVM) data sync issue

Pawan Shriwas
 

Hi Boxuan,

Please see my inline response

1. What do you mean by creating some data? For example, do you mean by creating new vertices or just updating existing vertices? If it’s the latter case, then you could try turning off cache.db-cache option as it might lead to stale data read.
[Pawan] - Creating data means vertex/edge creation and updation as well. 

2. What is your typical “duration” after which data gets reflected?  
[Pawan] - Seems to be within a 1 or two min.

3. What is your cql replication factor and read & write consistency levels? Are they default values? Also, how many Cassandra nodes do you have and are they in the same data center?
[Pawan] - These should be defaults, I am using only those graph properties which are mentioned in the below mail.  There are 8 nodes cluster(3 master + 5 nodes ). All cassandra nodes are there in the same data center.

Thanks,
Pawan

On Thu, Jan 6, 2022 at 5:39 PM Boxuan Li <liboxuan@...> wrote:
Hi Pawan,

A couple of questions:

1. What do you mean by creating some data? For example, do you mean by creating new vertices or just updating existing vertices? If it’s the latter case, then you could try turning off cache.db-cache option as it might lead to stale data read.

2. What is your typical “duration” after which data gets reflected?

3. What is your cql replication factor and read & write consistency levels? Are they default values? Also, how many Cassandra nodes do you have and are they in the same data center?

Best,
Boxuan

On Thu, Jan 6, 2022 at 3:03 PM Pawan Shriwas <shriwas.pawan@...> wrote:
Hi All,

I am facing one problem for synchronization of data stored between multiple embedded mode janusgraph instances.

If we are creating some data into graph using JVM 1 and after committing when we get same data from JVM 2 its not reflecting for some duration.

I want to avail the same information to all instances after any CRUD operation once it gets committed.

I am using the same graph property in all instances of embedded janusgraph.

##############graph.properties#####################
gremlin.graph=org.janusgraph.core.JanusGraphFactory
storage.backend=cql
storage.hostname=cql-dns
storage.cql.keyspace=janusgraphdbks
storage.port=30808
storage.username=user123
storage.password=user12345
schema.default=none
schema.constraints=true

index.search-central-graph.backend=elasticsearch
index.search-central-graph.hostname=api-es-instance1:9200
index.search-central-graph.index-name=search-central-graph
index.search-central-graph.elasticsearch.http.auth.type=basic
index.search-central-graph.elasticsearch.http.auth.basic.username=admin
index.search-central-graph.elasticsearch.http.auth.basic.password=admin 

cache.db-cache = true
cache.db-cache-clean-wait = 20
cache.db-cache-time = 180000
cache.db-cache-size = 0.25
query.batch = true
query.fast-property = true
query.batch-property-prefetch = true
storage.buffer-size=1024
######################property file end############################

Please let me know if someone faces this and how to prevent this. 

Thanks,
Pawan




--
Thanks & Regard

PAWAN SHRIWAS


Re: Janusgraph embedded multi instance(JVM) data sync issue

Pawan Shriwas
 

Same case also happened with two or more gremlin consoles as well where we are creating/updating something on console 1 and not reflecting on others.


On Thu, Jan 6, 2022 at 12:32 PM Pawan Shriwas <shriwas.pawan@...> wrote:
Hi All,

I am facing one problem for synchronization of data stored between multiple embedded mode janusgraph instances.

If we are creating some data into graph using JVM 1 and after committing when we get same data from JVM 2 its not reflecting for some duration.

I want to avail the same information to all instances after any CRUD operation once it gets committed.

I am using the same graph property in all instances of embedded janusgraph.

##############graph.properties#####################
gremlin.graph=org.janusgraph.core.JanusGraphFactory
storage.backend=cql
storage.hostname=cql-dns
storage.cql.keyspace=janusgraphdbks
storage.port=30808
storage.username=user123
storage.password=user12345
schema.default=none
schema.constraints=true

index.search-central-graph.backend=elasticsearch
index.search-central-graph.hostname=api-es-instance1:9200
index.search-central-graph.index-name=search-central-graph
index.search-central-graph.elasticsearch.http.auth.type=basic
index.search-central-graph.elasticsearch.http.auth.basic.username=admin
index.search-central-graph.elasticsearch.http.auth.basic.password=admin 

cache.db-cache = true
cache.db-cache-clean-wait = 20
cache.db-cache-time = 180000
cache.db-cache-size = 0.25
query.batch = true
query.fast-property = true
query.batch-property-prefetch = true
storage.buffer-size=1024
######################property file end############################

Please let me know if someone faces this and how to prevent this. 

Thanks,
Pawan




--
Thanks & Regard

PAWAN SHRIWAS


Re: Janusgraph embedded multi instance(JVM) data sync issue

Boxuan Li
 

Hi Pawan,

A couple of questions:

1. What do you mean by creating some data? For example, do you mean by creating new vertices or just updating existing vertices? If it’s the latter case, then you could try turning off cache.db-cache option as it might lead to stale data read.

2. What is your typical “duration” after which data gets reflected?

3. What is your cql replication factor and read & write consistency levels? Are they default values? Also, how many Cassandra nodes do you have and are they in the same data center?

Best,
Boxuan

On Thu, Jan 6, 2022 at 3:03 PM Pawan Shriwas <shriwas.pawan@...> wrote:
Hi All,

I am facing one problem for synchronization of data stored between multiple embedded mode janusgraph instances.

If we are creating some data into graph using JVM 1 and after committing when we get same data from JVM 2 its not reflecting for some duration.

I want to avail the same information to all instances after any CRUD operation once it gets committed.

I am using the same graph property in all instances of embedded janusgraph.

##############graph.properties#####################
gremlin.graph=org.janusgraph.core.JanusGraphFactory
storage.backend=cql
storage.hostname=cql-dns
storage.cql.keyspace=janusgraphdbks
storage.port=30808
storage.username=user123
storage.password=user12345
schema.default=none
schema.constraints=true

index.search-central-graph.backend=elasticsearch
index.search-central-graph.hostname=api-es-instance1:9200
index.search-central-graph.index-name=search-central-graph
index.search-central-graph.elasticsearch.http.auth.type=basic
index.search-central-graph.elasticsearch.http.auth.basic.username=admin
index.search-central-graph.elasticsearch.http.auth.basic.password=admin 

cache.db-cache = true
cache.db-cache-clean-wait = 20
cache.db-cache-time = 180000
cache.db-cache-size = 0.25
query.batch = true
query.fast-property = true
query.batch-property-prefetch = true
storage.buffer-size=1024
######################property file end############################

Please let me know if someone faces this and how to prevent this. 

Thanks,
Pawan



Re: Can I use spark computer on CQL without hadoop cluster

Pawan Shriwas
 

Thanks team,

I will check it based on your response and let you know if anything is needed.


On Wed, Jan 5, 2022 at 8:24 PM <hadoopmarc@...> wrote:
It is also possible to run spark on kubernetes (in combination with distributed storage like S3 or minio):
https://spark.apache.org/docs/latest/running-on-kubernetes.html

It will require some time to get your head around this, but note that you can do this with or without the spark operator installed on your kubernetes cluster.

Marc



--
Thanks & Regard

PAWAN SHRIWAS


Janusgraph embedded multi instance(JVM) data sync issue

Pawan Shriwas
 

Hi All,

I am facing one problem for synchronization of data stored between multiple embedded mode janusgraph instances.

If we are creating some data into graph using JVM 1 and after committing when we get same data from JVM 2 its not reflecting for some duration.

I want to avail the same information to all instances after any CRUD operation once it gets committed.

I am using the same graph property in all instances of embedded janusgraph.

##############graph.properties#####################
gremlin.graph=org.janusgraph.core.JanusGraphFactory
storage.backend=cql
storage.hostname=cql-dns
storage.cql.keyspace=janusgraphdbks
storage.port=30808
storage.username=user123
storage.password=user12345
schema.default=none
schema.constraints=true

index.search-central-graph.backend=elasticsearch
index.search-central-graph.hostname=api-es-instance1:9200
index.search-central-graph.index-name=search-central-graph
index.search-central-graph.elasticsearch.http.auth.type=basic
index.search-central-graph.elasticsearch.http.auth.basic.username=admin
index.search-central-graph.elasticsearch.http.auth.basic.password=admin 

cache.db-cache = true
cache.db-cache-clean-wait = 20
cache.db-cache-time = 180000
cache.db-cache-size = 0.25
query.batch = true
query.fast-property = true
query.batch-property-prefetch = true
storage.buffer-size=1024
######################property file end############################

Please let me know if someone faces this and how to prevent this. 

Thanks,
Pawan



Re: Slowing of janusgraph

hadoopmarc@...
 

Hmm, these figures seem perfectly reasonable. It seems I was also wrong about the db-cache heap region not being shared between graphs. So you have to gather more information about was is going wrong. Some ideas:

  1. does restarting nginx make a difference? Communication between JanusGraph server and gremlin clients runs over websockets. Make sure there is no issue with communication channels kept occupied.
  2. https://lists.lfaidata.foundation/g/janusgraph-users/topic/79935654#2886 This thread gives an nginx setup using a DNS virtual IP address. Possibly this makes a difference in websocket communication issues.
  3. disable the db-cache and see if this make a difference (this can only make a difference if there are many clients with transaction caches)
  4. check the logs of janusgraph server for other indications

Best wishes,      Marc


Re: Can I use spark computer on CQL without hadoop cluster

hadoopmarc@...
 

It is also possible to run spark on kubernetes (in combination with distributed storage like S3 or minio):
https://spark.apache.org/docs/latest/running-on-kubernetes.html

It will require some time to get your head around this, but note that you can do this with or without the spark operator installed on your kubernetes cluster.

Marc


Re: Can I use spark computer on CQL without hadoop cluster

Boxuan Li
 

Hi Pawan,

Do you want to run Spark traversal on a Spark standalone cluster rather than a Hadoop Yarn cluster? In that case, you could follow the JanusGraph documentation or check out this guide on Medium.

Best,
Boxuan


Can I use spark computer on CQL without hadoop cluster

Pawan Shriwas
 

Hi All,

I am checking the possibility of using a graph computer using spark on CQL backend without hadoop installation.

Please let me know if we can do this and how can i achieve this. I don't want to introduce hadoop cluster just because of this use case. I will appreciate  if anyone can share some resources around it.
  
Thanks,
Pawan

341 - 360 of 6656