Re: Database Level Caching


Boxuan Li
 

Thanks Nicolas, I am able to reproduce it using your configs & script. Created an issue at https://github.com/JanusGraph/janusgraph/issues/2369

Looks like a bug with calculating cache entries' size.

On Jan 15, 2021, at 11:53 PM, Nicolas Trangosi <nicolas.trangosi@...> wrote:

Hi Boxuan,
Issue seems to occurs when edge properties are retrieved: cache has expected size with g.V().outE().id() an not when I do g.V().outE().valueMap();

I am able to reproduce with following groovy script :
  • gremlin console  ( launched with JAVA_OPTS="-Xmx1G -Xms1G" ./bin/gremlin.sh)   on JG 0.5.3
  • conf/janusgraph-cache.properties:
gremlin.graph=org.janusgraph.core.JanusGraphFactory

storage.backend=cql
storage.hostname=127.0.0.1
storage.port=9042
schema.default=logging

cache.db-cache: true
cache.db-cache-size: 50000000
cache.db-cache-time: 6000000

  •   groovy script:

graph = JanusGraphFactory.open('conf/janusgraph-cache.properties')
g = graph.traversal()


// Schema creation
graph.tx().rollback()
mgmt = g.getGraph().openManagement()

try {
    deviceLabel = mgmt.makeVertexLabel('device').make()
    nameProperty = mgmt.makePropertyKey("name").dataType(java.lang.String).cardinality(org.janusgraph.core.Cardinality.SINGLE).make()
    mgmt.addProperties(deviceLabel, nameProperty)

    measurementLabel = mgmt.makeEdgeLabel('measurement').unidirected().make()
    deviceNameProperty = mgmt.makePropertyKey("deviceName").dataType(java.lang.String).cardinality(org.janusgraph.core.Cardinality.SINGLE).make()
    physicalQuantityProperty = mgmt.makePropertyKey("physicalQuantity").dataType(java.lang.String).cardinality(org.janusgraph.core.Cardinality.SINGLE).make()
    valueProperty = mgmt.makePropertyKey("value").dataType(java.lang.Double).cardinality(org.janusgraph.core.Cardinality.SINGLE).make()
    timestampProperty = mgmt.makePropertyKey("timestamp").dataType(java.util.Date).cardinality(org.janusgraph.core.Cardinality.SINGLE).make()

    mgmt.addProperties(measurementLabel, deviceNameProperty, physicalQuantityProperty, valueProperty, timestampProperty)
    mgmt.buildIndex("deviceByName", Vertex.class).indexOnly(deviceLabel).addKey(nameProperty).buildCompositeIndex();

    //mgmt.buildEdgeIndex(measurementLabel, 'measurementByTimestamp', Direction.OUT, Order.decr, timestampProperty);

    mgmt.commit()
} catch (Exception e) {
    mgmt.rollback();
    throw e;
}

// Load data
random = new Random();
startTs = System.currentTimeMillis();
for (i = 0; i < 100; i++) {
   deviceId = g.addV("device").property("name", "device-" + i).id().next();
   for (k = 0; k < 5000; k++) {
       g.V(deviceId).addE("measurement").
           property("deviceName",  "device-" + i).
           property("physicalQuantity", "physicalQuantity-" + random.nextInt(10)).
           property("value", random.nextDouble()).
           property("timestamp", new Date(startTs + k * 1000)).
           iterate();
       if (k % 1000 == 0) {
           g.tx().commit();
       }
   }
   log.info("Done i={}",i);
}
g.tx().commit();

// Request data 
for (i = 0; i < 100; i++) {
   measurementsList = g.V().has("device", "name", "device-" + i).outE().valueMap().toList();
   log.info("Got {} measurements for {}", measurementsList.size(), i);
}
g.tx().commit();



Le sam. 9 janv. 2021 à 05:21, BO XUAN LI <liboxuan@...> a écrit :
Hi Nicolas,

Looks interesting. Your configs look fine and I couldn’t reproduce your problem. Could you provide some sample code to reproduce it?

Best regards,
Boxuan


On Jan 4, 2021, at 10:20 PM, Nicolas Trangosi <nicolas.trangosi@...> wrote:

Hi Boxuan,

I have configured janusgraph with:

cache.db-cache-time: 600000  
cache.db-cache: true  
cache.db-cache-size: 50000000  
index.search.elasticsearch.create.ext.number_of_replicas: 0
storage.buffer-size: 1024
index.search.elasticsearch.create.ext.number_of_shards: 1
cache.cache.db-cache-time: 0
index.search.index-name: dcbrain
index.search.backend: elasticsearch
storage.port: 9042
ids.block-size: 1000000
schema.default: logging
storage.cql.batch-statement-size: 50
index.search.hostname: dfe-elasticsearch
storage.backend: cql
storage.hostname: dfe-cassandra
storage.cql.local-max-requests-per-connection: 4096
index.search.port: 9200


I have load some data on the graph and dump memory.
When I import this dump with jvisualVM, retained size for ExpirationKCVSCache 257 Mb when the limit should be 50 Mb.
<image.png>

Regards,
Nicolas

Le lun. 4 janv. 2021 à 13:11, BO XUAN LI <liboxuan@...> a écrit :
Hi Nicolas,

Can you provide your configurations and the memory usage you observed?

Regards,
Boxuan

On Jan 4, 2021, at 3:44 PM, Nicolas Trangosi <nicolas.trangosi@...> wrote:

Hi,
I try to use  Database Level Caching as described in https://docs.janusgraph.org/basics/cache/ but it seems to use more memory than the configured threshold ( cache.db-cache-size ). Does anyone use such a feature ? Is it production ready ?

Regards,
Nicolas


Ce message et ses pièces jointes peuvent contenir des informations confidentielles ou privilégiées et ne doivent donc pas être diffusés, exploités ou copiés sans autorisation. 
Si vous avez reçu ce message par erreur, veuillez le signaler a l'expéditeur et le détruire ainsi que les pièces jointes. 
Les messages électroniques étant susceptibles d'altération, DCbrain décline toute responsabilité si ce message a été altéré, déformé ou falsifié. Merci. 

This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, DCbrain is not liable for messages that have been modified, changed or falsified. Thank you.

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-users+unsubscribe@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/CAD7qnB4SYvXq5A3vkzu44fERkySr2kPhsoZC-5%3DbBoz9KvzPnw%40mail.gmail.com.


--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-users+unsubscribe@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/3B5EFE52-BE38-437B-B399-05AF4899F398%40connect.hku.hk.


--
  
Nicolas Trangosi
Lead back
+33 (0)6 77 86 66 44      
   



Ce message et ses pièces jointes peuvent contenir des informations confidentielles ou privilégiées et ne doivent donc pas être diffusés, exploités ou copiés sans autorisation. 
Si vous avez reçu ce message par erreur, veuillez le signaler a l'expéditeur et le détruire ainsi que les pièces jointes. 
Les messages électroniques étant susceptibles d'altération, DCbrain décline toute responsabilité si ce message a été altéré, déformé ou falsifié. Merci. 

This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, DCbrain is not liable for messages that have been modified, changed or falsified. Thank you.

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-users+unsubscribe@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/CAD7qnB7bY3bNwf3PVCLuuU%2BOT%2BAmGWnoEGtT00i6LQ8%2Bu5sHzw%40mail.gmail.com.


--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-users+unsubscribe@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/D7009A96-D9BB-4F05-A792-372EB3F5CE34%40connect.hku.hk.


--
  
Nicolas Trangosi
Lead back
+33 (0)6 77 86 66 44      
   



Ce message et ses pièces jointes peuvent contenir des informations confidentielles ou privilégiées et ne doivent donc pas être diffusés, exploités ou copiés sans autorisation. 
Si vous avez reçu ce message par erreur, veuillez le signaler a l'expéditeur et le détruire ainsi que les pièces jointes. 
Les messages électroniques étant susceptibles d'altération, DCbrain décline toute responsabilité si ce message a été altéré, déformé ou falsifié. Merci. 

This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, DCbrain is not liable for messages that have been modified, changed or falsified. Thank you.

Join janusgraph-users@lists.lfaidata.foundation to automatically receive all group messages.