1. Janusgraph-Users
  2. Messages

Re: Data Loading Script Optimization

Nicolas Trangosi <nicolas.trangosi@...>
 

Hi,
You could first create a local cache for ORG by retrying first all ORG:

Map<String, Long> orgCache = g.V().has('vertexLabel', 'ORG').project("name", "id").by("orgName").by(T.id)...

Then replace __.V().has('vertexLabel', 'ORG').has('orgName', orgName) by __V(orgCache.get(orgName))

Same trick, may be used for persons to remove the coalesce if you know that you import more users than already exist in db.

Nicolas


Le lun. 9 août 2021 à 10:07, Vinayak Bali <vinayakbali16@...> a écrit :
Hi Marc, 

To avoid confusion, including a new transaction at line number 39, as well as at line no 121.  
Line 39: GraphTraversalSource g = graph.newTransaction().traversal();
Line 121: g = ctx.g = graph.newTransaction().traversal();
The total time took was 11 mins. The maximum amount of cpu utilization was 40%. As the hardware configuration of the instance is at higher side, still we have enough RAM to increase the performance. 
Request you to share if it's possible to increase the performance further following some process(Hadoop/spark) etc based on your experience. 

Thanks & Regards,
Vinayak

On Sun, Aug 8, 2021 at 1:53 PM <hadoopmarc@...> wrote:
Hi Vinayak,

Yes, it should be possible to improve on the 3% CPU usage.

The newTransaction() should be added to line 39 (GraphTraversalSource g = graph.traversal();) as the global g from line 121 is not used.

Marc



--

  

Nicolas Trangosi

Lead back

+33 (0)6 77 86 66 44      

   



Ce message et ses pièces jointes peuvent contenir des informations confidentielles ou privilégiées et ne doivent donc pas être diffusés, exploités ou copiés sans autorisation. 
Si vous avez reçu ce message par erreur, veuillez le signaler a l'expéditeur et le détruire ainsi que les pièces jointes. 
Les messages électroniques étant susceptibles d'altération, DCbrain décline toute responsabilité si ce message a été altéré, déformé ou falsifié. Merci. 

This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, DCbrain is not liable for messages that have been modified, changed or falsified. Thank you.
Join janusgraph-users@lists.lfaidata.foundation to automatically receive all group messages.