Date
1 - 5 of 5
.JanusGraph/Elastic - Too many dynamic script compilations error for LIST type properties
Naresh Babu Y
Hi,
we are using janusgraph ( version 0.3.2) with elastic 6.
when updating a node/vertex with property of LIST cardinality which is mixed index frequently getting below exception and data is not stored/updated.
please let me know, if any other option to handle this in janusgraph?
Thanks,
Naresh
we are using janusgraph ( version 0.3.2) with elastic 6.
when updating a node/vertex with property of LIST cardinality which is mixed index frequently getting below exception and data is not stored/updated.
{type=illegal_argument_exception, reason=failed to execute script, caused_by={type=general_script_exception, reason=Failed to compile inline script
[if(ctx._source["property123"] == null) ctx._source["property123"] = [];ctx._source["property123"].add("jkkhhj#1");] using lang [painless], caused_by={type=circuit_breaking_exception, reason=[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting, bytes_wanted=0, bytes_limit=0}}}
we have requirement to update property of LIST type frequently, but changing max_compilations_rate to large number is not a good idea.please let me know, if any other option to handle this in janusgraph?
Thanks,
Naresh
hadoopmarc@...
Hi Naresh,
I guess that the script that the error message refers to, is the script that your client executes remotely at gremlin server. You may want to study:
https://tinkerpop.apache.org/docs/current/reference/#parameterized-scripts
which, depending on how you coded the frequent updates, can dramatically diminish the time spent on script compilation by gremlin server. This is also what the exception messages means with "use indexed, or scripts with parameters instead".
Best wishes, Marc
I guess that the script that the error message refers to, is the script that your client executes remotely at gremlin server. You may want to study:
https://tinkerpop.apache.org/docs/current/reference/#parameterized-scripts
which, depending on how you coded the frequent updates, can dramatically diminish the time spent on script compilation by gremlin server. This is also what the exception messages means with "use indexed, or scripts with parameters instead".
Best wishes, Marc
Naresh Babu Y
Hello Marc,
Thanks for quick reply.
Am not using gremlin server.
Am using spark, and read all messages per batch
Then open JanusGraph transaction add batch records and commit it.
Here is the details..
JanusGraph version: 0.3.2
Storage system: Hbase
Index : elastic
Please let me know if you have any clue at JanusGraph transaction level/any configuration (because am not using gremlin server)
Thanks,
Naresh
On Sat, 13 Mar 2021, 9:54 pm , <hadoopmarc@...> wrote:
Hi Naresh,
I guess that the script that the error message refers to, is the script that your client executes remotely at gremlin server. You may want to study:
https://tinkerpop.apache.org/docs/current/reference/#parameterized-scripts
which, depending on how you coded the frequent updates, can dramatically diminish the time spent on script compilation by gremlin server. This is also what the exception messages means with "use indexed, or scripts with parameters instead".
Best wishes, Marc
hadoopmarc@...
Hi Naresh,
Yes, elasticsearch, I should have recognized the "painless" scripting! This can mean the following things:
Yes, elasticsearch, I should have recognized the "painless" scripting! This can mean the following things:
- your use case is maybe unusual, would it be possible to introduce a groupby step in spark that first gathers all property updates for a vertex into one update call?
- the default value of script.max_compilations_rate may be really too low for your use case, so it is worth a try increasing it (elasticsearch docs do not discourage it). I think this should be done outside janusgraph, just using the elastic API's.
- the janusgraph code for calling elasticsearch with scripts is suboptimal; I did not investigate this other than checking for existing issues (none). This option will not help you now; if you want to create an issue on janusgraph github, please specify how your system setup is, what the update rates are, etc. You would also have to check whether your issue also holds for janusgraph 0.4.1 or 0.5.3 because 0.3.x is end of life.
Abhay Pandit
Hi Naresh,
I too used to get this exception. This was solved after moving to Janusgraph v0.5.2.
Hope this helps you.
Thanks,
Abhay
I too used to get this exception. This was solved after moving to Janusgraph v0.5.2.
Hope this helps you.
Thanks,
Abhay
On Sat, 13 Mar 2021 at 22:20, <hadoopmarc@...> wrote:
Hi Naresh,
Yes, elasticsearch, I should have recognized the "painless" scripting! This can mean the following things:
Best wishes, Marc
- your use case is maybe unusual, would it be possible to introduce a groupby step in spark that first gathers all property updates for a vertex into one update call?
- the default value of script.max_compilations_rate may be really too low for your use case, so it is worth a try increasing it (elasticsearch docs do not discourage it). I think this should be done outside janusgraph, just using the elastic API's.
- the janusgraph code for calling elasticsearch with scripts is suboptimal; I did not investigate this other than checking for existing issues (none). This option will not help you now; if you want to create an issue on janusgraph github, please specify how your system setup is, what the update rates are, etc. You would also have to check whether your issue also holds for janusgraph 0.4.1 or 0.5.3 because 0.3.x is end of life.