Date
1 - 6 of 6
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space while loading bulk data
Amyth Arora <aroras....@...>
Hi Everyone, I am trying to upload some dummy data for testing purposes to janusgraph (google cloud bigtable backend). I have a groovy script as follows that I execute while running the gremlin console that creates the schema, indexes, vertexes and edges.
I have already updated the heap size in the `JAVA_OPTIONS` environment variable to `-Xmx2048m`. Also here is how my configuration looks like.
I execute the script using the following command:
The problem is, even before the schema is created it throws out of memory error for java heap space. Following is the output.
Any help will be much appreciated. Thanks. Amyth (twiiter.com/mytharora) |
|
Robert Dale <rob...@...>
Looks like your file doesn't fit in memory. Robert Dale On Tue, Aug 8, 2017 at 9:23 AM, Amyth Arora <aroras....@...> wrote:
|
|
Amyth Arora <aroras....@...>
Hi Robert, The file is about 325 mb in size and contains info about a million vertices and a million edges. Also I forgot to mention that prior to testing on bigtable I tried the same script and file to test janus with cassandra backend on the same machine, which worked fine. while testing this with cassandra, I experienced a similar issue, which went away by the introduction of following configuration options: storage.batch-loading ids.block-size But in case of cassandra, the error was thrown while creating the edges. In case of bigtable, The exception is thrown as soon as the script is executed. On Tue, 8 Aug 2017 at 7:51 PM, Robert Dale <rob...@...> wrote:
-- Thanks & Regards ----------------------------------------------------- Amyth Arora ----------------------------------------------------- Web: ----------------------------------------------------- Email Addresses: aroras....@..., ----------------------------------------------------- Social Profiles: Twitter - @mytharora linkedin - https://in.linkedin.com/in/aroraamyth ----------------------------------------------------- |
|
Robert Dale <rob...@...>
Well, whatever, but your stacktrace points to: String fileContents = new File(jsonPath).getText('UTF-8') Thus, your file does not fit in memory - either available system memory or within jvm max memory. Robert Dale On Tue, Aug 8, 2017 at 10:38 AM, Amyth Arora <aroras....@...> wrote:
|
|
Amyth Arora <aroras....@...>
Thanks Robert, I am going to check the file size and its contents when I reach home and also will try to load the file through the shell and post the update here. On Tue, 8 Aug 2017 at 8:14 PM, Robert Dale <rob...@...> wrote:
-- Thanks & Regards ----------------------------------------------------- Amyth Arora ----------------------------------------------------- Web: ----------------------------------------------------- Email Addresses: aroras....@..., ----------------------------------------------------- Social Profiles: Twitter - @mytharora linkedin - https://in.linkedin.com/in/aroraamyth ----------------------------------------------------- |
|
Misha Brukman <mbru...@...>
You might consider using a format other than JSON which can easily read incrementally, such as CSV or a more compact binary encoding such as:
or you may want to use a streaming JSON reader which will read as little as possible to generate a meaningful callback. Several options exist (I have not tested these), e.g.,
If you search on Stack Overflow, you'll find others have had exactly the same issue as here with Python and JSON files, and the answers to those questions all pointed to incremental JSON parser libraries for Python as the solution to the OOMs. On Tue, Aug 8, 2017 at 10:49 AM, Amyth Arora <aroras....@...> wrote:
|
|