Date
1 - 4 of 4
Replacing EntryList by an iterable type
Hieu Nguyen <ntrh...@...>
When I look at the OrderedKeyValueStoreAdapter class, in methods getSlice() or getSlices(), objects of type RecordIterator<KeyValueEntry> are converted to objects of type EntryList (StaticArrayEntryList). This, basically, breaks the iterator fashion since all data are collected and materialized. In many cases, this is very inefficient. For example, with multi-query enabled and I want to do g.V().has("name", "xxx").out().limit(50) and assume g.V().has("name", "xxx") result has 1k vertices, out() would trigger 1k concurrent requests and may return thousands of vertices although it is likely that with much fewer requests it can satisfy the limit(50).
With an iterator fashion, it may try to do next() until it reaches the limit (50) and then stops.
I believe replacing EntryList by an iterable type to enable fully iterator-way of retrieving and processing data will benefit JanusGraph efficiency. My question is is there some reasons that EntryList was designed at the first place or there are some fundamental drawbacks of using iterator instead of EntryList? This would save time a lot since I plan to spend my effort to improve this.
Jerry He <jerr...@...>
Hi, Hieu
I am not aware of any special reason it is such way as you point out. Feel free to propose any improvement or new design.
The thing I can think of is that you probably would need to maintain a mini cache and mini batch (going to the backend storage) to feed into the Iterator.
Thanks.
Jerry
Thanks.
Jerry
On Fri, Jul 26, 2019 at 11:24 AM Hieu Nguyen <ntrh...@...> wrote:
When I look at the OrderedKeyValueStoreAdapter class, in methods getSlice() or getSlices(), objects of type RecordIterator<KeyValueEntry> are converted to objects of type EntryList (StaticArrayEntryList). This, basically, breaks the iterator fashion since all data are collected and materialized. In many cases, this is very inefficient. For example, with multi-query enabled and I want to do g.V().has("name", "xxx").out().limit(50) and assume g.V().has("name", "xxx") result has 1k vertices, out() would trigger 1k concurrent requests and may return thousands of vertices although it is likely that with much fewer requests it can satisfy the limit(50).--With an iterator fashion, it may try to do next() until it reaches the limit (50) and then stops.I believe replacing EntryList by an iterable type to enable fully iterator-way of retrieving and processing data will benefit JanusGraph efficiency. My question is is there some reasons that EntryList was designed at the first place or there are some fundamental drawbacks of using iterator instead of EntryList? This would save time a lot since I plan to spend my effort to improve this.
You received this message because you are subscribed to the Google Groups "JanusGraph developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-dev/1d6bd615-d355-4618-9a7e-5a3a687939ef%40googlegroups.com.
Hieu Nguyen <ntrh...@...>
I have the first version working. However, with a query like this:
g.V().has("PropA", "xxx").inE().outV().out().limit(50).valueMap(true).toList(), or
g.V().has("PropA", "xxx").inE().outV().as("a").out().as("b").limit(50).path().from("a").to("b")
the steps valueMap(true) and path() are still being executed sequentially (i.e., with the 50 results of the previous steps, for each vertex JanusGraph issues one getSlice().
I expected getSlices() should also be used here, but unfortunately it doesn't.
Is my understanding correct? Is it an issue?
On Sunday, July 28, 2019 at 7:11:22 PM UTC-7, Jerry He wrote:
Hi, HieuI am not aware of any special reason it is such way as you point out. Feel free to propose any improvement or new design.The thing I can think of is that you probably would need to maintain a mini cache and mini batch (going to the backend storage) to feed into the Iterator.Thanks.JerryThanks.JerryOn Fri, Jul 26, 2019 at 11:24 AM Hieu Nguyen <nt...@...> wrote:When I look at the OrderedKeyValueStoreAdapter class, in methods getSlice() or getSlices(), objects of type RecordIterator<KeyValueEntry> are converted to objects of type EntryList (StaticArrayEntryList). This, basically, breaks the iterator fashion since all data are collected and materialized. In many cases, this is very inefficient. For example, with multi-query enabled and I want to do g.V().has("name", "xxx").out().limit(50) and assume g.V().has("name", "xxx") result has 1k vertices, out() would trigger 1k concurrent requests and may return thousands of vertices although it is likely that with much fewer requests it can satisfy the limit(50).--With an iterator fashion, it may try to do next() until it reaches the limit (50) and then stops.I believe replacing EntryList by an iterable type to enable fully iterator-way of retrieving and processing data will benefit JanusGraph efficiency. My question is is there some reasons that EntryList was designed at the first place or there are some fundamental drawbacks of using iterator instead of EntryList? This would save time a lot since I plan to spend my effort to improve this.
You received this message because you are subscribed to the Google Groups "JanusGraph developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-dev/1d6bd615- d355-4618-9a7e-5a3a687939ef% 40googlegroups.com.