Date   

Re: New TSC member: Oleksandr Porunov

Misha Brukman <mbru...@...>
 

Congratulations, Oleksandr! Thank you for your extensive contributions to JanusGraph and welcome to the TSC!


On Tue, Aug 6, 2019 at 5:26 AM Florian Hockmann <f...@...> wrote:

On behalf of the JanusGraph Technical Steering Committee (TSC), I'm pleased to welcome a new Technical Steering Committee member on the project!

Oleksandr Porunov is one of the most active contributors on JanusGraph and he already managed the 0.3.2 release. Being a TSC member enables assistance with the project management and to guide the direction of the project.

Congratulations, Oleksandr!

--
You received this message because you are subscribed to the Google Groups "JanusGraph developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-dev/00fd01d54c38%24fbb30270%24f3190750%24%40florian-hockmann.de.


Re: New TSC member: Oleksandr Porunov

Abhay Pandit <abha...@...>
 

Congratulations, Oleksandr!

Thanks,
Abhay

On Tue, 6 Aug 2019 at 14:56, Florian Hockmann <f...@...> wrote:

On behalf of the JanusGraph Technical Steering Committee (TSC), I'm pleased to welcome a new Technical Steering Committee member on the project!

Oleksandr Porunov is one of the most active contributors on JanusGraph and he already managed the 0.3.2 release. Being a TSC member enables assistance with the project management and to guide the direction of the project.

Congratulations, Oleksandr!

--
You received this message because you are subscribed to the Google Groups "JanusGraph developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-dev/00fd01d54c38%24fbb30270%24f3190750%24%40florian-hockmann.de.


Re: New committer: Pavel Ershov

Abhay Pandit <abha...@...>
 

That's great!!
Congrats Pavel



On Tue, 6 Aug 2019 at 18:32, Florian Hockmann <f...@...> wrote:

On behalf of the JanusGraph Technical Steering Committee (TSC), I'm pleased to welcome a new committer on the project!

Pavel Ershov has been a solid contributor. He already made a significant number of contributions, including many bug fixes, but also performance improvements and improvements for the Lucene backend to get it on par with Elasticsearch and Solr.


Congratulations, Pavel!

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgra...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/010d01d54c57%2440887c90%24c19975b0%24%40florian-hockmann.de.


New committer: Pavel Ershov

"Florian Hockmann" <f...@...>
 

On behalf of the JanusGraph Technical Steering Committee (TSC), I'm pleased to welcome a new committer on the project!

Pavel Ershov has been a solid contributor. He already made a significant number of contributions, including many bug fixes, but also performance improvements and improvements for the Lucene backend to get it on par with Elasticsearch and Solr.


Congratulations, Pavel!


New TSC member: Oleksandr Porunov

"Florian Hockmann" <f...@...>
 

On behalf of the JanusGraph Technical Steering Committee (TSC), I'm pleased to welcome a new Technical Steering Committee member on the project!

Oleksandr Porunov is one of the most active contributors on JanusGraph and he already managed the 0.3.2 release. Being a TSC member enables assistance with the project management and to guide the direction of the project.

Congratulations, Oleksandr!


Re: [MEETUP] JanusGraph Community Online Meetup August 7

Ted Wilmes <twi...@...>
 

Hello,
Meetup #2 is almost here! If you haven't signed up already but are still interested, the link can be found here: https://training.experoinc.com/janus-meetup-august7/. We're looking forward to another great set of talks and community discussion. Hope to see you all there.

Thanks,
Ted

On Fri, Jul 19, 2019 at 11:26 AM Ted Wilmes <twi...@...> wrote:
Hello,
I would like to announce the second online JanusGraph community meetup. We have a great lineup of sessions that will be given by JanusGraph community members and lots of time for interactive Q&A. Session details and signup information can be found here: https://training.experoinc.com/janus-meetup-august7/

Thanks,
Ted


Re: Support for Partitioned Vertices in JanusGraphHadoop for OLAP queries

Florian Hockmann <f...@...>
 

I'm not aware of anyone working on this right now, but supernodes are definitely a big problem for graph databases, including JanusGraph, so any improvements in that area would be a great help for many users.
Regarding graph partitioning as a countermeasure for supernodes, I just want to point out that it depends on the size of your cluster of storage backend nodes how much it helps. This blog post by Ted Wilmes explains in greater detail. It talks about DSE Graph, but the same basically applies to JanusGraph.
So, you might need to implement something yourself to work around the supernode problem, like some bucketing approach where you split your supernodes up. If you want more information about supernodes and the impact they have on JanusGraph, we had a thread a while back on janusgraph-users on that topic.

Am Freitag, 2. August 2019 19:48:25 UTC+2 schrieb kes...@...:

Hey there,

I have been working with JanusGraph recently and unfortunately the dataset that I am dealing with is susceptible to supernodes (10+ mil edges into a single vertex). It seems that partitioning vertices with a particular vertex labels is the way to distribute these dense vertices in the storage backend: https://docs.janusgraph.org/latest/graph-partitioning.html but I see that these partitioned vertices must be filtered out for OLAP queries: https://github.com/JanusGraph/janusgraph/blob/master/janusgraph-hadoop-parent/janusgraph-hadoop-core/src/main/java/org/janusgraph/hadoop/config/JanusGraphHadoopConfiguration.java#L51-L57

Are there any plans to remove this restriction anytime soon/is there anyone currently working on this problem? 

Thanks,

Joseph


Support for Partitioned Vertices in JanusGraphHadoop for OLAP queries

kestin...@...
 

Hey there,

I have been working with JanusGraph recently and unfortunately the dataset that I am dealing with is susceptible to supernodes (10+ mil edges into a single vertex). It seems that partitioning vertices with a particular vertex labels is the way to distribute these dense vertices in the storage backend: https://docs.janusgraph.org/latest/graph-partitioning.html but I see that these partitioned vertices must be filtered out for OLAP queries: https://github.com/JanusGraph/janusgraph/blob/master/janusgraph-hadoop-parent/janusgraph-hadoop-core/src/main/java/org/janusgraph/hadoop/config/JanusGraphHadoopConfiguration.java#L51-L57

Are there any plans to remove this restriction anytime soon/is there anyone currently working on this problem? 

Thanks,

Joseph


Re: Not able to 'addEdge' to 'CacheVertex'

bodd...@...
 

Teena:

How do you fix this problem? I'm boring about it.


On Friday, June 30, 2017 at 12:43:27 PM UTC+8, Teena K wrote:
    
    JanusGraph graph = JanusGraphFactory.open("<path>/janusgraph/conf/janusgraph-cassandra.properties");
    JanusGraphTransaction tx = graph.newTransaction();    
    GraphTraversalSource g = graph.traversal();

    <code to add vertices and edges>

    tx.commit();

    The transaction is committed after all the code is executed.

On Thursday, June 29, 2017 at 10:04:20 PM UTC+5:30, Jason Plurad wrote:
You've left out some context here. How are g` and `tx` initialized? Where is the transaction committed?

On Thursday, June 29, 2017 at 10:33:49 AM UTC-4, Teena K wrote:


down votefavorite

I have the below code in my java program that queries janusgraph to create vertices and edges if they don't exist already.

Vertex v1 = g.V().has(<key1>,<value1>).tryNext().orElseGet(()->
tx.addVertex(T.label,<label1>,<key1>,<value1>));
Vertex v2 = g.V().has(<key2>,<value2>).tryNext().orElseGet(()->
tx.addVertex(T.label,<label2>,<key2>,<value2>));

Vertices v1 and v2 get created if they don't exist.

..code to check if an edge exists between the two vertices..
..if there is no edge between the two, create an edge
v1.addEdge(<label3>,v2,<key3>,<value3>)

If the vertices are newly created, the code works fine and the edge also gets created between the two vertices. But if the vertices already exist in the DB, the edge doesn't get created. The difference I could find between the two cases is that v1 and v2 are of 'StandardVertex' type when they are newly created and they are of 'CacheVertex' type when they already exist. 'addEdge' is a valid method in both cases. Yet the edge doesn't get created.


Re: Replacing EntryList by an iterable type

Jerry He <jerr...@...>
 

Hi, Hieu

I am not aware of any special reason it is such way as you point out. Feel free to propose any improvement or new design.  
The thing I can think of is that you probably would need to maintain a mini cache and mini batch (going to the backend storage) to feed into the Iterator.

Thanks.

Jerry

Thanks.

Jerry

On Fri, Jul 26, 2019 at 11:24 AM Hieu Nguyen <ntrh...@...> wrote:
When I look at the OrderedKeyValueStoreAdapter class, in methods getSlice() or getSlices(), objects of type RecordIterator<KeyValueEntry>  are converted to objects of type EntryList (StaticArrayEntryList). This, basically, breaks the iterator fashion since all data are collected and materialized. In many cases, this is very inefficient. For example, with multi-query enabled and I want to do g.V().has("name", "xxx").out().limit(50) and assume g.V().has("name", "xxx") result has 1k vertices, out() would trigger 1k concurrent requests and may return thousands of vertices although it is likely that with much fewer requests it can satisfy the limit(50). 

With an iterator fashion, it may try to do next() until it reaches the limit (50) and then stops.

I believe replacing EntryList by an iterable type to enable fully iterator-way of retrieving and processing data will benefit JanusGraph efficiency. My question is is there some reasons that EntryList was designed at the first place or there are some fundamental drawbacks of using iterator instead of EntryList? This would save time a lot since I plan to spend my effort to improve this.

--
You received this message because you are subscribed to the Google Groups "JanusGraph developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-dev/1d6bd615-d355-4618-9a7e-5a3a687939ef%40googlegroups.com.


Replacing EntryList by an iterable type

Hieu Nguyen <ntrh...@...>
 

When I look at the OrderedKeyValueStoreAdapter class, in methods getSlice() or getSlices(), objects of type RecordIterator<KeyValueEntry>  are converted to objects of type EntryList (StaticArrayEntryList). This, basically, breaks the iterator fashion since all data are collected and materialized. In many cases, this is very inefficient. For example, with multi-query enabled and I want to do g.V().has("name", "xxx").out().limit(50) and assume g.V().has("name", "xxx") result has 1k vertices, out() would trigger 1k concurrent requests and may return thousands of vertices although it is likely that with much fewer requests it can satisfy the limit(50). 

With an iterator fashion, it may try to do next() until it reaches the limit (50) and then stops.

I believe replacing EntryList by an iterable type to enable fully iterator-way of retrieving and processing data will benefit JanusGraph efficiency. My question is is there some reasons that EntryList was designed at the first place or there are some fundamental drawbacks of using iterator instead of EntryList? This would save time a lot since I plan to spend my effort to improve this.


Re: [DISCUSS] Don't use Scroll API for ElasticSearch requests

remi.g...@...
 

Hi !
I have strated to look at this point last week.
I haven't code anything, but this topic interests me.
In my searchs, the only problem using "search_after" wich is recommended now by elasticsearch is to find a good ordered properties.

I was thinking about using the ids (Vertex or Edges) but it's not recommanded.. 
In the worst scenario, we should add some kind of UUID to help us order the queries if no order is specified .

IMO, we should do what ES recommends ( https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-search-after.html ) and switch to search after.
In cases where the index queried returns only a few results, using the scroll api for this cases is to dismesured.






Le lundi 15 avril 2019 19:30:58 UTC+2, Oleksandr Porunov a écrit :
Currently we are using Scroll API for realtime search requests when using ElasticSearch as an index backend. In my experience it often creates more than 500 parallel cursors (sometimes more then 10000 cursors). Sure, we can decrease keep-alive parameter "index.[X].elasticsearch.scroll-keep-alive" to keep cursors for less than 60 seconds but I don't think that it is a wise solution. 

Statements from the ElasticSearch documentation:
>> Scrolling is not intended for real time user requests, but rather for processing large amounts of data, e.g. in order to reindex the contents of one index into a new index with a different configuration.

>> The Scroll api is recommended for efficient deep scrolling but scroll contexts are costly and it is not recommended to use it for real time user requests.

In addition, I can say that ElasticSearch 7.0.0 (released on 10 April) by default limits the amount of open cursors to 500.

Pros which I see if we remove usage of Scroll API in JanusGraph:
- All real time queries will be faster
- Less overhead on the ElasticSearch side (we don't keep open contexts)

Cons which I see if we remove usage of Scroll API in JanusGraph:
- The use would need to be aware of queries like .limit(1000000) or queries without limit because they may hit a lot of results and that is we may have some problems on ElasticSearch side.

Even considering the con of removing usage of Scroll API I think we should remove Scroll API usage because it is much simpler to write your Gremlin query with `limit` usage than dealing with too many open contexts (just by opinion).

Possible solutions to deal with the con:
- Warn users about possible problems for queries which hit many entities in ElasticSearch. Suggest to use "limit" and some data processing techniques.
- Imitate scroll usage by using "search_after" (this one could be hard to implement and not applicable to queries without sorting by unique parameter / parameters).

What other community members think about it?
Do you see any other pros of using Scroll API which I missed? Are you OK with removing usage of Scroll API?    


Re: [TSC ACTION REQUIRED] Unblock 0.4 branch

Misha Brukman <mbru...@...>
 

Sorry about that; I've fixed the branch protection rule so you should be good to go now.


On Sun, Jul 21, 2019 at 10:29 AM Oleksandr Porunov <alexand...@...> wrote:
Hello,

There is a branch protraction on `0.4` branch which should be removed. Please, remove branch protraction on `0.4` branch.

Currently the branch `0.4` is inconsistent with `0.3` and `master` branch due to impossibility to merge `0.3` branch into `0.4` branch. 

Please remove the branch protraction from GitHub because it violates our policy:
1. Lazy consensus by the PR from a commiter
2. Merging approved changes upstream

Best regards,
Oleksandr Porunov

--
You received this message because you are subscribed to the Google Groups "JanusGraph developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-dev/5a142e65-4a86-4918-8aab-4f50578166f4%40googlegroups.com.


[TSC ACTION REQUIRED] Unblock 0.4 branch

Oleksandr Porunov <alexand...@...>
 

Hello,

There is a branch protraction on `0.4` branch which should be removed. Please, remove branch protraction on `0.4` branch.

Currently the branch `0.4` is inconsistent with `0.3` and `master` branch due to impossibility to merge `0.3` branch into `0.4` branch. 

Please remove the branch protraction from GitHub because it violates our policy:
1. Lazy consensus by the PR from a commiter
2. Merging approved changes upstream

Best regards,
Oleksandr Porunov


[MEETUP] JanusGraph Community Online Meetup August 7

Ted Wilmes <twi...@...>
 

Hello,
I would like to announce the second online JanusGraph community meetup. We have a great lineup of sessions that will be given by JanusGraph community members and lots of time for interactive Q&A. Session details and signup information can be found here: https://training.experoinc.com/janus-meetup-august7/

Thanks,
Ted


ShortestPathVertexProgram OOM issue

Abhay Pandit <abha...@...>
 

Hi,

I am running SparkGraphComputer for ShortestPathVertexProgram and I am running into Out Of Memory issue with bigger clusters as well.

Total Nodes: 10 Million
Total Size Of Nodes: 10G approx
Total Edge: 5M approx

Cluster Config:
Total Executors: 60
Each Executors Size: 128G

I am using Janusgraph v0.4.0 backed by Cassandra and elasticseach
Data Storage and Query Protocol: CQL

Note: I am getting OOM where level of path is is more than 2.
My Observations:
 I can see this warning in logs
WARN  Executor:66 - Managed memory leak detected; size = 2873804338 bytes, TID = 1276

Old Generation Heap taking maximum memory and GC is not happening as its show live references in memory.

Any help is appreciated.

GC Snapshot from an executor:
image.png
Note: I tried running ConnectedComponentVertexProgram found same issue with that as well.



Thanks,
Abhay


Re: [ANNOUNCE] JanusGraph 0.4.0 Release

Misha Brukman <mbru...@...>
 

[ moving discussion from *-users@ to *-dev@ group ]

FYI, since future users who are not already subscribed to the mailing list will most likely find this release via GitHub release page and not from this email, I took the liberty of adding the 3 bullet points under "Notable new features" and added them to the release page as well, ahead of the "Tested compatibility" section.

This is something that users have been asking about on Twitter as well, so we should do this regularly in the future so folks know what to expect from each new release, right where they download the binaries.

On Tue, Jul 9, 2019 at 10:43 AM Florian Hockmann <f...@...> wrote:
The JanusGraph Technical Steering Committee is excited to announce the release of JanusGraph 0.4.0.

JanusGraph is an Apache TinkerPop enabled property graph database with support for a variety of storage and indexing backends. Thank you to all of the contributors.

Notable new features in this release include:
  • Updates to key dependencies for Apache TinkerPop 3.4, Apache Cassandra 2.2, Apache HBase 2.1, Berkeley DB JE 7.5, Google Cloud Bigtable 1.11
  • Support CQL for OLAP which completes our CQL support
  • Performance improvements for pre-fetching of properties
The release artifacts can be found at this location:
    https://github.com/JanusGraph/janusgraph/releases/tag/v0.4.0

A binary distribution is provided for user convenience:
    https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/janusgraph-0.4.0-hadoop2.zip

The online docs can be found here:
    https://docs.janusgraph.org/0.4.0/index.html

To view the resolved issues and commits check the milestone here:
        https://github.com/JanusGraph/janusgraph/milestone/8?closed=1

Thank you very much,
Florian Hockmann

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgra...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/2911c180-78d6-428e-ba86-77190113063e%40googlegroups.com.


[RESULT][VOTE] JanusGraph 0.4.0 release

Florian Hockmann <f...@...>
 

This vote is now closed with a total of 6 +1s, no +0s and no -1s. The results are:

BINDING VOTES:

+1  (3 -- Florian Hockmann, Ted Wilmes,  Misha Brukman)
0   (0)
-1  (0)

NON-BINDING VOTES:

+1 (3 -- Oleksandr Porunov, Abhay Pandit, Jan Jansen)
0  (0)
-1 (0)

Thank you very much,

Florian Hockmann


Re: [VOTE] JanusGraph 0.4.0 release

Misha Brukman <mbru...@...>
 

Checked signature, skimmed the docs and sanity-checked binaries.

VOTE: +1


On Tue, Jul 2, 2019 at 3:34 PM Florian Hockmann <f...@...> wrote:
Hello,

We are happy to announce that JanusGraph 0.4.0 is ready for release.

The release artifacts can be found at this location:
        https://github.com/JanusGraph/janusgraph/releases/tag/v0.4.0

A binary distribution is provided for user convenience:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/janusgraph-0.4.0-hadoop2.zip

The GPG key used to sign the release artifacts is available at:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/KEYS

The docs can be found here:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.4.0/janusgraph-0.4.0-hadoop2-doc.zip
        (See also documentation preview site)

The release tag in Git can be found here:
        https://github.com/JanusGraph/janusgraph/tree/v0.4.0

The release notes are available here:
        https://github.com/JanusGraph/janusgraph/blob/v0.4.0/docs/changelog.adoc

This [VOTE] will open for the next 3 days --- closing Friday, July 5, 2019 at 10:00 PM CEST (UTC+2).
All are welcome to review and vote on the release, but only votes from TSC members are binding.
My vote is +1.


Thank you,
Florian Hockmann

--
You received this message because you are subscribed to the Google Groups "JanusGraph developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-dev/9a08d67e-1fab-462a-aa44-5468b2cdc0b3%40googlegroups.com.


Re: [VOTE] JanusGraph 0.4.0 release

Jan Jansen <faro...@...>
 

Release looks good to me. JanusGraph 0.4 is running as readonly instance for some time without any problem.

VOTE: +1

Thanks,
Jan


421 - 440 of 1596