[VOTE] JanusGraph 0.6.0 release


Oleksandr Porunov
 

Hello,

We are happy to announce that JanusGraph 0.6.0 is ready for release.

The release artifacts can be found at this location:
        https://github.com/JanusGraph/janusgraph/releases/tag/v0.6.0

A full binary distribution is provided for user convenience:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.6.0/janusgraph-full-0.6.0.zip
 
A truncated binary distribution is provided:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.6.0/janusgraph-0.6.0.zip
 
The GPG key used to sign the release artifacts is available at:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.6.0/KEYS

The docs can be found here:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.6.0/janusgraph-0.6.0-doc.zip

The release tag in Git can be found here:
        https://github.com/JanusGraph/janusgraph/tree/v0.6.0

The release notes are available here:
        https://github.com/JanusGraph/janusgraph/blob/master/docs/changelog.md#version-060-release-date-august-11-2021

This [VOTE] will open for the next 3 days --- closing Saturday, August 14, 2021 at 3:45 PM GMT+3.
All are welcome to review and vote on the release, but only votes from TSC members are binding.
My vote is +1.

Thank you,
Oleksandr Porunov


Rafael Fernandes
 

+1
Tested the binaries in our docker instances..

Rafa


On Wed, Aug 11, 2021 at 8:43 AM Oleksandr Porunov <alexandr.porunov@...> wrote:
Hello,

We are happy to announce that JanusGraph 0.6.0 is ready for release.

The release artifacts can be found at this location:
        https://github.com/JanusGraph/janusgraph/releases/tag/v0.6.0

A full binary distribution is provided for user convenience:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.6.0/janusgraph-full-0.6.0.zip
 
A truncated binary distribution is provided:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.6.0/janusgraph-0.6.0.zip
 
The GPG key used to sign the release artifacts is available at:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.6.0/KEYS

The docs can be found here:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.6.0/janusgraph-0.6.0-doc.zip

The release tag in Git can be found here:
        https://github.com/JanusGraph/janusgraph/tree/v0.6.0

The release notes are available here:
        https://github.com/JanusGraph/janusgraph/blob/master/docs/changelog.md#version-060-release-date-august-11-2021

This [VOTE] will open for the next 3 days --- closing Saturday, August 14, 2021 at 3:45 PM GMT+3.
All are welcome to review and vote on the release, but only votes from TSC members are binding.
My vote is +1.

Thank you,
Oleksandr Porunov


Clement de Groc
 

Hey,

I've performed the following tests:
- Built the source code and a docker image from janusgraph-dist/Dockerfile.
- Updated an existing configuration/graph. Followed the upgrade instructions.
- Started JanusGraph and performed simple requests (hitting composite and mixed indices).
- Enabled and checked the presence of CQL metrics and threadpool metrics.

+1 from me.

Clement

On Wed, Aug 11, 2021 at 6:17 PM Rafael Fernandes <luizrafael@...> wrote:
+1
Tested the binaries in our docker instances..

Rafa


On Wed, Aug 11, 2021 at 8:43 AM Oleksandr Porunov <alexandr.porunov@...> wrote:
Hello,

We are happy to announce that JanusGraph 0.6.0 is ready for release.

The release artifacts can be found at this location:
        https://github.com/JanusGraph/janusgraph/releases/tag/v0.6.0

A full binary distribution is provided for user convenience:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.6.0/janusgraph-full-0.6.0.zip
 
A truncated binary distribution is provided:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.6.0/janusgraph-0.6.0.zip
 
The GPG key used to sign the release artifacts is available at:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.6.0/KEYS

The docs can be found here:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.6.0/janusgraph-0.6.0-doc.zip

The release tag in Git can be found here:
        https://github.com/JanusGraph/janusgraph/tree/v0.6.0

The release notes are available here:
        https://github.com/JanusGraph/janusgraph/blob/master/docs/changelog.md#version-060-release-date-august-11-2021

This [VOTE] will open for the next 3 days --- closing Saturday, August 14, 2021 at 3:45 PM GMT+3.
All are welcome to review and vote on the release, but only votes from TSC members are binding.
My vote is +1.

Thank you,
Oleksandr Porunov


Boxuan Li
 

I downloaded https://github.com/JanusGraph/janusgraph/releases/download/v0.6.0/janusgraph-0.6.0.zip, tried a simple traversal using SparkGraphComputer on my laptop but failed:

:plugin use tinkerpop.hadoop
:plugin use tinkerpop.spark
graph = GraphFactory.open('conf/hadoop-graph/read-cql.properties')
g = graph.traversal().withComputer(SparkGraphComputer)
g.V().count()
with error `java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;`

I don't have scala or Spark installed locally. The same traversal worked well on the 0.5.3 version. I personally never ran any workload on Spark before, so I am not sure if this is expected, or due to a library conflict in Janusgraph distribution. If anyone familiar with Spark can shed some light or test by themselves, it would be very helpful. Otherwise, I am afraid 0.6.0 might not work well with Spark.

Best,
Boxuan


Clement de Groc
 

Hey Boxuan,

Good catch !

To the best of my knowledge, this type of error usually means there is a mix of scala 2.11 and scala 2.12 libraries.

TinkerPop 3.5 switched to Spark 3 and Scala 2.12 (https://github.com/apache/tinkerpop/blob/master/spark-gremlin/pom.xml#L88).

But I see (at least one) reference to Scala 2.11 libraries in janusgraph-hadoop (https://github.com/JanusGraph/janusgraph/blob/master/janusgraph-hadoop/pom.xml#L77).

I would try chasing down libraries pulling Scala 2.11 and upgrading them.
I can look into that on Monday if noone has before.

Best,
Clement

Le sam. 14 août 2021 à 06:36, Boxuan Li <liboxuan@...> a écrit :
I downloaded https://github.com/JanusGraph/janusgraph/releases/download/v0.6.0/janusgraph-0.6.0.zip, tried a simple traversal using SparkGraphComputer on my laptop but failed:

:plugin use tinkerpop.hadoop
:plugin use tinkerpop.spark
graph = GraphFactory.open('conf/hadoop-graph/read-cql.properties')
g = graph.traversal().withComputer(SparkGraphComputer)
g.V().count()
with error `java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;`

I don't have scala or Spark installed locally. The same traversal worked well on the 0.5.3 version. I personally never ran any workload on Spark before, so I am not sure if this is expected, or due to a library conflict in Janusgraph distribution. If anyone familiar with Spark can shed some light or test by themselves, it would be very helpful. Otherwise, I am afraid 0.6.0 might not work well with Spark.

Best,
Boxuan


Oleksandr Porunov
 

Thank you Boxuan for the catch!

I will try to look at it today's evening. I have no luck I hope Clement will be able to check it on Monday.

Thus, I'm extending the voting process until the issue is fixed. After the fix I will re-upload artifacts to Sonatype and update release artifacts and a tag.

Best regards,
Oleksandr


Boxuan Li
 

Hi thanks for the quick responses. Followed by Clement’s advice, I made a fix https://github.com/li-boxuan/janusgraph/commit/aec47d44e6b0a57eac5fb5acab4c11d365d0ecac that worked on my local laptop. If CI pass I will submit a PR.

Best regards,
Boxuan

On Aug 14, 2021, at 8:15 PM, Oleksandr Porunov <alexandr.porunov@...> wrote:

Thank you Boxuan for the catch!

I will try to look at it today's evening. I have no luck I hope Clement will be able to check it on Monday.

Thus, I'm extending the voting process until the issue is fixed. After the fix I will re-upload artifacts to Sonatype and update release artifacts and a tag.

Best regards,
Oleksandr


vinay1331@...
 

After the upgrade to testing getting following exception using remote traversal. Used tinkerpop 3.5.1 with janusgrah drive-0.6.0


WARN  org.apache.tinkerpop.gremlin.server.handler.WsGremlinBinaryRequestDecoder  - 
Gremlin Server is not configured with a serializer for the 
requested mime type [application/vnd.gremlin-v1.0+gryo] 
- using org.apache.tinkerpop.gremlin.driver.ser.GraphBinaryMessageSerializerV1 by default
2889684 [gremlin-server-worker-1]
 WARN  org.apache.tinkerpop.gremlin.server.handler.OpSelectorHandler  - 
 Invalid OpProcessor requested [null]
org.apache.tinkerpop.gremlin.server.op.OpProcessorException: Invalid OpProcessor requested [null]
        at org.apache.tinkerpop.gremlin.server.handler.OpSelectorHandler.decode(OpSelectorHandler.java:85)
        at org.apache.tinkerpop.gremlin.server.handler.OpSelectorHandler.decode(OpSelectorHandler.java:48)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
        at io.netty.handler.codec.http.websocketx.Utf8FrameValidator.channelRead(Utf8FrameValidator.java:82)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)


Oleksandr Porunov
 

Hi @vinay1331,

This is expected. You must upgrade your Gremlin Server client first. See upgrade instructions regarding your issue here: https://docs.janusgraph.org/master/changelog/#serialization-of-janusgraph-predicates-has-changed
Also, for gremlin settings breaking changes see the following instructions: 
https://docs.janusgraph.org/master/changelog/#breaking-change-for-gremlin-server-configs
https://docs.janusgraph.org/master/changelog/#introduction-of-a-janusgraph-server-startup-class-as-a-replacement-for-gremlin-server-startup
https://docs.janusgraph.org/master/changelog/#automatic-configurations-of-dynamic-graph-binding

Also, notice, the binaries will be updated soon. Thus, you *may* potentially need to update your gremlin server multiple times.

Thanks for taking part in testing.

Best regards,
Oleksandr


Oleksandr Porunov
 

I dropped previous staging release from Sonatype account and reuploaded a new staring release.
I checked and successfully closed the staging release.
GitHub tag v0.6.0 has been updated to point on the latest commit.
All the GitHub artifacts has been rebuilt and updated.

The issue discovered by Boxuan Li has been resolved as well as several other issues. Moreover one new feature has been added to the release (possibility to configure internal Cassandra driver).

I'm reopening the voting process.

This [VOTE] will open for the next 3 days --- closing Tuesday, August 31, 2021 at 11:55 PM GMT+3.
All are welcome to review and vote on the release, but only votes from TSC members are binding.
My vote is +1.

Thank you,
Oleksandr Porunov


Boxuan Li
 

I downloaded janusgraph-0.6.0.zip and got this error when using the gremlin console:

gremlin> graph = JanusGraphFactory.open("inmemory")
java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
Type ':help' or ':h' for help.
Display stack trace? [yN]y
java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
at org.janusgraph.diskstorage.util.WriteByteBuffer.getStaticBufferFlipBytes(WriteByteBuffer.java:148)


Oleksandr Porunov
 

Thank you Boxuan! Confirm this issue. Most likely I made a mistake during the build process because I can reproduce this issue with my build but can't reproduce this issue with the next build (https://github.com/JanusGraph/janusgraph/actions/runs/1177782203) see `distribution-builds` artifact. I will check that tomorrow and will try to fix that.
I think we will need to automate the releasing process soon to eliminate such issues in the future and build releases in deterministic way.


Oleksandr Porunov
 

I realized that I was building with Java 11 instead of Java 8. I switched to the Java 8 (OpenJDK 1.8.0_302), dropped previous staging repository, rebuilt JanusGraph using Java 8, re-uploaded new artifacts to the staging repository and to GitHub release. Checked that `graph = JanusGraphFactory.open("inmemory")` now works in Gremlin console. I believe the above issue should be fixed now.


Oleksandr Porunov
 

I also executed all TinkerPop tests again with `mvn clean install -Dtest.skip.tp=false -DskipTests=true`. All tests passed.


Boxuan Li
 

I did the following tests:

  1. Connected to AWS keyspace, did some simple traversals
  2. Compatibility test: using 0.5.3 CQL + ES to create a graph with 10 million nodes (https://gist.github.com/li-boxuan/d8059d1c40652c3c5308670ce383e96a), able to load using 0.6.0
  3. Did some simple performance comparison (using the graph created in the previous test) and saw significant performance improvements (up to 2x) in some queries, compared to 0.5.3:

0.5.3 (query.batch=true):

gremlin>  clock(10) { graph.traversal().V().has("name", "inner0").toList() }

==>478.1922667999999

gremlin>  clock(10) { graph.traversal().V().has("age", 0).toList() }

==>17873.4094548

gremlin> clock(10) {graph.traversal().V().has("name", "outer").out().out().toList()}

==>14500.0969486

 

 

0.6.0 (query.batch = true, query.limit-batch-size=true)

gremlin>  clock(10) { graph.traversal().V().has("name", "inner0").toList() }

==>267.27335619999997

gremlin> clock(10) { graph.traversal().V().has("age", 0).toList() }

==>7432.816540099999

gremlin> clock(10) {graph.traversal().V().has("name", "outer").out().out().toList()}

==>11786.6717737

 

  1. Basic SparkGraphComputer traversal works in Spark-local, Spark standalone cluster, and Spark yarn cluster modes. I also wrote a blog to record all configurations needed: https://li-boxuan.medium.com/configure-janusgraph-0-6-0-for-spark-64bd7628fb77

My vote is +1.


Florian Hockmann
 

I just did a quick test of the two distribution archives which just consists of starting JanusGraph Server and connecting to it via the remote console to create a vertex. For the full distribution, I then also restarted the server to verify that the vertex was persisted in Cassandra. This unfortunately failed for 0.6.0. The graph was completely empty after executing ‘bin/janusgraph.sh stop’ and ‘start’.

I’m not sure what’s causing this, but I think that we should look into it. At a first glance, the problem seems to be related to Cassandra as that didn’t create a directory ‘db/Cassandra/data/janusgraph’ for the JanusGraph keyspace. I only see directories for the system keyspaces there, but for 0.5.3 where this still worked, there was also a janusgraph keyspace directory.

 

Other then that, I also noticed that the Changelog mentions the removal of Thrift and refers to this repository where the Thrift code should be moved to: https://github.com/JanusGraph/janusgraph-cassandra

Unfortunately, the PR that would actually add the code to master branch there is still open: https://github.com/JanusGraph/janusgraph-cassandra/pull/3

I don’t think that this should stop the release, but we should try to merge it at least soon after the release.

 

Von: janusgraph-dev@... <janusgraph-dev@...> Im Auftrag von Boxuan Li
Gesendet: Dienstag, 31. August 2021 09:32
An: janusgraph-dev@...
Betreff: Re: [janusgraph-dev] [VOTE] JanusGraph 0.6.0 release

 

I did the following tests:

  1. Connected to AWS keyspace, did some simple traversals
  2. Compatibility test: using 0.5.3 CQL + ES to create a graph with 10 million nodes (https://gist.github.com/li-boxuan/d8059d1c40652c3c5308670ce383e96a), able to load using 0.6.0
  3. Did some simple performance comparison (using the graph created in the previous test) and saw significant performance improvements (up to 2x) in some queries, compared to 0.5.3:

0.5.3 (query.batch=true):

gremlin>  clock(10) { graph.traversal().V().has("name", "inner0").toList() }

==>478.1922667999999

gremlin>  clock(10) { graph.traversal().V().has("age", 0).toList() }

==>17873.4094548

gremlin> clock(10) {graph.traversal().V().has("name", "outer").out().out().toList()}

==>14500.0969486

 

 

0.6.0 (query.batch = true, query.limit-batch-size=true)

gremlin>  clock(10) { graph.traversal().V().has("name", "inner0").toList() }

==>267.27335619999997

gremlin> clock(10) { graph.traversal().V().has("age", 0).toList() }

==>7432.816540099999

gremlin> clock(10) {graph.traversal().V().has("name", "outer").out().out().toList()}

==>11786.6717737

 

  1. Basic SparkGraphComputer traversal works in Spark-local, Spark standalone cluster, and Spark yarn cluster modes. I also wrote a blog to record all configurations needed: https://li-boxuan.medium.com/configure-janusgraph-0-6-0-for-spark-64bd7628fb77

 

My vote is +1.


Oleksandr Porunov
 

Confirming the issue which is found by Florian Hockmann.

I was trying to load `GraphOfTheGodsFactory.load(graph)` and couldn't:
GraphOfTheGodsFactory.load(graph)
The indexing backend with name "search" is not defined. Specify an existing indexing backend or use GraphOfTheGodsFactory.loadWithoutMixedIndex(graph,true) to load without the use of an indexing backend.

So, it looks like `janusgraph-server.sh` is using inmemory option without index backend instead of using Cassandra + ElasticSearch as requested by `janusgraph.sh` script.
Quickly checking `janusgraph.sh` script I see it starts `janusgraph-server.sh` with the following options:
```
"$BIN"/janusgraph-server.sh console conf/gremlin-server/gremlin-server-cql-es.yaml &
```

Checking `janusgraph-server.sh` script I see the following code:
```
if [[ -z "$JANUSGRAPH_YAML" ]]; then
  JANUSGRAPH_YAML="$JANUSGRAPH_CONF/gremlin-server/gremlin-server.yaml"
fi
 
if [[ ! -r "$JANUSGRAPH_YAML" ]]; then
  # try relative to conf
  JANUSGRAPH_YAML="$JANUSGRAPH_CONF/gremlin-server/gremlin-server.yaml"
  if [[ ! -r "$JANUSGRAPH_YAML" ]]; then
    echo WARNING: $JANUSGRAPH_YAML is unreadable
  fi
fi

...

    if [[ -n "$1" ]] ; then
      if [[ -r "$1" ]]; then
        JANUSGRAPH_YAML="$1"
        startForeground
      elif [[ -r "$JANUSGRAPH_HOME/$1" ]] ; then
        JANUSGRAPH_YAML="$JANUSGRAPH_HOME/$1"
        startForeground
      fi
      echo Configuration file not found.
    fi

...
```

So, it looks like this script never expects path to be passed as a second argument. Thus, it looks like a second argument `conf/gremlin-server/gremlin-server-cql-es.yaml` is simply ignored and the default `$JANUSGRAPH_CONF/gremlin-server/gremlin-server.yaml` is used. But `$JANUSGRAPH_CONF/gremlin-server/gremlin-server.yaml` is using inmemory option by default. 
I believe this is the root cause for this problem but didn't had a chance to actually verify that yet. If anyone has time today to fix this issue, please, submit a PR, otherwise I will try to fix it in a day or two.


Oleksandr Porunov
 

Opened a PR here which fixes the issue with `janusgraph.sh`: https://github.com/JanusGraph/janusgraph/pull/2781


Oleksandr Porunov
 

I re-uploaded a new staring release.
I checked and successfully closed the staging release (1035).
GitHub tag v0.6.0 has been updated to point on the latest commit.
All the GitHub artifacts has been rebuilt and updated.

The issue found by Florian Hockmann is now fixed.

I'm reopening the voting process.

This [VOTE] will open for the next 3 days --- closing Monday, September 6, 2021 at 3:45 PM GMT+3.
All are welcome to review and vote on the release, but only votes from TSC members are binding.
My vote is +1.

Thank you,
Oleksandr Porunov


Florian Hockmann
 

Thanks for fixing this!

 

I performed the same tests again and now everything looks good. So:

 

VOTE +1

 

Von: janusgraph-dev@... <janusgraph-dev@...> Im Auftrag von Oleksandr Porunov
Gesendet: Freitag, 3. September 2021 14:45
An: janusgraph-dev@...
Betreff: Re: [janusgraph-dev] [VOTE] JanusGraph 0.6.0 release

 

I re-uploaded a new staring release.
I checked and successfully closed the staging release (1035).
GitHub tag v0.6.0 has been updated to point on the latest commit.
All the GitHub artifacts has been rebuilt and updated.

The issue found by Florian Hockmann is now fixed.

I'm reopening the voting process.

This [VOTE] will open for the next 3 days --- closing Monday, September 6, 2021 at 3:45 PM GMT+3.
All are welcome to review and vote on the release, but only votes from TSC members are binding.
My vote is +1.

Thank you,
Oleksandr Porunov