janusgraph-users@lists.lfaidata.foundation | HadoopGraph / SparkGraphComputer - Cannot Connect to Cassandra Storage with SSL

Home Messages Hashtags Subgroups Calendar

Date Date 1 - 3 of 3

HadoopGraph / SparkGraphComputer - Cannot Connect to Cassandra Storage with SSL

Zachary Blumenfeld <zblu...@...> #4943 Thank you Marc, I think your intuition was correct. Though I am still working through some errors. It appears the ssl configurations should be passed without the janusgraphmr.ioformat prefix but the property keys differ from those used in OLTP. Here is what I added in my HadoopGraph config file to get me past the above error. `cassandra.input.native.ssl.trust.store.path=<trust store location> cassandra.input.native.ssl.trust.store.password=<trust store password>` Full set of cassandra configs can be inferred in the the cassandra source code https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/hadoop/cql3/CqlConfigHelper.java Now it appears I am running into this JanusGraph bug https://github.com/JanusGraph/janusgraph/issues/1727. Do we know if there are any plans to resolve in the near future? Otherwise I am not sure it is possible to use SSL. Thank you, Zach toggle quoted message Show quoted text On Thursday, June 25, 2020 at 6:08:28 AM UTC-4, HadoopMarc wrote: I read this one a second time becase I see that no one reacted so far. Did you also try to provide the ssl properties without the janusgraphmr.ioformat prefix, like for OLTP queries? I must admit that I do not fully understand what the division of roles is between spark driver and spark executors in getting the data from Cassandra, so I say just try it. I remember I once had similar problems in reaching a secure HBase cluster. HTH, Marc Op dinsdag 23 juni 2020 om 04:01:28 UTC+2 schreef zb...@...: JanusGraph Version: 0.5.2 Cassandra Version: 3.11.6 Hello All, I am having trouble running OLAP while using a Cassandra storage layer with SSL. I included some exceptions and debugging below which boil down to a NotSslRecordException. To me, I think it appears that the client (in this case the spark executors) are not using SSL, suggesting that some configuration(s) are not being properly set . Please let me know if you have any ideas/suggestions. Also, please let me know if I can contribute to documentation (or source code if this happens to be a bug) as I have become familiar at this point :) Note: I do not run into any issues when using OLTP with the same graph/cassandra cluster. For OLTP I can follow the documentation at https://docs.janusgraph.org/basics/configuration-reference/ , specifically setting, ssl.enabled, ssl.trustore.location and ssl.trustore.password, and everything works as expected. The following exceptions only happen when I try to use JanusGraph OLAP. Below is the HadoopGraph Configuration I am using. I also tried the same config replacing "cql" with "cassandra",and following https://docs.janusgraph.org/advanced-topics/hadoop/ docs but ran into the same issue # #Example CQL connection # # Gremlin gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph gremlin.graph.graphname=fakergraph_4_olap0 gremlin.hadoop.graphReader=org.janusgraph.hadoop.formats.cql.CqlInputFormat gremlin.hadoop.graphWriter=org.apache.hadoop.mapreduce.lib.output.NullOutputFormat gremlin.hadoop.jarsInDistributedCache=true gremlin.hadoop.inputLocation=none gremlin.hadoop.outputLocation=output # Storage - cassandra janusgraphmr.ioformat.conf.storage.backend=cql janusgraphmr.ioformat.conf.storage.hostname=<hosts> janusgraphmr.ioformat.conf.storage.username=<username> janusgraphmr.ioformat.conf.storage.password=<password> janusgraphmr.ioformat.conf.storage.cql.ssl.enabled=true janusgraphmr.ioformat.conf.storage.cql.ssl.truststore.location=<trustStore location> janusgraphmr.ioformat.conf.storage.cql.ssl.truststore.password=<trustStore password> janusgraphmr.ioformat.conf.storage.cql.keyspace=graph_dev_fakergraph_4_olap0 janusgraphmr.ioformat.conf.schema.default=none # Apache Cassandra InputFormat configuration cassandra.input.partitioner.class=org.apache.cassandra.dht.Murmur3Partitioner cassandra.input.widerows=true # SparkGraphComputer Configuration spark.master=local[4] spark.serializer=org.apache.spark.serializer.KryoSerializer spark.kryo.registrator=org.janusgraph.hadoop.serialize.JanusGraphKryoRegistrator Here is the exception I get when attempting a simple traversal with SparkGraphComputer \,,,/ (o o) -----oOOo-(3)-oOOo----- SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/data/janusgraph/janusgraph-full-0.5.2/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/data/janusgraph/janusgraph-full-0.5.2/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] plugin activated: tinkerpop.server plugin activated: tinkerpop.tinkergraph plugin activated: tinkerpop.hadoop plugin activated: tinkerpop.spark plugin activated: tinkerpop.utilities plugin activated: janusgraph.imports gremlin> graph = GraphFactory.open('hadoop-graph-configs/read-faker-cql.properties') ==>hadoopgraph[cqlinputformat->gryooutputformat] gremlin> g = graph.traversal().withComputer(SparkGraphComputer) ==>graphtraversalsource[hadoopgraph[cqlinputformat->gryooutputformat], sparkgraphcomputer] gremlin> g.V().count() com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: <host>:9042 (com.datastax.driver.core.exceptions.TransportException: [<host>:9042] Connection has been closed)) I would show the entire stack trace, but it is quite long. Ultimately when searching the gremlin logs, i get the below, the connection was “reset by peer”. 23:33:25 DEBUG com.datastax.driver.core.Connection - Connection[<host>:9042-1, inFlight=0, closed=false] connection error io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer 23:33:25 DEBUG com.datastax.driver.core.Host.STATES - Defuncting Connection[<host>:9042-1, inFlight=0, closed=false] because: [<host>:9042] Unexpected exception triggered (io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer) 23:33:25 DEBUG com.datastax.driver.core.Host.STATES - [<host>:9042] preventing new connections for the next 1000 ms 23:33:25 DEBUG com.datastax.driver.core.Host.STATES - [<host>:9042] Connection[<host>:9042-1, inFlight=0, closed=false] failed, remaining = 0 23:33:25 DEBUG com.datastax.driver.core.Connection - Connection[<host>:9042-1, inFlight=0, closed=true] closing connection 23:33:25 DEBUG com.datastax.driver.core.Connection - Not terminating Connection[<host>:9042-1, inFlight=0, closed=true]: there are still pending requests 23:33:25 DEBUG com.datastax.driver.core.ControlConnection - [Control connection] error on <host>:9042 connection, no more host to try com.datastax.driver.core.exceptions.TransportException: [<host>:9042] Connection has been closed at com.datastax.driver.core.Connection$ConnectionCloseFuture.force(Connection.java:1427) at com.datastax.driver.core.Connection$ConnectionCloseFuture.force(Connection.java:1408) at com.datastax.driver.core.Connection.defunct(Connection.java:585) at com.datastax.driver.core.Connection$Dispatcher.exceptionCaught(Connection.java:1330) at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:300) at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:279) at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:271) at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377) at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:300) at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:279) at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.handleReadException(AbstractEpollStreamChannel.java:728) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:818) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748 So I pulled the Cassandra logs from the host in question and get a NotSslRecordException io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: INFO [epollEventLoopGroup-2-3] 2020-06-21 23:33:25,017 Message.java:827 - Unexpected exception during request; channel = [id: 0x9a183987, L:/<hostIp>:9042 ! R:/<clientIp>:8370] io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0400000001000000500003000b43514c5f56455253494f4e0005332e302e30000e4452495645525f56455253494f4e0005332e392e30000b4452495645525f4e414d4500144461746153746178204a61766120447269766572 at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1057) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:934) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:397) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:302) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) [netty-all-4.0.44.Final.jar:4.0.44.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_252] Note Sure if it is relevant, but here is the Hex from the above exception decoded to ASCII P CQL_VERSION3.0.0 DRIVER_VERSION3.9.0 DRIVER_NAME DataStax Java Driver Again. Any help/suggestions would be greatly appreciated.!
More
HadoopMarc <bi...@...> #4933 I read this one a second time becase I see that no one reacted so far. Did you also try to provide the ssl properties without the janusgraphmr.ioformat prefix, like for OLTP queries? I must admit that I do not fully understand what the division of roles is between spark driver and spark executors in getting the data from Cassandra, so I say just try it. I remember I once had similar problems in reaching a secure HBase cluster. HTH, Marc Op dinsdag 23 juni 2020 om 04:01:28 UTC+2 schreef zb...@...: toggle quoted message Show quoted text JanusGraph Version: 0.5.2 Cassandra Version: 3.11.6 Hello All, I am having trouble running OLAP while using a Cassandra storage layer with SSL. I included some exceptions and debugging below which boil down to a NotSslRecordException. To me, I think it appears that the client (in this case the spark executors) are not using SSL, suggesting that some configuration(s) are not being properly set . Please let me know if you have any ideas/suggestions. Also, please let me know if I can contribute to documentation (or source code if this happens to be a bug) as I have become familiar at this point :) Note: I do not run into any issues when using OLTP with the same graph/cassandra cluster. For OLTP I can follow the documentation at https://docs.janusgraph.org/basics/configuration-reference/ , specifically setting, ssl.enabled, ssl.trustore.location and ssl.trustore.password, and everything works as expected. The following exceptions only happen when I try to use JanusGraph OLAP. Below is the HadoopGraph Configuration I am using. I also tried the same config replacing "cql" with "cassandra",and following https://docs.janusgraph.org/advanced-topics/hadoop/ docs but ran into the same issue # #Example CQL connection # # Gremlin gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph gremlin.graph.graphname=fakergraph_4_olap0 gremlin.hadoop.graphReader=org.janusgraph.hadoop.formats.cql.CqlInputFormat gremlin.hadoop.graphWriter=org.apache.hadoop.mapreduce.lib.output.NullOutputFormat gremlin.hadoop.jarsInDistributedCache=true gremlin.hadoop.inputLocation=none gremlin.hadoop.outputLocation=output # Storage - cassandra janusgraphmr.ioformat.conf.storage.backend=cql janusgraphmr.ioformat.conf.storage.hostname=<hosts> janusgraphmr.ioformat.conf.storage.username=<username> janusgraphmr.ioformat.conf.storage.password=<password> janusgraphmr.ioformat.conf.storage.cql.ssl.enabled=true janusgraphmr.ioformat.conf.storage.cql.ssl.truststore.location=<trustStore location> janusgraphmr.ioformat.conf.storage.cql.ssl.truststore.password=<trustStore password> janusgraphmr.ioformat.conf.storage.cql.keyspace=graph_dev_fakergraph_4_olap0 janusgraphmr.ioformat.conf.schema.default=none # Apache Cassandra InputFormat configuration cassandra.input.partitioner.class=org.apache.cassandra.dht.Murmur3Partitioner cassandra.input.widerows=true # SparkGraphComputer Configuration spark.master=local[4] spark.serializer=org.apache.spark.serializer.KryoSerializer spark.kryo.registrator=org.janusgraph.hadoop.serialize.JanusGraphKryoRegistrator Here is the exception I get when attempting a simple traversal with SparkGraphComputer \,,,/ (o o) -----oOOo-(3)-oOOo----- SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/data/janusgraph/janusgraph-full-0.5.2/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/data/janusgraph/janusgraph-full-0.5.2/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] plugin activated: tinkerpop.server plugin activated: tinkerpop.tinkergraph plugin activated: tinkerpop.hadoop plugin activated: tinkerpop.spark plugin activated: tinkerpop.utilities plugin activated: janusgraph.imports gremlin> graph = GraphFactory.open('hadoop-graph-configs/read-faker-cql.properties') ==>hadoopgraph[cqlinputformat->gryooutputformat] gremlin> g = graph.traversal().withComputer(SparkGraphComputer) ==>graphtraversalsource[hadoopgraph[cqlinputformat->gryooutputformat], sparkgraphcomputer] gremlin> g.V().count() com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: <host>:9042 (com.datastax.driver.core.exceptions.TransportException: [<host>:9042] Connection has been closed)) I would show the entire stack trace, but it is quite long. Ultimately when searching the gremlin logs, i get the below, the connection was “reset by peer”. 23:33:25 DEBUG com.datastax.driver.core.Connection - Connection[<host>:9042-1, inFlight=0, closed=false] connection error io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer 23:33:25 DEBUG com.datastax.driver.core.Host.STATES - Defuncting Connection[<host>:9042-1, inFlight=0, closed=false] because: [<host>:9042] Unexpected exception triggered (io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer) 23:33:25 DEBUG com.datastax.driver.core.Host.STATES - [<host>:9042] preventing new connections for the next 1000 ms 23:33:25 DEBUG com.datastax.driver.core.Host.STATES - [<host>:9042] Connection[<host>:9042-1, inFlight=0, closed=false] failed, remaining = 0 23:33:25 DEBUG com.datastax.driver.core.Connection - Connection[<host>:9042-1, inFlight=0, closed=true] closing connection 23:33:25 DEBUG com.datastax.driver.core.Connection - Not terminating Connection[<host>:9042-1, inFlight=0, closed=true]: there are still pending requests 23:33:25 DEBUG com.datastax.driver.core.ControlConnection - [Control connection] error on <host>:9042 connection, no more host to try com.datastax.driver.core.exceptions.TransportException: [<host>:9042] Connection has been closed at com.datastax.driver.core.Connection$ConnectionCloseFuture.force(Connection.java:1427) at com.datastax.driver.core.Connection$ConnectionCloseFuture.force(Connection.java:1408) at com.datastax.driver.core.Connection.defunct(Connection.java:585) at com.datastax.driver.core.Connection$Dispatcher.exceptionCaught(Connection.java:1330) at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:300) at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:279) at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:271) at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377) at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:300) at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:279) at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.handleReadException(AbstractEpollStreamChannel.java:728) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:818) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748 So I pulled the Cassandra logs from the host in question and get a NotSslRecordException io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: INFO [epollEventLoopGroup-2-3] 2020-06-21 23:33:25,017 Message.java:827 - Unexpected exception during request; channel = [id: 0x9a183987, L:/<hostIp>:9042 ! R:/<clientIp>:8370] io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0400000001000000500003000b43514c5f56455253494f4e0005332e302e30000e4452495645525f56455253494f4e0005332e392e30000b4452495645525f4e414d4500144461746153746178204a61766120447269766572 at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1057) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:934) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:397) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:302) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) [netty-all-4.0.44.Final.jar:4.0.44.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_252] Note Sure if it is relevant, but here is the Hex from the above exception decoded to ASCII P CQL_VERSION3.0.0 DRIVER_VERSION3.9.0 DRIVER_NAME DataStax Java Driver Again. Any help/suggestions would be greatly appreciated.!
More
Zachary Blumenfeld <zblu...@...> #4923 JanusGraph Version: 0.5.2 Cassandra Version: 3.11.6 Hello All, I am having trouble running OLAP while using a Cassandra storage layer with SSL. I included some exceptions and debugging below which boil down to a NotSslRecordException. To me, I think it appears that the client (in this case the spark executors) are not using SSL, suggesting that some configuration(s) are not being properly set . Please let me know if you have any ideas/suggestions. Also, please let me know if I can contribute to documentation (or source code if this happens to be a bug) as I have become familiar at this point :) Note: I do not run into any issues when using OLTP with the same graph/cassandra cluster. For OLTP I can follow the documentation at https://docs.janusgraph.org/basics/configuration-reference/ , specifically setting, ssl.enabled, ssl.trustore.location and ssl.trustore.password, and everything works as expected. The following exceptions only happen when I try to use JanusGraph OLAP. Below is the HadoopGraph Configuration I am using. I also tried the same config replacing "cql" with "cassandra",and following https://docs.janusgraph.org/advanced-topics/hadoop/ docs but ran into the same issue # #Example CQL connection # # Gremlin gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph gremlin.graph.graphname=fakergraph_4_olap0 gremlin.hadoop.graphReader=org.janusgraph.hadoop.formats.cql.CqlInputFormat gremlin.hadoop.graphWriter=org.apache.hadoop.mapreduce.lib.output.NullOutputFormat gremlin.hadoop.jarsInDistributedCache=true gremlin.hadoop.inputLocation=none gremlin.hadoop.outputLocation=output # Storage - cassandra janusgraphmr.ioformat.conf.storage.backend=cql janusgraphmr.ioformat.conf.storage.hostname=<hosts> janusgraphmr.ioformat.conf.storage.username=<username> janusgraphmr.ioformat.conf.storage.password=<password> janusgraphmr.ioformat.conf.storage.cql.ssl.enabled=true janusgraphmr.ioformat.conf.storage.cql.ssl.truststore.location=<trustStore location> janusgraphmr.ioformat.conf.storage.cql.ssl.truststore.password=<trustStore password> janusgraphmr.ioformat.conf.storage.cql.keyspace=graph_dev_fakergraph_4_olap0 janusgraphmr.ioformat.conf.schema.default=none # Apache Cassandra InputFormat configuration cassandra.input.partitioner.class=org.apache.cassandra.dht.Murmur3Partitioner cassandra.input.widerows=true # SparkGraphComputer Configuration spark.master=local[4] spark.serializer=org.apache.spark.serializer.KryoSerializer spark.kryo.registrator=org.janusgraph.hadoop.serialize.JanusGraphKryoRegistrator Here is the exception I get when attempting a simple traversal with SparkGraphComputer \,,,/ (o o) -----oOOo-(3)-oOOo----- SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/data/janusgraph/janusgraph-full-0.5.2/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/data/janusgraph/janusgraph-full-0.5.2/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] plugin activated: tinkerpop.server plugin activated: tinkerpop.tinkergraph plugin activated: tinkerpop.hadoop plugin activated: tinkerpop.spark plugin activated: tinkerpop.utilities plugin activated: janusgraph.imports gremlin> graph = GraphFactory.open('hadoop-graph-configs/read-faker-cql.properties') ==>hadoopgraph[cqlinputformat->gryooutputformat] gremlin> g = graph.traversal().withComputer(SparkGraphComputer) ==>graphtraversalsource[hadoopgraph[cqlinputformat->gryooutputformat], sparkgraphcomputer] gremlin> g.V().count() com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: <host>:9042 (com.datastax.driver.core.exceptions.TransportException: [<host>:9042] Connection has been closed)) I would show the entire stack trace, but it is quite long. Ultimately when searching the gremlin logs, i get the below, the connection was “reset by peer”. 23:33:25 DEBUG com.datastax.driver.core.Connection - Connection[<host>:9042-1, inFlight=0, closed=false] connection error io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer 23:33:25 DEBUG com.datastax.driver.core.Host.STATES - Defuncting Connection[<host>:9042-1, inFlight=0, closed=false] because: [<host>:9042] Unexpected exception triggered (io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer) 23:33:25 DEBUG com.datastax.driver.core.Host.STATES - [<host>:9042] preventing new connections for the next 1000 ms 23:33:25 DEBUG com.datastax.driver.core.Host.STATES - [<host>:9042] Connection[<host>:9042-1, inFlight=0, closed=false] failed, remaining = 0 23:33:25 DEBUG com.datastax.driver.core.Connection - Connection[<host>:9042-1, inFlight=0, closed=true] closing connection 23:33:25 DEBUG com.datastax.driver.core.Connection - Not terminating Connection[<host>:9042-1, inFlight=0, closed=true]: there are still pending requests 23:33:25 DEBUG com.datastax.driver.core.ControlConnection - [Control connection] error on <host>:9042 connection, no more host to try com.datastax.driver.core.exceptions.TransportException: [<host>:9042] Connection has been closed at com.datastax.driver.core.Connection$ConnectionCloseFuture.force(Connection.java:1427) at com.datastax.driver.core.Connection$ConnectionCloseFuture.force(Connection.java:1408) at com.datastax.driver.core.Connection.defunct(Connection.java:585) at com.datastax.driver.core.Connection$Dispatcher.exceptionCaught(Connection.java:1330) at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:300) at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:279) at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:271) at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377) at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:300) at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:279) at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.handleReadException(AbstractEpollStreamChannel.java:728) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:818) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748 So I pulled the Cassandra logs from the host in question and get a NotSslRecordException io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: INFO [epollEventLoopGroup-2-3] 2020-06-21 23:33:25,017 Message.java:827 - Unexpected exception during request; channel = [id: 0x9a183987, L:/<hostIp>:9042 ! R:/<clientIp>:8370] io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0400000001000000500003000b43514c5f56455253494f4e0005332e302e30000e4452495645525f56455253494f4e0005332e392e30000b4452495645525f4e414d4500144461746153746178204a61766120447269766572 at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1057) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:934) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:397) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:302) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) [netty-all-4.0.44.Final.jar:4.0.44.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_252] Note Sure if it is relevant, but here is the Hex from the above exception decoded to ASCII P CQL_VERSION3.0.0 DRIVER_VERSION3.9.0 DRIVER_NAMEDataStax Java Driver Again. Any help/suggestions would be greatly appreciated.!
More

1 - 3 of 3

1

Previous Topic Next Topic

Home Hashtags Subgroups Calendar Terms