Tinkerpop 3.4.1 with Hadoop3


Hi All,
I am trying to run SparkGraphComputer using tinkerpop 3.4.1 on Spark2.3 having Hadoop3 but getting issues, not able to connect with Spark.
As per http://tinkerpop.apache.org/docs/3.4.1/upgrade/#_hadoop_gremlin , "Hadoop1 is no longer supported. Hadoop2 is now the only supported Hadoop version in TinkerPop." Does it means Tinkerpop is not supported with Hadoop3?

Thanks in advance.



Hi Anjani,

JanusGraph receives the hadoop dependency from Apache TinkerPop, so Apache TinkerPop will be in the driver seat regarding the upgrade of hadoop. Making a custom build of janusgraph with hadoop3 will be very time consuming because many library conflicts exist, which need to be resolved manually and tested. If you try, be sure to start from a janusgraph branch that uses TinkerPop 3.5.x with spark 3.0 (so janusgraph master or the future 0.6.0) to minimize the chances of library conflicts.

After a quick googling session, I find that hadoop3 services (resource manager, name node) support hadoop2 clients to a certain extent. What errors do you get when trying janusgraph-0.5.3 on a hadoop3 cluster? Note that TinkerPop/Janusgraph are not shipped with hadoop-yarn, see: https://tinkerpop.apache.org/docs/current/recipes/#olap-spark-yarn

Best wishes,     Marc

[Edited] Adding missing spark-yarn and hadoop-yarn jars is described here.