Re: JanusGraph 0.5.3 SparkGraph Computer with YARN Error - java.lang.ClassCastException: org.apache.hadoop.yarn.proto.YarnServiceProtos$GetNewApplicationRequestProto cannot be cast to org.apache.hadoop.hbase.shaded.com.google.protobuf.Message


hadoopmarc@...
 

Hi,

Because TinkerPop and JanusGraph do not ship with spark-yarn, all kinds of library conflict can occur when you start using spark-yarn. Since my blog from some years ago, where it was possible to avoid library conflicts, the hbase jars were repackaged leading to new problems. Basically, you have three options to proceed:

  1. forget about the gremlin console and build a java application where you avoid all conflicts by using the maven shade plugin. Some recent evidence that this approach can be successful (I remember another but could not find it):
    https://groups.google.com/g/janusgraph-users/c/Fh0ARPasw8s/m/tkdi_k5nAQAJ

  2. add spark-yarn to one or more janusgraph modules and make a custom build of janusgraph. Use the maven dependency-tree to judiciously exclude transitive dependencies to resolve library conflicts. This requires more maven-fu than the first option, but the advantage is that you end up with a working gremlin console.
  3. put on your blindfold and start deleting jars from the janusgraph lib folder. First candidates are all HBase jars, because they occur in the exception messages. Although the Yarn timeline server uses HBase, the yarn client should not depend on HBase, so the hbase jars can be safely removed. This approach requires some expertise but mostly luck (possibly it cannot succeed at all).

Best wishes,    Marc

Join {janusgraph-users@lists.lfaidata.foundation to automatically receive all group messages.