Date
1 - 4 of 4
java.io.EOFException in kryo+blvp error in bulk loading
|
Elizabeth <hlf...@...>
Hi all, I was using the Kryo format and BulkLoaderVertexProgram to load large files into Janusgraph, and encountered an error: gremlin> hdfs.copyFromLocal('data/test.kryo','data/test.kryo') ==>null gremlin> graph = GraphFactory.open('conf/hadoop-graph/hadoop-load.properties') ==>hadoopgraph[gryoinputformat->gryooutputformat] gremlin> gremlin> blvp = BulkLoaderVertexProgram.build().writeGraph('conf/janusgraph-hbase-es.properties').create(graph) ==>BulkLoaderVertexProgram[bulkLoader=IncrementalBulkLoader, vertexIdProperty=bulkLoader.vertex.id, userSuppliedIds=false, keepOriginalIds=true, batchSize=0] gremlin> gremlin> result = graph.compute(SparkGraphComputer).program(blvp).submit().get() 20:21:32 ERROR org.apache.spark.executor.Executor - Exception in task 0.0 in stage 0.0 (TID 0) java.io.EOFException at java.io.DataInputStream.readByte(DataInputStream.java:267) at org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoRecordReader.seekToHeader(GryoRecordReader.java:93) at org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoRecordReader.initialize(GryoRecordReader.java:85) at org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat.createRecordReader(GryoInputFormat.java:38) has anyone ever had this error before, please help me with this last step! Any idea is appreciated! Best, Meng |
|
|
|
marc.d...@...
Hi Eliz and Meng, Did the seqence of gremlin commands work for the tinkerpop-modern.kryo and grateful-dead.kryo example files? How did you create the test.kryo file? Marc Op woensdag 28 juni 2017 15:02:27 UTC+2 schreef Elizabeth:
|
|
|
|
Angelasweet <hlf...@...>
Hi Biko and Marc, Below is the method I used to create test.kryo, which was converted from a CSV file: vertices.txt vi vertices.txt 123 11692 11814 12473 14180 14319 14667 112228 vi CsvToJavaObject.java package mypkg; import java.io.BufferedReader; import java.io.FileNotFoundException; import java.io.FileReader; import java.io.IOException; import java.util.ArrayList; import java.util.List; import com.esotericsoftware.kryo.Kryo; import com.esotericsoftware.kryo.io.Input; import com.esotericsoftware.kryo.io.Output; import java.io.File; import java.io.FileOutputStream; import java.io.FileInputStream; import java.util.Arrays; public class CsvToJavaObject { public static void main(String[] args) { String csvFile = "/home/dev/wanmeng/kryo/mypkg/vertices.txt"; BufferedReader br = null; String line = ""; String csvSplitBy = "\n"; List<Long> list = new ArrayList<Long>(); File file; FileOutputStream fop = null; try { br = new BufferedReader(new FileReader(csvFile)); while ((line = br.readLine()) != null) { //trim the line breaks at the end of line. line = line.trim(); // adding Long objects to a list. list.add(Long.parseLong(line)); //System.out.println(list); } } catch (FileNotFoundException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } finally { if (br != null) { try { br.close(); } catch (IOException e) { e.printStackTrace(); } } } try{ Kryo kryo = new Kryo(); file = new File("test.kryo"); fop = new FileOutputStream(file); //if the file doesnt exist, then create it. if(!file.exists()) { file.createNewFile(); } Output output = new Output(fop); kryo.writeObject(output, list); output.close(); } catch (Exception ex) { ex.printStackTrace(); } finally { try{ if (fop != null) { fop.close(); } } catch (IOException e) { e.printStackTrace(); } } } public void deserialize() { File file; FileInputStream fip = null; // deserialize try{ Kryo kryo = new Kryo(); file = new File("test.kryo"); fip = new FileInputStream(file); if(!file.exists()) { file.createNewFile(); } Input input = new Input(fip); //ArrayList<Long> listin = kryo.readObject(input, Long.class); Long listin = kryo.readObject(input, Long.class); //System.out.println(Arrays.toString(listin.toArray())); System.out.println(listin); input.close(); } catch (Exception ex) { ex.printStackTrace(); } finally { try{ if (fip != null) { fip.close(); } } catch (IOException e) { e.printStackTrace(); } } } } javac -d . CsvToJavaObject.java java mypkg.CsvToJavaObject.java Thanks, Eliz 2017-07-02 23:14 GMT+08:00 <bi...@...>:
|
|
|
|
HadoopMarc <bi...@...>
Hi Eliz, You are supposed to use Tinkerpop's gryo writer to create kryo graph input files (I do not believe there is a formal spec of the format). There is also the BulkDumperVertexProgram to create a kryo outputfile from a HadoopGraph. If you have a large dataset on hdfs and you want a distributed load into JanusGraph it is also possible to use Spark mapPartitions and have each Spark task make its own connection to the same JanusGraph. HTH, Marc Op donderdag 6 juli 2017 04:45:22 UTC+2 schreef Elizabeth:
|
|
|