Lucene index long overflow


@inverseintegral
 

When I run the following code with janusgraph-inmemory and janusgraph-lucene both in version 0.6.2.

PropertiesConfiguration conf = ConfigurationUtil.loadPropertiesConfig("conf/test.properties");
JanusGraph graph = JanusGraphFactory.open(conf);

JanusGraphManagement m = graph.openManagement();
PropertyKey key = m.makePropertyKey("prop").dataType(Long.class).make();
VertexLabel label = m.makeVertexLabel("test").make();
m.buildIndex("propIndex", Vertex.class).addKey(key).indexOnly(label).buildMixedIndex("search");
m.commit();

graph.traversal()
.V()
.hasLabel("test")
.has("prop", P.neq(9223372036854775807L))
.count()
.next();


an unexpected exception is thrown:

Exception in thread "main" org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
    at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54)
    at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:488)
    at org.janusgraph.diskstorage.BackendTransaction.indexQueryCount(BackendTransaction.java:431)
    at org.janusgraph.graphdb.database.IndexSerializer.queryCount(IndexSerializer.java:603)
    at org.janusgraph.graphdb.query.graph.MixedIndexCountQueryBuilder.executeTotals(MixedIndexCountQueryBuilder.java:61)
    at org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphMixedIndexCountStep.processNextStart(JanusGraphMixedIndexCountStep.java:75)
    at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:135)
    at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:40)
    at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.next(DefaultTraversal.java:240)
    at Scratch.main(scratch.java:35)
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Permanent exception while executing backend operation indexQueryCount
    at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:79)
    at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:52)
    ... 9 more
Caused by: java.lang.ArithmeticException: long overflow
    at java.base/java.lang.Math.addExact(Math.java:848)
    at org.janusgraph.diskstorage.lucene.LuceneIndex.numericQuery(LuceneIndex.java:634)
    at org.janusgraph.diskstorage.lucene.LuceneIndex.convertQuery(LuceneIndex.java:766)
    at org.janusgraph.diskstorage.lucene.LuceneIndex.convertQuery(LuceneIndex.java:864)
    at org.janusgraph.diskstorage.lucene.LuceneIndex.queryCount(LuceneIndex.java:927)
    at org.janusgraph.diskstorage.indexing.IndexTransaction.queryCount(IndexTransaction.java:114)
    at org.janusgraph.diskstorage.BackendTransaction$7.call(BackendTransaction.java:434)
    at org.janusgraph.diskstorage.BackendTransaction$7.call(BackendTransaction.java:431)
    at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:66)
    ... 10 more

I would expect this query to run without any problem since the constant does not overflow.


@inverseintegral
 

A similar problem occurs when using the following traversal:

graph.traversal()
.V()
.hasLabel("test")
.has("prop", P.neq(-9223372036854775808L))
.count()
.next();


hadoopmarc@...
 
Edited

I am not sure what the trouble is in your approach. Using variable names key and label might be problematic (it is in the Gremlin Console). Rather than building out your example, I chose to rework the GraphOfTheGodsFactory with a Long property index for use in the console. See below, this provides you with a working playground (tested on JanusGraph-0.5.3).




graph = JanusGraphFactory.open('conf/janusgraph-berkeleyje-lucene.properties')


management = graph.openManagement();
name = management.makePropertyKey("name").dataType(String.class).make();
nameIndex = management.buildIndex("name", Vertex.class).addKey(name).unique().buildCompositeIndex();
management.setConsistency(nameIndex, ConsistencyModifier.LOCK);
age = management.makePropertyKey("age").dataType(Long.class).make();
management.buildIndex("vertices", Vertex.class).addKey(age).buildMixedIndex("search");

time = management.makePropertyKey("time").dataType(Integer.class).make();
reason = management.makePropertyKey("reason").dataType(String.class).make();
place = management.makePropertyKey("place").dataType(Geoshape.class).make();
management.buildIndex("edges", Edge.class).addKey(reason).addKey(place).buildMixedIndex("search");

management.makeEdgeLabel("father").multiplicity(Multiplicity.MANY2ONE).make();
management.makeEdgeLabel("mother").multiplicity(Multiplicity.MANY2ONE).make();
battled = management.makeEdgeLabel("battled").signature(time).make();
management.buildEdgeIndex(battled, "battlesByTime", Direction.BOTH, Order.desc, time);
management.makeEdgeLabel("lives").signature(reason).make();
management.makeEdgeLabel("pet").make();
management.makeEdgeLabel("brother").make();

management.makeVertexLabel("titan").make();
management.makeVertexLabel("location").make();
management.makeVertexLabel("god").make();
management.makeVertexLabel("demigod").make();
management.makeVertexLabel("human").make();
management.makeVertexLabel("monster").make();

management.commit();

tx = graph.newTransaction();
// vertices

saturn = tx.addVertex(T.label, "titan", "name", "saturn", "age", 10000);
sky = tx.addVertex(T.label, "location", "name", "sky");
sea = tx.addVertex(T.label, "location", "name", "sea");
jupiter = tx.addVertex(T.label, "god", "name", "jupiter", "age", 5000);
neptune = tx.addVertex(T.label, "god", "name", "neptune", "age", 4500);
hercules = tx.addVertex(T.label, "demigod", "name", "hercules", "age", 30);
alcmene = tx.addVertex(T.label, "human", "name", "alcmene", "age", 45);
pluto = tx.addVertex(T.label, "god", "name", "pluto", "age", 4000);
nemean = tx.addVertex(T.label, "monster", "name", "nemean");
hydra = tx.addVertex(T.label, "monster", "name", "hydra");
cerberus = tx.addVertex(T.label, "monster", "name", "cerberus");
tartarus = tx.addVertex(T.label, "location", "name", "tartarus");

// edges

jupiter.addEdge("father", saturn);
jupiter.addEdge("lives", sky, "reason", "loves fresh breezes");
jupiter.addEdge("brother", neptune);
jupiter.addEdge("brother", pluto);

neptune.addEdge("lives", sea).property("reason", "loves waves");
neptune.addEdge("brother", jupiter);
neptune.addEdge("brother", pluto);

hercules.addEdge("father", jupiter);
hercules.addEdge("mother", alcmene);
hercules.addEdge("battled", nemean, "time", 1, "place", Geoshape.point(38.1f, 23.7f));
hercules.addEdge("battled", hydra, "time", 2, "place", Geoshape.point(37.7f, 23.9f));
hercules.addEdge("battled", cerberus, "time", 12, "place", Geoshape.point(39f, 22f));

pluto.addEdge("brother", jupiter);
pluto.addEdge("brother", neptune);
pluto.addEdge("lives", tartarus, "reason", "no fear of death");
pluto.addEdge("pet", cerberus);

cerberus.addEdge("lives", tartarus);

// commit the transaction to disk
tx.commit();

g = graph.traversal()
g.V().has("age", P.lt(9223372036854775807L))

gremlin> g.V().has("age", P.lt(9223372036854775807L))
==>v[4264]
==>v[12536]
==>v[8344]
==>v[8440]
==>v[8272]
==>v[4344]


@inverseintegral
 

Your example indeed seems to work fine. If you change the traversal as such:

g.V().has("age", P.neq(9223372036854775807L))

it should throw an exception. The neq predicate seems to be the problem here.


hadoopmarc@...
 

OK, good catch, it is a bug. I had not recognized Long.MAX_VALUE in your query. The LuceneIndex class converts the neq to a range query and adds + 1 to the requested value without any check. The writer must have thought nobody would notice. It works for:
g.V().has("age", P.neq(9223372036854775806L))

If you want, you can report it as an issue.

Best wishes,   Marc