Hi all,
I've come across something with the .limit() step that seems like either a bug or missed optimization.
I have a graph loaded with ~5.5 million vertices and ~55 million edges. There are 4 vertex labels, "Server" (5,000), "Library" (50,000), "Table" (500,000), "Column" (5,000,000). Each vertex has a property "typeDef" that reflects that vertex's label. This property has a composite index on it.
I just want to return 10 Table vertices, for example. My query looks like
g.V().has('Table', 'typeDef', 'Table').limit(10).valueMap().toList()
This takes about close to 7 seconds to return, as you can see here in the profile:
gremlin> g.V().has('Table', 'typeDef', 'Table').limit(10).valueMap().profile()
==>Traversal Metrics
Step Count Traversers Time (ms) % Dur
=============================================================================================================
JanusGraphStep([],[~label.eq(Table), typeDef.eq... 10 10 6610.874 99.99
\_condition=(~label = Table AND typeDef = Table)
\_orders=[]
\_limit=10
\_isFitted=false
\_isOrdered=true
\_query=multiKSQ[1]@2147483647
\_index=typeDefIndex
optimization 0.144
optimization 0.890
backend-query 0.000
\_query=typeDefIndex:multiKSQ[1]@2147483647
PropertyMapStep(value) 10 10 0.364 0.01
>TOTAL - - 6611.239 -
Clearly it is using the index, as expected, but it is very slow for only needing to find and return 10 objects.
Interestingly, if I look for Columns (the most frequent label) instead it is even slower
gremlin> g.V().has('Column', 'typeDef', 'Column').limit(10).valueMap().profile()
==>Traversal Metrics
Step Count Traversers Time (ms) % Dur
=============================================================================================================
JanusGraphStep([],[~label.eq(Column), typeDef.e... 10 10 44905.231 100.00
\_condition=(~label = Column AND typeDef = Column)
\_orders=[]
\_limit=10
\_isFitted=false
\_isOrdered=true
\_query=multiKSQ[1]@2147483647
\_index=typeDefIndex
optimization 0.101
optimization 0.333
backend-query 0.000
\_query=typeDefIndex:multiKSQ[1]@2147483647
PropertyMapStep(value) 10 10 0.192 0.00
>TOTAL - - 44905.423 -
here...
Likewise, looking for Servers (the least frequent label) is the fastest
gremlin> g.V().has('Server', 'typeDef', 'Server').limit(10).valueMap().profile()
==>Traversal Metrics
Step Count Traversers Time (ms) % Dur
=============================================================================================================
JanusGraphStep([],[~label.eq(Server), typeDef.e... 10 10 40.789 99.19
\_condition=(~label = Server AND typeDef = Server)
\_orders=[]
\_limit=10
\_isFitted=false
\_isOrdered=true
\_query=multiKSQ[1]@2147483647
\_index=typeDefIndex
optimization 0.047
optimization 0.308
backend-query 0.000
\_query=typeDefIndex:multiKSQ[1]@2147483647
PropertyMapStep(value) 10 10 0.332 0.81
>TOTAL - - 41.121 -
I assumed these queries would execute such that the 'has' step only gathers as many vertices as the 'limit' step specifies. However, it looks like it's gathering all vertices with that label/typeDef, and then filtering down to the limit spec, hence why the most frequent labels take a lot longer to come back.
So what's going on here? Is there a better way to write these queries? Is there a bug/missing optimization in JanusGraphStep?
Thanks,
JP
