Multiple vertices generated for the same index value and vertex properties missing with RF3


sauverma
 

We are facing a weird issue with Replication factor 3 with scylla DB as backend.  

- we are using index for looking up V in the graph
- observation is that for many vertices with the same index value, multiple vertex IDs are generated as seen below



Has anyone else encountered the same issue with RF3, with RF1 the issue is not encountered

Thanks
Saurabh Verma


hadoopmarc@...
 

Hi Saurabh,

To be sure, in the valueMap(true) return value I see both the id token and an "id" property key appear (I am not even sure whether an "id" property key is allowed). Is this "id" property key something that was part of the input data of the graph? Can you also show the graph schema using:
mgmt = graph.openManagement()
mgmt.printSchema()
Best wishes,    Marc


sauverma
 

Hi Marc

==>------------------------------------------------------------------------------------------------
Vertex Label Name              | Partitioned | Static                                             |
---------------------------------------------------------------------------------------------------
node                           | false       | false                                              |
---------------------------------------------------------------------------------------------------
Edge Label Name                | Directed    | Unidirected | Multiplicity                         |
---------------------------------------------------------------------------------------------------
link                           | true        | false       | SIMPLE                               |
---------------------------------------------------------------------------------------------------
Property Key Name              | Cardinality | Data Type                                          |
---------------------------------------------------------------------------------------------------
country_LUX                    | SET         | class java.lang.Integer                            |
country_FRA                    | SET         | class java.lang.Integer                            |
country_CHE                    | SET         | class java.lang.Integer                            |
country_ESP                    | SET         | class java.lang.Integer                            |
dpts_0                         | SINGLE      | class java.lang.Long                               |
dpts_100                       | SINGLE      | class java.lang.Long                               |
dpts_101                       | SINGLE      | class java.lang.Long                               |
dpts_1011                      | SINGLE      | class java.lang.Long                               |
dpts_103                       | SINGLE      | class java.lang.Long                               |
dpts_1038                      | SINGLE      | class java.lang.Long                               |
dpts_104                       | SINGLE      | class java.lang.Long                               |
dpts_1045                      | SINGLE      | class java.lang.Long                               |
dpts_105                       | SINGLE      | class java.lang.Long                               |
dpts_1055                      | SINGLE      | class java.lang.Long                               |
dpts_1076                      | SINGLE      | class java.lang.Long                               |
dpts_1077                      | SINGLE      | class java.lang.Long                               |
dpts_1081                      | SINGLE      | class java.lang.Long                               |
dpts_1085                      | SINGLE      | class java.lang.Long                               |
dpts_1099                      | SINGLE      | class java.lang.Long                               |
dpts_1110                      | SINGLE      | class java.lang.Long                               |
dpts_1129                      | SINGLE      | class java.lang.Long                               |
dpts_1141                      | SINGLE      | class java.lang.Long                               |
dpts_1146                      | SINGLE      | class java.lang.Long                               |
dpts_1147                      | SINGLE      | class java.lang.Long                               |
dpts_1148                      | SINGLE      | class java.lang.Long                               |
dpts_1168                      | SINGLE      | class java.lang.Long                               |
dpts_118                       | SINGLE      | class java.lang.Long                               |
dpts_12                        | SINGLE      | class java.lang.Long                               |
dpts_120                       | SINGLE      | class java.lang.Long                               |
dpts_1201                      | SINGLE      | class java.lang.Long                               |
dpts_121                       | SINGLE      | class java.lang.Long                               |
dpts_1222                      | SINGLE      | class java.lang.Long                               |
dpts_123                       | SINGLE      | class java.lang.Long                               |
dpts_1249                      | SINGLE      | class java.lang.Long                               |
dpts_1252                      | SINGLE      | class java.lang.Long                               |
dpts_1257                      | SINGLE      | class java.lang.Long                               |
dpts_1258                      | SINGLE      | class java.lang.Long                               |
dpts_1263                      | SINGLE      | class java.lang.Long                               |
dpts_1271                      | SINGLE      | class java.lang.Long                               |
dpts_1290                      | SINGLE      | class java.lang.Long                               |
dpts_1292                      | SINGLE      | class java.lang.Long                               |
dpts_1295                      | SINGLE      | class java.lang.Long                               |
dpts_13                        | SINGLE      | class java.lang.Long                               |
dpts_1302                      | SINGLE      | class java.lang.Long                               |
dpts_1314                      | SINGLE      | class java.lang.Long                               |
dpts_1316                      | SINGLE      | class java.lang.Long                               |
dpts_1325                      | SINGLE      | class java.lang.Long                               |
dpts_1326                      | SINGLE      | class java.lang.Long                               |
dpts_1332                      | SINGLE      | class java.lang.Long                               |
dpts_1344                      | SINGLE      | class java.lang.Long                               |
dpts_1346                      | SINGLE      | class java.lang.Long                               |
dpts_1348                      | SINGLE      | class java.lang.Long                               |
dpts_135                       | SINGLE      | class java.lang.Long                               |
dpts_1353                      | SINGLE      | class java.lang.Long                               |
dpts_1354                      | SINGLE      | class java.lang.Long                               |
dpts_1357                      | SINGLE      | class java.lang.Long                               |
dpts_1361                      | SINGLE      | class java.lang.Long                               |
dpts_1363                      | SINGLE      | class java.lang.Long                               |
dpts_1380                      | SINGLE      | class java.lang.Long                               |
dpts_1382                      | SINGLE      | class java.lang.Long                               |
dpts_1396                      | SINGLE      | class java.lang.Long                               |
dpts_1401                      | SINGLE      | class java.lang.Long                               |
dpts_1406                      | SINGLE      | class java.lang.Long                               |
dpts_1408                      | SINGLE      | class java.lang.Long                               |
country_NLD                    | SET         | class java.lang.Integer                            |
dpts_1409                      | SINGLE      | class java.lang.Long                               |
dpts_1410                      | SINGLE      | class java.lang.Long                               |
dpts_1415                      | SINGLE      | class java.lang.Long                               |
dpts_1429                      | SINGLE      | class java.lang.Long                               |
dpts_1441                      | SINGLE      | class java.lang.Long                               |
dpts_1443                      | SINGLE      | class java.lang.Long                               |
dpts_1479                      | SINGLE      | class java.lang.Long                               |
dpts_1480                      | SINGLE      | class java.lang.Long                               |
dpts_1491                      | SINGLE      | class java.lang.Long                               |
dpts_18                        | SINGLE      | class java.lang.Long                               |
dpts_1801                      | SINGLE      | class java.lang.Long                               |
dpts_21                        | SINGLE      | class java.lang.Long                               |
dpts_212                       | SINGLE      | class java.lang.Long                               |
dpts_213                       | SINGLE      | class java.lang.Long                               |
dpts_215                       | SINGLE      | class java.lang.Long                               |
dpts_216                       | SINGLE      | class java.lang.Long                               |
dpts_22                        | SINGLE      | class java.lang.Long                               |
dpts_222                       | SINGLE      | class java.lang.Long                               |
dpts_226                       | SINGLE      | class java.lang.Long                               |
dpts_229                       | SINGLE      | class java.lang.Long                               |
dpts_233                       | SINGLE      | class java.lang.Long                               |
dpts_237                       | SINGLE      | class java.lang.Long                               |
dpts_239                       | SINGLE      | class java.lang.Long                               |
dpts_24                        | SINGLE      | class java.lang.Long                               |
dpts_255                       | SINGLE      | class java.lang.Long                               |
dpts_258                       | SINGLE      | class java.lang.Long                               |
dpts_26                        | SINGLE      | class java.lang.Long                               |
dpts_27                        | SINGLE      | class java.lang.Long                               |
dpts_28                        | SINGLE      | class java.lang.Long                               |
dpts_29                        | SINGLE      | class java.lang.Long                               |
dpts_307                       | SINGLE      | class java.lang.Long                               |
dpts_31                        | SINGLE      | class java.lang.Long                               |
dpts_313                       | SINGLE      | class java.lang.Long                               |
dpts_314                       | SINGLE      | class java.lang.Long                               |
dpts_320                       | SINGLE      | class java.lang.Long                               |
dpts_322                       | SINGLE      | class java.lang.Long                               |
dpts_329                       | SINGLE      | class java.lang.Long                               |
dpts_33                        | SINGLE      | class java.lang.Long                               |
dpts_331                       | SINGLE      | class java.lang.Long                               |
dpts_335                       | SINGLE      | class java.lang.Long                               |
dpts_336                       | SINGLE      | class java.lang.Long                               |
dpts_338                       | SINGLE      | class java.lang.Long                               |
dpts_34                        | SINGLE      | class java.lang.Long                               |
dpts_346                       | SINGLE      | class java.lang.Long                               |
dpts_35                        | SINGLE      | class java.lang.Long                               |
dpts_362                       | SINGLE      | class java.lang.Long                               |
dpts_37                        | SINGLE      | class java.lang.Long                               |
dpts_370                       | SINGLE      | class java.lang.Long                               |
dpts_372                       | SINGLE      | class java.lang.Long                               |
dpts_374                       | SINGLE      | class java.lang.Long                               |
dpts_376                       | SINGLE      | class java.lang.Long                               |
dpts_391                       | SINGLE      | class java.lang.Long                               |
dpts_395                       | SINGLE      | class java.lang.Long                               |
dpts_41                        | SINGLE      | class java.lang.Long                               |
dpts_416                       | SINGLE      | class java.lang.Long                               |
dpts_45                        | SINGLE      | class java.lang.Long                               |
dpts_50                        | SINGLE      | class java.lang.Long                               |
dpts_516                       | SINGLE      | class java.lang.Long                               |
dpts_521                       | SINGLE      | class java.lang.Long                               |
dpts_522                       | SINGLE      | class java.lang.Long                               |
dpts_525                       | SINGLE      | class java.lang.Long                               |
dpts_529                       | SINGLE      | class java.lang.Long                               |
dpts_532                       | SINGLE      | class java.lang.Long                               |
dpts_533                       | SINGLE      | class java.lang.Long                               |
dpts_556                       | SINGLE      | class java.lang.Long                               |
dpts_56                        | SINGLE      | class java.lang.Long                               |
dpts_566                       | SINGLE      | class java.lang.Long                               |
dpts_568                       | SINGLE      | class java.lang.Long                               |
dpts_575                       | SINGLE      | class java.lang.Long                               |
dpts_58                        | SINGLE      | class java.lang.Long                               |
dpts_59                        | SINGLE      | class java.lang.Long                               |
dpts_598                       | SINGLE      | class java.lang.Long                               |
dpts_611                       | SINGLE      | class java.lang.Long                               |
dpts_613                       | SINGLE      | class java.lang.Long                               |
dpts_620                       | SINGLE      | class java.lang.Long                               |
dpts_625                       | SINGLE      | class java.lang.Long                               |
dpts_627                       | SINGLE      | class java.lang.Long                               |
dpts_63                        | SINGLE      | class java.lang.Long                               |
dpts_634                       | SINGLE      | class java.lang.Long                               |
dpts_637                       | SINGLE      | class java.lang.Long                               |
dpts_648                       | SINGLE      | class java.lang.Long                               |
dpts_654                       | SINGLE      | class java.lang.Long                               |
dpts_667                       | SINGLE      | class java.lang.Long                               |
dpts_67                        | SINGLE      | class java.lang.Long                               |
dpts_678                       | SINGLE      | class java.lang.Long                               |
dpts_68                        | SINGLE      | class java.lang.Long                               |
dpts_685                       | SINGLE      | class java.lang.Long                               |
dpts_69                        | SINGLE      | class java.lang.Long                               |
dpts_697                       | SINGLE      | class java.lang.Long                               |
dpts_7123                      | SINGLE      | class java.lang.Long                               |
dpts_735                       | SINGLE      | class java.lang.Long                               |
dpts_738                       | SINGLE      | class java.lang.Long                               |
dpts_75                        | SINGLE      | class java.lang.Long                               |
dpts_751                       | SINGLE      | class java.lang.Long                               |
dpts_754                       | SINGLE      | class java.lang.Long                               |
dpts_76                        | SINGLE      | class java.lang.Long                               |
dpts_768                       | SINGLE      | class java.lang.Long                               |
dpts_78                        | SINGLE      | class java.lang.Long                               |
dpts_784                       | SINGLE      | class java.lang.Long                               |
dpts_799                       | SINGLE      | class java.lang.Long                               |
dpts_800                       | SINGLE      | class java.lang.Long                               |
dpts_801                       | SINGLE      | class java.lang.Long                               |
dpts_806                       | SINGLE      | class java.lang.Long                               |
dpts_807                       | SINGLE      | class java.lang.Long                               |
dpts_808                       | SINGLE      | class java.lang.Long                               |
dpts_809                       | SINGLE      | class java.lang.Long                               |
dpts_810                       | SINGLE      | class java.lang.Long                               |
dpts_82                        | SINGLE      | class java.lang.Long                               |
dpts_820                       | SINGLE      | class java.lang.Long                               |
dpts_824                       | SINGLE      | class java.lang.Long                               |
dpts_84                        | SINGLE      | class java.lang.Long                               |
dpts_86                        | SINGLE      | class java.lang.Long                               |
dpts_87                        | SINGLE      | class java.lang.Long                               |
dpts_871                       | SINGLE      | class java.lang.Long                               |
dpts_876                       | SINGLE      | class java.lang.Long                               |
dpts_90901                     | SINGLE      | class java.lang.Long                               |
dpts_946                       | SINGLE      | class java.lang.Long                               |
dpts_95                        | SINGLE      | class java.lang.Long                               |
dpts_96                        | SINGLE      | class java.lang.Long                               |
dpts_963                       | SINGLE      | class java.lang.Long                               |
dpts_979                       | SINGLE      | class java.lang.Long                               |
dpts_98                        | SINGLE      | class java.lang.Long                               |
dpts_984                       | SINGLE      | class java.lang.Long                               |
dpts_988                       | SINGLE      | class java.lang.Long                               |
dpts_99                        | SINGLE      | class java.lang.Long                               |
device_deviceos                | SET         | class java.lang.String                             |
device_ipaddress               | SET         | class java.lang.String                             |
id                             | SINGLE      | class java.lang.String                             |
country_AUT                    | SET         | class java.lang.Integer                            |
type                           | SINGLE      | class java.lang.String                             |
soft_del                       | SINGLE      | class java.lang.Boolean                            |
country_IRL                    | SET         | class java.lang.Integer                            |
country_ITA                    | SET         | class java.lang.Integer                            |
country_GBR                    | SET         | class java.lang.Integer                            |
country_BEL                    | SET         | class java.lang.Integer                            |
country_DEU                    | SET         | class java.lang.Integer                            |
country_GEO                    | SET         | class java.lang.Integer                            |
---------------------------------------------------------------------------------------------------
Vertex Index Name              | Type        | Unique    | Backing        | Key:           Status |
---------------------------------------------------------------------------------------------------
idIndex                        | Composite   | true      | internalindex  | id:           ENABLED |
                               |             |           |                | type:         ENABLED |
---------------------------------------------------------------------------------------------------
Edge Index (VCI) Name          | Type        | Unique    | Backing        | Key:           Status |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
Relation Index                 | Type        | Direction | Sort Key       | Order    |     Status |
---------------------------------------------------------------------------------------------------

After modifying index and properties to ConsistenceModifier.LOCK as per https://docs.janusgraph.org/advanced-topics/eventual-consistency/
the issue seems to be fixed

Thanks


hadoopmarc@...
 

Hi Saurabh,

The workaround you found has implications for performance. So, if you can describe how  to reproduce the issue, you can make an issue for it on https://github.com/JanusGraph/janusgraph/issues
This might easily be a scenario that is not covered by the current janusgraph tests.

I see that "id" is indeed a property key in the schema. I assume your input data already had this "id" property and this was not generated by janusgraph. In the former case does replacing "id" by "user_id" make any difference (this will make your queries more readable anyway)?

Best wishes,    Marc


sauverma
 

Hi

The issue still persists, and the vertex metadata is still missing for some vertices, after enabling https://docs.janusgraph.org/advanced-topics/eventual-consistency/, has someone seen the same issue.

The issue is logged at https://github.com/JanusGraph/janusgraph/issues/2515

Thanks


sauverma
 

Another really strange observation

gremlin> g.V().has('id','131594d6a416666b401a9e48e54ebc8f22be75e2593c5d98e2d9ecfd719d5f29').has('type','email_sha256_lowercase').valueMap(true)
==>[dpts_678:[1595548800],label:vertex,id:201523209257056,id:[19df651e-90d5-47f6-af2e-35dcb59bcc0a],type:[id_mid_10],soft_del:[false],country_GBR:[678]]


Could you please have a look?


hadoopmarc@...
 

Hi,
You did not answer my questions about the "id" poperty. TinkerPop uses a Token.ID that has the value 'id', see:

https://github.com/apache/tinkerpop/blob/master/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/T.java

I suspect that you ingested data without schema validation ("automatic schema creation"), that your input data contains an "id¨ property key and that JanusGraph/TinkerPop get confused about which id is what. So I strongly suggest that you make sure that this is not the root cause of this issue. To be sure, it would still be an issue but not for you anymore :-)

Best wishes,    Marc