Titan to JanusGraph migration Elasticsearc| "janusgraph" index is created and used , not titan


akshay...@...
 

Refer https://github.com/JanusGraph/janusgraph/issues/228 for other related issue with migration. 
It seems the default name of elastic search index is also changed from 'titan' to  'janusgraph'. 
"index.search.index-name" property name option should help. 
However I am afraid its not working. I am using BerkeleyDB+Embedded ES option.  
1.  I created the DB with Titan. 
`storage.backend=berkeleyje
storage.directory=data/titan-janus/berkeley
index.search.backend=elasticsearch
index.search.directory=data/titan-janus/es
index.search.elasticsearch.client-only=false
index.search.elasticsearch.local-mode=true
`
2. At this point titan index is created. http://localhost:9200/titan returns index definition.  http://localhost:9200/janusgraph/ returns IndexMissingException. 
3. Data is created in titan index. 
4. Now I migrate the code to Janus. I thought index-name should work and added - 
'index.search.index-name=titan'
5. Now when I start , storage backend works seamlessly. I can query nodes added with Titan. And new nodes are added. 
6. The index backend does not work well. "janusgraph" index is created ( even when index-name is titan). The new nodes never go to "titan" , they are created in "janusgraph" index. 

Akshya


Jason Plurad <plu...@...>
 

Hi Akshya,

Thanks for the info, and unfortunately it looks like some scenarios were missed. Note that if you created the Titan graph originally with the `index.search.index-name` property set, it would work without additional configuration in JanusGraph.

In your scenario, when opening the graph in JanusGraph with  `index.search.index-name=titan`, this warning is shown:

WARN  org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration  - Local setting index.search.index-name=titan (Type: GLOBAL_OFFLINE) is overridden by globally managed value (janusgraph).  Use the ManagementSystem interface instead of the local configuration to control this setting.

The additional work you need to do is:

mgmt = graph.openManagement()
mgmt
.set('index.search.index-name', 'titan')
mgmt
.commit()
// at this point the graph is closed, so reopen it with the same properties



On Tuesday, April 25, 2017 at 11:18:11 AM UTC-4, Akshaya Rawat wrote:
Refer https://github.com/JanusGraph/janusgraph/issues/228 for other related issue with migration. 
It seems the default name of elastic search index is also changed from 'titan' to  'janusgraph'. 
"index.search.index-name" property name option should help. 
However I am afraid its not working. I am using BerkeleyDB+Embedded ES option.  
1.  I created the DB with Titan. 
`storage.backend=berkeleyje
storage.directory=data/titan-janus/berkeley
index.search.backend=elasticsearch
index.search.directory=data/titan-janus/es
index.search.elasticsearch.client-only=false
index.search.elasticsearch.local-mode=true
`
2. At this point titan index is created. http://localhost:9200/titan returns index definition.  http://localhost:9200/janusgraph/ returns IndexMissingException. 
3. Data is created in titan index. 
4. Now I migrate the code to Janus. I thought index-name should work and added - 
'index.search.index-name=titan'
5. Now when I start , storage backend works seamlessly. I can query nodes added with Titan. And new nodes are added. 
6. The index backend does not work well. "janusgraph" index is created ( even when index-name is titan). The new nodes never go to "titan" , they are created in "janusgraph" index. 

Akshya


Akshaya Rawat <akshay...@...>
 

Hi Jason,
Thanks for quick response.
Will try this and it should work.

Thanks
Akshaya


Akshaya Rawat <akshay...@...>
 

Hi Jason, 
I tried to do that. 
JanusGraphManagement janusGraphManagement=janusGraph.openManagement();
janusGraphManagement.set("index.search.index-name", "titan");
janusGraphManagement.commit();


But I got error - 
Caused by: java.lang.IllegalArgumentException: Cannot change offline config option [root.index.index-name] since multiple instances are currently open: [c0a838679548-NOIARAWA3569221, c0a8386711772-NOIARAWA3569221, c0a8386712068-NOIARAWA3569221, c0a8386712596-NOIARAWA3569221]
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
at org.janusgraph.graphdb.database.management.ManagementSystem$1.verifyModification(ManagementSystem.java:180)
at org.janusgraph.diskstorage.configuration.UserModifiableConfiguration.set(UserModifiableConfiguration.java:99)
at org.janusgraph.graphdb.database.management.ManagementSystem.set(ManagementSystem.java:1316)

If I close the graph, then set , I get another error - 
janusGraph.close();
JanusGraphManagement janusGraphManagement=janusGraph.openManagement();
janusGraphManagement.set("index.search.index-name", "titan");
janusGraphManagement.commit();

Caused by: java.lang.IllegalStateException: Environment is closed.
at com.sleepycat.je.Environment.getNonNullEnvImpl(Environment.java:2496)
at com.sleepycat.je.Environment.checkOpen(Environment.java:2472)
at com.sleepycat.je.Environment.openDatabase(Environment.java:643)
at org.janusgraph.diskstorage.berkeleyje.BerkeleyJEStoreManager.openDatabase(BerkeleyJEStoreManager.java:185)
at org.janusgraph.diskstorage.berkeleyje.BerkeleyJEStoreManager.openDatabase(BerkeleyJEStoreManager.java:48)
at org.janusgraph.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreManagerAdapter.openDatabase(OrderedKeyValueStoreManagerAdapter.java:89)
at org.janusgraph.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreManagerAdapter.openDatabase(OrderedKeyValueStoreManagerAdapter.java:38)
at org.janusgraph.diskstorage.log.kcvs.KCVSLogManager.openLog(KCVSLogManager.java:225)
at org.janusgraph.diskstorage.Backend.getSystemMgmtLog(Backend.java:337)
at org.janusgraph.graphdb.database.StandardJanusGraph.openManagement(StandardJanusGraph.java:303)


I am not sure how to move ahead. 

Thanks
Akshaya


On Tuesday, 25 April 2017 14:19:50 UTC-7, Akshaya Rawat wrote:
Hi Jason,
Thanks for quick response.
Will try this and it should work.

Thanks
Akshaya


Jason Plurad <plu...@...>
 

The first error message said that there were multiple open instances. In order to close them, again you must work through the management system (do not close the graph you are already connect to).

Try something like this:

mgmt = graph.openManagement()
open
= mgmt.openInstances() // returns a set of unique instance ids
for (String inst : open) { if (!inst.endsWith("(current)")) mgmt.forceCloseInstance(inst) }
mgmt
.set('index.search.index-name', 'titan')
mgmt
.commit()

https://github.com/JanusGraph/janusgraph/blob/v0.1.0/janusgraph-core/src/main/java/org/janusgraph/core/schema/JanusGraphManagement.java#L364

On Wednesday, April 26, 2017 at 2:47:43 AM UTC-4, Akshaya Rawat wrote:
Hi Jason, 
I tried to do that. 
JanusGraphManagement janusGraphManagement=janusGraph.openManagement();
janusGraphManagement.set("index.search.index-name", "titan");
janusGraphManagement.commit();


But I got error - 
Caused by: java.lang.IllegalArgumentException: Cannot change offline config option [root.index.index-name] since multiple instances are currently open: [c0a838679548-NOIARAWA3569221, c0a8386711772-NOIARAWA3569221, c0a8386712068-NOIARAWA3569221, c0a8386712596-NOIARAWA3569221]
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
at org.janusgraph.graphdb.database.management.ManagementSystem$1.verifyModification(ManagementSystem.java:180)
at org.janusgraph.diskstorage.configuration.UserModifiableConfiguration.set(UserModifiableConfiguration.java:99)
at org.janusgraph.graphdb.database.management.ManagementSystem.set(ManagementSystem.java:1316)

If I close the graph, then set , I get another error - 
janusGraph.close();
JanusGraphManagement janusGraphManagement=janusGraph.openManagement();
janusGraphManagement.set("index.search.index-name", "titan");
janusGraphManagement.commit();

Caused by: java.lang.IllegalStateException: Environment is closed.
at com.sleepycat.je.Environment.getNonNullEnvImpl(Environment.java:2496)
at com.sleepycat.je.Environment.checkOpen(Environment.java:2472)
at com.sleepycat.je.Environment.openDatabase(Environment.java:643)
at org.janusgraph.diskstorage.berkeleyje.BerkeleyJEStoreManager.openDatabase(BerkeleyJEStoreManager.java:185)
at org.janusgraph.diskstorage.berkeleyje.BerkeleyJEStoreManager.openDatabase(BerkeleyJEStoreManager.java:48)
at org.janusgraph.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreManagerAdapter.openDatabase(OrderedKeyValueStoreManagerAdapter.java:89)
at org.janusgraph.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreManagerAdapter.openDatabase(OrderedKeyValueStoreManagerAdapter.java:38)
at org.janusgraph.diskstorage.log.kcvs.KCVSLogManager.openLog(KCVSLogManager.java:225)
at org.janusgraph.diskstorage.Backend.getSystemMgmtLog(Backend.java:337)
at org.janusgraph.graphdb.database.StandardJanusGraph.openManagement(StandardJanusGraph.java:303)


I am not sure how to move ahead. 

Thanks
Akshaya

On Tuesday, 25 April 2017 14:19:50 UTC-7, Akshaya Rawat wrote:
Hi Jason,
Thanks for quick response.
Will try this and it should work.

Thanks
Akshaya


Akshaya Rawat <akshay...@...>
 

This solutions worked for me. Commit had to be done after the force closure. 
JanusGraphManagement jMgmt = jGraph.openManagement();
Set<String> open = jMgmt.getOpenInstances();
for (String string : open) {
if (!string.endsWith("(current)")) {
jMgmt.forceCloseInstance(string);
}
jMgmt.commit();
jMgmt = jGraph.openManagement();
jMgmt.set("index.search.index-name", "titan");
jMgmt.commit();

The graph has to be initialized all over again after this since its closed after this. 
This looks to be disruptive solution though the graph is closed and has to be initialized all over again. However this only has to be done once. Once this configuration is forced, it persists. So this could be done as one time migration may be using a script and then it works then on. 

Thanks for the solution. 

Akshaya


On Wednesday, 26 April 2017 02:36:48 UTC-7, Jason Plurad wrote:
The first error message said that there were multiple open instances. In order to close them, again you must work through the management system (do not close the graph you are already connect to).

Try something like this:

mgmt = graph.openManagement()
open
= mgmt.openInstances() // returns a set of unique instance ids
for (String inst : open) { if (!inst.endsWith("(current)")) mgmt.forceCloseInstance(inst) }
mgmt
.set('index.search.index-name', 'titan')
mgmt
.commit()

On Wednesday, April 26, 2017 at 2:47:43 AM UTC-4, Akshaya Rawat wrote:
Hi Jason, 
I tried to do that. 
JanusGraphManagement janusGraphManagement=janusGraph.openManagement();
janusGraphManagement.set("index.search.index-name", "titan");
janusGraphManagement.commit();


But I got error - 
Caused by: java.lang.IllegalArgumentException: Cannot change offline config option [root.index.index-name] since multiple instances are currently open: [c0a838679548-NOIARAWA3569221, c0a8386711772-NOIARAWA3569221, c0a8386712068-NOIARAWA3569221, c0a8386712596-NOIARAWA3569221]
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
at org.janusgraph.graphdb.database.management.ManagementSystem$1.verifyModification(ManagementSystem.java:180)
at org.janusgraph.diskstorage.configuration.UserModifiableConfiguration.set(UserModifiableConfiguration.java:99)
at org.janusgraph.graphdb.database.management.ManagementSystem.set(ManagementSystem.java:1316)

If I close the graph, then set , I get another error - 
janusGraph.close();
JanusGraphManagement janusGraphManagement=janusGraph.openManagement();
janusGraphManagement.set("index.search.index-name", "titan");
janusGraphManagement.commit();

Caused by: java.lang.IllegalStateException: Environment is closed.
at com.sleepycat.je.Environment.getNonNullEnvImpl(Environment.java:2496)
at com.sleepycat.je.Environment.checkOpen(Environment.java:2472)
at com.sleepycat.je.Environment.openDatabase(Environment.java:643)
at org.janusgraph.diskstorage.berkeleyje.BerkeleyJEStoreManager.openDatabase(BerkeleyJEStoreManager.java:185)
at org.janusgraph.diskstorage.berkeleyje.BerkeleyJEStoreManager.openDatabase(BerkeleyJEStoreManager.java:48)
at org.janusgraph.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreManagerAdapter.openDatabase(OrderedKeyValueStoreManagerAdapter.java:89)
at org.janusgraph.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreManagerAdapter.openDatabase(OrderedKeyValueStoreManagerAdapter.java:38)
at org.janusgraph.diskstorage.log.kcvs.KCVSLogManager.openLog(KCVSLogManager.java:225)
at org.janusgraph.diskstorage.Backend.getSystemMgmtLog(Backend.java:337)
at org.janusgraph.graphdb.database.StandardJanusGraph.openManagement(StandardJanusGraph.java:303)


I am not sure how to move ahead. 

Thanks
Akshaya

On Tuesday, 25 April 2017 14:19:50 UTC-7, Akshaya Rawat wrote:
Hi Jason,
Thanks for quick response.
Will try this and it should work.

Thanks
Akshaya


Akshaya Rawat <akshay...@...>
 

I tried this with Cassandra + Embedded Elastic configuration. It works there as well, however one has to add configuration to use "titan" keyspace name. 
storage.cassandra.keyspace=titan

In this case there is not need to force configuration like it needs to be done for Elastic search index name. 
Just checking if there is a good reason for different approaches. 

Akshaya


On Wednesday, 26 April 2017 23:35:22 UTC-7, Akshaya Rawat wrote:
This solutions worked for me. Commit had to be done after the force closure. 
JanusGraphManagement jMgmt = jGraph.openManagement();
Set<String> open = jMgmt.getOpenInstances();
for (String string : open) {
if (!string.endsWith("(current)")) {
jMgmt.forceCloseInstance(string);
}
jMgmt.commit();
jMgmt = jGraph.openManagement();
jMgmt.set("index.search.index-name", "titan");
jMgmt.commit();

The graph has to be initialized all over again after this since its closed after this. 
This looks to be disruptive solution though the graph is closed and has to be initialized all over again. However this only has to be done once. Once this configuration is forced, it persists. So this could be done as one time migration may be using a script and then it works then on. 

Thanks for the solution. 

Akshaya

On Wednesday, 26 April 2017 02:36:48 UTC-7, Jason Plurad wrote:
The first error message said that there were multiple open instances. In order to close them, again you must work through the management system (do not close the graph you are already connect to).

Try something like this:

mgmt = graph.openManagement()
open
= mgmt.openInstances() // returns a set of unique instance ids
for (String inst : open) { if (!inst.endsWith("(current)")) mgmt.forceCloseInstance(inst) }
mgmt
.set('index.search.index-name', 'titan')
mgmt
.commit()

On Wednesday, April 26, 2017 at 2:47:43 AM UTC-4, Akshaya Rawat wrote:
Hi Jason, 
I tried to do that. 
JanusGraphManagement janusGraphManagement=janusGraph.openManagement();
janusGraphManagement.set("index.search.index-name", "titan");
janusGraphManagement.commit();


But I got error - 
Caused by: java.lang.IllegalArgumentException: Cannot change offline config option [root.index.index-name] since multiple instances are currently open: [c0a838679548-NOIARAWA3569221, c0a8386711772-NOIARAWA3569221, c0a8386712068-NOIARAWA3569221, c0a8386712596-NOIARAWA3569221]
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
at org.janusgraph.graphdb.database.management.ManagementSystem$1.verifyModification(ManagementSystem.java:180)
at org.janusgraph.diskstorage.configuration.UserModifiableConfiguration.set(UserModifiableConfiguration.java:99)
at org.janusgraph.graphdb.database.management.ManagementSystem.set(ManagementSystem.java:1316)

If I close the graph, then set , I get another error - 
janusGraph.close();
JanusGraphManagement janusGraphManagement=janusGraph.openManagement();
janusGraphManagement.set("index.search.index-name", "titan");
janusGraphManagement.commit();

Caused by: java.lang.IllegalStateException: Environment is closed.
at com.sleepycat.je.Environment.getNonNullEnvImpl(Environment.java:2496)
at com.sleepycat.je.Environment.checkOpen(Environment.java:2472)
at com.sleepycat.je.Environment.openDatabase(Environment.java:643)
at org.janusgraph.diskstorage.berkeleyje.BerkeleyJEStoreManager.openDatabase(BerkeleyJEStoreManager.java:185)
at org.janusgraph.diskstorage.berkeleyje.BerkeleyJEStoreManager.openDatabase(BerkeleyJEStoreManager.java:48)
at org.janusgraph.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreManagerAdapter.openDatabase(OrderedKeyValueStoreManagerAdapter.java:89)
at org.janusgraph.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreManagerAdapter.openDatabase(OrderedKeyValueStoreManagerAdapter.java:38)
at org.janusgraph.diskstorage.log.kcvs.KCVSLogManager.openLog(KCVSLogManager.java:225)
at org.janusgraph.diskstorage.Backend.getSystemMgmtLog(Backend.java:337)
at org.janusgraph.graphdb.database.StandardJanusGraph.openManagement(StandardJanusGraph.java:303)


I am not sure how to move ahead. 

Thanks
Akshaya

On Tuesday, 25 April 2017 14:19:50 UTC-7, Akshaya Rawat wrote:
Hi Jason,
Thanks for quick response.
Will try this and it should work.

Thanks
Akshaya