Custermized Docker for janusgraph 1.0.0. rc2
Yingjie Li
I did some resource cleaning up and now the docker compose up of all these component are successful without below addtional env setup. What I need help is to confirm whether there are anything wrong with these configs and if so, shall I build a docker from based by manually change the config files like what I did before with janusgprah-full version.
gremlinserver.graphs.ConfigurationManagementGraph=/etc/opt/janusgraph/janusgraph.properties
janusgraph.gremlin.graph=org.janusgraph.core.ConfiguredGraphFactory
janusgraph.graph.graphname=ConfigurationManagementGraph
gremlinserver%d.graphs.graph=
gremlinserver.evaluationTimeout=300000 |
|
Yingjie Li
Hello Florian,
Is below the right env setup for janusgraph docker-compose if I want to change the default? Thank you. gremlinserver.graphs.ConfigurationManagementGraph=/etc/opt/janusgraph/janusgraph.properties janusgraph.gremlin.graph=org.janusgraph.core.ConfiguredGraphFactory
janusgraph.graph.graphname=ConfigurationManagementGraph
gremlinserver%d.graphs.graph=
gremlinserver.evaluationTimeout=300000
I think I was able to start Janusgraph 1.1.0CR2 without these configs when docker compose cassandra and opensearch. There are some issues like out of memeory for cassandra and opensearch that I need to take care of seperately. But when I add these configures to the janusgraph.env. I was not abe to start janusgprah. Any suggestions? I am attaching the complete code with verbose output from both runs. 3.janus1.0rc2-config-cs4.0.0-opensearch.zip is the one with added configuration 2.janus1.0rc2-cs4.0.0-opensearch.zip is the one without. |
|
Yingjie Li
I tried with https://github.com/JanusGraph/janusgraph-docker/blob/master/docker-compose-cql-es.yml by replacing the images with Janusgraph1.0.0RC2 and Cassandra (4.0.0) and ElasticSearch *8.6.0) that are closest to the versions that are listed in the tested pltform of the Annoucement . So just to cofirm. the test were done with local installation but not at container level. If so, Fwhat are the most recent versions that have been tested?
Thanks, Yingjie |
|
Yingjie Li
Hello Florian,
I will also try the docker comopose yml link you share and see.
|
|
Florian Hockmann
Your log shows that JanusGraph tries to connect to Cassandra under 127.0.0.1 which doesn’t make much sense if you are using Docker compose (and don’t use the host network). If you are already using Docker Compose, then why don’t you try out the example Docker Compose files provided by JanusGraph first? You can start with one of that, see if you get it up and running and then you can start to change some of its config to what you need. If you want to use Cassandra and Elasticsearch (/Opensearch), then you can simply use the docker-compose-cql-es.yml: https://github.com/JanusGraph/janusgraph-docker/blob/master/docker-compose-cql-es.yml
If you get that working, then you can for example try out to update Cassandra to version 4 and update Elasticsearch or use Opensearch instead.
Von: janusgraph-users@... <janusgraph-users@...> Im Auftrag von Yingjie Li
In the Janusgraph 1.0.0RC2 annoucement, it is said that "support Csssandra 4", So there should be a way to docker compose them. Is there something wrong/missing in my code? |
|
Yingjie Li
In the Janusgraph 1.0.0RC2 annoucement, it is said that "support Csssandra 4", So there should be a way to docker compose them. Is there something wrong/missing in my code?
Thanks |
|
Yingjie Li
Actually while trying to add more env variables to the janusgraph.env file (posted earlier). I realized that my docker compose of latest Janusgraph (janusgraph/janusgraph:1.0.0-rc2), cassandra:4.1 and openshift2.6 failed. One of the reasons is Cassandra server can not started successfully, so does the Junagarh server. I am attaching the related files and the output of docker-compose --verbose up (Janusgraph-1.0.0-rc2-cassandra-4.1-openshift2.6-out.txt) . Any pointers of how to fix the issue? Thanks |
|
Yingjie Li
Thank you Florian. Will try and report back!
Yingjie |
|
Florian Hockmann
1) I assume you mean the ConfigurationManagementGraph. You can enable that by setting this env var: gremlinserver.graphs.ConfigurationManagementGraph="/etc/opt/janusgraph/janusgraph.properties"
To additionally set the required options in the properties file, you can also set these env vars:
janusgraph.gremlin.graph="org.janusgraph.core.ConfiguredGraphFactory" janusgraph.graph.graphname="ConfigurationManagementGraph"
You might then want to remove the statically configured graph from the JanusGraph Server YAML file which can be done by also setting this env var:
gremlinserver%d.graphs.graph=
(without any value as this removes the option from the file).
2) JanusGraph 1.0.0 uses Java 11 so it uses a file “jvm-11.options” instead of “jvm-8.options”. The easiest way to replace that file is by mounting it into the container or by creating your own Docker image that uses the JanusGraph image as the base image and then contains a COPY like this:
COPY --chown=janusgraph:janusgraph src/jvm-11.options ${JANUS_HOME}/conf/jvm-11.options
3) scriptEvaluationTimeout has been replaced by evaluationTimeout. You can set that just like any other JanusGraph Server options through an env var:
gremlinserver.evaluationTimeout=30000
4) Just mount your script or COPY it into the container, exactly like for the `jvm-11.options` file. The default config of the server already contains a script that will be executed on startup so you can for example just replace that: scripts/empty-sample.groovy
Von: janusgraph-users@... <janusgraph-users@...> Im Auftrag von Yingjie Li
I was able to create a docker-compose that starts four containers Janusgraph 1.0.0-rc2, Cassandra and OpenSearch based on a repo https://gist.github.com/Phate334/9fd90ddce0eaea74086c1a2e5d419402 from an earlier discussion in Janusgraph git: https://github.com/JanusGraph/janusgraph/discussions/2837
3). How to change the scriptEvaluationTime in some of the files, for example, in all-in-one case, I replace the sring in the related files like below sed "s/evaluationTimeout: 30000/evaluationTimeout: $SCRIPT_EVALUATION_TIMEOUT/g" $JANUSGRPH_DIR/conf/gremlin-server/gremlin-server-cql-es.yaml > $JANUSGRPH_DIR/conf/gremlin-server/gremlin-server-cql-es.yaml
//Create a sessioned connection to the Gremlin Server
:remote connect tinkerpop.server conf/remote.yaml session
:remote console
//define graph name
graph_name = "graph1"
map = new HashMap();
map.put("storage.backend", "cql");
map.put("storage.hostname", "127.0.0.1")
map.put("index.search.backend", "elasticsearch")
map.put("index.search.hostname", "127.0.0.1")
map.put("graph.graphname", graph_name)
map.put('query.batch', true)
ConfiguredGraphFactory.createConfiguration(new MapConfiguration(map))
graph = ConfiguredGraphFactory.open(graph_name)
//Creat graph schema
//Close all transactions
graph.getOpenTransactions().forEach { tx -> tx.rollback() }
mgmt = graph.openManagement()
mgmt.getOpenInstances().forEach {
if (it.reverse().take(1) != ")") {
mgmt.forceCloseInstance(it)
}
}
pk_id = mgmt.makePropertyKey('id').dataType(String.class).cardinality(org.janusgraph.core.Cardinality.LIST).make()
pk_name = mgmt.makePropertyKey('name').dataType(String.class).cardinality(org.janusgraph.core.Cardinality.LIST).make()
mgmt.buildIndex('vby_name_m', Vertex.class).addKey(pk_name).buildMixedIndex('search')
mgmt.commit()
ConfiguredGraphFactory.close(graph_name)
|
|
Yingjie Li
Would prefer to using existing docker if I could somehow achieve the above with ENV settting etc, if not , could anybody point me to the exact process that I can use to build the docker image from januspgrah 1.10 RC2 source code in the same style as the office docker image? I have been mostly a user so far but not a docker builder, so any help in this regards is highly appreciated.
Yingjie |
|
Yingjie Li
I was able to create a docker-compose that starts four containers Janusgraph 1.0.0-rc2, Cassandra and OpenSearch based on a repo https://gist.github.com/Phate334/9fd90ddce0eaea74086c1a2e5d419402 from an earlier discussion in Janusgraph git: https://github.com/JanusGraph/janusgraph/discussions/2837
3). How to change the scriptEvaluationTime in some of the files, for example, in all-in-one case, I replace the sring in the related files like below sed "s/evaluationTimeout: 30000/evaluationTimeout: $SCRIPT_EVALUATION_TIMEOUT/g" $JANUSGRPH_DIR/conf/gremlin-server/gremlin-server-cql-es.yaml > $JANUSGRPH_DIR/conf/gremlin-server/gremlin-server-cql-es.yaml
4). How to start my janusgraph docker run with a default graph & schema (groovy), a sample of my groovy as below ( somehow can not attache it) : //Create a sessioned connection to the Gremlin Server
:remote connect tinkerpop.server conf/remote.yaml session
:remote console
//define graph name
graph_name = "graph1"
map = new HashMap();
map.put("storage.backend", "cql");
map.put("storage.hostname", "127.0.0.1")
map.put("index.search.backend", "elasticsearch")
map.put("index.search.hostname", "127.0.0.1")
map.put("graph.graphname", graph_name)
map.put('query.batch', true)
ConfiguredGraphFactory.createConfiguration(new MapConfiguration(map))
graph = ConfiguredGraphFactory.open(graph_name)
//Creat graph schema
//Close all transactions
graph.getOpenTransactions().forEach { tx -> tx.rollback() }
mgmt = graph.openManagement()
mgmt.getOpenInstances().forEach {
if (it.reverse().take(1) != ")") {
mgmt.forceCloseInstance(it)
}
}
pk_id = mgmt.makePropertyKey('id').dataType(String.class).cardinality(org.janusgraph.core.Cardinality.LIST).make()
pk_name = mgmt.makePropertyKey('name').dataType(String.class).cardinality(org.janusgraph.core.Cardinality.LIST).make()
mgmt.buildIndex('vby_name_m', Vertex.class).addKey(pk_name).buildMixedIndex('search')
mgmt.commit()
ConfiguredGraphFactory.close(graph_name)
|
|
Yingjie Li
I guess if I can start the janusgraph 1.0.0-rc2 docker with the configuration that alignes with my customization (excluding the removing extention/jar parts, intially I have to do it due to hgh CVE number for certain jars. which might be fixed already), I can create a sepreate image for my flask service and docker compose all the related images.
Please advise. Thanks |
|
Boxuan Li
I am unfamiliar with packaging so I will wait to see if other experts can chime in.
My two cents: you can look at https://github.com/JanusGraph/janusgraph-docker/blob/master/docker-compose-cql-es.yml and come up with your own docker-compose setting. Cheers, Boxuan |
|
Yingjie Li
Hello Boxuan. To add a little bit of context here. I have been using Janusgraph all in one package with the cql-es backend. In my docker, I first do certain customization to janusgraph-all-in-one package :1) Configure Janusgraph to use ConfiguredGraphFactory instead of JanusgraphFactory 2) change the JVM heap Size 3) change the default evaluationTimeout time for gremlin server, and 4) remove certain extensions/jars that I don't use.
|
|
Boxuan Li
Hi Yingjie,
janusgraph/janusgraph:1.0.0-rc2 should be available. You can find it here: https://hub.docker.com/layers/janusgraph/janusgraph/1.0.0-rc2/images/sha256-048c7a3470156c3018c58a20674cf2cb2ea1bb58f8bca57684360317e9e92516?context=explore Cheers, Boxuan |
|
Yingjie Li
Hello,
It appears that the latest janusgraph docker is 0.6.3, and I will need to create a customized docker for Janusgraph 1.0.0.rc2. Any pointers of the steps I should take? Thank you. Yingjie |
|