Custermized Docker for janusgraph 1.0.0. rc2


Yingjie Li
 

I did some resource cleaning up and now the docker compose up of all these component are successful without below addtional env setup. What I need help is to confirm whether there are anything wrong with these configs and if so, shall I build a docker from based by manually change the config files like what I did before with janusgprah-full version. 

Thanks

 

gremlinserver.graphs.ConfigurationManagementGraph=/etc/opt/janusgraph/janusgraph.properties
janusgraph.gremlin.graph=org.janusgraph.core.ConfiguredGraphFactory
janusgraph.graph.graphname=ConfigurationManagementGraph
gremlinserver%d.graphs.graph=
gremlinserver.evaluationTimeout=300000


Yingjie Li
 

Hello Florian,

Is below the right env setup  for janusgraph docker-compose  if I want to change the default? Thank you.


gremlinserver.graphs.ConfigurationManagementGraph=/etc/opt/janusgraph/janusgraph.properties
janusgraph.gremlin.graph=org.janusgraph.core.ConfiguredGraphFactory
janusgraph.graph.graphname=ConfigurationManagementGraph
gremlinserver%d.graphs.graph=
gremlinserver.evaluationTimeout=300000


I think I was able to start Janusgraph 1.1.0CR2 without these configs when docker compose cassandra and opensearch.  There are some issues like  out of memeory  for cassandra and opensearch that I need to take care of seperately. But when I add these configures to the janusgraph.env. I was not abe to start janusgprah. Any suggestions? I am attaching the complete code with verbose output from both runs. 

3.janus1.0rc2-config-cs4.0.0-opensearch.zip  is the one with added configuration
2.janus1.0rc2-cs4.0.0-opensearch.zip is the one without.
 


Yingjie Li
 

I tried with https://github.com/JanusGraph/janusgraph-docker/blob/master/docker-compose-cql-es.yml by replacing the images with Janusgraph1.0.0RC2 and   Cassandra (4.0.0) and ElasticSearch *8.6.0) that are closest to the versions that are listed in the tested pltform of the Annoucement . So just to cofirm. the test were done with local installation but not at container level.  If so, Fwhat are the most recent versions that have been tested?

Thanks,
Yingjie


Yingjie Li
 

Hello Florian,

Thanks for the pointers. I started with this one as I need to replace Elastic Search With other index backends that have Apache license, prefer to be OpenSearch. I found an example (the one and only) that worked  (I tried) with  older images (janusgraph/janusgraph:0.6.2, cassandra:3,docker.io/opensearchproject/opensearch:2.4.0)  from  https://gist.github.com/Phate334/9fd90ddce0eaea74086c1a2e5d419402    that was  emebed in an earlier discussion in Janusgraph git:  https://github.com/JanusGraph/janusgraph/discussions/2837


As old images contains security issues such as log4j etc, so I swapped old images with the latest  in the docker compose yml,  but kept the env files the same, and encounter the error. 

As for "Your log shows that JanusGraph tries to connect to Cassandra under 127.0.0.1 which doesn’t make much sense if you are using Docker compose (and don’t use the host network)"  can you elaborate on it? 

I will also try the docker comopose yml link you share and see.

Thanks

 

 

 


Florian Hockmann
 

Your log shows that JanusGraph tries to connect to Cassandra under 127.0.0.1 which doesn’t make much sense if you are using Docker compose (and don’t use the host network).

If you are already using Docker Compose, then why don’t you try out the example Docker Compose files provided by JanusGraph first? You can start with one of that, see if you get it up and running and then you can start to change some of its config to what you need.

If you want to use Cassandra and Elasticsearch (/Opensearch), then you can simply use the docker-compose-cql-es.yml: https://github.com/JanusGraph/janusgraph-docker/blob/master/docker-compose-cql-es.yml

 

If you get that working, then you can for example try out to update Cassandra to version 4 and update Elasticsearch or use Opensearch instead.

 

Von: janusgraph-users@... <janusgraph-users@...> Im Auftrag von Yingjie Li
Gesendet: Donnerstag, 16. März 2023 11:23
An: janusgraph-users@...
Betreff: Re: [janusgraph-users] Custermized Docker for janusgraph 1.0.0. rc2

 

In the  Janusgraph 1.0.0RC2 annoucement, it is said that "support Csssandra 4", So there should be a way to docker compose them. Is there something wrong/missing  in my code?


Thanks


Yingjie Li
 

In the  Janusgraph 1.0.0RC2 annoucement, it is said that "support Csssandra 4", So there should be a way to docker compose them. Is there something wrong/missing  in my code?


Thanks


Yingjie Li
 

Actually while trying to add more env variables  to the janusgraph.env file (posted earlier). I realized that my docker compose of latest Janusgraph (janusgraph/janusgraph:1.0.0-rc2), cassandra:4.1 and openshift2.6 failed. One of the reasons is Cassandra server can not started successfully, so does the Junagarh server. I am attaching the related files and the output of  docker-compose --verbose up  (Janusgraph-1.0.0-rc2-cassandra-4.1-openshift2.6-out.txt) . Any pointers of how to fix the issue?  Thanks 



Yingjie Li
 

Thank you Florian. Will try and report back!

Yingjie


Florian Hockmann
 

1) I assume you mean the ConfigurationManagementGraph. You can enable that by setting this env var:

gremlinserver.graphs.ConfigurationManagementGraph="/etc/opt/janusgraph/janusgraph.properties"

 

To additionally set the required options in the properties file, you can also set these env vars:

 

janusgraph.gremlin.graph="org.janusgraph.core.ConfiguredGraphFactory"

janusgraph.graph.graphname="ConfigurationManagementGraph"

 

You might then want to remove the statically configured graph from the JanusGraph Server YAML file which can be done by also setting this env var:

 

gremlinserver%d.graphs.graph=

 

(without any value as this removes the option from the file).

 

2) JanusGraph 1.0.0 uses Java 11 so it uses a file “jvm-11.options” instead of “jvm-8.options”. The easiest way to replace that file is by mounting it into the container or by creating your own Docker image that uses the JanusGraph image as the base image and then contains a COPY like this:

 

COPY --chown=janusgraph:janusgraph src/jvm-11.options ${JANUS_HOME}/conf/jvm-11.options

 

3) scriptEvaluationTimeout has been replaced by evaluationTimeout. You can set that just like any other JanusGraph Server options through an env var:

 

gremlinserver.evaluationTimeout=30000

 

4) Just mount your script or COPY it into the container, exactly like for the `jvm-11.options` file. The default config of the server already contains a script that will be executed on startup so you can for example just replace that: scripts/empty-sample.groovy

 

 

Von: janusgraph-users@... <janusgraph-users@...> Im Auftrag von Yingjie Li
Gesendet: Mittwoch, 15. März 2023 12:39
An: janusgraph-users@...
Betreff: Re: [janusgraph-users] Custermized Docker for janusgraph 1.0.0. rc2

 

I was able to create a docker-compose that starts four containers Janusgraph 1.0.0-rc2, Cassandra and OpenSearch  based on a repo https://gist.github.com/Phate334/9fd90ddce0eaea74086c1a2e5d419402   from an earlier discussion in Janusgraph git:  https://github.com/JanusGraph/janusgraph/discussions/2837

The docker-compose  and janusgraph.env files are attached. 

Here are my question is in this new context and apprecaite any pointers. 


1) how can I enable Janusgraph container to use ConfiguredGraphManager but not default JanusgraphFactory. In all-in-one case. I simply do

cp $JANUSGRPH_DIR/conf/gremlin-server/gremlin-server-configuration.yaml $JANUSGRPH_DIR/conf/gremlin-server/gremlin-server-cql-es.yaml

2) how can i change the default Janusgraph JVM heap size. In all-in-one case, I replace the sring in the related file

sed "s/Xmx4096m/$JVM_HEAP_MAX/g" $JANUSGRPH_DIR/conf/jvm-8.options > $JANUSGRPH_DIR/conf/jvm-8.options

3). How to change the scriptEvaluationTime in some of the files, for example, in all-in-one case, I replace the sring in the related files like below


sed "s/evaluationTimeout: 30000/evaluationTimeout: $SCRIPT_EVALUATION_TIMEOUT/g" $JANUSGRPH_DIR/conf/gremlin-server/gremlin-server-cql-es.yaml > $JANUSGRPH_DIR/conf/gremlin-server/gremlin-server-cql-es.yaml

4). How to start my janusgraph docker run with a default graph & schema (groovy), a sample of my groovy as below ( somehow can not attache it) :


 

//Create a sessioned connection to the Gremlin Server

 

:remote connect tinkerpop.server conf/remote.yaml session

 

:remote console

 



//define graph name

 

graph_name = "graph1"

 

map = new HashMap();

 

map.put("storage.backend", "cql");

 

map.put("storage.hostname", "127.0.0.1")

 

map.put("index.search.backend", "elasticsearch")

 

map.put("index.search.hostname", "127.0.0.1")

 

map.put("graph.graphname", graph_name)

 

map.put('query.batch', true)

 

ConfiguredGraphFactory.createConfiguration(new MapConfiguration(map))

 

graph = ConfiguredGraphFactory.open(graph_name)

 

//Creat graph schema

 

//Close all transactions

 

graph.getOpenTransactions().forEach { tx -> tx.rollback() }

 

mgmt = graph.openManagement()

 

mgmt.getOpenInstances().forEach {

 

if (it.reverse().take(1) != ")") {

 

mgmt.forceCloseInstance(it)

 

}

 

}

 

pk_id = mgmt.makePropertyKey('id').dataType(String.class).cardinality(org.janusgraph.core.Cardinality.LIST).make()

 

pk_name = mgmt.makePropertyKey('name').dataType(String.class).cardinality(org.janusgraph.core.Cardinality.LIST).make()

 

mgmt.buildIndex('vby_name_m', Vertex.class).addKey(pk_name).buildMixedIndex('search')

 

mgmt.commit()

 

ConfiguredGraphFactory.close(graph_name)

 

 



 

 

 

 

 

 


Yingjie Li
 

Would prefer to using existing docker if I could somehow achieve  the above with ENV settting etc, if not , could anybody point me to the exact process that I can use to build the docker image  from januspgrah 1.10 RC2 source code  in the same style as the office docker image? I have been mostly a user so far but not a docker builder, so any help in this regards is highly appreciated.

Yingjie


Yingjie Li
 

I was able to create a docker-compose that starts four containers Janusgraph 1.0.0-rc2, Cassandra and OpenSearch  based on a repo https://gist.github.com/Phate334/9fd90ddce0eaea74086c1a2e5d419402   from an earlier discussion in Janusgraph git:  https://github.com/JanusGraph/janusgraph/discussions/2837

The docker-compose  and janusgraph.env files are attached. 

Here are my question is in this new context and apprecaite any pointers. 


1) how can I enable Janusgraph container to use ConfiguredGraphManager but not default JanusgraphFactory. In all-in-one case. I simply do

cp $JANUSGRPH_DIR/conf/gremlin-server/gremlin-server-configuration.yaml $JANUSGRPH_DIR/conf/gremlin-server/gremlin-server-cql-es.yaml

2) how can i change the default Janusgraph JVM heap size. In all-in-one case, I replace the sring in the related file

sed "s/Xmx4096m/$JVM_HEAP_MAX/g" $JANUSGRPH_DIR/conf/jvm-8.options > $JANUSGRPH_DIR/conf/jvm-8.options

3). How to change the scriptEvaluationTime in some of the files, for example, in all-in-one case, I replace the sring in the related files like below

sed "s/evaluationTimeout: 30000/evaluationTimeout: $SCRIPT_EVALUATION_TIMEOUT/g" $JANUSGRPH_DIR/conf/gremlin-server/gremlin-server-cql-es.yaml > $JANUSGRPH_DIR/conf/gremlin-server/gremlin-server-cql-es.yaml

4). How to start my janusgraph docker run with a default graph & schema (groovy), a sample of my groovy as below ( somehow can not attache it) :

//Create a sessioned connection to the Gremlin Server
:remote connect tinkerpop.server conf/remote.yaml session
:remote console

//define graph name
graph_name = "graph1"
map = new HashMap();
map.put("storage.backend", "cql");
map.put("storage.hostname", "127.0.0.1")
map.put("index.search.backend", "elasticsearch")
map.put("index.search.hostname", "127.0.0.1")
map.put("graph.graphname", graph_name)
map.put('query.batch', true)
ConfiguredGraphFactory.createConfiguration(new MapConfiguration(map))
graph = ConfiguredGraphFactory.open(graph_name)
//Creat graph schema
//Close all transactions
graph.getOpenTransactions().forEach { tx -> tx.rollback() }
mgmt = graph.openManagement()
mgmt.getOpenInstances().forEach {
if (it.reverse().take(1) != ")") {
mgmt.forceCloseInstance(it)
}
}
pk_id = mgmt.makePropertyKey('id').dataType(String.class).cardinality(org.janusgraph.core.Cardinality.LIST).make()
pk_name = mgmt.makePropertyKey('name').dataType(String.class).cardinality(org.janusgraph.core.Cardinality.LIST).make()
mgmt.buildIndex('vby_name_m', Vertex.class).addKey(pk_name).buildMixedIndex('search')
mgmt.commit()
ConfiguredGraphFactory.close(graph_name)



 
 


Yingjie Li
 

I guess if I can start the janusgraph 1.0.0-rc2 docker with the configuration that alignes with my customization (excluding the removing extention/jar parts, intially I have to do it due to hgh CVE number for certain jars. which might be fixed already), I can create a sepreate image for my flask service and docker compose all the related images.

Please advise. 

Thanks


Boxuan Li
 

I am unfamiliar with packaging so I will wait to see if other experts can chime in.

My two cents: you can look at https://github.com/JanusGraph/janusgraph-docker/blob/master/docker-compose-cql-es.yml and come up with your own docker-compose setting.

Cheers,
Boxuan


Yingjie Li
 

Hello Boxuan.

Thanks for the pointers.   Will it allow me to run with certain configuration,  such as use ConfiguredGraphFactory instead of JanusgraphFactory etc? 


To add a little bit of context here. I have been using Janusgraph all in one package with the cql-es backend.  In my docker, I first do certain customization to janusgraph-all-in-one package :1) Configure Janusgraph to use ConfiguredGraphFactory instead of JanusgraphFactory 2) change the JVM heap Size 3) change the default evaluationTimeout time for gremlin server, and 4) remove certain extensions/jars that I don't use.

I then initialize it with a default graph with own defined schema (groovy script) if there is no graph defined.


The last part is a flask service that insert data to the graph and query the graph (gremlin-python or gremlin command). The entry point of my docker is the flask service. So basically the user interacts with the graph via flask APIs.


Now I need to split my docker into three separate ones, Janusgraph + flask (can split if necessary), Cassandra (latest docker) OpenSearch(latest docker, replacing Elasticsearch as the ElasticSearch license in Janusgraph all in one is not sspl).

I am looking for pointers/samples that help me to do the above.

Thanks,
Yingjie


Boxuan Li
 


Yingjie Li
 

Hello,

 It appears that the latest janusgraph docker is 0.6.3, and  I will need to create a customized docker for  Janusgraph 1.0.0.rc2. Any pointers of the  steps  I should take?

Thank you.

Yingjie