Hi all,

During my work on Testcontainers (https://github.com/GDATASoftwareAG/janusgraph/tree/testcontainers), I needed to modified all tests files multiple times. These multiple changes can follows to mistakes in implementations. Besides, if something in the configuration has changed, you probably need to update all test classes.

Janusgraph test classes should become better manageable, for example, add new test classes, split up tests or detect difference between backend (completeness). 

I found two possible solution to improve:

1. Combine all tests into test suite, see PR https://github.com/JanusGraph/janusgraph/pull/1259 . My proof of concept based on solution from Tinkerpop with some changes for Janusgraph.

+ Reduced boilerplate, one provider file for one database backend and a suite file to define tests.

+ Improved ignoring of tests, currently tests are overwritten with an empty method. These method doesn't prevent to execute the startup and cleanup method for these ignored tests. (Performance improvement)

+ Changes of database configuration need to be done only once.

- Tests are only executable behind a test suite. (Can be probably fixed)

- It could only exists one provider per test project. (This need to be fixed)

2. Using Junit Rule for configuration.

+ Reduced boilerplate, one provider file for one database backend. 

+ Less code change (Smaller review).

+ Changes of database configuration need to be done only once.

-  All test classes need to be exists for all backends. (Current state) (An idea to write all of theses classes would be to auto generate these test classes.)

Greetings,

Jan