Jan, thanks for digging into this. In the #2 option, you mentioned that "All test classes need to be exists for all backends. An idea to write all of theses classes would be to auto generate these test classes." Would the generated test classes be done dynamically at build time or would it be done once and checked into source control? If it is the latter, my concern would be that the test classes would get out of sync across different backends.

On Tuesday, October 16, 2018 at 6:54:23 AM UTC-4, Florian Hockmann wrote:

I talked briefly with Jan about this after he created his WIP PR and recommended to ask here on the list before refactoring basically the whole test suite of JanusGraph for recommendations on how to proceed.

So, if anyone has a strong opinion on any of the two options Jan presented or maybe even against making such big changes to the test suite in general, then please come forward and express your concerns / your preferences.

Otherwise, I guess he can only decide on his own which option makes most sense for JanusGraph and then create a PR with that changes (ideally maybe a minimal first PR with changes for just 1-2 backends or test classes so we have something so to review).

Am Dienstag, 9. Oktober 2018 14:11:04 UTC+2 schrieb Ja...@...:

Hi all,

During my work on Testcontainers (https://github.com/GDATASoftwareAG/janusgraph/tree/testcontainers), I needed to modified all tests files multiple times. These multiple changes can follows to mistakes in implementations. Besides, if something in the configuration has changed, you probably need to update all test classes.

Janusgraph test classes should become better manageable, for example, add new test classes, split up tests or detect difference between backend (completeness). 

I found two possible solution to improve:

1. Combine all tests into test suite, see PR https://github.com/JanusGraph/janusgraph/pull/1259 . My proof of concept based on solution from Tinkerpop with some changes for Janusgraph.

+ Reduced boilerplate, one provider file for one database backend and a suite file to define tests.

+ Improved ignoring of tests, currently tests are overwritten with an empty method. These method doesn't prevent to execute the startup and cleanup method for these ignored tests. (Performance improvement)

+ Changes of database configuration need to be done only once.

- Tests are only executable behind a test suite. (Can be probably fixed)

- It could only exists one provider per test project. (This need to be fixed)

2. Using Junit Rule for configuration.

+ Reduced boilerplate, one provider file for one database backend. 

+ Less code change (Smaller review).

+ Changes of database configuration need to be done only once.

-  All test classes need to be exists for all backends. (Current state) (An idea to write all of theses classes would be to auto generate these test classes.)

Besides, an upgrade to Junit 5 from Junit 4 can be a benefit in both cases, see https://junit.org/junit5/docs/current/user-guide/ .

I would appreciate any feedback to both present solution or do you have another possible solution? What should be the next steps to improve the current situation?

 

Greetings,

Jan