1 reply to this topic

[dstockto]

Hello to everyone. I am posting this at the request of Ray Stoeckicht. He asked me to take a look at the project and let him know what I thought and then later to post that here.

First of all, I am super happy to see OSS built using TDD with continuous integration. Since I don't have a lot of time, I took a look at the project's running statistics from the CI servers and noticed a few things. Please feel free to ignore anything I say as it's just my opinion.

The first thing I noticed was that it seems to be taking a rather long time to run through the test suites and also there are a very large number of assertions for the number of tests being run. I made some guesses/assumptions about why that may be.

Typically if you're running unit tests (testing a single "unit" of software and mocking out any of the dependencies) you should see the unit tests running very very quickly and have a low number of assertions per test.

If you're running full system tests, then I'd expect to see longer running tests (you're controlling a browser and waiting on things like network latency, full database interaction, etc) and a lot more assertions per test since it's just not practical to test a single thing in one of these longer running tests.

It is important to have unit tests as well as full integrated system tests. I would recommend breaking these two types of tests apart. The unit tests should be able to run in a matter of seconds or in a few minutes if there are a lot. You should be able to quickly and easily see all the problems that exist and know where to go to fix the problem in these sorts of tests. This is due to a low number of assertions per test. The importance of this is that when PHPUnit runs, if it hits a failed assertion, the other assertions (which could also fail) do not get run which means it can take a number of iterations of fixing code and re-running tests.

The system tests that control a browser are expected to run longer and have more assertions. The unit tests should provide very rapid feedback to a developer to let them know if they've broken something.

Anyway, I am very happy to see that this project is using CI and TDD and I wish you all the best of luck and many green builds.

Thanks,
David Stockton

[Jason]

David,
Thanks for the feedback. It is very much appreciated. There is a special type of unit test that I call a 'walkthrough' test. It is more of a hybrid test half way between a unit test and a functional test via selenium. If you look at the options when running unit tests you can see that you can exclude or just run the walkthrough tests. These are probably the tests you are seeing take a long time to run. We will review your other suggestions as well.