Friday, October 29, 2010

Revenge of the unit test

I wrote some unit tests a while ago, but I was too lazy to run them. I looked into FOSS Continuous Integration (CI) testing solutions, but couldn't really find anything that really caught my eye.
I played around some more with CDash (http://www.cdash.org/), which was at the top of my list. Unfortunately, it currently still seems pretty SVN/CVS oriented and without previous experience with CDash, the entry barrier to use it seems relatively high. Its natural zone of comfort seems to be SVN/CMake/Doxygen. There is a test server on the Kitware website at http://my.cdash.org/index.php?project=uvudec . I also tried to setup a local server which I may have had more luck with, but I couldn't get one of the dependencies installed. I'm told that Kitware is moving to git for one of their projects, so support for git might be cleaner in the near future.
I was also recommended to look into CIJoe (http://github.com/defunkt/cijoe). Unfortunately, it seems to crash for me. This may be because Fedora runs an olllld version of Ruby. Since this will be a dedicated virtual server anyway, I'll try to setup a VM to give it a better shot. They have a cool logo thought and even sell merchandise with it:

In the end, I decided my current needs are very modest and it would be better to get some crude hacked together server running than nothing at all. So, enter UVBuilder. It can be found at util/uvbuilder . Basically it uses a JSON config file to checkout, update, build, and run the code. It then e-mails results if there seems to be some change in status. Its very simplistic and has some dumb features like needing to checkout two copies of the code that can be solved with minor effort if I care.
I also played some more with using Valgrind to extensivly test the code. I had noticed that sometimes I would hexdump a data object and valgrind would trigger on that object if there was an error. So, I added a function that I called "poke" that would iterate over a block of memory that effectively does nothing. However, it makes the control flow appear to depend on the values by executing a statement like if( *ptr ) doNothing(); on each value. I found that it seems that std::map might leave uninitialized values though, so I'll have to make more custom Valgrind ignore files if I want testing to be truly effective at these tests.
So, next steps are to get all of the existing unit tests to pass (3 regressions, they all seem related to the same issue), and then beef up the unit tests now that I don't have to keep manually running them.

EDIT: all original unit tests pass after various fixes. Now to write a bunch of FLIRT tests.
EDIT: obj2pat unit tests created. And I'll try to get back to the comment below in the near future

1 comment:

  1. We are glad you are experimenting with CDash. We would be happy to help making sure CDash works for you locally or on my.cdash.org.
    Let us know which issues you had while installing the local CDash (which dependency was missing?). We are including features to support git and we appreciate any feedback.

    Julien

    ReplyDelete