Unit Testing's False Security

Unit Testing is software development where developers write code to test other code.  They key point here is the word unit in unit test case! Let us take a very idealized case.  Suppose I have a calculator class (unit).
class Calculator    int add( int a, int b)    int divide (int a, int b);
   int sqroot( int a);
To unit test the 'calculator' unit, You can write some unit test cases for it.

Calculator_test {     bool test_add()
    {
         assert( add(5,3) == 8 );
         assert( add(-1, 6) == 5);
    }

    bool test_divide()...
    bool test_sqroot()...
}

It is very nice and clean.  I start out with this clean example as this how people end up on the wrong path. This is so clean and testable... let's just apply it to everything.  It's a symptom of the plague of computer science which has spent that past decades obsessing on formally proving a program 'correct'. Unit testing works very well when the code under test is very organized and INDEPENDENT. 
Unit testing poorly organized code generally doesn't improve its quality.  Make sure the code is organized properly into self contained units that can be tested independently.

If you're working on a poor code base that is not nicely organized into units, I'd forgo the unit testing on it.  I would suggest simply make sure new code added is created in units and has appropriate unit testing.  Focus on cleaning up the poor code itself.

Independence is another big point.  Let me write that sentence again.  INDEPENDENCE IS ANOTHER BIG POINT.  Unit testing is most effective when you can test each unit independently.  Notice in the calculator class above, I am not calling anything outside the calculator class.  I can test the calculator without thinking of anything else.  There are no external dependencies.

If you are developing an API, this tends to come naturally and you unit testing is more useful.

If you're dealing with an application, finding code that is so well organized into units is... hmmm.. challenging...  Finding code that is independent is even rarer.  Suppose we are dealing with a server.  It gets commands from a client.  To actually test the server, you need to get input from the client.  This is normally done via a simulator of some kind... things get even messier if the server or client need to maintain state information.  To create an accurate simulator, you would actually need to build a fully functional client.  You're now venturing well passed the ideal unit case above... and well into functional testing. Now, you can use mock objects.  Mock objects simulate what the a real object would.  For example, suppose you are communicating with a database.  Instead of actually talking to a database, you simply talk to your mock object which would return some data that you create. Writing your code to support 'mock' objects is good practice and generally makes for better object oriented design.  However, mock objects do impose significant overhead as you must now write them.  You will probably spend more time coding the simulators/mock objects of some kind than actually fixing problems in the code.

Which brings me to my next point.  Unit Testing is code.  Why are we unit testing?  Because developers have bugs in their code.... Is there a chance the unit test code will be buggy?  Absolutely!  I get nervous when my unit test starts to look complicated or it is getting too big.  You don't want be be spending your time fixing unit test bugs instead of fixing actual bugs in the application.

On a side note... these cases tend to be example of venturing into functional testing and just trying to hack it onto a unit testing framework.  There is nothing wrong with this.  I write functional tests all the time and just attach them onto a unit testing framework as it is very convenient to run and get reports.

The final point is very important.  Unit testing only checks conditions you code it to test for.
So, if you are not doing good error checking in your regular code, chances are the same checks will be missing in the unit test code.

A quick example of this.  In the unit test for the add function in the Calculator above, I do not test the border conditions.
If a do add( 2147483647, 1), the result will not be 2147483648.  It will actually be -2147483648 due to the overflow.
If I didn't think up this scenario the coding the actual add() function, it is unlikely I would test for it in the unit test for it.

Unit testing should not be an afterthought.  It should be done as an aid to the developer as they are writing software.
Many developers do unit testing informally typically coding a function and then running the debugger and inspecting variables or logs to make sure things are correct.  Unit testing is just a formalization of this process.


In general, I like unit testing, but it should not be confused with functional or regression testing.  I personally find it very useful when building APIs or libraries, but less usefull on the application end.  Generally speaking, on the application side, you tend to need simulators and there are various dependencies that make unit testing less desirable.  It is much more in the realm of general function testing and automation.

I always find it useful to have a unit-testing framework (nunit, junit..) from the start of building an application.  Many times, you can ever hook onto this framework to run functional tests or various automated tests.  However, let's not lose sight of the reality that they are not unit tests in the idealized sense. At the end of the day, you must handle the reality that you have limited developers, limited time, limited testers... You have to choose how much effort you put in all areas of software development and testing.  Focus on what unit-testing is good for.  If you are writing an API or purely computational math, then unit-testing will work wonders.  If you're writing applications or interacting with a lot of 3rd party components, you probably need to focus much more on functional testing.