|
Nov 08
2009
|
Unit Testing as the Last Resort for Code VerificationPosted by Christopher Diggins in Testing, Software Development Methodology and Management, Agile   |
 |
I love code which is stable and rigorous, I just don't feel that unit testing is the most effective way to achieve it.
I stumbled upon the following question posted on StackOverflow, "What's your most controversial programming opinion? ", and it made me think about my views on unit testing.
I have mixed feelings on unit testing. I do use it often, but I find that it does not give me sufficient peace of mind, compared to other code verification techniques. The following is my list of code verification techniques in order of importance:
- System testing
- Use the compiler's type checker
- Use assertions
- Code reviews
- Unit testing
First, any code verification technique is pointless without system testing. While it may seem almost obvious to include, it does happen where people get caught up in the minutiae of code verification and overlook the big picture of whether or not a system is behaving as intended.
My preferred code verification technique is to leverage the language's type system to express assumptions, requirements, and invariants. It is nice to know that if something compiles, then it is correct. Some examples of how this can be done are:
- using unsigned types instead of signed types so you don't have to worry about non-negativity. Consider for example the example of using an unsigned type instead of a signed type in a square root function.
- using id types instead of integers to identify objects. I recently ran into problems in my code, because I violated this rule. I confused the index of a vertex, with the index of an index array into the vertex array. The fix was easy: use a new VertexID type.
- using smart pointer classes with the appropriate semantics. E.g. null-checking pointers, deletion prevention pointers, pointers with ownership semantics, etc.
In this bucket I also include any static contract verification tools like those found in Spec# and Eiffel. This is because the techniques of type checking and contract verification are closely related, and both based on theorem proving techniques.
When the type system can't be used easily, I find that the next most powerful code verification technique is to use assertions to check assumptions, requirements and invariants at run-time. Assertions turn regular system testing into a much more powerful tool for uncovering design errors and potential defects. In addition assertions occur directly in the code, as a kind of documentation that is useful during code reviews.
Code reviews are another technique which I find is more powerful than unit testing, if it is done properly. In order to really leverage the power of a code review, code has to be written so that verification is as simple as possible. I will often sacrifice performance and ignore any popular idiom of the week if it means making my code's meaning obvious and unambiguous.
When I find unit testing to really become valuable as a tool, is when code can't be proven correct by the compiler or fully verified through assertions, and that a code review can't give a high confidence that the code is correct because of inherent complexity.
Comments (6)
Subscribe to this comment's feed
Show/Hide comments
written by Nathan Ernst, November 09, 2009
I like the concept of assertions. The problem I have with assertions is that most implementations are done in debug versions of executables/runtimes/vms only. This is great for verifying your code in a development/QA type environment, but what about a production environment where assertions are effectively ifdef'd out?
I know it sucks, but the most effective means I've found is to throw an exception, rather than assert for all but the most trivial of checks. My reasoning: if it's important enough to check in development/QA, it's important enough to check in a production environment and to fail to a known state.
The problem you'll run into assertions in a production, optimized compile of C++, is that the checks are not performed, and when they fail execution is not stopped. Immediately you're into undefined behavior. Now, I'm sure you could argue that this should never happen with sufficient testing, and you'd likely be correct. But, excepting companies where software is the business and profit center, I'll counter that reproducible and extensive testing is not the norm and thus assertions, while well intended, will not catch the errors like they are intended to do.
I know it sucks, but the most effective means I've found is to throw an exception, rather than assert for all but the most trivial of checks. My reasoning: if it's important enough to check in development/QA, it's important enough to check in a production environment and to fail to a known state.
The problem you'll run into assertions in a production, optimized compile of C++, is that the checks are not performed, and when they fail execution is not stopped. Immediately you're into undefined behavior. Now, I'm sure you could argue that this should never happen with sufficient testing, and you'd likely be correct. But, excepting companies where software is the business and profit center, I'll counter that reproducible and extensive testing is not the norm and thus assertions, while well intended, will not catch the errors like they are intended to do.
written by Christopher Diggins, November 09, 2009
I am a big fan of throwing exceptions. They make code more robust and tolerant of error.
However, always using exceptions instead of assertions means that the programmer will think twice about how they use them.
They may start to think about performance, or they may say, "I am completely confident that I coded this correctly, and my assumption is true".
The point of assertions is that they explicitly state our assumptions programmatically to make sure we catch silly mistakes, and assure that things don't break when we refactor.
In summary, if you find that your assertions can all be replaced with exceptions, this is exactly what you should be doing. However you probably are not using assertions in all the places where you need to.
However, always using exceptions instead of assertions means that the programmer will think twice about how they use them.
They may start to think about performance, or they may say, "I am completely confident that I coded this correctly, and my assumption is true".
The point of assertions is that they explicitly state our assumptions programmatically to make sure we catch silly mistakes, and assure that things don't break when we refactor.
In summary, if you find that your assertions can all be replaced with exceptions, this is exactly what you should be doing. However you probably are not using assertions in all the places where you need to.
written by Michelle Prather, November 13, 2009
"...excepting companies where software is the business and profit center, I'll counter that reproducible and extensive testing is not the norm..."
Hoo, boy, you got that right. My "testers" have other jobs to do. They'll run a few things they know and call it good. It's up to me to try to figure out all the different ways users might try to use the program. I can't seem to convince them of the value of their outsider (at least relative to the code) perspective.
Hoo, boy, you got that right. My "testers" have other jobs to do. They'll run a few things they know and call it good. It's up to me to try to figure out all the different ways users might try to use the program. I can't seem to convince them of the value of their outsider (at least relative to the code) perspective.
written by Christopher Yeleighton, November 14, 2009
Why would you implement a square root function? 
It should be in the run-time library of your programming language of choice, probably just a proxy to the C library.
It should be in the run-time library of your programming language of choice, probably just a proxy to the C library. written by Yun Li, November 18, 2009
Initially, I didn't think unit test is helpful for my code quality (even life). But after I took unit test as a way to verify my code on a project. I entirely understood why unit test is so advocated.
For code review, from my point of view the result of it isn't as good as I expected. It depends on the experience of reviewers and attitude. It's a little bit unreliable.
How about assertion? I agree it's a good way for debugging but for verifying the code quality it's not enough. To verify the quality of my code, the essence is I have to make my code being run under ALL test cases. The purpose of unit test is trying to write as more as possible test cases to run to verify the behavior of my code.
I'm a huge fun who believes "Quality comes from design", with good design the chance for making mistake is small. However, I still deem unit test is good to make sure the success of projects. Of course how to balance the effort for unit test is considerable.
For code review, from my point of view the result of it isn't as good as I expected. It depends on the experience of reviewers and attitude. It's a little bit unreliable.
How about assertion? I agree it's a good way for debugging but for verifying the code quality it's not enough. To verify the quality of my code, the essence is I have to make my code being run under ALL test cases. The purpose of unit test is trying to write as more as possible test cases to run to verify the behavior of my code.
I'm a huge fun who believes "Quality comes from design", with good design the chance for making mistake is small. However, I still deem unit test is good to make sure the success of projects. Of course how to balance the effort for unit test is considerable.
Write comment
Show/Hide comment formYou must be logged in to a comment. Please register if you do not have an account yet.
