This is why testing is hard: You can't test everything, you can't test enough. So when are you going to stop?... What's the answer? Don't write tests!
— John Hughes, co-creator of QuickCheck in Testing the Hard Stuff and Staying Sane
Defining all the possible scenarios to cover our system under test is not always easy. For some parts of our software, writing all the possible combinations of input data and the state associated when the SUT is stimulated is not trivial at all. But, what if we could forget about the input data and the state of the software when it's evaluated in our tests? What if we could stay focused on the correct state of the software at the end of each test execution? Using property-based testing, we are going to generate random input data and check if using this input our software validates some properties.
Property-based testing is yet another mechanism to validate the correctness of software, that is, another testing strategy. The main difference between property-based testing and other approaches is that with the former, you need to define what properties your code needs to satisfy and with the latter, you are forced to define how those properties are going to be tested.
Is not a surprise that property-based testing was developed having declarative programming in mind, more specifically, functional programming with Haskell. QuickCheck was the first property-based testing framework and it was created by Koen Claessen and John Hughes. Here you have an example of the famous FizzBuzz kata tested using QuickCheck for Haskell.
Disclaimer! Remember this approach we are going to evaluate during these blog posts is not a silver bullet. Property-based testing is going to be just a new shiny tool in our toolbox. We are not going to replace our unit tests based on this new testing strategy.
To better understand what property-based testing is we are going to start defining our use case. A simple sum function implementation written in Swift and tested using SwiftCheck.
Given two integers, our sum function returns the addition of these values passed as parameters.
func sum(_ a: Int,_ b: Int) -> Int {
return a + b
}
We need to check our sum function implementation is correct. Using a classic testing approach we could check it writing tons of unit tests like this:
class SumSpec: XCTestCase {
func test1() {
expect(sum(1, 1)).to(equal(2))
}
func test2() {
expect(sum(1, 2)).to(equal(3))
}
func test3() {
expect(sum(3, 4)).to(equal(7))
}
func test4() {
expect(sum(5, 2)).to(equal(7))
}
...
}
But, what about negative values? What about big integers? What about the number 0? Using this approach we can't ensure our software is working for the test cases we didn't write.
What if, instead of staying focused on the input data, we forget about the input values and stay focused on the properties? What if we use property-based testing? Let's see the result.
Based on the addition properties, we are going to specify the sum function behavior. The three properties we are going to test are:
- Commutative property: A binary operation is commutative if changing the order of the operands does not change the result.
- Associative property: Within an expression containing two or more occurrences in a row of the same associative operator, the order in which the operations are performed does not matter as long as the sequence of the operands is not changed.
- Identity property: An identity element or neutral element is a special type of element of a set with respect to a binary operation on that set, which leaves other elements unchanged when combined with them.
We are going to check if our sum function matches these three properties using random integers as input.
class SumSpec: XCTestCase {
func testSumProperties() {
property("Commutative property. When I add two numbers, the result should not depend on parameter order") <- forAll { (a: Int, b: Int) in
let result1 = sum(a, b)
let result2 = sum(b, a)
return result1 == result2
}
property("Associative property. Adding 1 twice is the same than adding 2") <- forAll { (a: Int) in
let result1 = sum(sum(a, 1), 1)
let result2 = sum(a, 2)
return result1 == result2
}
property("Identity property. Adding 0 is the same as adding nothing") <- forAll { (a: Int) in
return a == sum(a, 0)
}
}
}
Here you have the Java implementation:
Our sum function written in Java:
private int sum(int a, int b) {
return a + b;
}
The sum function properties specified using JUnit-QuickCheck:
@RunWith(JUnitQuickcheck.class) public class SumProperties {
//Commutative property
@Property public void whenIAddTwoNumbersTheResultShouldNotDependOnTheOrder(int a, int b) {
int result1 = sum(a, b);
int result2 = sum(b, a);
assertEquals(result1, result2);
}
//Associative property
@Property public void addingOneTwiceIsTheSameThanAddingTwo(int a) {
int result1 = sum(sum(a, 1), 1);
int result2 = sum(a, 2);
assertEquals(result1, result2);
}
//Identity property
@Property public void addingZeroIsTheSameThanAddingNothing(int a) {
assertEquals(a, sum(a, 0));
}
}
We have simplified the example to only accept integers but a solution accepting any numeric value is also possible.
Staying focused on the three sum function properties and forgetting about the input data we are validating this implementation for many different values. The property-based testing framework chosen is going to evaluate our properties 100 times by default using random integers.
But what about real-world software? What about non-pure software full of side effects? During the next blog post, we review a more realistic example. Take a look!
This post is just a small part of the content shown during our testing training. If you are interested in performing a training in your company or to be part of our open trainings, all the information can be found in our web site.