A beginners’ look at Mutation Testing and how to get started with Infection, the Mutation Testing Framework for PHP to improve the quality of your web applications.
You are developing your web applications with automated unit, acceptance, end-to-end, resilience and security tests and even your deployment is fully automated with build pipelines. Super! Great Job! You’ve come a long way.
In my discussions with other developers I learned that small coding errors still slip through the cracks and make their way into production because the unit tests didn’t pick up these small, insignificant changes that causes major problems in a live environment.
Infection, the Mutation Testing Framework for PHP
Say hello to Infection, the Mutation Testing Framework for PHP! I was first introduced to Infection through the newsletter of Chris Hartjes of November 2017 where Chris explained what Mutation Testing is and how Infection applied this technique for PHP applications.
To get started with Infection, just grab it from Packagist and install with Composer.
composer require --dev infection/infection
The first time you run Infection, it will ask you to set the path to your source code (e.g. src/), what your timeout in seconds for your mutation tests will be and where you want to store your mutation test log. Infection will store these settings in infection.json.dist. You can override it for local development in infection.json.
For this example, my infection.json.dist looks like the following:
{
"timeout": 5,
"source": {
"directories": [
"src"
]
},
"logs": {
"text": "build\/logs\/infection.txt"
}
}
What is Mutation Testing?
Mutation Testing is an advanced testing technique to make small changes in the source code (“Mutations”) and see if your unit tests are failing because of these changes (“Killing the Mutant”).
To give you an example, consider the following code which is a small application that will classify a person based on their age in a funny group name.
<?php
namespace In2it\Blog\Example;
class AgeCalculator
{
/**
* @var int
*/
protected $age;
public function __construct(int $age)
{
$this->age = $age;
}
public function wisdomGroup(): string
{
if (15 > $this->age) {
return 'Little Chipmunk';
} elseif (35 > $this->age) {
return 'Wild Tiger';
} elseif (55 > $this->age) {
return 'Majestic Elephant';
} elseif (75 > $this->age) {
return 'Grey Monkey';
} else {
return 'Sportive Snail';
}
}
}
This is all tested with unit tests (of course).
<?php
namespace In2it\Blog\Test\Example;
use In2it\Blog\Example\AgeCalculator;
use PHPUnit\Framework\TestCase;
class AgeCalculatorTest extends TestCase
{
public function testExceptionThrownWhenNoAgeProvided()
{
$this->expectException(\ArgumentCountError::class);
$ac = new AgeCalculator();
$this->fail('Expected exception was not thrown');
}
public function widsomGroupProvider(): array
{
return [
[8, 'Little Chipmunk'],
[24, 'Wild Tiger'],
[44, 'Majestic Elephant'],
[72, 'Grey Monkey'],
[84, 'Sportive Snail'],
];
}
/**
* @dataProvider widsomGroupProvider
*/
public function testWisdomGroupIsCalculatedOnAge(
int $age,
string $group
) {
$ac = new AgeCalculator($age);
$actual = $ac->wisdomGroup();
$this->assertSame($group, $actual);
}
}
For PHPUnit this example is no problem at all.
PHPUnit 7.1.5 by Sebastian Bergmann and contributors.
...... 6 / 6 (100%)
Time: 102 ms, Memory: 4.00MB
OK (6 tests, 6 assertions)
But when we fire up Infection, we get another view on the matter:
You are running Infection with xdebug enabled.
____ ____ __ _
/ _/___ / __/__ _____/ /_(_)___ ____
/ // __ \/ /_/ _ \/ ___/ __/ / __ \/ __ \
_/ // / / / __/ __/ /__/ /_/ / /_/ / / / /
/___/_/ /_/_/ \___/\___/\__/_/\____/_/ /_/
Running initial test suite...
PHPUnit version: 7.1.5
12 [============================] < 1 sec
Generate mutants...
Processing source code files: 1/1
Creating mutated files and processes: 17/17
.: killed, M: escaped, S: uncovered, E: fatal error, T: timed out
MMM.MMM.MMM.MMM.. (17 / 17)
17 mutations were generated:
5 mutants were killed
0 mutants were not covered by tests
12 covered mutants were not detected
0 errors were encountered
0 time outs were encountered
Metrics:
Mutation Score Indicator (MSI): 29%
Mutation Code Coverage: 100%
Covered Code MSI: 29%
Please note that some mutants will inevitably be harmless (i.e. false positives).
To understand really what Infection did and what escaped, killed and uncovered mutants are, we need to look at the log that was generated by Infection.
Escaped mutants:
================
1) /Users/dragonbe/workspace/in2it/blog/mutation-testing/src/AgeCalculator.php:19 [M] DecrementInteger
exec /opt/php7/bin/php -c /private/var/folders/m0/72rv98hs1514ndlk6q_fjqy40000gn/T/infectionANFGZu /Users/dragonbe/workspace/in2it/blog/mutation-testing/vendor/phpunit/phpunit/phpunit --configuration /var/folders/m0/72rv98hs1514ndlk6q_fjqy40000gn/T/infection/phpunitConfiguration.ed2627bb99e6f47a27aed5c4f6431300.infection.xml --stop-on-failure
--- Original
+++ New
@@ @@
}
public function wisdomGroup() : string
{
- if (15 > $this->age) {
+ if (14 > $this->age) {
return 'Little Chipmunk';
} elseif (35 > $this->age) {
return 'Wild Tiger';
PHPUnit 7.1.5 by Sebastian Bergmann and contributors.
...... 6 / 6 (100%)
Time: 37 ms, Memory: 4.00MB
OK (6 tests, 6 assertions)
In this scenario Infection modified the upper limit value for age by one less, which in our example didn’t made any significant change, and executed our unit tests on it.
Because this was not captured by PHPUnit as a failure and the application didn’t fail either, we can say this was an “escaped mutant”.
Killed mutants:
===============
1) /Users/dragonbe/workspace/in2it/blog/mutation-testing/src/AgeCalculator.php:19 [M] GreaterThanNegotiation
exec /opt/php7/bin/php -c /private/var/folders/m0/72rv98hs1514ndlk6q_fjqy40000gn/T/infectionANFGZu /Users/dragonbe/workspace/in2it/blog/mutation-testing/vendor/phpunit/phpunit/phpunit --configuration /var/folders/m0/72rv98hs1514ndlk6q_fjqy40000gn/T/infection/phpunitConfiguration.7637ec0ce6ed7114f3319059939a3fe4.infection.xml --stop-on-failure
--- Original
+++ New
@@ @@
}
public function wisdomGroup() : string
{
- if (15 > $this->age) {
+ if (15 <= $this->age) {
return 'Little Chipmunk';
} elseif (35 > $this->age) {
return 'Wild Tiger';
PHPUnit 7.1.5 by Sebastian Bergmann and contributors.
.F
Time: 55 ms, Memory: 4.00MB
There was 1 failure:
1) In2it\Blog\Test\Example\AgeCalculatorTest::testWisdomGroupIsCalculatedOnAge with data set #0 (8, 'Little Chipmunk')
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'Little Chipmunk'
+'Wild Tiger'
/Users/dragonbe/workspace/in2it/blog/mutation-testing/tests/AgeCalculatorTest.php:35
FAILURES!
Tests: 2, Assertions: 2, Failures: 1.
In this case we change < into >=, flipping the comparison of values all together. The fact that PHPUnit expected a different result, the test failed and thus the “Mutant was killed”.
What does this tells us? In our application we’re relying on hard-coded values for thresholds (15, 35, 55 and 75) and rely on a static comparison between these values and the values we passed on the during our tests.
Code should not have these hard-coded values within a method, it would be better to define these thresholds as constants to the class.
const TRESHOLD_CHIPMUNK = 15;
const TRESHOLD_TIGER = 35;
const TRESHOLD_ELEPHANT = 55;
const TRESHOLD_MONKEY = 75;
Another step to improve our code is to define a method to do the comparison.
private function lessThan(int $treshold, int $age): bool
{
return ($treshold > $age);
}
We now just need to refactor our wisdomGroup method to make use of our changes.
public function wisdomGroup(): string
{
if ($this->lessThan(self::TRESHOLD_CHIPMUNK, $this->age)) {
return 'Little Chipmunk';
} elseif ($this->lessThan(self::TRESHOLD_TIGER, $this->age)) {
return 'Wild Tiger';
} elseif ($this->lessThan(self::TRESHOLD_ELEPHANT, $this->age)) {
return 'Majestic Elephant';
} elseif ($this->lessThan(self::TRESHOLD_MONKEY, $this->age)) {
return 'Grey Monkey';
} else {
return 'Sportive Snail';
}
}
Time to run Infection again. This time we get a full 100%. You also see that we reduced the amount of mutations from 17 to 3 because we now have less chances to have our code mutated.
You are running Infection with xdebug enabled.
____ ____ __ _
/ _/___ / __/__ _____/ /_(_)___ ____
/ // __ \/ /_/ _ \/ ___/ __/ / __ \/ __ \
_/ // / / / __/ __/ /__/ /_/ / /_/ / / / /
/___/_/ /_/_/ \___/\___/\__/_/\____/_/ /_/
Running initial test suite...
PHPUnit version: 7.1.5
15 [============================] < 1 sec
Generate mutants...
Processing source code files: 1/1
Creating mutated files and processes: 3/3
.: killed, M: escaped, S: uncovered, E: fatal error, T: timed out
... (3 / 3)
3 mutations were generated:
3 mutants were killed
0 mutants were not covered by tests
0 covered mutants were not detected
0 errors were encountered
0 time outs were encountered
Metrics:
Mutation Score Indicator (MSI): 100%
Mutation Code Coverage: 100%
Covered Code MSI: 100%
Please note that some mutants will inevitably be harmless (i.e. false positives).
Just to give you a clear idea how the code and the tests look like, I’ve included them here.
AgeCalculator class
<?php
namespace In2it\Blog\Example;
class AgeCalculator
{
const TRESHOLD_CHIPMUNK = 15;
const TRESHOLD_TIGER = 35;
const TRESHOLD_ELEPHANT = 55;
const TRESHOLD_MONKEY = 75;
/**
* @var int
*/
protected $age;
public function __construct(int $age)
{
$this->age = $age;
}
public function wisdomGroup(): string
{
if ($this->lessThan(self::TRESHOLD_CHIPMUNK, $this->age)) {
return 'Little Chipmunk';
} elseif ($this->lessThan(self::TRESHOLD_TIGER, $this->age)) {
return 'Wild Tiger';
} elseif ($this->lessThan(self::TRESHOLD_ELEPHANT, $this->age)) {
return 'Majestic Elephant';
} elseif ($this->lessThan(self::TRESHOLD_MONKEY, $this->age)) {
return 'Grey Monkey';
} else {
return 'Sportive Snail';
}
}
/**
* Comparing the age against a treshold
*
* @param int $treshold The treshold for age comparison
* @param int $age The age to compare
* @return bool
*/
private function lessThan(int $treshold, int $age): bool
{
return ($treshold > $age);
}
}
AgeCalculatorTest class
<?php
namespace In2it\Blog\Test\Example;
use In2it\Blog\Example\AgeCalculator;
use PHPUnit\Framework\TestCase;
class AgeCalculatorTest extends TestCase
{
public function testExceptionThrownWhenNoAgeIsProvidedToConstructor()
{
$this->expectException(\ArgumentCountError::class);
$ac = new AgeCalculator();
$this->fail('Expected exception was not thrown for missing argument');
}
public function widsomGroupProvider(): array
{
return [
[8, 'Little Chipmunk'],
[24, 'Wild Tiger'],
[44, 'Majestic Elephant'],
[72, 'Grey Monkey'],
[84, 'Sportive Snail'],
];
}
/**
* @covers \In2it\Blog\Example\AgeCalculator::__construct()
* @covers \In2it\Blog\Example\AgeCalculator::wisdomGroup()
* @dataProvider widsomGroupProvider
*/
public function testWisdomGroupIsCalculatedOnAge(int $age, string $group)
{
$ac = new AgeCalculator($age);
$actual = $ac->wisdomGroup();
$this->assertSame($group, $actual);
}
/**
* @covers \In2it\Blog\Example\AgeCalculator::__construct()
* @covers \In2it\Blog\Example\AgeCalculator::lessThan()
*/
public function testAgeLessThanReturnsFalseForHigherAge()
{
$lessThan = new \ReflectionMethod(AgeCalculator::class, 'lessThan');
$lessThan->setAccessible(true);
$value = $lessThan->invokeArgs(new AgeCalculator(1), [0,1]);
$this->assertFalse($value);
}
/**
* @covers \In2it\Blog\Example\AgeCalculator::__construct()
* @covers \In2it\Blog\Example\AgeCalculator::lessThan()
*/
public function testAgeLessThanReturnsTrueForLowerAge()
{
$lessThan = new \ReflectionMethod(AgeCalculator::class, 'lessThan');
$lessThan->setAccessible(true);
$value = $lessThan->invokeArgs(new AgeCalculator(1), [1,0]);
$this->assertTrue($value);
}
/**
* @covers \In2it\Blog\Example\AgeCalculator::__construct()
* @covers \In2it\Blog\Example\AgeCalculator::lessThan()
*/
public function testAgeLessThanReturnsFalseForEqualAge()
{
$lessThan = new \ReflectionMethod(AgeCalculator::class, 'lessThan');
$lessThan->setAccessible(true);
$value = $lessThan->invokeArgs(new AgeCalculator(1), [1,1]);
$this->assertFalse($value);
}
}
The return values are still hard coded, which I’m not fond of. Nonetheless I’m going to keep them for what they are in this simple example. In more complex applications these values should come from a different source like a database or web service.
In this small example I have given you some insights into Mutation Testing and why it can help you improve your code and tests to ensure small unwanted changes will not break your application. I would love to see how your experiences are when you add Infection to your applications.
If you’re interested in seeing this in action, come and see me give a presentation about it at the PHPLeuven Meetup on June 7, 2018. I’m looking forward seeing you there.
