Voxxed Days Athens - Improve your tests with Mutation Testing

42
MUTATION TESTING TO THE RESCUE OF YOUR TESTS ΝΙΚΟΛΑΣ ΦΡΑΝΚΕΛ

Transcript of Voxxed Days Athens - Improve your tests with Mutation Testing

Page 1: Voxxed Days Athens - Improve your tests with Mutation Testing

MUTATION TESTINGTO THE RESCUE OF YOUR TESTS

ΝΙΚΟΛΑΣ ΦΡΑΝΚΕΛ

Page 2: Voxxed Days Athens - Improve your tests with Mutation Testing

ME, MYSELF AND I

@nicolas_frankel #mutationtesting

By day

• Consultant ($$$)

By night

•Developer

• Blogger

• Book author

• Teacher/trainer

2

Page 3: Voxxed Days Athens - Improve your tests with Mutation Testing

HYBRIS, AN SAP COMPANY

@nicolas_frankel #mutationtesting 3

Page 4: Voxxed Days Athens - Improve your tests with Mutation Testing

MANY KINDS OF TESTING

@nicolas_frankel #mutationtesting

Unit Testing

Integration Testing

End-to-end Testing

Performance Testing

Penetration Testing

Exploratory Testing

etc.

4

Page 5: Voxxed Days Athens - Improve your tests with Mutation Testing

THEIR ONLY SINGLE GOAL

@nicolas_frankel #mutationtesting

Ensure the quality of the production code

5

Page 6: Voxxed Days Athens - Improve your tests with Mutation Testing

THE PROBLEM

@nicolas_frankel #mutationtesting

How to check the quality of the testing code?

6

Page 7: Voxxed Days Athens - Improve your tests with Mutation Testing

CODE COVERAGE

@nicolas_frankel #mutationtesting

“Code coverage is a measure used to describe the degree to which the source code of a program is tested”

--Wikipediahttp://en.wikipedia.org/wiki/Co

de_coverage

7

Page 8: Voxxed Days Athens - Improve your tests with Mutation Testing

MEASURING CODE COVERAGE

@nicolas_frankel #mutationtesting

Check whether a source code line is executed during a test

•Or Branch Coverage

8

Page 9: Voxxed Days Athens - Improve your tests with Mutation Testing

COMPUTING CODE COVERAGE

CC =Lexecuted

Ltotal*100

CC: Code Coverage(in percent)

Lexecuted: Number of executed lines of code

Ltotal: Number of total lines of code

@nicolas_frankel #mutationtesting 9

Page 10: Voxxed Days Athens - Improve your tests with Mutation Testing

JAVA TOOLS FOR CODE COVERAGE

@nicolas_frankel #mutationtesting

JaCoCo

Clover

Cobertura

etc.

10

Page 11: Voxxed Days Athens - Improve your tests with Mutation Testing

100% CODE COVERAGE?

@nicolas_frankel #mutationtesting

“Is 100% code coverage realistic? Of course it is. If you can write a line of code, you can write another that tests it.”

Robert Martin (Uncle Bob)https://twitter.com/unclebobmartin/status/5596662050966732

8

11

Page 12: Voxxed Days Athens - Improve your tests with Mutation Testing

ASSERT-LESS TESTING

@Test

public void add_should_add() {

new Math().add(1, 1);

}

@nicolas_frankel #mutationtesting

But, where is the

assert?As long as the Code Coverage is

OK…

12

Page 13: Voxxed Days Athens - Improve your tests with Mutation Testing

CODE COVERAGE AS A MEASURE OF QUALITY

@nicolas_frankel #mutationtesting

Any metric can be gamed!

Code coverage is a metric…

⇒ Code coverage can be gamed

• On purpose

• Or by accident

13

Page 14: Voxxed Days Athens - Improve your tests with Mutation Testing

CODE COVERAGE AS A MEASURE OF QUALITY

@nicolas_frankel #mutationtesting

Code Coverage lulls you into a false sense of security…

14

Page 15: Voxxed Days Athens - Improve your tests with Mutation Testing

THE PROBLEM STILL STANDS

@nicolas_frankel #mutationtesting

Code coverage cannot ensure test quality

• Is there another way?

Mutation Testing to the rescue!

15

Page 16: Voxxed Days Athens - Improve your tests with Mutation Testing

THE CAST

@nicolas_frankel #mutationtesting

William Stryker

Original Source Code

Jason Stryker

Modified Source Code

a.k.a “The Mutant”

16

Page 17: Voxxed Days Athens - Improve your tests with Mutation Testing

public class Math {

public int add(int i1, int i2) {

return i1 + i2;

}

}

@nicolas_frankel #mutationtesting

public class Math {

public int add(int i1, int i2) {

return i1 - i2;

}

}

17

Page 18: Voxxed Days Athens - Improve your tests with Mutation Testing

STANDARD TESTING

@nicolas_frankel #mutationtesting

✔Execute Test

18

Page 19: Voxxed Days Athens - Improve your tests with Mutation Testing

MUTATION TESTING

@nicolas_frankel #mutationtesting

?Execute SAME Test

MUTATION

19

Page 20: Voxxed Days Athens - Improve your tests with Mutation Testing

MUTATION TESTING

@nicolas_frankel #mutationtesting

✔Execute SAME Test

Execute SAME Test

Mutant

Killed

Mutant

Survived

20

Page 21: Voxxed Days Athens - Improve your tests with Mutation Testing

KILLED OR SURVIVING?

@nicolas_frankel #mutationtesting

Surviving means changing the source code did notchange the test result

• It’s bad!

Killed means changing the source code changed the test result

• It’s good

21

Page 22: Voxxed Days Athens - Improve your tests with Mutation Testing

TEST THE CODE

@nicolas_frankel #mutationtesting

✔Execute Test

public class Math {

public int add(int i1, int i2) {

return i1 + i2;

}

}

@Test

public void add_should_add() {

new Math().add(1, 1);

}

22

Page 23: Voxxed Days Athens - Improve your tests with Mutation Testing

✔Execute Test

SURVIVING MUTANT

@nicolas_frankel #mutationtesting

public class Math {

public int add(int i1, int i2) {

return i1 - i2;

}

}

@Test

public void add_should_add() {

new Math().add(1, 1);

}

23

Page 24: Voxxed Days Athens - Improve your tests with Mutation Testing

TEST THE CODE

@nicolas_frankel #mutationtesting

@Test

public void add_should_add() {

int sum = new Math().add(1, 1);

Assert.assertEquals(sum, 2);

}

✔Execute Test

public class Math {

public int add(int i1, int i2) {

return i1 + i2;

}

}

24

Page 25: Voxxed Days Athens - Improve your tests with Mutation Testing

KILLED MUTANT

@nicolas_frankel #mutationtesting

✗Execute SAME Test

@Test

public void add_should_add() {

int sum = new Math().add(1, 1);

Assert.assertEquals(sum, 2);

}

public class Math {

public int add(int i1, int i2) {

return i1 - i2;

}

}

25

Page 26: Voxxed Days Athens - Improve your tests with Mutation Testing

MUTATION TESTING IN JAVA

@nicolas_frankel #mutationtesting

PIT is a tool for Mutation testing

Available as

• Command-line tool

•Ant target

•Maven plugin

26

Page 27: Voxxed Days Athens - Improve your tests with Mutation Testing

MUTATORS

@nicolas_frankel #mutationtesting

Mutators are patterns applied to source code to produce mutations

27

Page 28: Voxxed Days Athens - Improve your tests with Mutation Testing

PIT MUTATORS SAMPLE

Name Example source Result

Conditionals Boundary > >=

Negate Conditionals == !=

Remove Conditionals foo == bar true

Math + -

Increments foo++ foo--

Invert Negatives -foo foo

Inline Constant static final FOO= 42 static final FOO = 43

Return Values return true return false

Void Method Call System.out.println("foo")

Non Void Method Call long t = System.currentTimeMillis() long t = 0

Constructor Call Date d = new Date() Date d = null;

@nicolas_frankel #mutationtesting 28

Page 29: Voxxed Days Athens - Improve your tests with Mutation Testing

IMPORTANT MUTATORS

@nicolas_frankel #mutationtesting

Conditionals Boundary

• Potential serious bug hiding there

•if (foo > bar)

29

Page 30: Voxxed Days Athens - Improve your tests with Mutation Testing

IMPORTANT MUTATORS

@nicolas_frankel #mutationtesting

Void Method Call• Assert.checkNotNull()

• connection.close()

30

Page 31: Voxxed Days Athens - Improve your tests with Mutation Testing

REMEMBER

@nicolas_frankel #mutationtesting

It’s not because the IDE generates code safely that it will never change

• equals()

• hashCode()

31

Page 32: Voxxed Days Athens - Improve your tests with Mutation Testing

ENOUGH TALK…

@nicolas_frankel #mutationtesting 32

Page 33: Voxxed Days Athens - Improve your tests with Mutation Testing

SH… HAPPENS

@nicolas_frankel #mutationtesting

False positives Imperfect Sloooooooooooooooow

33

Page 34: Voxxed Days Athens - Improve your tests with Mutation Testing

PIT IS IMPERFECT

if (p < 0)

...

// changed condition boundary -> survived:

if (p > 0)

...

return 0;

@nicolas_frankel #mutationtesting 34

Page 35: Voxxed Days Athens - Improve your tests with Mutation Testing

PIT IS IMPERFECT

void rebootMachine() {

// removed method call:

checkUserPermissions();

Runtime.getRuntime().exec("reboot");

}

@nicolas_frankel #mutationtesting 35

Page 36: Voxxed Days Athens - Improve your tests with Mutation Testing

WHY SO SLOW?

@nicolas_frankel #mutationtesting

Analyze test code

For each class under test

• For each mutator

• Create mutation

• For each mutation

• Run test

• Analyze result

• Aggregate results

36

Page 37: Voxxed Days Athens - Improve your tests with Mutation Testing

WORKAROUNDS

Increase number of threads

Set a limited a set of mutators

Limit scope of target classes

Limit number of tests

Limit dependency distance

Use incremental analysis

Don’t bind to the test phase

Use scmMutationCoverage

@nicolas_frankel #mutationtesting 37

Page 38: Voxxed Days Athens - Improve your tests with Mutation Testing

INCREMENTAL ANALYSIS

Metadata stored between runs

Mutant will not be checked again, if:

• timed out, and class has not changed• killed, and neither class nor test have changed• survived, and there are no new/changed tests for it

@nicolas_frankel #mutationtesting 38

Page 39: Voxxed Days Athens - Improve your tests with Mutation Testing

THE SILVER BULLET?

@nicolas_frankel #mutationtesting

Checks the relevance of your unit tests

Points out potential bugs

39

Page 40: Voxxed Days Athens - Improve your tests with Mutation Testing

THE REST IS UP TO YOU

@nicolas_frankel #mutationtesting

Validate the assembled application

• Integration Testing

Check the performance

• Performance Testing

Look out for display bugs

• End-to-end testing

Etc.

40

Page 41: Voxxed Days Athens - Improve your tests with Mutation Testing

TESTING IS ABOUT ROI

@nicolas_frankel #mutationtesting

Don’t test to achieve 100% coverage

Test because it saves money in the long run

Prioritize:

• Business-critical code

• Complex code

41

Page 42: Voxxed Days Athens - Improve your tests with Mutation Testing

Q&A

@nicolas_frankel #mutationtesting

http://blog.frankel.ch/@nicolas_frankelhttp://frankel.in/https://git.io/vznQK

42