FREE: Read a new Swift article every day – click here! >>

How to measure code coverage in Xcode

Paul Hudson       @twostraws

Code coverage is an easy metric to add to our projects: Xcode launches our tests, runs them them all, then tracks how many lines of our code were executed. In this article we’re going to look at how it works – specifically why it’s less than perfect as a metric! – then look at how to use it in your own projects.

 

  • Advent of Swiftmas Day Eight: Today’s Swiftmas surprise is 50% off my book Hacking with macOS – if you can already make iOS apps, you'll find macOS apps a breeze! Click here to check it out!

     

A warning before we start

Running code and testing code aren’t the same thing, which makes code coverage an easy metric to game if you’re so inclined.

For example, here’s a User struct that can be created with a username and password, and during initialization we encrypt the password using some sort of algorithm:

struct User {
    var username: String
    var password: String

    init(username: String, password: String) {
        self.username = username

        // super-secret algorithm!
        self.password = String(password.reversed())
    }

    func authenticate() -> Bool {
        if username.count > 5 && password.count >= 12 {
            return true
        } else {
            return false
        }
    }
}

To test out that code, we could write a test like this:

func testUserAuthenticationSucceeds() {
    // given
    let user = User(username: "twostraws", password: "i_heart_chris_lattner")

    // when
    let result = user.authenticate()

    // then
    XCTAssertTrue(result, "User authentication should succeed.")
}

That test successfully evaluates most of the lines in the User struct, and if we added a second test with a short username then all the code would be run.

However, we’ve written nothing to test that our super secret encryption algorithm actually works. So, even though Xcode will tell us that we have 100% code coverage, that doesn’t mean we have 100% test coverage.

With that massive proviso out of the way, let’s take a look at how to track coverage in Xcode…

Tracking code coverage

To try out code coverage for real, create a new iOS app using the Single View App template, naming it Coverage, and making sure you check the “Include Unit Tests” box.

Xcode’s code coverage tracking is disabled by default, so the first thing we’ll do is turn it on. So, go to the Product menu and choose Scheme > Edit Scheme. Next, choose the Test scheme and activate its Options tab. Finally, check the Code Coverage box, so that Xcode will gather code coverage data while your tests are running.

Xcode’s template gives us a variety of method stubs, but I’d like you to add in the User example I gave earlier so we have something more interesting to work with. So, create a new Swift file called User.swift in your new project, then give it this content:

struct User {
    var username: String
    var password: String

    init(username: String, password: String) {
        self.username = username
        self.password = String(password.reversed())
    }

    func authenticate() -> Bool {
        if username.count > 5 && password.count >= 12 {
            return true
        } else {
            return false
        }
    }
}

Now open CoverageTests.swift and paste in this test method:

func testUserAuthenticationSucceeds() {
    // given
    let user = User(username: "twostraws", password: "i_heart_chris_lattner")

    // when
    let result = user.authenticate()

    // then
    XCTAssertTrue(result, "User authentication should succeed.")
}

Finally, press Cmd+U to run all our tests, collecting code coverage along the way. This is a nice example to work with when learning about code coverage, because it shows us three important things.

Press Cmd+9 to open the report navigator, and look for “Coverage” under the most recent test run. This will show the code coverage data that was just collected – open the disclosure indicators so you can see inside User.swift. You should see three things:

  • An unnamed function (the green “F” with nothing next to it) is actually the initializer for User along with its properties. It’s marked as having 100% code coverage even though it’s not directly being tested, because we initialize a User before calling authenticate().
  • The call to authenticate() has 57.1% coverage, which is a bit of a strange number given that it only contains 5 lines of code including braces.
  • There’s a third entry: “implicit closure #1 : @autoclosure () throws -> Swift.Bool in Coverage.User.authenticate………” This should have 100% coverage.

All our property and initializer lines were run, which makes sense because we had to run them just to create a user to test with, but those other two are odd.

Let’s start with the authenticate() method. Our test passes because we created a user with a sufficiently long username and password, but it means we only tested one part of the condition. Here’s the code again:

func authenticate() -> Bool {
    if username.count > 5 && password.count >= 12 {
        return true
    } else {
        return false
    }
}

Xcode counts all the lines there as being part of the method, so there are seven lines in total. Of those, only the follow were run:

func authenticate() -> Bool {
    if username.count > 5 && password.count >= 12 {
        return true
    } else {

So, four lines out of seven – that explains where the coverage of 57.1% comes from, because each of the four lines of code is “worth” 14.29%.

You can see this in action by enabling Xcode’s visual feedback of code coverage: open User.swift, then go to the Editor menu and choose Show Code Coverage. This will cause Xcode to show colored bands along the right edge of your source code, telling you which code was and was not run – no color means it was run, a salmon color means it was not run, and salmon stripes mean it was partially run, such as in the case of } else {.

Now for the more complex part: “implicit closure #1”. This is shown as having 100% code coverage, which is lovely, but what is it? You might think that our code doesn’t have any closures, but that’s why it says implicit – this isn’t a closure we created by hand.

If we look inside Swift’s standard library things ought to become clearer:

public static func && (lhs: Bool, rhs: @autoclosure () throws -> Bool) rethrows -> Bool {
    return lhs ? try rhs() : false
}

That’s the source code for the && operator, which is most commonly used to check multiple conditions in a single if statement. It’s what we use in our authenticate() method:

if username.count > 5 && password.count >= 12 {

What && does is take some sort of boolean on its left, and some sort of code that returns a boolean on the right, and returns true if both of them return true. But if the thing on the left is false then the thing on the right is never actually checked. This behavior is made possible by autoclosures: Swift silently wraps the right-hand side of the code in a miniature closure so that it’s only run when needed.

So, when we wrote username.count > 5 && password.count >= 12 both the username.count > 5 and the password.count >= 12 parts were run. Try changing the authenticate() method to use || rather than &&, like this:

func authenticate() -> Bool {
    if username.count > 5 || password.count >= 12 {
        return true
    } else {
        return false
    }
}

Press Cmd+U to re-run all tests, then view the new coverage report - this time you’ll see the implicit closure (this time caused by ||) has 0% coverage, because the right-hand side of || is never being evaluated.

So, that “implicit closure #1” line might look weird, but it’s extremely helpful for pointing out subtle places where you don’t have full coverage.

What is a good amount of code coverage?

There’s no one single figure that you should aim for in terms of code coverage, although I have to admit when I see folks talking about 100% coverage I do wonder what hacks they did to make that happen. If you want to have something to aim for, try 85%, but honestly the difference between 75% and 85% isn’t as great as you might expect.

There’s a well-respected test author called Brian Marick, who once wrote this about code coverage: “If you treat their clues as commands, you'll end up in the fable of the Sorcerer's Apprentice: causing a disaster because your tools do something very precisely, very enthusiastically, and with inhuman efficiency - but that something is only what you thought you wanted.”

 

 

MASTER SWIFT NOW
Buy Testing Swift Buy Practical iOS 12 Buy Pro Swift Buy Swift Design Patterns Buy Swift Coding Challenges Buy Server-Side Swift (Vapor Edition) Buy Server-Side Swift (Kitura Edition) Buy Hacking with macOS Buy Advanced iOS Volume One Buy Advanced iOS Volume Two Buy Hacking with watchOS Buy Hacking with tvOS Buy Hacking with Swift Buy Dive Into SpriteKit Buy Swift in Sixty Seconds Buy Objective-C for Swift Developers Buy Beyond Code

About the author

Paul Hudson is the creator of Hacking with Swift, the most comprehensive series of Swift books in the world. He's also the editor of Swift Developer News, the maintainer of the Swift Knowledge Base, and Mario Kart world champion. OK, so that last part isn't true. If you're curious you can learn more here.

Was this page useful? Let me know!

Average rating: 5.0/5

Click here to visit the Hacking with Swift store >>