UPGRADE YOUR SKILLS: Learn advanced Swift and SwiftUI on Hacking with Swift+! >>

How to measure code coverage in Xcode

Add some metrics to your tests, but do so carefully!

Paul Hudson       @twostraws

Code coverage is an easy metric to add to our projects: Xcode launches our tests, runs them them all, then tracks how many lines of our code were executed. In this article we’re going to look at how it works – specifically why it’s less than perfect as a metric! – then look at how to use it in your own projects.

Hacking with Swift is sponsored by RevenueCat

SPONSORED Take the pain out of configuring and testing your paywalls. RevenueCat's Paywalls allow you to remotely configure your entire paywall view without any code changes or app updates.

Learn more here

Sponsor Hacking with Swift and reach the world's largest Swift community!

A warning before we start

Running code and testing code aren’t the same thing, which makes code coverage an easy metric to game if you’re so inclined.

For example, here’s a User struct that can be created with a username and password, and during initialization we encrypt the password using some sort of algorithm:

struct User {
    var username: String
    var password: String

    init(username: String, password: String) {
        self.username = username

        // super-secret algorithm!
        self.password = String(password.reversed())
    }

    func authenticate() -> Bool {
        if username.count > 5 && password.count >= 12 {
            return true
        } else {
            return false
        }
    }
}

To test out that code, we could write a test like this:

func testUserAuthenticationSucceeds() {
    // given
    let user = User(username: "twostraws", password: "i_heart_chris_lattner")

    // when
    let result = user.authenticate()

    // then
    XCTAssertTrue(result, "User authentication should succeed.")
}

That test successfully evaluates most of the lines in the User struct, and if we added a second test with a short username then all the code would be run.

However, we’ve written nothing to test that our super secret encryption algorithm actually works. So, even though Xcode will tell us that we have 100% code coverage, that doesn’t mean we have 100% test coverage.

With that massive proviso out of the way, let’s take a look at how to track coverage in Xcode…

Tracking code coverage

To try out code coverage for real, create a new iOS app using the Single View App template, naming it Coverage, and making sure you check the “Include Unit Tests” box.

Xcode’s code coverage tracking is disabled by default, so the first thing we’ll do is turn it on. So, go to the Product menu and choose Scheme > Edit Scheme. Next, choose the Test scheme and activate its Options tab. Finally, check the Code Coverage box, so that Xcode will gather code coverage data while your tests are running.

Xcode’s template gives us a variety of method stubs, but I’d like you to add in the User example I gave earlier so we have something more interesting to work with. So, create a new Swift file called User.swift in your new project, then give it this content:

struct User {
    var username: String
    var password: String

    init(username: String, password: String) {
        self.username = username
        self.password = String(password.reversed())
    }

    func authenticate() -> Bool {
        if username.count > 5 && password.count >= 12 {
            return true
        } else {
            return false
        }
    }
}

Now open CoverageTests.swift and paste in this test method:

func testUserAuthenticationSucceeds() {
    // given
    let user = User(username: "twostraws", password: "i_heart_chris_lattner")

    // when
    let result = user.authenticate()

    // then
    XCTAssertTrue(result, "User authentication should succeed.")
}

Finally, press Cmd+U to run all our tests, collecting code coverage along the way. This is a nice example to work with when learning about code coverage, because it shows us three important things.

Press Cmd+9 to open the report navigator, and look for “Coverage” under the most recent test run. This will show the code coverage data that was just collected – open the disclosure indicators so you can see inside User.swift. You should see three things:

  • An unnamed function (the green “F” with nothing next to it) is actually the initializer for User along with its properties. It’s marked as having 100% code coverage even though it’s not directly being tested, because we initialize a User before calling authenticate().
  • The call to authenticate() has 57.1% coverage, which is a bit of a strange number given that it only contains 5 lines of code including braces.
  • There’s a third entry: “implicit closure #1 : @autoclosure () throws -> Swift.Bool in Coverage.User.authenticate………” This should have 100% coverage.

All our property and initializer lines were run, which makes sense because we had to run them just to create a user to test with, but those other two are odd.

Let’s start with the authenticate() method. Our test passes because we created a user with a sufficiently long username and password, but it means we only tested one part of the condition. Here’s the code again:

func authenticate() -> Bool {
    if username.count > 5 && password.count >= 12 {
        return true
    } else {
        return false
    }
}

Xcode counts all the lines there as being part of the method, so there are seven lines in total. Of those, only the follow were run:

func authenticate() -> Bool {
    if username.count > 5 && password.count >= 12 {
        return true
    } else {

So, four lines out of seven – that explains where the coverage of 57.1% comes from, because each of the four lines of code is “worth” 14.29%.

You can see this in action by enabling Xcode’s visual feedback of code coverage: open User.swift, then go to the Editor menu and choose Show Code Coverage. This will cause Xcode to show colored bands along the right edge of your source code, telling you which code was and was not run – no color means it was run, a salmon color means it was not run, and salmon stripes mean it was partially run, such as in the case of } else {.

Now for the more complex part: “implicit closure #1”. This is shown as having 100% code coverage, which is lovely, but what is it? You might think that our code doesn’t have any closures, but that’s why it says implicit – this isn’t a closure we created by hand.

If we look inside Swift’s standard library things ought to become clearer:

public static func && (lhs: Bool, rhs: @autoclosure () throws -> Bool) rethrows -> Bool {
    return lhs ? try rhs() : false
}

That’s the source code for the && operator, which is most commonly used to check multiple conditions in a single if statement. It’s what we use in our authenticate() method:

if username.count > 5 && password.count >= 12 {

What && does is take some sort of boolean on its left, and some sort of code that returns a boolean on the right, and returns true if both of them return true. But if the thing on the left is false then the thing on the right is never actually checked. This behavior is made possible by autoclosures: Swift silently wraps the right-hand side of the code in a miniature closure so that it’s only run when needed.

So, when we wrote username.count > 5 && password.count >= 12 both the username.count > 5 and the password.count >= 12 parts were run. Try changing the authenticate() method to use || rather than &&, like this:

func authenticate() -> Bool {
    if username.count > 5 || password.count >= 12 {
        return true
    } else {
        return false
    }
}

Press Cmd+U to re-run all tests, then view the new coverage report - this time you’ll see the implicit closure (this time caused by ||) has 0% coverage, because the right-hand side of || is never being evaluated.

So, that “implicit closure #1” line might look weird, but it’s extremely helpful for pointing out subtle places where you don’t have full coverage.

What is a good amount of code coverage?

There’s no one single figure that you should aim for in terms of code coverage, although I have to admit when I see folks talking about 100% coverage I do wonder what hacks they did to make that happen. If you want to have something to aim for, try 85%, but honestly the difference between 75% and 85% isn’t as great as you might expect.

There’s a well-respected test author called Brian Marick, who once wrote this about code coverage: “If you treat their clues as commands, you'll end up in the fable of the Sorcerer's Apprentice: causing a disaster because your tools do something very precisely, very enthusiastically, and with inhuman efficiency - but that something is only what you thought you wanted.”

Hacking with Swift is sponsored by RevenueCat

SPONSORED Take the pain out of configuring and testing your paywalls. RevenueCat's Paywalls allow you to remotely configure your entire paywall view without any code changes or app updates.

Learn more here

Sponsor Hacking with Swift and reach the world's largest Swift community!

BUY OUR BOOKS
Buy Pro Swift Buy Pro SwiftUI Buy Swift Design Patterns Buy Testing Swift Buy Hacking with iOS Buy Swift Coding Challenges Buy Swift on Sundays Volume One Buy Server-Side Swift Buy Advanced iOS Volume One Buy Advanced iOS Volume Two Buy Advanced iOS Volume Three Buy Hacking with watchOS Buy Hacking with tvOS Buy Hacking with macOS Buy Dive Into SpriteKit Buy Swift in Sixty Seconds Buy Objective-C for Swift Developers Buy Beyond Code

Was this page useful? Let us know!

Average rating: 5.0/5

 
Unknown user

You are not logged in

Log in or create account
 

Link copied to your pasteboard.