Favorite Bugs: errno and Idempotence

I love bugs. Well, some bugs. Some bugs are so insidious, or their solution so simple, that they're worth remembering. This is one of those bugs.

Recently I wrote a type in Swift to represent a SHA-256 hash. To generate hashes from data in iOS, you can use the CommonCrypto library, but there's no function that I'm aware of that will parse a checksum from a hexadecimal string representation. I wanted a nice Swift wrapper that could perform that translation and hide the internal, byte-array nature of the checksum from callers.

Thus, my initial (simplified) implementation of SHA256Checksum:

public struct SHA256Checksum {

    fileprivate let value: [UInt8]

    public init?(hexString: String) {
        guard hexString.utf8.count == Int(CC_SHA256_DIGEST_LENGTH * 2) else {
            return nil
        }
        var valueBytes = [UInt8]()
        for i in stride(from: 0, to: hexString.characters.count, by: 2) {
            let startIndex = hexString.index(hexString.startIndex, offsetBy: i)
            let endIndex = hexString.index(startIndex, offsetBy: 2)
            let byteString = hexString.substring(with: startIndex..<endIndex)
            let value = strtoul(byteString, nil, 16)
            if value == 0 && errno == EINVAL { // conversion failed
                return nil
            }
            valueBytes.append(UInt8(value))
        }
        self.value = valueBytes
    }

}

The core loop in the initializer uses the C function strtoul to convert a two-character hexadecimal string to an unsigned long. That function is documented as such:

If no conversion could be performed, 0 is returned and the global variable errno is set to EINVAL.

So, naturally, the code checks for those conditions and returns nil in such cases. Simple enough, right? I wrote some unit tests, and everything worked as expected. Ship it!

Some time later, this code started failing to parse valid hashes. Specifically, it would fail on 115b1577b7b517405200445cfb2f8b5fe9a1ba3e7e68229ad927bd3cfbe135e2. Why would that happen?

Well, it turns out to be a case of missing idempotence. In the failing value, did you notice the pesky 00? Passing that string to strtoul, one would expect to get back 0 as a result. Since that would be a successful conversion, one would not expect errno to equal EINVAL, right? And yet, using the debugger, I was able to verify that this is exactly what was happening.

strtoul is like many C functions in that it sets errno as a side effect. Unfortunately, it's not guaranteed to do so except in the case of failure. Therefore, if you're expecting a specific value of errno after calling it, you need to set errno to a sane value before that. Otherwise, some other function (anywhere prior to your call) may have set it to something you didn't expect, like EINVAL!

The fix is ridiculously simple, but not obvious to the uninitiated:

    ...
    let byteString = hexString.substring(with: startIndex..<endIndex)
    errno = 0 // FIX: preset errno to avoid catastrophe
    let value = strtoul(byteString, nil, 16)
    if value == 0 && errno == EINVAL { // conversion failed
        return nil
    }
    valueBytes.append(UInt8(value))
    ...

Old C pros have known this for years. It's so common that it's part of CERT's secure C coding guidelines. It's a great example of why idempotence is important (and side effects are bad). And that's what makes this one of my favorite bugs.

Continuous Integration and Automatic Code Signing in Xcode 8

With Xcode 8, Apple introduced a new code signing and provisioning feature called Automatic Signing. For day-to-day development and distribution in Xcode, this is the preferred way to configure your apps and avoid the hassle of maintaining certificates and provisioning profiles. If you're not familiar with it yet, check out this WWDC 2016 Session for an introduction.

Unfortunately, Automatic Signing doesn't play well with headless builds, as typically executed on a continuous integration (CI) system. We're using Jenkins for our builds, and in the process of upgrading our systems to use Xcode 8 and Automatic Signing, code signing broke on Jenkins.

I wanted the best of both worlds, and this is the story of how I got there.

You can play along with the sample project on GitHub and see build output on TravisCI.

The New Way

Automatic Signing in Xcode

Configuring Automatic Signing for a project is pretty straightforward. In short, it involves just two steps:

  1. Check the "Automatically manage signing" checkbox in the target's General -> Signing UI.
  2. Select your development team from the list below that.

In an ideal world, Xcode will do the rest for you. It really is impressively streamlined. For the app target, it will use development code signing with your account's personal certificate by default. For an embedded framework target, it will know not to apply code signing.

Everyone on your team should use the same settings, and there should be no need to commit specific values for traditional code signing settings to source control. With that out of the way, you can move on to getting things working on CI.

The Old Way

Code signing in a continuous integration environment has historically been achieved using a combination of two variables in the build configuration: CODE_SIGN_IDENTITY and PROVISIONING_PROFILE. The former is the Common Name of your distribution certificate, and the latter is the UUID of the profile you're signing with. No matter how you specify them, to xcodebuild they look something like this:

CODE_SIGN_IDENTITY="iPhone Distribution: The Omega Corp., LLC (OMGWTFBBQ)"
PROVISIONING_PROFILE="3c351988-58bd-475c-8586-5d397860c19f"

The tricky part is that if you have Automatic Signing enabled, and you try to use this familiar manual code signing method, you'll see a frustrating error:

YourFancyApp has conflicting provisioning settings. YourFancyApp is automatically signed, but code signing identity iPhone Distribution: ... has been manually specified. Set the code signing identity value to "iPhone Developer" in the build settings editor, or switch to manual signing in the project editor.

Here's an example build on TravisCI that shows the error in context.

You could try to take that error message's advice, but you'll end up running in code signing circles. To properly work around this issue, you have to do two things:

  1. Programmatically set the code signing mode to manual in the project file.
  2. Provide PROVISIONING_PROFILE in the old way, but only for the targets that need it.

Switching to Manual, Captain!

If you're familiar with Xcode's build system, you might imagine there'd be a build configuration variable that allows you to switch between automatic and manual signing mode. Mysteriously, that is not the case in Xcode 8. The place where it's specified is in the rather-obscure project-level target attributes. These can't be edited directly in Xcode, but are controlled by the "Automatically manage signing" checkbox and friends in the target's "General" settings UI.

Checking and unchecking the box changes the value of the ProvisioningStyle attribute for the affected target in the project's TargetAttributes attribute. For headless/CI builds, you need to set its value to Manual, which is the same as unchecking the box in Xcode.

There are a few ways you could do this, but given the finicky nature of Xcode project files, I prefer to play it safe. There's an excellent Ruby gem called Xcodeproj that provides a clean way to manipulate the project file without resorting to something like sed.

Here's a simple ruby method (as used in a Fastlane Fastfile) that can make the switch happen:

# Switch ProvisioningStyle to Manual for all targets in the given project
def enable_manual_provisioning(project_file)
  UI.important "Switching ProvisioningStyle to Manual."
  fastlane_require 'xcodeproj'
  project = Xcodeproj::Project.open(project_file)
  target_attributes = project.root_object.attributes["TargetAttributes"]
  target_attributes.each { |id, attrs| attrs["ProvisioningStyle"] = 'Manual' }
  project.save
end

For more detail, see the sample project's Fastfile in context.

With that in place, you need only to specify the provisioning profile to use for signing.

Providing the Provisioning Profile

In a simple, single-target project, you could use the PROVISIONING_PROFILE build setting to specify the UUID of your desired profile and that would work fine. However, few significant apps these days use only a single target. You will likely have an app target, potentially some embedded frameworks, and perhaps an Extension target or two.

If you used a "global" setting for PROVISIONING_PROFILE, xcodebuild would try to sign all of the targets with that profile, and that may not be what you want. Embedded frameworks, for example, do not require code signing because the embedding target will wrap them and the whole package will be signed.

In order to associate the correct provisioning profile with the target(s) to be signed, some indirection is required. Fortunately, Xcode build settings can use the value of other build settings and nested expressions to define their value. Taking advantage of this, you can define a target's PROVISIONING_PROFILE setting in terms of a static prefix and the expanded value of some other settings.

But which other settings should those be? Well, if the goal is (as in the sample project) to sign an app target but avoid signing a framework, you can use WRAPPER_EXTENSION to distinguish the two. For an app target, WRAPPER_EXTENSION evaluates to "app" at build time. As in "MyCoolApp.app".

Combining this value with a static prefix can define the name of another variable that will ultimately have as its value the UUID of the desired provisioning profile. In practice, the definition looks like this:

PROVISIONING_PROFILE=$(PROVISIONING_PROFILE_$(WRAPPER_EXTENSION))

If you set the value above for your app target's PROVISIONING_PROFILE setting (in Xcode), you can then use the variable PROVISIONING_PROFILE_app to provide the UUID of the profile in your CI system. If PROVISIONING_PROFILE_app is not defined (e.g. outside of CI on your development machine), there will be no value set for PROVISIONING_PROFILE, and Automatic Signing will kick in and do its thing.

You can see this in action in the sample project by looking at the project file and the Fastfile. A successful build on TravisCI is also available as a reference.

If you need to get more specific, for example to use different provisioning profiles for multiple apps or extensions, you can just include another variable in the definition:

PROVISIONING_PROFILE=$(PROVISIONING_PROFILE_$(PRODUCT_NAME)_$(WRAPPER_EXTENSION))

This will give you the ability to specify the profile differently on a per-target basis, using something like PROVISIONING_PROFILE_MyCoolApp_app for the app, and PROVISIONING_PROFILE_MyCoolFramework_framework for a framework. You could even set the above value for PROVISIONING_PROFILE at the project level and rely on the default empty value for targets you don't want to sign.

Signing Off

This technique makes it possible for our team to use the new Automatic Signing features of Xcode 8 in development while still using manual signing on our CI machines. I'd love to see future versions of Xcode support both signing modes more gracefully, without resorting to such gymnastics.

If you found this post useful, let me know in the comments! I couldn't have achieved this so quickly without the fine example provided by Matt Hamilton. Samantha Marshall has a comprehensive write-up of the whole code signing system that's worth bookmarking as a reference, as well.