Xcode 4: Run tests from the command line (xcodebuild)?

Question

I've created a brand new iOS project in Xcode 4, and included unit tests. The default app has 2 targets, the main application and the unit test bundle. Using "Product > Test" (Command-U) builds the application, builds the unit test bundle, launches the iOS simulator and runs the tests. Now I'd like to be able to do the same thing from the command line. The command line tool (xcodebuild) doesn't have a "test" action, but it seems like I should be able to build the unit test bundle target directly, since it depends on the application itself. However, running:

xcodebuild -target TestAppTests -sdk iphonesimulator4.3 -configuration Debug build

gives the following message:

/Developer/Platforms/iPhoneSimulator.platform/Developer/Tools/Tools/RunPlatformUnitTests:95: warning: Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set).

That seems like a lie, since Test Host is set for my unit test bundle target when I run Command-U from the GUI. I've seen previous posts about the separation between logic tests and application tests, but it seems like Xcode 4 does away with that distinction. Any clue how I can run my tests from the command line?

Answer 1

Important Note

With Xcode 5.1 (perhaps earlier Xcode as well) test is a valid build action.

We were able to replace the entire hack below with a call to xcodebuild using the build action of test and with appropriate -destination options. man xcodebuild for more info.

The information below is left here for posterity

---

I tried hacking Apple's scripts to run unit tests as mentioned in

Running Xcode 4 unit tests from the command line

and

Xcode4: Running Application Tests From the Command Line in iOS

and numerous similar postings across the web.

However, I ran into a problem with those solutions. Some of our unit tests exercised the iOS Keychain and those calls, when running in the environment that comes from hacking Apple's scripts, failed with an error (errSecNotAvailable[-25291] for the morbidly curious). As a result, the tests always failed... an undesirable feature in a test.

I tried a number of solutions based on information I found elsewhere on the web. Some of those solutions involved trying to launch the iOS simulator's security services daemon, for example. After struggling with those, My best bet seemed to be to run in the iOS simulator with the full benefit of the simulator's environment.

What I did, then was get ahold of the iOS Simulator launching tool ios-sim. This command line tool uses private Apple frameworks to launch an iOS application from the command line. Of particular use to me, however, was the fact that it allows me to pass both Environment Variables and Command Line Arguments to the app that it is launching.

Though the Environment variables, I was able to get my Unit Testing bundle injected into my Application. Through the command line arguments, I can pass the "-SenTest All" needed to get the app to run the unit tests and quit.

I created a Scheme (which I called "CommandLineUnitTests") for my unit testing bundle and checked the "Run" action in the build section as described in the posts above.

Rather than hacking Apple's scripts, though, I replaced the script with one that launches the application using ios-sim and sets up the environment to inject my unit testing bundle into the application separately.

My script is written in Ruby which is more familiar to me than BASH scripting. Here's that script:

if ENV['SL_RUN_UNIT_TESTS'] then
    launcher_path = File.join(ENV['SRCROOT'], "Scripts", "ios-sim")
    test_bundle_path= File.join(ENV['BUILT_PRODUCTS_DIR'], "#{ENV['PRODUCT_NAME']}.#{ENV['WRAPPER_EXTENSION']}")

    environment = {
        'DYLD_INSERT_LIBRARIES' => "/../../Library/PrivateFrameworks/IDEBundleInjection.framework/IDEBundleInjection",
        'XCInjectBundle' => test_bundle_path,
        'XCInjectBundleInto' => ENV["TEST_HOST"]
    }

    environment_args = environment.collect { |key, value| "--setenv #{key}=\"#{value}\""}.join(" ")

    app_test_host = File.dirname(ENV["TEST_HOST"])
    system("#{launcher_path} launch \"#{app_test_host}\" #{environment_args} --args -SenTest All #{test_bundle_path}")
else
    puts "SL_RUN_UNIT_TESTS not set - Did not run unit tests!"
end

Running this from the command line looks like:

xcodebuild -sdk iphonesimulator -workspace iPhoneApp.xcworkspace/ -scheme "CommandLineUnitTests" clean build SL_RUN_UNIT_TESTS=YES

After looking for the SL_RUN_UNIT_TESTS environment variable, the script finds the "launcher" (the iOS-sim executable) within the project's source tree. It then constructs the path to my Unit Testing Bundle based on build settings that Xcode passes in environment variables.

Next, I create the set of runtime Environment Variables for my running application that inject the unit testing bundle. I set up those variables in the environment hash in the middle of the script then use some ruby grunge to join them into a series of command line arguments for the ios-sim application.

Near the bottom I grab the TEST_HOST from the environment as the app I want to launch and the system command actually executes ios-sim passing the application, the command arguments to set up the environment, and the arguments -SenTest All and the test bundle path to the running application.

The advantage of this scheme is that it runs the unit tests in the simulator environment much as I believe Xcode itself does. The disadvantage of the scheme is that it relies on an external tool to launch the application. That external tool uses private Apple frameworks, so it may be fragile with subsequent OS releases, but it works for the moment.

P.S. I used "I" a lot in this post for narrative reasons, but a lot of the credit goes to my partner in crime, Pawel, who worked through these problems with me.