Xcode Server Hacks: 1. Under the Hood of Xcode Server

This article is Part 1 of a whole series called Xcode Server Hacks. New posts are always tweeted by me (@czechboy0) on Twitter.


As part of building a project called Buildasaur, I had a chance to explore the ins and outs of Xcode Server. Xcode Server is a combination of two of Apple’s apps, OS X Server and Xcode. Together, they can provide a continuous integration server for your repository. Today, I’ll talk about how Xcode Server works under the hood, which open source frameworks it uses internally and I’ll even show you how to connect to its hidden API. We have a lot to cover, so let’s get started!

Back in 2013, when Xcode 5 was released, Apple referred to the system as “Xcode Bots”, but technically, it’s called Xcode Server. Each job/plan on the server is called a “Bot” and each run of a job is called an “Integration”.

Buildasaur supports the newer version of Xcode Server, released in 2014 with Xcode 6, so everything described in this article relates to a setup with Xcode 6.x, OS X Server 4.x.

This article assumes you have such an environment setup, if you want to try some of the hacks yourself. It’s fine if you don’t, I’ll be writing an article on how to set up Xcode Server from scratch in the next couple of weeks.

So, let’s get started!

Logs

First we’ll look at where Xcode Server prints its logs, because that should be the go-to place when something breaks. All the logs can be found at /Library/Developer/XcodeServer/Logs. You’ll find multiple log and pid files there.

  • xcsbuildd.log
    This is the build daemon’s log, so when a Bot is building your code, this is where all the output goes. It should match what you see when you just run xcodebuild on your project from Terminal. This is a very useful log, because sometimes Xcode Server isn’t verbose enough about a build failure, at those times, head right for xcsbuildd.log!
  • xcscontrol.log
    This is the log of the main process which controls Xcode Server under the hood.
  • xcscouch.log
    CouchDB’s log, more on the role of CouchDB in Xcode Server later.
  • xcsd.log
    API Server’s log, one of the most important ones. Every request, processing of it and response are logged here. This is extremely useful when trying to figure out how the API talks to Xcode or to your app. But more on xcsd later.
  • xcsdeviced.log
    Contains xcsdeviced‘s logs, the daemon which handles the communication with connected iOS devices. If you can’t figure out why your build isn’t being tested on your device, this is the place to go. You’ll find messages like "The device is passcode protected." if you forgot to unlock your iPhone, etc.
  • xcsredis.log
    Redis’s log, as with CouchDB, more on its role later.

The Logs folder also contains several .pid files, which just tell you the process ids of its running daemons. This can be useful if you need to force quit one of them.

You might have noticed that all the file names have a prefix of xcs - I can only guess that this is an abbreviation of Xcode Server. This prefix is present in the majority of files and folders that Xcode Server keeps around.

Persistence

Xcode Server runs under multiple users on your system (like _xcsbuildd, _xcsd) and it persists all files in the /Library/Developer/XcodeServer folder (which is where the logs are as well). Even if you remove both Xcode and OS X Server from your system and then reinstall them, all preferences and data will get picked up, because this location remained untouched. Now, let’s look at what’s in there for us. Except for the Logs folder, you’ll need to go into superuser mode by using $ sudo -s in Terminal before you start digging into the folders below.

  • Certificates
    Contains the server’s certificates, used for signing various resources (not code, however). I’m not yet clear on the exact purpose of what’s in there.
  • ConfigurationProfiles
    Contains just ota.mobileconfig, which is a signed property list allowing OTA (over the air) installations of your app, when you archive it with Xcode Server. This is one of my favorite features of Xcode Server, because it lets non-technical members of your team install nightly builds in the morning without ever touching Xcode.
  • CurrentXcodeSymlink
    As the name suggests, this is a symlink to your currently used Xcode in OS X Server. The fact that a different Xcode can be plugged in easily makes Xcode Server very flexible when it comes to the ever-changing versions of Xcode.
  • Database
    Contains the persisted data from CouchDB, Xcode Server’s database and a file called couch.uri, which contains the address and port of the running CouchDB instance.
  • HostedRepositories, HostedRepositoriesHTTPCGIScriptSymlink
    Might be useful when you host your git repo on Xcode Server. However, I have never tried that, so if anyone has experience with hosting code on Xcode Server, let me know.
  • IntegrationAssets
    Now we’re getting to the good stuff. This folder contains everything about the integration of each Bot. This is described in detail later.
  • Integrations
    This folder contains the assets of the currently running integration (useful for observing the build log of a currently running integration) and a Caches folder, which contains the latest assets for each Bot, so that if you start another integration on an existing Bot, it already has its DerivedData and source folders cached.
  • Keychains
    This folder contains Xcode Server’s keychains, which contain SSH keys and other sensitive information. If you sign your team into Xcode Server, this is where it downloads its certificates and keys.
  • Logs
    Were described in detail at the beginning of this article.
  • ProvisioningProfiles
    Folder containing all provisioning profiles downloaded by the server from the developer portal. Also the place to put any provisioning profiles that you want the server to use during code signing (more on how to set this up in a future article).
  • SharedSecrets
    Folder containing shared secrets used by Xcode Server to unlock the keychains in Keychains and to securely talk to the developer portal.

Integration Assets

Both IntegrationAssets and Integrations folders contain something I called “integration assets”. But what are those exactly?

Integrations

Integrations contains cached and running integrations, so it needs to keep the files necessary to build your project. The nice thing about Xcode Server is that it keeps DerivedData and Source files per Bot. So if one of your Bots has a broken ModuleCache (classic) in its DerivedData, it won’t affect other Bots. The file structure of Buildasaur’s Bot’s folder in Caches looks like this

.
|-- DerivedData
|   |-- Build
|   |-- Logs
|   |-- ModuleCache
|   |-- TestResults
|   `-- info.plist
`-- Source
    `-- Buildasaur

The Source folder contains the checked out repository and DerivedData we’re all too familiar with - this folder contains the intermediate files created by the compiler to speed up future compilations and also the final built products, together with logs.

IntegrationAssets

In contrast to Integrations, which contained still active integrations and caches, IntegrationAssets only contains build results such as archives, test results and logs. This is exactly what you get when you hit “Download All Logs” in the Bot’s detail screen in Xcode’s Report navigator.

The structure of that folder looks like this:

6b3de48352a8126ce7e08ecf85093613-Builda\ Archiver/
|-- 1
|   |-- Archive.xcarchive.zip
|   |-- Session-2015-05-04_00:24:01-MKjX3a.log
|   |-- build.log
|   |-- buildService.log
|   |-- sourceControl.log
|   `-- xcodebuild_result.bundle.zip
`-- 2
    |-- Archive.xcarchive.zip
    |-- Session-2015-05-04_13:37:55-YQAc8p.log
    |-- build.log
    |-- buildService.log
    |-- sourceControl.log
    `-- xcodebuild_result.bundle.zip

Each integration’s assets are in a folder matching its number and contains

  • Archive.xcarchive.zip
    The built Xcode archive, if you ticked the archive action as part of your Bot setup.
  • Session-[timestamp]-[random string].log
    Containing the log of a test run. This one is very important when hunting down a test failure.
  • build.log
    Has the complete build log, the output of running xcodebuild on your project.
  • buildService.log
    Contains information about the high level actions performed as part of your integration, such as building, testing and input parameters of those actions.
  • sourceControl.log
    Will prove helpful if you are having problems with git authentication, such as when you have invalid SSH keys. It has the whole output of all git commands performed as part of the checkout step.
  • xcodebuild_result.bundle.zip
    Contains results of the Bot run, mostly in binary plists, which, in my best guess, are used by Xcode to present you with the pretty printed, interactive result pane.

The unzipped structure looks like this:

xcodebuild_result.bundle
|-- 1_Analyze
|   `-- build.xcactivitylog
|-- 2_Test
|   |-- action.xcactivitylog
|   |-- action_TestSummaries.plist
|   `-- build.xcactivitylog
|-- Info.plist
`-- TestSummaries.plist

Source Code

Now that you know where Xcode Server persists its data, you might be curious where its source code lives. Well, luckily, the engineers in Cupertino are indeed using the same open source tools as we do, like Node.js, Express, Redis and CouchDB, among others. And all of that lives inside of Xcode’s bundle, specifically in

/Applications/Xcode.app/Contents/Developer/usr/share/xcs

This is a goldmine of the source code of all the important parts of Xcode Server, in addition to

/Applications/Xcode.app/Contents/Developer/usr/bin

, where the closed source binaries live, e.g. xcscontrol, xcsbuildd, xcsbridge, xcssecurity and others.

This is a 2-level structure of the xcs directory:

.
├── CouchDB
│   ├── bin
│   ├── etc
│   ├── lib
│   ├── sbin
│   ├── share
│   └── var
├── Migration
│   ├── ArchiveXCSv1ServiceData.rb
│   ├── ExportXCSv1JSONData.rb
│   └── MigrateXCSv1HostedRepositories.rb
├── Node
│   └── bin
├── Redis
│   └── bin
├── httpd_xcs.conf
├── newsyslog.d
│   └── com.apple.xcs.conf
├── xcscouch
│   ├── _design
│   └── config.ini
├── xcsd
│   ├── app.js
│   ├── bootstrap_xcs.js
│   ├── classes
│   ├── com.apple.xcsd.plist
│   ├── config
│   ├── constants.js
│   ├── diagnostics.md
│   ├── error_handler.js
│   ├── node_modules
│   ├── package.json
│   ├── public
│   ├── routes
│   ├── socket.js
│   ├── util
│   └── views
├── xcsredis
│   └── redis.conf
└── xcswebui
    ├── base
    ├── bigscreen
    └── webui

and I’d like to point out just a couple of interesting parts.

  • CouchDB and Redis
    Contain the binaries of both databases, so that Xcode Server doesn’t have to pull any dependencies when launched.
  • Migration
    Contains scripts to migrate v1 to v2 (currently Xcode Server is on v2 and might be upgraded again to v3 this year).
  • Node
    Contains a binary of an old version of Node.js, specifically v0.10.26 in Xcode 6.3. But hey, who knew that Xcode contained Node.js?
  • httpd_xcs.conf
    Is an Apache proxy config file for the Server.
  • newsyslog.d/com.apple.xcs.conf
    Contains the list of daemons and where to find their .pid files.
  • xcscouch and xcsredis
    Contain configuration files of Xcode Server’s CouchDB and Redis instances, on which ports to run etc.
  • xcswebui
    Has the source of all the Javascript and HTML files used in Xcode Server’s browser UI, so feel free to take a look and modify to your liking.

  • and finally xcsd
    Contains the full source to xcsd‘s Node.js Express app. Most interesting is the routes/routes.js file, which describes all the API endpoints that xcsd responds to, and that’s how you can write an app talking to those APIs, like I did with Buildasaur.

Components

I have discovered a couple (possibly not all) of important subsystems of Xcode Server, which I’ll talk about now. If you know about more of them, let me know and I’ll gladly amend the article and learn more in the process.

All of Xcode Server’s daemons’ users are members of the group _xcs, which is useful when managing permissions. If you don’t care whether xcsd (the API) or xcsbuildd (the worker/builder) need to access your resource, just allow the whole group _xcs.

xcsd

The API server is called xcsd and runs under the user _xcsd. The actual process is a Node.js Express app and it handles talking to all the Xcodes on your local network, in addition to any browser in which you go to your [servers-address]/xcode, to see Xcode Server’s status page. It also handles talking to its caching database, Redis and its persistence and map-reduce database, CouchDB.

  • Log:
    /Library/Developer/XcodeServer/Logs/xcsd.log
    
  • Process Id:
    /Library/Developer/XcodeServer/Logs/xcsd.pid
    
  • Source:
    /Applications/Xcode.app/Contents/Developer/usr/share/xcs/xcsd
    

xcsbuildd

This is the worker process, the one that actually takes the scheduled Bots from the database and runs xcodebuild on your source code. It first needs to checkout the code, based on your Bot’s blueprint. It also runs pre and post-build scripts, which can be used for running CocoaPods, version increment or any other scripts, like uploading your archives to Dropbox and emailing of build results to your team members.

  • Log:
    /Library/Developer/XcodeServer/Logs/xcsbuildd.log
    
  • Process Id: (seems to run on demand)
  • Binary:
    /Applications/Xcode.app/Contents/Developer/usr/bin/xcsbuildd
    

xcscontrol

Its man page says it’s a “utility for managing Xcode Server”. Basically it’s the manager of the whole Xcode Server system. It starts and stops the databases and all the Xcode Server daemons. When you turn on Xcode Server in OS X Server, that giant green button translates to xcrun xcscontrol --start.

  • Log:
    /Library/Developer/XcodeServer/Logs/xcscontrol.log
    
  • Process Id:
    /Library/Developer/XcodeServer/Logs/xcscontrol.pid
    
  • Binary:
    /Applications/Xcode.app/Contents/Developer/usr/bin/xcscontrol
    

xcsdeviced

Handles connected devices and makes sure they are ready for testing. Presumably takes care of waking them up for running tests and other responsibilities. I haven’t found many more details about it.

  • Log:
    /Library/Developer/XcodeServer/Logs/xcsdeviced.log
    
  • Process Id:
    /Library/Developer/XcodeServer/Logs/xcsdeviced.pid
    
  • Binary:
    /Applications/Xcode.app/Contents/Developer/usr/bin/xcsdeviced
    

CouchDB

Xcode Server uses CouchDB as its main persistence and a fast lookup of items through map-reduce queries. You can actually browse the data in your web browser by going to http://localhost:10355/_utils/.

  • Log:
    /Library/Developer/XcodeServer/Logs/xcscouch.log
    
  • Port:
    /Applications/Xcode.app/Contents/Developer/usr/share/xcs/xcscouch/config.ini
    
    and grep for “port =”, usually 10355
  • Binary:
    /Applications/Xcode.app/Contents/Developer/usr/share/xcs/CouchDB/bin/couchdb
    

Redis

Xcode Server seems to use Redis as an in-memory cache only (its redis.conf doesn’t specify any save rules, so we know the data has to be inserted at runtime). Xcode engineers probably have good reasons to use Redis in addition to CouchDB, my guess would be its simplicity of the single level key-value storage, performance and great features for inter-process communication (like blocking pop on list for worker queues etc). Again, you can browse the data by running $ redis-cli -p 10356 in Terminal.

  • Log:
    /Library/Developer/XcodeServer/Logs/xcsredis.log
    
  • Port:
    /Applications/Xcode.app/Contents/Developer/usr/share/xcs/xcsredis/redis.conf
    
    and grep for “port “, usually 10356
  • Binary:
    /Applications/Xcode.app/Contents/Developer/usr/share/xcs/Redis/bin/redis-server
    

Troubleshooting

When you hack too much and Xcode Server goes crazy, you need a way to reset all of its state and start over. When I was originally developing Buildasaur, I would reset Xcode Server several times a day. There is one command which stops all the running daemons of Xcode Server and deletes all the contents of /Library/Developer/XcodeServer. Be careful, all your bots and integration data will be deleted as well. During debugging it’s an indispensable tool, but be careful in production to not accidentally delete all your Bots and Integration assets.

Ok, since you now read the small print and I warned you, the command is sudo xcrun xcscontrol --reset.

Complaints

I don’t have many, just one big one. Xcode Server cannot run multiple integrations at the same time. From how much I’ve seen in its internals, there is nothing inherently preventing mutliple instances of xcsbuildd running at the same time on two different devices. Especially since it uses Redis, a database perfect for managing a worker queue with multiple instances. I’d love to get my hands on the source code of xcsbuildd and find out what the problem is or if there is a way to make it work. That’s my one and only feature request!

Why would I want to know all this?

I don’t know, to be honest. I like finding out how things work under the hood. However, this might still prove useful to you if you use Xcode Server in your team and something goes south. Or if you’re like me and want to build more tools on top of its “hidden” API. However, I really appreciate that Apple has done the heavy lifting for us. Xcode Server provides an amazing, always-compatible-with-Xcode way to have a very simple CI server, for free.

That is why I wanted to build on top of Xcode Server and I’m always looking for a way to put a bit of energy into making our workflow even more efficient, without buying into a completely new 3rd party solution. Those usually take ages to support the latest Xcode and Xcode’s features. We have a simple, easy to use CI system handed to us by Apple, so you should have a damn good reason not to use it.

So whether you’re thinking of using Xcode Server’s now-unhidden APIs or you just wanted to know a bit more about the tool you depend on, I hope you learned something new. Now, let’s perform an experiment - if you read all the way till the end, tweet at me with the word “gelato” used in a sentence, so that I know you got through the whole thing :)

Ping me on Twitter with your success/failure stories of using Xcode Server and check out my Xcode Server Tutorial series!


I hope you found this useful or interesting. For criticism, praise and future articles, I’m @czechboy0 on Twitter.