04. May 2015
This article is Part 1 of a whole series called Xcode Server Hacks. New posts are always tweeted by me (@czechboy0) on Twitter.
As part of building a project called Buildasaur, I had a chance to explore the ins and outs of Xcode Server. Xcode Server is a combination of two of Apple’s apps, OS X Server and Xcode. Together, they can provide a continuous integration server for your repository. Today, I’ll talk about how Xcode Server works under the hood, which open source frameworks it uses internally and I’ll even show you how to connect to its hidden API. We have a lot to cover, so let’s get started!
Back in 2013, when Xcode 5 was released, Apple referred to the system as “Xcode Bots”, but technically, it’s called Xcode Server. Each job/plan on the server is called a “Bot” and each run of a job is called an “Integration”.
Buildasaur supports the newer version of Xcode Server, released in 2014 with Xcode 6, so everything described in this article relates to a setup with Xcode 6.x, OS X Server 4.x.
This article assumes you have such an environment setup, if you want to try some of the hacks yourself. It’s fine if you don’t, I’ll be writing an article on how to set up Xcode Server from scratch in the next couple of weeks.
So, let’s get started!
First we’ll look at where Xcode Server prints its logs, because that should be the go-to place when something breaks. All the logs can be found at
/Library/Developer/XcodeServer/Logs. You’ll find multiple log and
pid files there.
This is the build daemon’s log, so when a Bot is building your code, this is where all the output goes. It should match what you see when you just run
xcodebuildon your project from Terminal. This is a very useful log, because sometimes Xcode Server isn’t verbose enough about a build failure, at those times, head right for
This is the log of the main process which controls Xcode Server under the hood.
CouchDB’s log, more on the role of CouchDB in Xcode Server later.
API Server’s log, one of the most important ones. Every request, processing of it and response are logged here. This is extremely useful when trying to figure out how the API talks to Xcode or to your app. But more on
xcsdeviced‘s logs, the daemon which handles the communication with connected iOS devices. If you can’t figure out why your build isn’t being tested on your device, this is the place to go. You’ll find messages like
"The device is passcode protected."if you forgot to unlock your iPhone, etc.
Redis’s log, as with CouchDB, more on its role later.
Logs folder also contains several
.pid files, which just tell you the process ids of its running daemons. This can be useful if you need to force quit one of them.
You might have noticed that all the file names have a prefix of
xcs - I can only guess that this is an abbreviation of Xcode Server. This prefix is present in the majority of files and folders that Xcode Server keeps around.
Xcode Server runs under multiple users on your system (like
_xcsd) and it persists all files in the
/Library/Developer/XcodeServer folder (which is where the logs are as well). Even if you remove both Xcode and OS X Server from your system and then reinstall them, all preferences and data will get picked up, because this location remained untouched. Now, let’s look at what’s in there for us. Except for the
Logs folder, you’ll need to go into superuser mode by using
$ sudo -s in Terminal before you start digging into the folders below.
Contains the server’s certificates, used for signing various resources (not code, however). I’m not yet clear on the exact purpose of what’s in there.
ota.mobileconfig, which is a signed property list allowing OTA (over the air) installations of your app, when you archive it with Xcode Server. This is one of my favorite features of Xcode Server, because it lets non-technical members of your team install nightly builds in the morning without ever touching Xcode.
As the name suggests, this is a symlink to your currently used Xcode in OS X Server. The fact that a different Xcode can be plugged in easily makes Xcode Server very flexible when it comes to the ever-changing versions of Xcode.
Contains the persisted data from CouchDB, Xcode Server’s database and a file called
couch.uri, which contains the address and port of the running CouchDB instance.
Might be useful when you host your git repo on Xcode Server. However, I have never tried that, so if anyone has experience with hosting code on Xcode Server, let me know.
Now we’re getting to the good stuff. This folder contains everything about the integration of each Bot. This is described in detail later.
This folder contains the assets of the currently running integration (useful for observing the build log of a currently running integration) and a
Cachesfolder, which contains the latest assets for each Bot, so that if you start another integration on an existing Bot, it already has its
DerivedDataand source folders cached.
This folder contains Xcode Server’s keychains, which contain SSH keys and other sensitive information. If you sign your team into Xcode Server, this is where it downloads its certificates and keys.
Were described in detail at the beginning of this article.
Folder containing all provisioning profiles downloaded by the server from the developer portal. Also the place to put any provisioning profiles that you want the server to use during code signing (more on how to set this up in a future article).
Folder containing shared secrets used by Xcode Server to unlock the keychains in
Keychainsand to securely talk to the developer portal.
Integrations folders contain something I called “integration assets”. But what are those exactly?
Integrations contains cached and running integrations, so it needs to keep the files necessary to build your project. The nice thing about Xcode Server is that it keeps
Source files per Bot. So if one of your Bots has a broken ModuleCache (classic) in its
DerivedData, it won’t affect other Bots. The file structure of Buildasaur’s Bot’s folder in
Caches looks like this
. |-- DerivedData | |-- Build | |-- Logs | |-- ModuleCache | |-- TestResults | `-- info.plist `-- Source `-- Buildasaur
Source folder contains the checked out repository and
DerivedData we’re all too familiar with - this folder contains the intermediate files created by the compiler to speed up future compilations and also the final built products, together with logs.
In contrast to
Integrations, which contained still active integrations and caches,
IntegrationAssets only contains build results such as archives, test results and logs. This is exactly what you get when you hit “Download All Logs” in the Bot’s detail screen in Xcode’s Report navigator.
The structure of that folder looks like this:
6b3de48352a8126ce7e08ecf85093613-Builda\ Archiver/ |-- 1 | |-- Archive.xcarchive.zip | |-- Session-2015-05-04_00:24:01-MKjX3a.log | |-- build.log | |-- buildService.log | |-- sourceControl.log | `-- xcodebuild_result.bundle.zip `-- 2 |-- Archive.xcarchive.zip |-- Session-2015-05-04_13:37:55-YQAc8p.log |-- build.log |-- buildService.log |-- sourceControl.log `-- xcodebuild_result.bundle.zip
Each integration’s assets are in a folder matching its number and contains
The built Xcode archive, if you ticked the archive action as part of your Bot setup.
Containing the log of a test run. This one is very important when hunting down a test failure.
Has the complete build log, the output of running
xcodebuildon your project.
Contains information about the high level actions performed as part of your integration, such as building, testing and input parameters of those actions.
Will prove helpful if you are having problems with git authentication, such as when you have invalid SSH keys. It has the whole output of all git commands performed as part of the checkout step.
Contains results of the Bot run, mostly in binary plists, which, in my best guess, are used by Xcode to present you with the pretty printed, interactive result pane.
The unzipped structure looks like this:
xcodebuild_result.bundle |-- 1_Analyze | `-- build.xcactivitylog |-- 2_Test | |-- action.xcactivitylog | |-- action_TestSummaries.plist | `-- build.xcactivitylog |-- Info.plist `-- TestSummaries.plist
Now that you know where Xcode Server persists its data, you might be curious where its source code lives. Well, luckily, the engineers in Cupertino are indeed using the same open source tools as we do, like Node.js, Express, Redis and CouchDB, among others. And all of that lives inside of Xcode’s bundle, specifically in
This is a goldmine of the source code of all the important parts of Xcode Server, in addition to
, where the closed source binaries live, e.g.
xcssecurity and others.
This is a 2-level structure of the
. ├── CouchDB │ ├── bin │ ├── etc │ ├── lib │ ├── sbin │ ├── share │ └── var ├── Migration │ ├── ArchiveXCSv1ServiceData.rb │ ├── ExportXCSv1JSONData.rb │ └── MigrateXCSv1HostedRepositories.rb ├── Node │ └── bin ├── Redis │ └── bin ├── httpd_xcs.conf ├── newsyslog.d │ └── com.apple.xcs.conf ├── xcscouch │ ├── _design │ └── config.ini ├── xcsd │ ├── app.js │ ├── bootstrap_xcs.js │ ├── classes │ ├── com.apple.xcsd.plist │ ├── config │ ├── constants.js │ ├── diagnostics.md │ ├── error_handler.js │ ├── node_modules │ ├── package.json │ ├── public │ ├── routes │ ├── socket.js │ ├── util │ └── views ├── xcsredis │ └── redis.conf └── xcswebui ├── base ├── bigscreen └── webui
and I’d like to point out just a couple of interesting parts.
Contain the binaries of both databases, so that Xcode Server doesn’t have to pull any dependencies when launched.
Contains scripts to migrate v1 to v2 (currently Xcode Server is on v2 and might be upgraded again to v3 this year).
Contains a binary of an old version of Node.js, specifically v0.10.26 in Xcode 6.3. But hey, who knew that Xcode contained Node.js?
Is an Apache proxy config file for the Server.
Contains the list of daemons and where to find their
Contain configuration files of Xcode Server’s CouchDB and Redis instances, on which ports to run etc.
- and finally
Contains the full source to
xcsd‘s Node.js Express app. Most interesting is the
routes/routes.jsfile, which describes all the API endpoints that
xcsdresponds to, and that’s how you can write an app talking to those APIs, like I did with Buildasaur.
I have discovered a couple (possibly not all) of important subsystems of Xcode Server, which I’ll talk about now. If you know about more of them, let me know and I’ll gladly amend the article and learn more in the process.
All of Xcode Server’s daemons’ users are members of the group
_xcs, which is useful when managing permissions. If you don’t care whether
xcsd (the API) or
xcsbuildd (the worker/builder) need to access your resource, just allow the whole group
The API server is called
xcsd and runs under the user
_xcsd. The actual process is a Node.js Express app and it handles talking to all the Xcodes on your local network, in addition to any browser in which you go to your
[servers-address]/xcode, to see Xcode Server’s status page. It also handles talking to its caching database, Redis and its persistence and map-reduce database, CouchDB.
- Process Id:
This is the worker process, the one that actually takes the scheduled Bots from the database and runs
xcodebuild on your source code. It first needs to checkout the code, based on your Bot’s blueprint. It also runs pre and post-build scripts, which can be used for running CocoaPods, version increment or any other scripts, like uploading your archives to Dropbox and emailing of build results to your team members.
- Process Id: (seems to run on demand)
Its man page says it’s a “utility for managing Xcode Server”. Basically it’s the manager of the whole Xcode Server system. It starts and stops the databases and all the Xcode Server daemons. When you turn on Xcode Server in OS X Server, that giant green button translates to
xcrun xcscontrol --start.
- Process Id:
Handles connected devices and makes sure they are ready for testing. Presumably takes care of waking them up for running tests and other responsibilities. I haven’t found many more details about it.
- Process Id:
Xcode Server uses CouchDB as its main persistence and a fast lookup of items through map-reduce queries. You can actually browse the data in your web browser by going to
and grep for “port =”, usually 10355
Xcode Server seems to use Redis as an in-memory cache only (its
redis.conf doesn’t specify any save rules, so we know the data has to be inserted at runtime). Xcode engineers probably have good reasons to use Redis in addition to CouchDB, my guess would be its simplicity of the single level key-value storage, performance and great features for inter-process communication (like blocking pop on list for worker queues etc). Again, you can browse the data by running
$ redis-cli -p 10356 in Terminal.
and grep for “port “, usually 10356
When you hack too much and Xcode Server goes crazy, you need a way to reset all of its state and start over. When I was originally developing Buildasaur, I would reset Xcode Server several times a day. There is one command which stops all the running daemons of Xcode Server and deletes all the contents of
/Library/Developer/XcodeServer. Be careful, all your bots and integration data will be deleted as well. During debugging it’s an indispensable tool, but be careful in production to not accidentally delete all your Bots and Integration assets.
Ok, since you now read the small print and I warned you, the command is
sudo xcrun xcscontrol --reset.
I don’t have many, just one big one. Xcode Server cannot run multiple integrations at the same time. From how much I’ve seen in its internals, there is nothing inherently preventing mutliple instances of
xcsbuildd running at the same time on two different devices. Especially since it uses Redis, a database perfect for managing a worker queue with multiple instances. I’d love to get my hands on the source code of
xcsbuildd and find out what the problem is or if there is a way to make it work. That’s my one and only feature request!
Why would I want to know all this?
I don’t know, to be honest. I like finding out how things work under the hood. However, this might still prove useful to you if you use Xcode Server in your team and something goes south. Or if you’re like me and want to build more tools on top of its “hidden” API. However, I really appreciate that Apple has done the heavy lifting for us. Xcode Server provides an amazing, always-compatible-with-Xcode way to have a very simple CI server, for free.
That is why I wanted to build on top of Xcode Server and I’m always looking for a way to put a bit of energy into making our workflow even more efficient, without buying into a completely new 3rd party solution. Those usually take ages to support the latest Xcode and Xcode’s features. We have a simple, easy to use CI system handed to us by Apple, so you should have a damn good reason not to use it.
So whether you’re thinking of using Xcode Server’s now-unhidden APIs or you just wanted to know a bit more about the tool you depend on, I hope you learned something new. Now, let’s perform an experiment - if you read all the way till the end, tweet at me with the word “gelato” used in a sentence, so that I know you got through the whole thing :)
Ping me on Twitter with your success/failure stories of using Xcode Server and check out my Xcode Server Tutorial series!
I hope you found this useful or interesting. For criticism, praise and future articles, I’m @czechboy0 on Twitter.