04. May 2015
This article is Part 1 of a whole series called Xcode Server Hacks. New posts are always tweeted by me (@czechboy0.dev) on Bluesky.
As part of building a project called Buildasaur, I had a chance to explore the ins and outs of Xcode Server. Xcode Server is a combination of two of Apple’s apps, OS X Server and Xcode. Together, they can provide a continuous integration server for your repository. Today, I’ll talk about how Xcode Server works under the hood, which open source frameworks it uses internally and I’ll even show you how to connect to its hidden API. We have a lot to cover, so let’s get started!
Back in 2013, when Xcode 5 was released, Apple referred to the system as “Xcode Bots”, but technically, it’s called Xcode Server. Each job/plan on the server is called a “Bot” and each run of a job is called an “Integration”.
Buildasaur supports the newer version of Xcode Server, released in 2014 with Xcode 6, so everything described in this article relates to a setup with Xcode 6.x, OS X Server 4.x.
This article assumes you have such an environment setup, if you want to try some of the hacks yourself. It’s fine if you don’t, I’ll be writing an article on how to set up Xcode Server from scratch in the next couple of weeks.
So, let’s get started!
Logs
First we’ll look at where Xcode Server prints its logs, because that should be the go-to place when something breaks. All the logs can be found at /Library/Developer/XcodeServer/Logs
. You’ll find multiple log and pid
files there.
xcsbuildd.log
This is the build daemon’s log, so when a Bot is building your code, this is where all the output goes. It should match what you see when you just runxcodebuild
on your project from Terminal. This is a very useful log, because sometimes Xcode Server isn’t verbose enough about a build failure, at those times, head right forxcsbuildd.log
!xcscontrol.log
This is the log of the main process which controls Xcode Server under the hood.xcscouch.log
CouchDB’s log, more on the role of CouchDB in Xcode Server later.xcsd.log
API Server’s log, one of the most important ones. Every request, processing of it and response are logged here. This is extremely useful when trying to figure out how the API talks to Xcode or to your app. But more onxcsd
later.xcsdeviced.log
Containsxcsdeviced
‘s logs, the daemon which handles the communication with connected iOS devices. If you can’t figure out why your build isn’t being tested on your device, this is the place to go. You’ll find messages like"The device is passcode protected."
if you forgot to unlock your iPhone, etc.xcsredis.log
Redis’s log, as with CouchDB, more on its role later.
The Logs
folder also contains several .pid
files, which just tell you the process ids of its running daemons. This can be useful if you need to force quit one of them.
You might have noticed that all the file names have a prefix of xcs
- I can only guess that this is an abbreviation of Xcode Server. This prefix is present in the majority of files and folders that Xcode Server keeps around.
Persistence
Xcode Server runs under multiple users on your system (like _xcsbuildd
, _xcsd
) and it persists all files in the /Library/Developer/XcodeServer
folder (which is where the logs are as well). Even if you remove both Xcode and OS X Server from your system and then reinstall them, all preferences and data will get picked up, because this location remained untouched. Now, let’s look at what’s in there for us. Except for the Logs
folder, you’ll need to go into superuser mode by using $ sudo -s
in Terminal before you start digging into the folders below.
Certificates
Contains the server’s certificates, used for signing various resources (not code, however). I’m not yet clear on the exact purpose of what’s in there.ConfigurationProfiles
Contains justota.mobileconfig
, which is a signed property list allowing OTA (over the air) installations of your app, when you archive it with Xcode Server. This is one of my favorite features of Xcode Server, because it lets non-technical members of your team install nightly builds in the morning without ever touching Xcode.CurrentXcodeSymlink
As the name suggests, this is a symlink to your currently used Xcode in OS X Server. The fact that a different Xcode can be plugged in easily makes Xcode Server very flexible when it comes to the ever-changing versions of Xcode.Database
Contains the persisted data from CouchDB, Xcode Server’s database and a file calledcouch.uri
, which contains the address and port of the running CouchDB instance.HostedRepositories
,HostedRepositoriesHTTPCGIScriptSymlink
Might be useful when you host your git repo on Xcode Server. However, I have never tried that, so if anyone has experience with hosting code on Xcode Server, let me know.IntegrationAssets
Now we’re getting to the good stuff. This folder contains everything about the integration of each Bot. This is described in detail later.Integrations
This folder contains the assets of the currently running integration (useful for observing the build log of a currently running integration) and aCaches
folder, which contains the latest assets for each Bot, so that if you start another integration on an existing Bot, it already has itsDerivedData
and source folders cached.Keychains
This folder contains Xcode Server’s keychains, which contain SSH keys and other sensitive information. If you sign your team into Xcode Server, this is where it downloads its certificates and keys.Logs
Were described in detail at the beginning of this article.ProvisioningProfiles
Folder containing all provisioning profiles downloaded by the server from the developer portal. Also the place to put any provisioning profiles that you want the server to use during code signing (more on how to set this up in a future article).SharedSecrets
Folder containing shared secrets used by Xcode Server to unlock the keychains inKeychains
and to securely talk to the developer portal.
Integration Assets
Both IntegrationAssets
and Integrations
folders contain something I called “integration assets”. But what are those exactly?
Integrations
Integrations
contains cached and running integrations, so it needs to keep the files necessary to build your project. The nice thing about Xcode Server is that it keeps DerivedData
and Source
files per Bot. So if one of your Bots has a broken ModuleCache (classic) in its DerivedData
, it won’t affect other Bots. The file structure of Buildasaur’s Bot’s folder in Caches
looks like this
.
|-- DerivedData
| |-- Build
| |-- Logs
| |-- ModuleCache
| |-- TestResults
| `-- info.plist
`-- Source
`-- Buildasaur
The Source
folder contains the checked out repository and DerivedData
we’re all too familiar with - this folder contains the intermediate files created by the compiler to speed up future compilations and also the final built products, together with logs.
IntegrationAssets
In contrast to Integrations
, which contained still active integrations and caches, IntegrationAssets
only contains build results such as archives, test results and logs. This is exactly what you get when you hit “Download All Logs” in the Bot’s detail screen in Xcode’s Report navigator.
The structure of that folder looks like this:
6b3de48352a8126ce7e08ecf85093613-Builda\ Archiver/
|-- 1
| |-- Archive.xcarchive.zip
| |-- Session-2015-05-04_00:24:01-MKjX3a.log
| |-- build.log
| |-- buildService.log
| |-- sourceControl.log
| `-- xcodebuild_result.bundle.zip
`-- 2
|-- Archive.xcarchive.zip
|-- Session-2015-05-04_13:37:55-YQAc8p.log
|-- build.log
|-- buildService.log
|-- sourceControl.log
`-- xcodebuild_result.bundle.zip
Each integration’s assets are in a folder matching its number and contains
Archive.xcarchive.zip
The built Xcode archive, if you ticked the archive action as part of your Bot setup.Session-[timestamp]-[random string].log
Containing the log of a test run. This one is very important when hunting down a test failure.build.log
Has the complete build log, the output of runningxcodebuild
on your project.buildService.log
Contains information about the high level actions performed as part of your integration, such as building, testing and input parameters of those actions.sourceControl.log
Will prove helpful if you are having problems with git authentication, such as when you have invalid SSH keys. It has the whole output of all git commands performed as part of the checkout step.xcodebuild_result.bundle.zip
Contains results of the Bot run, mostly in binary plists, which, in my best guess, are used by Xcode to present you with the pretty printed, interactive result pane.
The unzipped structure looks like this:
xcodebuild_result.bundle
|-- 1_Analyze
| `-- build.xcactivitylog
|-- 2_Test
| |-- action.xcactivitylog
| |-- action_TestSummaries.plist
| `-- build.xcactivitylog
|-- Info.plist
`-- TestSummaries.plist
Source Code
Now that you know where Xcode Server persists its data, you might be curious where its source code lives. Well, luckily, the engineers in Cupertino are indeed using the same open source tools as we do, like Node.js, Express, Redis and CouchDB, among others. And all of that lives inside of Xcode’s bundle, specifically in
/Applications/Xcode.app/Contents/Developer/usr/share/xcs
This is a goldmine of the source code of all the important parts of Xcode Server, in addition to
/Applications/Xcode.app/Contents/Developer/usr/bin
, where the closed source binaries live, e.g. xcscontrol
, xcsbuildd
, xcsbridge
, xcssecurity
and others.
This is a 2-level structure of the xcs
directory:
.
├── CouchDB
│ ├── bin
│ ├── etc
│ ├── lib
│ ├── sbin
│ ├── share
│ └── var
├── Migration
│ ├── ArchiveXCSv1ServiceData.rb
│ ├── ExportXCSv1JSONData.rb
│ └── MigrateXCSv1HostedRepositories.rb
├── Node
│ └── bin
├── Redis
│ └── bin
├── httpd_xcs.conf
├── newsyslog.d
│ └── com.apple.xcs.conf
├── xcscouch
│ ├── _design
│ └── config.ini
├── xcsd
│ ├── app.js
│ ├── bootstrap_xcs.js
│ ├── classes
│ ├── com.apple.xcsd.plist
│ ├── config
│ ├── constants.js
│ ├── diagnostics.md
│ ├── error_handler.js
│ ├── node_modules
│ ├── package.json
│ ├── public
│ ├── routes
│ ├── socket.js
│ ├── util
│ └── views
├── xcsredis
│ └── redis.conf
└── xcswebui
├── base
├── bigscreen
└── webui
and I’d like to point out just a couple of interesting parts.
CouchDB
andRedis
Contain the binaries of both databases, so that Xcode Server doesn’t have to pull any dependencies when launched.Migration
Contains scripts to migrate v1 to v2 (currently Xcode Server is on v2 and might be upgraded again to v3 this year).Node
Contains a binary of an old version of Node.js, specifically v0.10.26 in Xcode 6.3. But hey, who knew that Xcode contained Node.js?httpd_xcs.conf
Is an Apache proxy config file for the Server.newsyslog.d/com.apple.xcs.conf
Contains the list of daemons and where to find their.pid
files.xcscouch
andxcsredis
Contain configuration files of Xcode Server’s CouchDB and Redis instances, on which ports to run etc.xcswebui
Has the source of all the Javascript and HTML files used in Xcode Server’s browser UI, so feel free to take a look and modify to your liking.
- and finally
xcsd
Contains the full source toxcsd
‘s Node.js Express app. Most interesting is theroutes/routes.js
file, which describes all the API endpoints thatxcsd
responds to, and that’s how you can write an app talking to those APIs, like I did with Buildasaur.
Components
I have discovered a couple (possibly not all) of important subsystems of Xcode Server, which I’ll talk about now. If you know about more of them, let me know and I’ll gladly amend the article and learn more in the process.
All of Xcode Server’s daemons’ users are members of the group _xcs
, which is useful when managing permissions. If you don’t care whether xcsd
(the API) or xcsbuildd
(the worker/builder) need to access your resource, just allow the whole group _xcs
.
xcsd
The API server is called xcsd
and runs under the user _xcsd
. The actual process is a Node.js Express app and it handles talking to all the Xcodes on your local network, in addition to any browser in which you go to your [servers-address]/xcode
, to see Xcode Server’s status page. It also handles talking to its caching database, Redis and its persistence and map-reduce database, CouchDB.
- Log:
/Library/Developer/XcodeServer/Logs/xcsd.log
- Process Id:
/Library/Developer/XcodeServer/Logs/xcsd.pid
- Source:
/Applications/Xcode.app/Contents/Developer/usr/share/xcs/xcsd
xcsbuildd
This is the worker process, the one that actually takes the scheduled Bots from the database and runs xcodebuild
on your source code. It first needs to checkout the code, based on your Bot’s blueprint. It also runs pre and post-build scripts, which can be used for running CocoaPods, version increment or any other scripts, like uploading your archives to Dropbox and emailing of build results to your team members.
- Log:
/Library/Developer/XcodeServer/Logs/xcsbuildd.log
- Process Id: (seems to run on demand)
- Binary:
/Applications/Xcode.app/Contents/Developer/usr/bin/xcsbuildd
xcscontrol
Its man page says it’s a “utility for managing Xcode Server”. Basically it’s the manager of the whole Xcode Server system. It starts and stops the databases and all the Xcode Server daemons. When you turn on Xcode Server in OS X Server, that giant green button translates to xcrun xcscontrol --start
.
- Log:
/Library/Developer/XcodeServer/Logs/xcscontrol.log
- Process Id:
/Library/Developer/XcodeServer/Logs/xcscontrol.pid
- Binary:
/Applications/Xcode.app/Contents/Developer/usr/bin/xcscontrol
xcsdeviced
Handles connected devices and makes sure they are ready for testing. Presumably takes care of waking them up for running tests and other responsibilities. I haven’t found many more details about it.
- Log:
/Library/Developer/XcodeServer/Logs/xcsdeviced.log
- Process Id:
/Library/Developer/XcodeServer/Logs/xcsdeviced.pid
- Binary:
/Applications/Xcode.app/Contents/Developer/usr/bin/xcsdeviced
CouchDB
Xcode Server uses CouchDB as its main persistence and a fast lookup of items through map-reduce queries. You can actually browse the data in your web browser by going to http://localhost:10355/_utils/
.
- Log:
/Library/Developer/XcodeServer/Logs/xcscouch.log
- Port:
and grep for “port =”, usually 10355/Applications/Xcode.app/Contents/Developer/usr/share/xcs/xcscouch/config.ini
- Binary:
/Applications/Xcode.app/Contents/Developer/usr/share/xcs/CouchDB/bin/couchdb
Redis
Xcode Server seems to use Redis as an in-memory cache only (its redis.conf
doesn’t specify any save rules, so we know the data has to be inserted at runtime). Xcode engineers probably have good reasons to use Redis in addition to CouchDB, my guess would be its simplicity of the single level key-value storage, performance and great features for inter-process communication (like blocking pop on list for worker queues etc). Again, you can browse the data by running $ redis-cli -p 10356
in Terminal.
- Log:
/Library/Developer/XcodeServer/Logs/xcsredis.log
- Port:
and grep for “port “, usually 10356/Applications/Xcode.app/Contents/Developer/usr/share/xcs/xcsredis/redis.conf
- Binary:
/Applications/Xcode.app/Contents/Developer/usr/share/xcs/Redis/bin/redis-server
Troubleshooting
When you hack too much and Xcode Server goes crazy, you need a way to reset all of its state and start over. When I was originally developing Buildasaur, I would reset Xcode Server several times a day. There is one command which stops all the running daemons of Xcode Server and deletes all the contents of /Library/Developer/XcodeServer
. Be careful, all your bots and integration data will be deleted as well. During debugging it’s an indispensable tool, but be careful in production to not accidentally delete all your Bots and Integration assets.
Ok, since you now read the small print and I warned you, the command is
sudo xcrun xcscontrol --reset
.
Complaints
I don’t have many, just one big one. Xcode Server cannot run multiple integrations at the same time. From how much I’ve seen in its internals, there is nothing inherently preventing mutliple instances of xcsbuildd
running at the same time on two different devices. Especially since it uses Redis, a database perfect for managing a worker queue with multiple instances. I’d love to get my hands on the source code of xcsbuildd
and find out what the problem is or if there is a way to make it work. That’s my one and only feature request!
Why would I want to know all this?
I don’t know, to be honest. I like finding out how things work under the hood. However, this might still prove useful to you if you use Xcode Server in your team and something goes south. Or if you’re like me and want to build more tools on top of its “hidden” API. However, I really appreciate that Apple has done the heavy lifting for us. Xcode Server provides an amazing, always-compatible-with-Xcode way to have a very simple CI server, for free.
That is why I wanted to build on top of Xcode Server and I’m always looking for a way to put a bit of energy into making our workflow even more efficient, without buying into a completely new 3rd party solution. Those usually take ages to support the latest Xcode and Xcode’s features. We have a simple, easy to use CI system handed to us by Apple, so you should have a damn good reason not to use it.
So whether you’re thinking of using Xcode Server’s now-unhidden APIs or you just wanted to know a bit more about the tool you depend on, I hope you learned something new. Now, let’s perform an experiment - if you read all the way till the end, tweet at me with the word “gelato” used in a sentence, so that I know you got through the whole thing :)
Ping me on Bluesky with your success/failure stories of using Xcode Server and check out my Xcode Server Tutorial series!
I hope you found this useful or interesting. For criticism, praise and future articles, I’m @czechboy0.dev on Bluesky.