Every project I’m working with starts with a setup of continuous integration pipeline. I’m a big fan of cloud services, that’s why I was always using Travis. A few of my clients questioned this choice recently, mostly because of the price. So I decided to make a brief analysis of the market.
I configured Rultor, an open source project, in every CI service I managed to find. All of them are free for open source projects. All of them are hosted and do not require any server installation Here they are, in order of my personal preference (first four are the best and highly recommended):
| Minimum price | Linux1 | Windows2 | MacOS3 | Pull requests4 | Log compress5 | Docker6 | Build | |
| Shippable☺ | free (!) | | ||||||
| Travis | $69/mo |  | ||||||
| AppVeyor | $39/mo | ? | --- | | ||||
| Wercker | $350/mo | | ||||||
| SemaphoreApp | $29/mo | |||||||
| Snap-CI☺ | $30/mo | ? | ||||||
| Codeship☺ | $49/mo | ? | ||||||
| CircleCI☺ | $19/mo | |||||||
| SonoLabs | $15/mo | ? | ||||||
| DeployBot | $15/mo | ? | ? | |||||
| Vexor | ¢90/hr | ? | ? | |||||
| GreenHouseCI | $49/mo | ? | ? | |||||
If you know any other good continuous integration services, email me, I’ll review and add them to this list. BTW, here is a “full” list of continuous integration software and services.
By the way, a few platforms from this list contacted me and asked to review them again. Some even offered me money to put them higher in the list (kidding). Anyway, it’s up to you to decide whether it’s a good sign or not (I think it’s good, they care about their PR). I marked them in the list with ☺ emoji.
Best Four
Shippable was easy to configure since it understands .travis.yml out of the box. The user interface is easy to navigate since it doesn’t have “settings” page at all (or I didn’t find it). Everything is configured via shippable.yml file in the repository. The service looks stable and robust, no complains so far. What is especially cool about them is that they allow you to build in a Docker container.
Travis is the best platform I’ve seen so far. Mostly because it is the most popular. Perfectly integrates with GitHub and has proper documentation. One important downside is the price of $129 per month. “With this money you can get a dedicated EC2 instance and install Jenkins there”—some of my clients say. I strongly disagree, since Jenkins will require a 24x7 administration, which costs way more than $129, but it’s always difficult to explain.
Wercker is a European product from Amsterdam, which is still in beta and that’s why free for all projects. The platform looks very promising. It is still free for private repositories and is backed up by investments. They also have an interesting concept of build “boxes,” which can be pre-configured similar to Docker containers. It works rather stable for the last few months, no complains so far.
AppVeyor is the only one that runs Windows builds. Even though I’m working mostly with Java and Ruby, which are expected to be platform independent, they very often appear to be exactly the opposite. When your build succeeds on Linux, there is almost no guarantee it will pass on Windows or Mac. I’m planning to use AppVeyor in every project, in combination with some other CI service. Here is how I integrate Maven builds with AppVeyor.
Others
SemaphoreApp is easy to configure and work with. It makes an impression of a light-weight system, which I generally appreciate. As a downside, they don’t have any Maven pre-installed have an old version of Maven, but this was solved easily with a short custom script that downloads and unpacks the latest Maven. Another downside is that they are not configurable through a file (like .travis.yml)—you should do everything through a UI. They also support caching between builds.
Snap-CI is a product of ThoughtWorks, an author of Go, an open source continuous integration server. It looks a bit more complicated than others, giving you an ability to define “stages” and combine them into pipelines. I’m not sure yet how these mechanisms may help in small and medium size projects we’re mostly working with, but they look “cool.” There is also a very unfortunate limitation of 2Gb RAM per build—some of my Java projects fail because of that. Besides that, they don’t give full access to the build server, for example we can’t modify anything in /etc—it is a show-stopper for us.
Codeship works fine, but their web UI looks a bit out-dated. Besides that, they promise to work with pull requests, but I didn’t manage to configure them. They simply don’t notify our pull requests in GitHub, even though they build them. Maybe I’ll find a way, so far it’s not clear.
CircleCI I still don’t know why my build fails there. Really difficult to configure and understand what’s going on. Trying to figure it out…
SolanoLabs looks rather immature and difficult to configure. They don’t even support automatic GitHub hook configuration when new repository is added. However, their sales spams me rather aggressively.
Hosted-CI is for iOS/OSX only. They don’t give anything for free, even for open source projects. I didn’t have a chance to test them yet.
CloudBees is basically a hosted Jenkins. I don’t really like Jenkins, that’s why can’t recommend this platform.
DeployBot doesn’t even allow me to login via GitHub, huh? They seem to be more “deployment” oriented, not just continuous integration.
Vexor looks nice and offers a rather unique billing model—they charge per build, not per month. I would definitely recommend to give it a try. I couldn’t make it work though…
GreenHouseCI is a CI platform for mobile apps (iOS, Android, etc.) Seems to be interesting, I just don’t have a full scale mobile app to test it against.
gitlab-ci: will review soon
coverity.com: will review it soon and add to the list.
buddy.works: will review soon.
AWS Code Build and AWS Code Deploy: will review soon.
Ship.io is dead (as of 20-Sep-2016).
ZeroCI.com is dead (as of 28-Aug-2016).
Drone.io is not hosted any more, but is open source (as of 23-Jan-2017).
Hosted-ci doesn’t look alive (as of 16-Apr-2017).
Magnum-CI.com doesn’t look alive (as of 30-May-2017).
BTW, if you don’t like the idea of keeping continuous integration in cloud, consider these on-premise software packages (in order or preference): Jenkins, TeamCity, Go, Strider, BuildBot.
Keep in mind that no matter how good and expensive your continuous integration service is, it won’t help you unless you make your master branch read-only.
1 This means that the platform can build your repo in Linux environment. Almost all of them do that by default, unless you configure them otherwise.
2 Some of them can build on Windows platform.
3 MacOS support means that an Objective-C/Swift product can be built there.
4 I mean GitHub pull request support here. Some of them can be integrated with GitHub and will build pull requests before they are merged. Build status will be visible in GitHub. A pretty convenient feature.
5 Logs compression is a critical feature, at least for me. Most of my logs are from Maven and without col -b they look too long and unreadable.
6 Looks like Docker containers must be supported by all of them, but unfortunately it’s not the case. Ideally, all builds should run in containers.
