24 January 2019
There are 2 systems in any company that are critical: the payroll system, and the CI/CD system. Why? You may ask…
If the payroll system doesn’t work, people will leave the company and the company (may) face legal problems; the CI/CD system is the gateway to production. If it is down and there is a bug in production, it will affect your business; loss of revenue, loss of customers, loss of money, just to name a few.
Usually, I find these problems regarding the CI/CD tooling:
To tackle these problems, I usually suggest the use of a SaaS solution where they shift the responsibility of maintaining and securing the tool to the vendor; also, I recommend to select a tool which supports containers where the software is built. It can address several of the issues, such as providing isolation and decreasing the lead time to provide new capabilities. Also, it allows teams to create their own containers, shifting the responsibilities of the platform team to the development teams. Using containers as computing unit, it allows for granular capabilities (comparing to the typical full flagged machine), having a cleaner environment based on specific needs.
As an added benefit, the platform team is no longer a constraint, since the development teams do not need to request new capabilities for the build/deploy agents. The platform team is responsible for a service provided to the organisation, and the service needs to be stable and secure. It is a paradigm and mindset shift for the typical operations team, which can become a platform team, offering outstanding services to the developers.
The pivotal change is shifting the responsibilities of the platform team to the development teams. Well, this calls for a great life lesson from Ben Parker:
“With great power, comes great responsibilities!
Allowing the teams to build their own containers, who guarantees the security of the containers? Although it ticks some boxes, new challenges arose…
The platform team, who offers the service, is in the perfect spot to implement (dogfooding) a pipeline where the containers are tested and scanned for vulnerabilities. When everything passes, the container image is stored in the container registry, and it is continuously monitored. As CVE’s can be disclosed at any moment these images are constantly monitored. Using this approach, the platform team is in control of the CI/CD environment, and it is flexible enough for the development teams to deliver value quickly.
In the end, it’s 0’s and 1’s. Also known as Infrastructure as Code, and like any piece of code, it should be built, tested, stored in a central artifact store and deployed.
Putting all together, the pipeline can be described by the previous image. The code is in a git repository and a build agent uses an image from the container registry to kick the pipeline. Once the pipeline starts, it builds the code (another container image in this case), testing and scanning it. Finally, if all is sound and safe, it pushes the new image to the container registry. All of this using the typical Trunk Based Development, with Pull Requests to enforce the 4-eyes principle (traceability for audit purposes).
A possible implementation can be founded on this open source project. It uses free tools for open source projects, but there are other alternatives.
The implementation is simple and uses the following tools:
The decisions behind the tooling were:
Taking a closer look at the implementation you can see it in action:
The implementation empowers the development teams to create their own containers to be used in the build environments, without interfering with other teams. It also enables the platform team to be in control without constraint the development teams, and dictate the “holy rules” of “the platform”. However, provides the correct level of security controls, since everything is automated, PR’s are enforced, and it is possible to know exactly what and when it was used. It makes the auditors lives easier, and creates a safer environment!
If you have alternative implementations or ideas, feel free to shout! This implementation uses some tooling, and perhaps you have better alternatives. I will love to discuss it!