How Three Organisations Matured their DevOps Practices

Insights

Q&A with Jeremy Ford

Everyone is doing DevOps. But there’s huge variation in the way companies are going about it, their levels of maturity and what their goals are. Cloud Solution Architect Jeremy Ford has consulted on a large number of DevOps optimisation projects, and has seen both the good and the bad. We sat down and had a chat with him about how organisations of very different size and scale are tackling DevOps challenges and different goals for their DevOps practice. 

Q: You’ve worked across a number of DevOps projects. Can you describe some common scenarios you’ve encountered? 

A: Sure, so I’ve got three examples that are all pretty unique. One of them was a huge international corporation. In this scenario, we came in to develop a new product for them, meaning I got to set up pipelines and everything from scratch, to best practice. A real greenfield situation which was great, though obviously we had to work with their security architects to ensure we had the appropriate checks in place. But important to note that the DevOps improvements we made there only influenced that particular project; the rest of the organisation still operates to a different standard. 

The second use case / scenario was a company (I’ll call them Organisation 2) that had relatively good DevOps maturity. They had pipelines for everything and they were very good at the ‘build’ stage but their operations side wasn’t where it could be. They had dramatically increased the number of releases they were doing and the output of their IT team, but hadn’t set up some of the things they needed to run and manage at that scale. They had a mixture of AWS SAM (Serverless Application Model) as the application framework, or sometimes they would use custom build scripts to put stuff together, so it was a bit ad hoc. 

Organisation 3 were already very mature; they were entirely serverless, they had the bitbucket pipelines set up and they had good infrastructure. We were originally engaged just to build a couple of extra APIs and integrations for a new client they were bringing on, and as part of that I did a review and pointed out some improvements we could make with the security stance and testing and things like that. The technology evolves so quickly so a review every 3-6 months is generally a good idea to ensure optimal setup. 

Q: What were the key struggles/pain points these organisations were facing? 

Organisation 1

A: Well because this organisation was greenfield it was about getting everything started in the right way. They did, however (and still do) face issues getting alignment to new ways of working. Being a huge organisation, many IT staff had been able to get away with some pretty heavy specialisation and a certain way of doing things for a long time. Whereas in Organisation 2 we’d been able to optimise things like the branching method to align with the way they deploy and test, in Organisation 1 we weren’t able to change as much. Introducing DevOps sort of required the staff to get their hands dirty with some new areas which naturally caused a little bit of friction. I think this company would like to be a bit more mature than they are today, but they’re moving in the right direction. 

Organisation 2

This organisation had issues with consistency and testing. As I said before they were scaling up their output quite considerably, but they didn’t have mature error handling, they had very few tests and standards implemented for code quality, they had very few alarms at all, and they hadn’t yet implemented proper naming conventions for those alarms or lambda functions. The impacts of that were that their DevOps team only found out that something had gone wrong in one of their own processes when a customer or someone from an external-facing team told them. Very reactive rather than proactive.

Organisation 3

This organisation was facing mainly capacity issues. They’d had one key employee who had set up and managed most of the DevOps practice, but there’s only so much he could do on his own. They had infrastructure that did the job, but we were able to introduce some better patterns and more secure ways of access, as well as introducing code quality checks into their pipelines. Prior to us coming in they didn’t really have any unit tests, which made it difficult to build any new integrations with confidence. So we set up that whole testing framework, security policies and more comprehensive libraries, with the goal that they have more confidence in their code going forward.

Q: It sounds like some of these organisations were already strong in various aspects of DevOps. What was their incentive to improve it and bring in an external partner? 

A: One of the main reasons is if they’ve ramped up their customer focus, they’re looking at more customer-facing applications and services. Obviously the bar is higher for anything externally-facing, for 24/7 operational services. You want better security, you want rigorous automated testing and high quality code going out into production. There’s also more imperative for faster, more frequent deployments; at one of my clients we’re now deploying multiple times a day, we can push a jira ticket into production the same day we receive it depending on the other tickets. And that’s critical to minimise the time bugs and issues are out there, means we can get new features to customers sooner, better customer experience, all those things. 

Also scale – if you’re growing, things like a naming convention across alarms and Lambda functions (like we implemented in Organisation 2) can make a huge difference. If you’ve only got one project you can write alarms for it and be done with it. But when you’ve got hundreds of repositories all doing the same thing, you need to optimise a bit and make it into a library. 

And then sometimes we get called in just for capacity, like in Organisation 3. Especially in the market lately, it can be tricky to get staff with the right skills, and if you’re able to set and forget, automate things…it’s a great fill in solution. 

Q: You mention a couple of key roadblocks to a mature DevOps practice above. Can you list any others you’ve encountered? 

A: One of the core things I’ve been focussing on with both Organisations 1 & 2 is that they don’t really have a strong operations team as part of the project team. So if you look at Organisation 1, pretty much everyone on that project is more of a developer, and few have much of that broader AWS skillset we’re using on the operations side. Some of them can do DevOps, but they’re more interested in the dev side of things. So we’re tackling that by aiming for a set-and-forget wherever possible; automating what we can and using standards like CDK (Cloud Development Kit) that are more developer friendly and abstract a lot of the infrastructure away from the devs. 

Another roadblock or challenge might be like Organisation 2, who are using an offshore DevOps team where the focus is more on delivery than quality. We knew it was important to introduce code quality checks and automated processes for security and to ensure what goes out is meeting the organisation’s standard – it can be tricky from a communication point of view across time zones, automation just makes sure it’s all standardised. We also introduced a different branching method that better aligns with how they deploy to the environments and how they test, sort of optimised for how they work to help keep the integrity of the organisation’s processes. 

Q: Where are these organisations at now, and what benefits are they seeing from improved DevOps maturity? 

A: As I said earlier, I think Organisation 1 would like to be a bit further down the road than they are, but it’s always a long change management process in complex enterprise organisations. At the moment they’re making a company-wide switch to Google Cloud Platform (GCP), so much of their DevOps work will be about migrating onto that platform. We’ll be migrating everything across to bitbucket pipelines and Terraform rather than the current backend which is Amazon CodePipeline and CloudFormation. 

Organisation 2 are now using their pipelines to the fullest – I looked recently and they’ve had almost 30 different repositories changed in the last 24 hours. Now the pipelines are all standardised, the infrastructure is standardised, there are shared libraries for alarms and stuff like that, we’ve refined their branching model so there’s consistent branching naming standards across everything they want to do. Alarms are in place for everything. Now that they’ve got CDK for all their repositories a lot of the developers are actually doing that themselves as well, so it’s not just ‘ah I need the DevOps team to do something,’ it’s actually ‘I can just go and do this myself.’ More efficiency, better security, faster development and everything’s just much more automated and organised. 

Organisation 3 is benefiting from security improvements and testing changes, better quality code and checks that are automated into the pipelines just from some fairly minor tweaks. With such a small team going forward that’s going to make a big difference to what they’re going to be able to deliver.

Author Details

Jeremy Ford
Jeremy has over 16 years’ experience in software engineering. He specialises in Java and Web technologies and has extensive experience across all phases of the SDLC. Jeremy has led the successful delivery of multiple solutions for our clients utilising agile principles and processes. Jeremy is known for his exceptional technical knowledge, as well as his outstanding ability to apply this to achieve optimal solutions for clients; he is a certified AWS solutions architect and is highly experienced utilising the diverse AWS ecosystem. Jeremy is also a member of Intelligent Pathways’ internal consulting group, which identifies and recommends suitable technologies, coding practices and standards across the company.

You might be interested in these related insights