Hierarchy in DevOps?

If you are moving to DevOps in your organisation, then bringing together different people from different silos and then expecting them (under their own steam) to magically cooperate and agree on everything is a little unrealistic.

If you are dealing with historical teams and employees - which I suspect most of you are - then breaking down silos is likely to be a significant challenge.

Aligning incentives makes things a lot easier. If both your devs and ops guys are paid a bonus depending on how much "change" they do in a year is a simple way of getting them to agree on whether they should automate releases for example.  That's a little too simplistic but hopefully you get what I mean.

This may well come at the cost of something else though.  The first thing that springs to mind in the last example is quality.

Let's put everything straight into live!  We get paid for it!

Ok so that isn't going to work.

Sometimes, at least to begin with, you need to bring people together and ultimately empower someone to make decisions when one can't be reached through consensus.   I've seen a real danger in DevOps of death by committee.  The more silos you bring together then potentially the greater the possibility of them not making a decision.

You may have heard about this idea of DevOps rockstars.  Guys (and gals) that get DevOps inside and out.  People that understand all sides of the coin (development, DBA, infrastructure etc).

They try and influence through education and persuasion and point people in the right direction.  They help the silos reach the right decisions themselves rather than a top-down "do what I say".

All said and done sometimes this open inclusiveness doesn't always work and someone needs to just stand up and be counted.

So what's my point? 

If you are trying to move towards DevOps then hire a rockstar.  Someone that has done it before.  Someone with a very mature sense of what DevOps is and preferably someone that has done most of the jobs he/she is trying to influence.

Moreover, hire someone who's first instinct is to try and bring people around to their way of thinking.  And lastly, someone who isn't ultimately afraid to put their privates on the line and make a goddam decision.

DevOps isn't automation

Perhaps I'm getting older and more and more like my father.  More intolerant or, at least, less likely to hide discontent or my  view of the world.

Perhaps this is why it makes me an angry and frustrated bear when I constantly hear DevOps being used as a word for automation.

LinkedIn for example is full of DevOps "experts" and forums talking about "doing DevOps". 

Deployment Automation != DevOps

DevOps is a culture of shared goals and incentives.

Deployment Automation may help you to get there if your shared goals are to get things into production faster and in a more consistent state.  However there are lots of other things to consider which also affect this.  If your Release Manager only let's 1 deployment into prod a month then it doesn'tatter how much faster you've done your non production systems. As I've said before DevOps is much more than 'dev' and 'ops'.

I need a little lie down and to get out of the heat.

Sorry for shouting 

DevOps change challenges

One of the challenges of DevOps in the setting of a traditional (i.e. siloed, waterfall etc) Enterprise is where to begin.

I'm a keen advocate that DevOps isn't just Dev and Ops (see http://enterprisedevops.blogspot.co.uk/2014/07/devbiztestreleasethingyops.html).

With that in mind, potentially you are talking about some fairly radical changes across a wide number of people of various skills, experience and personality types.  I guess it really depends on what your organisation is structured like, the skills of your team and many other factors.  For us however, it's meant some fundamental change.

That fundamental change has been sponsored by some early and small wins by implementing things like Continuous Integration and labelling it "DevOps".  Ok so it's not DevOps but if using that term starts to get visibility and movement within your organisation then I don't think that's a bad thing.

I always we believed we needed a banner for the modernisation of the department to fly behind.  If that banner reads "DevOps" then so be it.

That early small success was really vital to us. 

When we started out we had a look of push back and negativity....

Why are you telling me how to do my job?
You are using the wrong tools
I don't see the benefit
I know more about this than you do
What you are trying to do is easy
We don't have the right people
This is a hipster idea for the likes of Facebook
There are regulatory reasons this won't work


Basically everything under the sun.

That's stressful when you are trying to do the right thing and move the incumbent forward.

I don't think we helped ourselves too much to begin with.  There was too much of my team trying to pick up and change other teams technologies and processes.  Not enough leading the other team to water and helping them implement something themselves.  Lesson learnt.

Don't do DevOps to teams.  Help them do the doing.


More about what changes we've made a little later.

Failing early

I'm sure you are familiar with the idea of failing early.

It's best to fail as close to the point of originating a change because it's much easier and cheaper to fix.

When people talk about 'failing early' it's often as an approach to testing code - most noticeably through TDD/unit testing.

However, you can really apply the idea to most things. 

I was thinking today about deployments. 

Wouldn't it be good if you knew that your deployment was going to be work before you even started it? How many times have you had issues because there wasn't enough disk space or you couldn't reach all the servers you needed to?

What we are going to do is a series of pre-deployment checks. I want to fail before a deployment not mid way through.

I think that's a pretty good practise. Automatically test everything before you start with a set of environment readiness scripts. Is the environment in a state that you expect? Does it have the version and configuration that you expect?

If it doesn't fail the deployment and investigate. It will save you all kinds of headaches.

It's also a good idea to kickoff a load of automated tests after a deployment. I'd rather know if there was an issue BEFORE the business/test team get their little mitts on it...

Failing is good. You learn, you fix.

If DevOps isn't a job title then it can't be a team? Can it?

I blogged earlier how DevOps is cultural idea and not a job title.  Definitely not.  Unimaginable.

So how is it I run a "DevOps" team?   Surely that's a collective of "DevOps engineers"?!

Ok so I appreciate that this contradictory in the extreme.

Or is it......?

My team "Platform Services" supports the idea of the change platform at my company.  The change platform consists of tools such as Jenkins, Puppet, uDeploy, Teamcity, TFS and a few others.

Someone ultimately needs to "own" the tools making sure they are maintained, creating best practices and establishing a roadmap for the future.

My team goes beyond this and helps different change functions adapt to the change platform and are evangelists for things like DevOps and Continuous Delivery.

With DevOps being concerned with breaking down silos its contradictory to have a team that bottleneck of knowledge and skills.  To this end, Platform Services is the silo-buster - we help others pick up the tools & processes and implement them.  

For example, we don't do deployments ourselves but we do sell the benefits to other teams and help them to adopt the tools and processes.

I think this is a good model and helps spread the success of DevOps around the IT function.

It helps that consultancy from my guys is effectively free to other parts of the IT team (thanks to an enlightened CIO & CTO).  

Why wouldn't you take the teams advice?  Why wouldn't you follow the patterns of success established elsewhere?

Go create a DevOps team; just don't let them become a silo.

DevBizTestReleaseThingyOps

The traditional definition of DevOps relates to changing the culture between the development and operation teams so they can reach shared end goals.

Thoroughly bought in that. Great.

But what about everyone else? Dev's and Ops aren't the only people involved in making change.

In my eyes, the term "DevOps" isn't not really just about Dev vs. Ops.
 
It’s about all the functions that come together to achieve meaningful change.
 
You might as well call it DevBizTestReleaseThingyOps.
 
To that, I'm starting to evolve my definition and understanding of this "DevOps" thing.
 
The business and techies (inc testers, BA's etc) should have shared goals and have shared incentives (not necessarily exclusively). Where possible they should be sat around the same desk. The team should rise and fall collectively (there is no "us and them"). The team should manage change from cradle to grave (requirements, build, test, deploy, support).
 
Ok so there is a lot of where "possible" in that paragraph but hopefully you get my point...

Why DevOps isn't a job

Buzzwords.  Gotta hate them.  I love the idea of this thing called "DevOps" though but I come out in a cold sweat when  I hear this word being thrown about with abandon.

My official position is that really there is no such thing as DevOps – at least not as a technical discipline.  I also happen to lead the DevOps initiative at a large financial services enterprise.

So not much contradiction there then...!

The ever increasing popularity of the Cloud has really driven a lot of ideas about new how to create and deploy ‘change, how to collaborate and how to manage mega sized infrastructures.  The massive scale of companies like Facebook, Amazon and Netflix have caused companies to look for new ways to scale their teams, technology and business processes. These types of companies often evangelise this term “DevOps” as part of the solution to their woes.

What they are really talking about though is culture.  Not a collection of funky hipster tools and technologies such as Puppet, Chef and Vagrant.

So, here are my five reasons why you shouldn't ever be able to get a "DevOps" job.

1.    DevOps means producing good quality software and systems that are easy to maintain and quick to change is the responsibility of everyone

That’s not new – just how software should be adopted in the first place.

Its not a developer responsibility.  Or a DBA.  Or a BA.  Or the Operations guys.  Its everyone involved in producing and managing change.

It doesn't just apply to funky new web startups out of Silicon Valley.  It's as applicable to traditional enterprises (including financial services where I work) as it is to those cool Etsy-esque Cloud companies.

2. You can’t get rid of organizational barriers by creating a new organizational silo and calling it DevOps

As companies grow they tend to put people in technology focused "teams".  A database team, or a network team, a development team, a test team, a release management team and so on.

By doing this, these teams often become silos, incentivised in different ways – not focusing on the same end goals.  That’s why you can’t become a DevOps engineer.  Silo’s aren’t fixed by creating a new silo and calling it “DevOps”.

This is why the scale of Cloud has really driven the idea behind DevOps as a “new” discipline.  The idea is to get everyone to collaborate effectively to achieve common goals.  If you want to deploy code to 100,000 servers there are some pretty massive engineering challenges.  Do we update 10% first?  How do we ensure that the change hasn’t broken anything?  Has the login rate dropped?  How do we quickly roll it back if something goes wrong?

3.  One of the misconceptions about DevOps is that it involves automated releases into production.

DevOps is about bringing people together and having the culture in place to achieve shared goals.  Once you have the culture in place, processes and tools become a means rather than an end.  How do developers and infrastructure engineers both ensure that they can test their scripts?  Can they use common techniques such as version control and unit testing?  These are all important questions, but the inherent technical skill sets are not new per say.

DevOps is about breaking down whatever barriers are necessary in order to achieve your goals.  Happy with your pace of change but want to increase quality?  Break down the silos between Devs and Test to get them aligned in a collaborative environment.  Have them work within the same team and incentivised the same way. By sharing tools and techniques, you can avoid developers falling into the trap of throwing code over the wall and moving on.

4. You can’t create a new discipline simply by cobbling up a cool portmanteau

With that in mind, it's not really just about Dev vs. Ops. It’s about the functions that come together to achieve meaningful change.  You might as well call it DevBizTestReleaseThingyOps (more to come on this a later date!)

5. Business culture cannot be changed by creating a new position category nobody really understands in the first place

DevOps is whatever you want to make it.  It’s a culture that extends far beyond technology.