There are already thousands of controversies on the Internet about what DevOps is. We decided to approach differently: not to impose on you the point of view “understand this word this way”, but to look back into the past and trace the history of its origin. What led to DevOps? What people were the first to use this word, and what did they mean by it? What has changed during this time, and what has remained unchanged? And what’s next?

And having dealt with all this, in the end you can find that now you are answering yourself more clearly to the question “what is DevOps”.

Background: how not drown in a waterfall?

Before talking directly about DevOps, let's remember what preceded it.

As you know, the main idea of ​​DevOps is the effective collaboration of development ("Dev") and operations ("Ops"). Of course, effective cooperation between different specialists is always good, but why is it given so much attention in this particular case? Because here it helps the business to achieve many important goals for it: to bring solutions to the market faster, to recover more quickly in case of failures, and so on.

And people pursued these goals long before the words "development" and "operations" came together. Let's turn to history.

We can say that developers in the modern sense (they sat at the terminal and wrote the code) appeared after 1957 when they invented the first high-level programming language FORTRAN. Before that, there were no managerial approaches to development as such – writing programs for those very computers the size of a house was a piece product. But then everything began to change.

By the late 1970s and 1980s, developers could regularly sit at workstations, write code, compile it, and test it on the same computer. Deployment consisted of compiling and copying to dozens of floppy disks. Companies have begun to open up programmers' jobs to write "source code," a term that has only recently entered the lexicon. The market for commercial software and mass development began to grow.

The software became more complex, the systems developed, a crisis broke out in the industry: the deadlines were violated, and as a result, the budgets were exceeded, the software came out of poor quality, and the projects were unmanageable. Specialization of skills and distribution of roles as needed.

In 1970, the world learned about the waterfall (or waterfall) development model. It was proposed by the American Winston Royce, director of the Lockheed Software Technology Center. The described approach resembled a conveyor belt – all work was divided into stages (analysis, design, coding, testing, release), and the next stage could only be moved after the previous one was completed. No late changes were allowed.

It was a breakthrough – at last, there was some kind of system that allows you to control the development. The problem was that this approach was not suitable for complex projects that required iterative development (as Royce himself said).

As Robert Martin wrote in Clean Agile: Back to Basics (Peter, 2020): “Big work isn't done by big teams. In fact, a lot of work is done by a lot of small teams, which in turn do a lot of small tasks. In the 1950s and 60s, programmers understood this instinctively. But in the 1970s, this was simply forgotten. "

The Waterfall Model: Expectation and Reality

Until 1990, the developers worked mainly on the waterfall model, trying to modify it in some way. But even it didn’t save us from problems – deadlines were still thwarted, budgets were inadequately estimated, and the end result was not always predictable. In fact, until the end of the project, the customer did not know what he would receive in the end. By the end of the 80s, the so-called iterative incremental development took shape.

And by the mid-90s, the community, through pain and tears, finally realized that the principles of development needed to be reloaded.

In 2001, the Agile Manifesto was compiled in the town of Snowbird. You've probably heard about the manifesto itself, but have you paid attention to how close the 12 Agile Development Principles are to DevOps? “Deliver frequently”, “continuous delivery” – it seems that if you replace the title with “12 principles of good DevOps” and post on Habr, many will not notice any catch.

Later, the authors of the manifesto developed his ideas. Among the authors of the Agile Manifesto, in addition to the same Robert Martin and other famous people like Martin Fowler, was Alistair Cockburn – and in 2004 he described the methodology for developing Crystal Clear software. And then the ideas were picked up and developed by other people: for example, in 2006, the system administrator Marcel Wegermann wrote an essay in German on the use of the principles of software development Crystal Clear, Scrum, and Agile in the field of system administration. You can watch his short 2008 talk based on this work at the Chaos Communication Congress.

Marcel's message did not become very popular, but it shows that there is a demand for "Agile in operation": it was not only in development that they wanted to improve everything. In the same 2008, system administrator and Agile practitioner, Patrick Debois spoke at the Agile Conference in Toronto with a report “Agile infrastructure and operations”, and played a much larger role. But more on that later.

The road to the clouds

And what, besides Agile, contributed to the emergence of DevOps? The changing IT landscape: the emergence of new technologies and the industry's response to them.

For example, in the 90s, the Internet and cloud technologies began to develop rapidly. This changed a lot – a new way of development and delivery became necessary. In 1995, MySQL was also introduced to help build dynamic websites and applications.

Virtualization of software and hardware environments originated in 1964, with the beginning of the development of the IBM CP-40 OS, but it became massively in demand later. Virtualization introduced an additional layer of abstraction between executable code and system software, separating development and operations.

The advancement of virtualization, the cloud, and the Internet has led to the emergence of managing IT infrastructure as code and has allowed us to do things differently by testing applications in production environments early in the development cycle.

With all this, "fast software delivery" has become an urgent need. Now the business demanded that software be created quickly, but at the same time with high quality and with minimal costs.

To do this, it was not enough just to write the code, it was necessary to maintain it efficiently. The degree of automation was growing, but the exploitation was clearly not keeping up with the development. And in the end, although Agile originated from the development side, DevOps was primarily spawned by people who were clearly aware of the problems with operations.

Debois and others

One of these people was Patrick Debois. He helped the Belgian ministry with data center migration and worked in conjunction with several Agile teams. Debois had to coordinate activities between the application development teams and the operational teams (server, database network).

Patrick was faced with a lack of understanding and coherence between the teams. At the same time, the development team worked on Agile and showed high productivity. Then Debois decided to conduct an experiment and introduce Scrum into the work of the operations team.

At the Agile 2008 conference in Toronto, he gave a talk “Agile Operations and Infrastructure: How Infra-gile are You?”, Where he talked about his experience of changing operational work. He talked about applying Agile principles to infrastructure, not writing code. But in the Agile community, the report did not receive a wide response.

Much of his content focuses on three events: data center migrations, disaster recovery improvements, and application server upgrades. The most successful attempt was a data center migration, but the team failed to address a key SLA issue. And the failure lay in management and cultural interactions (both between Agile teams and traditional waterfall teams, and between Operations, Infrastructure, and Development).

Patrick Debois talk at DevOops 2020

And the movement owes its further development to a curious incident. Programmer Andrew Clay Schafer, who was interested in IT problems, suggested holding a discussion meeting on Agile Infrastructure at the same conference. However, the topic did not attract the attention of the audience so much that he did not even go to his event himself, considering it useless to anyone. But Patrick Debois came. And for him, this “fan meeting where no one came”, on the contrary, became a signal that since someone invented it, it means that someone still needs it! He contacted Andrew and proceeded to discuss Agile Infrastructure with him directly.

In addition to Debois, DevOps is associated with the name of John Allspaw, who was then at Flickr. The company started implementing new processes and sharing its experience with the community. Allow was the lead operations engineer tasked with handling new project scaling and data migration. Paul Hammond joined Flickr in 2007 and became CTO in 2008, heading the development department with Allspaw.

At Velocity Santa Clara 2009, Hammond and Allspow jointly presented 10+ Deploys per Day: Dev and Ops Cooperation at Flickr. As you can see, although the name did not yet have the word “DevOps”, it was literally half a step away from it. Here is this report:

In the talk, they described the main conflict between Dev and Ops: developers devote their lives to making changes, and for operation, any change is a possible cause of failures and failures. However, change is vital for business, which means that instead of waging wars, the task of Ops is to give businesses the opportunity to grow.

Hammond and Allspaw have proposed creating tools to reduce the risk of change and tools to quickly recover from disruptions.

They also suggested the following:

• Automated infrastructure -> consistent development and production platforms (infrastructure as code, role, and configuration management).
• General version control for both development and production.
• Setting up a one-step build system.
• Deploy in one step.
• Deployment log (Who? When? What?).

Most importantly, it is important for the company to create a culture of trust so that operations know what the development is doing and vice versa. Allspow asked to pay close attention to this point, as he saw that the audience was more interested in the technical side of the report. People, processes, and software are much more interconnected than is commonly believed.

It cannot be said that this report immediately turned the industry upside down: the words about "10 deployments a day" sounded for many either a lie, or madness, or empty bragging of a large company ("well, they can afford this there, but why should we?" ).

But the report inspired a specific person – Patrick Debois. He didn't like the fact that there were development conferences and infrastructure conferences then, but there was no event where two audiences would intersect and communicate. And the joint report by Allspoe and Hammond convinces him that such an event is necessary. Debois decides to create a conference for both dev and ops.

He calls it DevOpsDays ("because the Agile System Administration was too long") and is holding it for the first time in 2009 in Belgium. After the end of the conference, participants continued to discuss the raised topics on Twitter, where the name of the event was shortened to the hashtag #DevOps. As a result, the word, which spontaneously emerged as a hashtag, stuck so much that today Google finds it on millions of pages. So we owe this name to Patrick, although he himself did not at all expect such an effect.

In 2010, the DevOpsDays conference was held for the first time in the United States, and there it also gave fertile ground for discussions. Damon Edwards and John Willis then coined the acronym CAMS (Culture, Automation, Measurement, Sharing) – culture, automation, measurement, and sharing. Jez Hamble later added L (Lean) for lean. Thus, the key principles of DevOps were identified and a framework for assessing readiness for the transition to DevOps appeared.

By the way, John Willis became one of the key people for the whole phenomenon (for example, as a co-author of the book "DevOps Guide"), spoke a lot in connection with him, and one of these speeches was at our DevOops conference:

To be continued…