Let’s check why attempts to “implement DevOps” do not make sense without a specific goal and how to optimize the work of an IT company when there is a real goal.

In different technical teams you can find different formulations of DevOps and their skills. Most come to the fact that this is a certain person who is “in the same room” with development and IT operations (sometimes also QA) and coordinates their work. Of course, such a definition is very, very arbitrary.

In technology startups with few people, non-programming DevOps is nonsense. Why is he there? In fact, in such commands, ops are lowercase letters and DEV are large letters in the beginning. In stable medium-sized teams, OPS are something “wow” and many simple developers are just “around”, so the “weight” of letters is noticeably changing.

We never set ourselves the goal of “implementing DevOps” or “hiring DevOps”. Because – well, who is it? Why should they make us better than anyone? How do they differ from those employees that have always been with the company? We created a team of motivated operations who know and delve into the logic of the product, working side by side with developers and testers. And realized that many in the global IT community call such people DevOps.

Why do we need DevOps

DevOps is a set of practices and tools, they must solve real problems. But first you define the goal, and then you are already looking for tools.

Our goal is the most effective business. Therefore, we are flexible and do not hesitate to choose simple solutions. Where others do according to established patterns, we do it primarily optimally, but still good. This approach requires the constant cultivation of ever new life hacks on the technical field. We need to design and administer environments so that they provide the greatest possible flexibility for the business. If at the same time they are simple, sometimes even primitive in their implementation – so what?

We don’t need people with the title of DevOps and the ability to talk about Kubernetes for 20 minutes in a row, unless they can explain how a TCP packet gets from one OS to another.

We are a product company, and everything revolves around our product. We never had “system administrators” because there were no systems that would make sense and value without a product. At the beginning of the 2010s, when we entered the market, system administrators at product companies were people who perfectly understood both: the business logic of their products and the complexity of their mass supply and maintenance in production. The same people are now becoming good DevOps: they understand what, how and why to do it.

We did not set ourselves the goal of “implementing DevOps”. Instead, they asked themselves the question: what do we need in operations and who can implement it. The answer came in three parts: people (team), the approach to IT operations and the environment.

What (and who) makes the team effective

First of all, we described the people we want to see in the team: their hard and soft skills, mindset and the work of the whole team. Here are the signs that turned out to be highlighted:

• The structure should be as flat as possible, no bosses. Operations should not care about politics. They should not think about “who asked to do this,” and evaluate tasks based on hierarchy. What is logical is legitimate. “More right” is the one who better substantiates.
• People must be independent. The ratio of goal driven versus process oriented among employees should be maximally in favor of the former, but the latter also need balance.
• They should be able to find out extraordinary solutions like gods! But to keep track of siuch “solutions”, and correctly write them in technical duty.
• Cloud skills are less important to us than on-premise. A person who knows how to turn on-premise DC into a decent private IAAS, in our eyes, is more valuable than a pro in an AWS environment, for example.
• Good skills in templating. We need someone with desire to see the template in disparate entities and processes.
• “Automation” and “axiom” should not be synonyms in the head of the operator. Actually, we return to the third point: they should be able to find out extraordinary solutions like gods!
• Logic is more important than continuity. In an interview in game technical tasks, we try to put a person in a situation which he cannot solve due to encyclopedic knowledge. And we look at how he thinks in these conditions, whether he can explain the direction of his thoughts. By the way, we allow you to use during the interview the phone with the Internet and google.

Which characteristics should IT operations have

• The product is used by the user. And he must be satisfied. We do not use the phrase “well, let those who did that, to solve it.” Our operations are a hosts of any pain for which a specific reason has not yet been found.
• Convenient – it is transparent and intuitive. The less you need to remember and know about services and systems, the better. Ideally, you don’t need to remember and know anything about services. But transparent and intuitive is not equally unsafe (more about it below).
• We do not plan for years, but still plan. And we do not spend significant resources on planning, we save time.
• If someone wants to try a new product, he simply sets himself the task and installs PoC. Then he tells others.
• New entities in production appear for two reasons:

a) it can be designed and put into the described scheme: everything should answer the question, “how to create another hundred of such and make them the same and convenient”;

b) because it is necessary “right now”, you need to catch the moment.

• less meetings, more ad hoc (i.e. fewer group meetings, more specific technical dialogs and disputes).
• If possible, the lack of non-interactive communications (such as emails).
• If possible, the presence of the ticket system.
• The sooner ops appears among dev during the implementation of something new, the better. Ideally – to be in time from the very beginning. Therefore, instead of working on a single pool of tasks, our guys work each in their development teams.
• In customer support, say no to Agile. If you leave the phone support team with customers in private, then there will be no quality improvement. In this area, the ITSM approach is our everything: dividing into L1-L2-L3 levels, if possible – an independent incident and problem management, working with incident statistics … We are investing resources in this more and more. By the way, in our company, monitoring and alerting are closely related to customer support.
• Security must be transparent. It doesn’t mean to hide information, no! And it should not burden operations. This does not mean that security can be neglected or violate the principle of defense in depth. This means that security – whether it is implemented in some way by dev, DevOps or SecOps – should not break the design, processes and procedures in general.

Probably, every IT specialist who worked in the corporate sector was faced with a situation where security department become self-contained and self-sufficient structure. They make rules, the meaning of which can no longer be clearly explained and how to prove their effectiveness by statistics is also a question. In this case, the PCI DSS approach is better for us, as well as its main postulate: before defending, narrow the scope.

In what environment do people work effectively

What characteristics should environments have in order for the people described above working in the style described above to be able effectively administer and develop them?

• Infrastructure as a code.
• Horizontal scaling and dynamic everything: a catalog of services, service mesh, etc.
• If during design in the column “recovery scenario” there is a choice between “recreate” or “from backup”, then you need to choose the first. Definitely.
• Transparent structures can be documented in less detail, or even not documented at all, with enough sketches (therefore, we are strongly criticized for indentation from naming conventions and similar).

When we all formulated this, it turned out that much of the above fits the manifest of Agile in general and DevOps in particular. The order is very important here: we didn’t implement the Agile philosophy. We worked, identified the most effective approaches – and found that our philosophy is called like this.

DevOps tools are an abstraction

Now about the tools with which all this is implemented. There are no universal DevOps solutions. No, it cannot be. You can, of course, list some trending technologies and products, but why?

For example, in online articles in the list of DevOps tools, is often mentioned Jenkins. Yes, Jenkins appeared already in all the days of DevOps. But before, good old system administrators were quietly setting it up for themselves.

The Internet is full of articles with the topic “Company X uses Y.” But you should not be interested in the fact that such a business uses Prometheus, Grafana, or something else. You should be wondering how much this company resembles yours, what tasks it solves and why exactly with these solutions. You need to look at what they choose from, how long they use, what else they use from similar products, how deep the implementation is. Maybe they are already going to abandon this tool. In general, you need little things. And the names of the products themselves are not needed. Formulate a specific problem and look for tools to solve it.

And even if you are lucky to see a list of normal tools, what is its value? Any tools today become so old that it makes no sense to list them.

What conclusions can be made from all of this?

There is no “instruction” for implementing DevOps. Moreover, it doesn’t make any sense. There is no single system by which DevOps should be organized in a company. For example, we have “DevOps  for DevOps” – a kind of core-team that develops things common for all teams. But overall, this is a virtual structure: system tasks and subprojects move from person to person. There is no technical support and DevOps separately. Devs did not lose the ITIL classification of system administrators, like the L3 tech support.

There is a certain division into enterprise and production, but both meta-environments are supported by the same people. Therefore, our task is to make the enterprise as flexible and relaxed as possible: we try not to produce unnecessary services so that they do not need to be supported. And we do it.

Once again: there is no “step-by-step plan for implementing DevOps.” But if you still try to formulate general advice, then there are only two of them:

• Don’t be afraid to redo it. A very big mistake is to refactor the work in everything and always. Because remodeling is essentially life.
• Know the price of mistakes. Making a mistake is often cheaper than not making it.