Attacking Ansible (2): How to quickly get the layout of the Ansible project in the production environment?

Tips: Similar to the previous article "Ansible (1): Ansible Quick Start", the Ansible version 2.5.4 used in this article, the project demonstration environment MacOS. Since the active version of the Ansible project is updated quickly and many API interfaces are not backward compatible, please make sure that the versions used are the same when you practice this article.

After studying the previous article "Attack on Ansible (1): Ansible Quick Start", it is more than enough to publish a single project. However, in the actual production environment, a service often has multiple components. For example, when deploying a big data service, it is often necessary to deploy a "big data family bucket": Hadoop, Zookeeper, Hive, Mysql, Flink, etc. At this time, only relying on the knowledge in the previous article is a bit stretched. The dependence of a large number of yaml files and other configuration files is complicated. If the directory organization project structure cannot be divided correctly, it is very unfavorable for later maintenance. So today's article focuses on solving this problem: How to scientifically and correctly divide the directory structure of Ansible applications?

Think of Ansible as a programming language

First of all, we must establish the concept: "Think of Ansible as a programming language." We can understand Ansible as a DSL specifically used to manage automated releases. Its basic grammatical rules are approximately equal to the yaml language rules, such as synchronize, pip, template, etc. At the same time, the Ansible module can be equivalent to the language's built-in functions or built-in packages, that is, when writing a playbook, you are writing code blocks for the Ansible language. In this way, as long as we understand Ansible along the line of programming language, many doubts will be easily solved: for example, Ansible supports variables, module when is the flow control statement if in the programming language, and module loop or with_* is the loop iteration statement in the programming language for or while.

In the process of project development using programming languages, how do we reduce the complexity of the project? Of course it is "modularization". Different functions are encapsulated in different packages or files, so that the smallest unit that constitutes a business function is called a "module". In the entry file of the project, we load the required modules through keywords such as import (this keyword is used in languages such as Python, Golang) or require (this keyword is used in languages such as Node.js), and then you can proceed Business logic orchestration.

The advantages of this are obvious. A module is a specific realization of a business function. When there is a need for modification in the later period, only the relevant module needs to be modified. This is exactly what the "SRP" (Single Responsibility Principle) advocates in the SOLID principle. . In addition, modularization also supports the flexible organization of business processes based on business needs like "building blocks", so that the current modules can be reused to the greatest extent, which is also in line with the "DRY" (Don't Repeat Yourself) in the programming principle.

Therefore, when we regard Ansible as a programming language (DSL), we can find that in the Ansible document, Roles is used to represent the concept of "module", and import_tasks, include_tasks, etc. are used to represent the concept of import. Here is a digression. I wonder if anyone thinks that the concepts of "playbook" and "roles" invented in Ansible are confusingly translated into "scripts" and "roles"?

Crucial concept: Roles

Now we have established the concept of "Roles = Module". Let's take a look at the definition of Roles in the document to enrich the details:

Roles let you automatically load related vars, files, tasks, handlers, and other Ansible artifacts based on a known file structure. After you group your content in roles, you can easily reuse them and share them with other users.

"Roles allow you to automatically load related variables, files, tasks, handlers, and other Ansible artifacts based on a known file structure. After the content is grouped by roles, you can easily reuse them and share them with other users."

To put it bluntly, Roles is an encapsulation of variables, files, tasks, etc., for the purpose of module reuse.

How to use Roles?

In the best practice of software design paradigm, there is a rule called "Convention is greater than configuration". Simply put, "There are some prerequisite assumptions made in the software. These assumptions are the agreement between the software developer and the software user. As a software user, you abide by these The agreement is sufficient, and it is no longer necessary (or not supported) to configure these agreements."-To a certain extent, it can be understood as the default configuration in the software.

When Ansible uses Roles, there are also "conventions greater than configuration", and these conventions are hard (that is, customization in the configuration file is not supported).

First of all, the directory structure of Role is fixed.

An Ansible role has a defined directory structure with eight main standard directories. You must include at least one of these directories in each role. You can omit any directories the role does not use.

Ansible roles have a well-defined directory structure with eight main standard directories. The specified directory structure, an example is as follows:

├── defaults
│   └── main.yml
├── files
├── handlers
│   └── main.yml
├── meta
│   └── main.yml
├── tasks
│   └── main.yml
├── templates
├── tests
│   ├── inventory
│   └── test.yml
└── vars
    └── main.yml

By default Ansible will look in each directory within a role for a main.yml file for relevant content.

The functions of these eight directories are:

tasks/main.yml: Place the files used by the role to execute tasks.
handlers/main.yml: handlers, which can be used inside or outside the role
library/my_module.py: Module, which can be used in roles (for more information, see Embedding Modules and Plugins in rroles).
defaults/main.yml: The default variables for the role (for more information, see Using Variables). These variables have the lowest priority of all available variables and can be easily overridden by any other variables (including inventory variables).
vars/main.yml: Other variables in the role. (It has the same function as vars in the Ansible module, except that vars here means a directory.)
files/main.yml: The file used in role deployment.
templates/main.yml: The template used during role deployment. It has the same function as the templates in the Ansible module, except that the templates here represent directories. )
meta/main.yml: Metadata used by the role.

At least one such directory must be included in each role. Of course, we can omit any directories that are not used by the role, but the main.yml file in each directory is the entry file of the directory. Ansible will look for this file by default when reading it. . So this main.yml file cannot be omitted.

Next, store and find roles.

We have learned about the internal directory structure of a role above, but this is far from meeting the needs of actual production. At the beginning of the article, we also took a big data project as an example, which often requires the deployment of multiple components such as Flink and Hadoop. Each of these components can be regarded as a role. So how does Ansible find roles?

By default, there are 2 ways:

In Ansible's release project, create a directory called roles.
By default, Ansible will automatically find the roles in the /etc/ansible/roles directory.

For example, we are going to use Ansible to create a bigdata project that publishes big data "family buckets". The project should contain three roles: Flink, Mysql, and Hive. Then the directory structure of the bigdata project is roughly as follows:

➜  bigdata tree -L 3
.
└── roles
    ├── flink
    │   ├── defaults
    │   ├── files
    │   ├── handlers
    │   ├── meta
    │   ├── tasks
    │   ├── templates
    │   ├── tests
    │   └── vars
    ├── hive
    │   ├── defaults
    │   ├── files
    │   ├── handlers
    │   ├── meta
    │   ├── tasks
    │   ├── templates
    │   ├── tests
    │   └── vars
    └── mysql
        ├── defaults
        ├── files
        ├── handlers
        ├── meta
        ├── tasks
        ├── templates
        ├── tests
        └── vars

Obviously, you can see that the sub-directory structure of Flink, hive, and Mysql under roles is the eight directories mentioned above.

ansible-galaxy

Quickly create a Role

It can be seen from the above that every role created must contain at least one of the eight major directories. Therefore, Ansible has built-in a command line tool ansible-galaxy to quickly create eight directories of roles, reducing our workload.

Assuming that the role name is flink, use the following command to generate the relevant directory:

ansible-galaxy init flink

Use the tree command to see that the directories generated by ansible-galaxy are exactly the eight standard directories required by the role.

➜  tree flink
flink
├── README.md
├── defaults
│   └── main.yml
├── files
├── handlers
│   └── main.yml
├── meta
│   └── main.yml
├── tasks
│   └── main.yml
├── templates
├── tests
│   ├── inventory
│   └── test.yml
└── vars
    └── main.yml

Ansible-galaxy init several other useful parameters:

ansible-galaxy init -force role_name, if the role created by default has the same name as the file existing in the current working directory, an exception will be thrown. Using the -force option will force the creation of the role directory and replace the directories or files with the same name under the role directory.
ansible-galaxy init --role-skeleton=/path/to/skeleton role_name, students who have used Maven should know that Maven supports using other projects as skeletons to create new projects. Ansible-galaxy also supports this function, taking the role under the /path/to/skeleton path as the skeleton, and copying all files to create a new role.

Galaxy: role online sharing community

In addition, similar to Docker Hub and Grafana Dashboards, Ansible Galaxy also has an online community Galaxy[ https://galaxy.ansible.com/home], which has various developed roles shared by developers. It is convenient for us to search for ready-made roles to download, and we can also upload roles developed by ourselves to Galaxy.

To download or upload a role to the Galaxy website, you also need to use the command line tool ansible-galaxy. By default, the address of the Galaxy server called by ansible-galaxy is https://galaxy.ansible.com, The address of Galaxy can be reconfigured through the -server option or in the ansible.cfg file.

download roles

The syntax template for downloading roles is:

$ ansible-galaxy install username.role_name

By default, ansible-galaxy will download the role to the environment variable ANSIBLE_ROLES_PATH. Ansible-galaxy provides the parameter --role_path to specify the download address of the role.

ansible-galaxy Quick overview of other commands

ansible-galaxy search elasticsearch, find the role elasticsearch in the Galaxy website
ansible-galaxy info username.role_name, view the detailed information of username.role_name
ansible-galaxy list, view installed roles
ansible-galaxy remove username.role_name, uninstall the installed username.role_name
ansible-galaxy login, log in to the Galaxy website

Overall layout: Suit is best

If you insist on reading the above part, then you must know how to use roles. Simply put, a directory called roles needs to exist under the current Ansible application. Next, let's talk about the layout of other directories in addition to the roles directory under the Ansible application? The official Ansible best practice document [ https://docs.ansible.com/ansible/2.5/user_guide/playbooks_best_practices.html#content-organization] suggests this:

Your usage of Ansible should fit your needs, however, not ours, so feel free to modify this approach and organize as you see fit.

One crucial way to organize your playbook content is Ansible’s “roles” organization feature, which is documented as part of the main playbooks page. You should take the time to read and understand the roles documentation which is available here: Roles.

Ansible's overall directory structure does not have a certain rule, just suit your current needs. But the concept of Roles is crucial.

Ansible of conscience has officially opened a project ansible-examples [ https://github.com/ansible/ansible-examples] on Github to collect excellent best practices. You can learn from your actual needs. Let me share my commonly used project layout:


.
├── Makefile
├── README.md
├── deploy.retry
├── deploy.yml
├── files
│   ├── apache-maven-3.8.3-bin.tar.gz
│   ├── apache-zookeeper-3.7.0-bin.tar.gz
│   ├── flink-1.14.0-bin-scala_2.11.tgz
│   ├── hadoop-2.7.5.tar
│   ├── hadoop-3.3.1.tar.gz
│   ├── mysql-connector-java-8.0.26-1.el7.noarch.rpm
│   ├── mysql-connector-java-8.0.26.jar
│   └── openjdk-11.0.2_linux-x64_bin.tar.gz
├── inventories
│   └── hosts
└── roles
    ├── flink
    ├── hadoop
    ├── hadoop3
    ├── hive
    ├── java
    ├── linux
    ├── mvn
    ├── mysql
    └── zookeeper

Makefile, used to encapsulate Ansible's release commands
deploy.ym is the entry file when executing Ansible commands
Files, used to store role-related deployment packages, are generally large in size and will not use git for version management.
inventories, used to manage deployment machines.
roles, the components used for deployment. As can be seen from the above catalog, it is currently mainly used to deploy big data-related components.

Make a small advertisement for yourself: In the next practical chapter, you will use this project layout to publish big data projects. What Ansible knowledge needs to be added in this process? Why don't I directly use the role on the Galaxy website but develop it from scratch? In deploying Hadoop3 and Flink projects, what pits have you stepped on when using Ansible?

Stay tuned "Ansible Attack (3): Ansible Big Data Practice"

Reference