👉 🖤 🏟️ Infrastruktur als Code in Avito: Lehren, die wir gezogen haben 🖖 👲🏻 🎬

Hallo Habr! Mein Name ist Sasha Kozlov. Ich habe in den letzten dreieinhalb Jahren die Infrastruktur- und Systemadministration bei Avito entwickelt. Ich werde Ihnen zeigen, wie wir unsere Arbeit mit Infrastrukturcode im Laufe der Zeit skaliert und modernisiert und auf ein qualitativ neues Niveau gebracht haben.

Wir sind für den gesamten Zyklus des Gerätemanagements verantwortlich: von der Beschaffung und Installation im DC bis zur Lieferung an den Endbenutzer. Hier geht es um die Lektionen, die wir in den letzten Jahren gelernt haben, als wir mit Tausenden von Geräten und mehreren Tausend Konfigurationen gearbeitet haben.

Wir werden über die IaC-Tools der "ersten Generation" wie Ansible, Chef, Salt, Puppet sprechen. Wenn Sie mit einer lokalen Infrastruktur zu tun haben und keine eigene Cloud für die Virtualisierung haben, verwenden Sie höchstwahrscheinlich bereits eines dieser Tools.

Puppet. - , , Puppetlabs . , , , , . , — .

, . , . - push-, - pull, ansible pull-. — Python, Jinja- YAML-, — DSL Ruby. : , PuppetDB, , .

, . git revert, , . , , . → ’ , 0 → ’. .

, Puppet. - — , , . , :

, ;
;
CI-, ;
;
, , ;
«» Configuration Drift.

, , . — . .

. : , , .

Puppet , . , .. pull-. — . 30 , , , Puppet . , event-driven , Salt, .

, . stateful-, , . .

: Puppet stateful-, , . , DNS- Puppet, . , , , API.

, . , . , , git . . .

: control repo . git-. , Puppet-, "The roles and profiles method".

Control repo — . , CI- - . — , . — , « ?». : k8s- , Kafka , ClickHouse- ..

— , . - . Puppet -, , . , , , .

control repo, .

— , , , . control repo, semver . 50 : , , Kubernetes .

, , Puppet. . , , . , control repo?

, , . control repo , . , . , , . Docker, . control repo, . , .

, , :

Building a Functional Puppet Workflow Part 2: Roles and Profiles
Roles and Profiles in a Control Repo?
Workflows Evolved: Even Besterer Practices
Profiles and the Path to Hiera Data
Puppetlabs Best Practices Docs

- control repo, . , , , :

puppet-controlrepo-template
puppet-module-template

External Node Classifier

, , — , ?

node definitions , , . , - :

node /^avi-ceph(2[1-9]|3[0-9]|4[0-9]|5[0-9]|6[0-9]|7[0-9]|8[0-9]|9[0-9])/ {
...
}

. — , . , , . Puppetlabs External Node Classifier.

External Node Classifier — , , . , . ENC , top-scope variables. control repo , , , node definition:

node default {
  include base #    ,    control repo
  if $::role != '' {
    notify{ "Node ${::fqdn} has role ${::role}": loglevel => info }
    include "role::${role}"
  } else {
    notify{ "Node ${::fqdn} has no role": loglevel => warning }
  }
}

ENC , . CMDB, . CMDB netbox Digital Ocean, . razor, , , -.

CMDB - , 15-20 , , .

«» ? , Configuration Drift, . Configuration Drift — , , . — , , , - Puppet.

, , , . , . , k8s-.

, API-. netbox API, . PXE netboot- Debian, preseed' , API, IPMI Redfish API. , , , .

, , . , .

. : , , , , .

, — , . — . . “write-only” , .

3.7 6 Puppet, , . :

, ..
, , duplicate resource declaration. : , , .
Docker-, , inspec.

Kubernetes

Kubernetes Puppet "Kubernetes The Hard Way", .

, , smoke-, . , , .

, Beaker , , -, . ~~ smoke-: ~~ , .

, , :

  context 'application deployment' do
    it 'can deploy an application into a namespace and expose it' do
      shell('systemctl restart kubelet')
      shell('count=0;
        while [[ $(kubectl get pods -n tiller -l name=tiller -o \'jsonpath={..status.conditions[?(@.type=="Ready")].status}\') != "True" ]];
          do
            if [[ count -gt 180 ]]; then
              break;
            fi;
            sleep 1 && ((count++));
          done')
      shell('kubectl create -f /tmp/nginx.yaml', acceptable_exit_codes: [0]) do |r|
        expect(r.stdout).to match(%r{namespace/nginx created\nconfigmap/my-nginx-config created\ndeployment.apps/my-nginx created\nservice/my-nginx created\n})
      end
    end

    it 'can access the deployed service' do
      shell('count=0;
        while [[ $(kubectl get pods -n nginx -l run=my-nginx -o \'jsonpath={..status.conditions[?(@.type=="Ready")].status}\') != "True" ]];
          do
            if [[ count -gt 180 ]]; then
              break;
            fi;
            sleep 1 && ((count++));
          done')
      shell('curl --connect-timeout 1 --retry-delay 1 --retry-max-time 300 --retry 150 -s --retry-connrefused 10.100.10.5', acceptable_exit_codes: [0]) do |r|
        expect(r.stdout).to match %r{Welcome to nginx!}
      end
    end
 end

, . CI .

CI PR. PR : . , , , CI PR, , .

, :

puppet-syntax
puppet-lint
rspec-puppet
puppetlabs-spec-helper
test-kitchen, Puppet: kitchen-docker-puppet-example
beaker

Unit testing with rspec-puppet — for beginners

Development kit

, CI/CD — , workflow.

- — PDK, . , , . PDK control repo Kitchen Docker. PDK Beaker, .

, :

, : , CI .
, , -.
, -, .
control repo, , .
.
.

development tool, CI , .

(puppet-rspec, puppet-linter, test-kitchen) , . , , .

IDE Puppet

, IDE - VSCode, DSL, .

, , « ». , .

, , . , — . : Forge. Puppet Forge , .

Puppet . , : metadata.json Puppetfile. , - . . librarian-puppet, , r10k, . , . — , : acceptance- -.

. , git-, :

mod 'dba-clickhouse',
  :git => 'ssh://git@github.com/iac/dba-clickhouse.git',
  :ref => '1.2.2'

mod 'dba-kafka',
  :git => 'ssh://git@github.com/iac/dba-kafka.git',
  :ref => '1.2.0'

, , . Puppet Forge, , librarian-puppet.

semver, , . , , — :

# Puppetfile

mod 'arch-puppetserver', '0.20.5' #     
mod 'arch-vault', '~> 2.1' #      
mod 'si-lxc' #

, . , , , . :

[22:39:43] in dba-control on  production via  ruby-2.5.1 at ️  unstable 
$ iack dep show
[] Collecting modules metadata
FULL NAME            | CURRENT VERSION | LATEST VERSION | OUT OF DATE?  
---------------------|-----------------|----------------|---------------
si-lxc               | latest          | 0.3.2          | N/A           
si-base              | latest          | 1.3.1          | N/A           
petems-hiera_vault   | v0.4.1          |                | Major
arch-vault           | 2.1.0           | 2.1.0          | No   
dba-postgresql       | 0.1.2           | 0.1.3          | Tiny 
dba-pgbouncer        | 0.4.0           | 0.5.1          | Minor
si-grub              | 0.1.0           | 0.1.0          | No   
si-collectd          | 0.2.3           | 0.2.4          | Tiny 
si-confluent         | 0.3.0           | 0.3.0          | No   
dba-redis            | 0.2.3           | 0.2.3          | No   
dba-collectd_plugins | latest          | 0.2.0          | N/A           
dba-mongodb          | 0.2.1           | 0.2.1          | No   
dba-patroni          | 0.1.4           | 0.2.4          | Minor
dba-cruise_control   | 0.1.1           | 0.1.2          | Tiny 
dba-lxd              | 0.7.0           | 0.7.0          | No   
dba-clickhouse       | 1.2.1           | 1.2.2          | Tiny 
dba-zookeeper        | 2.0.0           | 2.0.0          | No   
si-td_agent          | 0.1.0           | 0.1.0          | No   
dba-kafka            | 1.1.6           | 1.2.1          | Minor
arch-puppetserver    | 0.20.1          | 0.20.2         | Tiny 
pcfens-filebeat      | 4.1.0           | 4.4.1          | Minor
KyleAnderson-consul  | 5.0.3           | 6.0.1          | Major
puppetlabs-apt       | 6.3.0           | 7.4.2          | Major
puppetlabs-stdlib    | 5.2.0           | 6.3.0          | Major

Managing environment content with a Puppetfile
puppet-forge-server — ruby Sinatra, Puppet Forge
librarian-puppet

code style

code style ? — , . , , , . review — , .

Puppet . The puppet language style guide . Puppet-lint, CI, .

, . control repo:

Avito Module Coding Standards
Avito Control Repo Coding Standards

development kit, «»:

;
: how to . ;
, .

—

, . Puppet Hiera Vault. hiera-backend, vault hiera_lookup.

-, , - . , :

$token_data = vault::secret_field('tokens.csv', 'data')

token_data 'data' tokens.csv, Vault. Vault , , Hiera:

$ vault-util ls puppet/arch/      
common/
nodes/
roles/

, , common. — roles/ nodes/.

— , -, , , . TLS- .

, , , . , Puppet:

avito-vault

Canary

Puppet , control repo. control repo , . ENC .

, . , . — - .

, , , .

Puppet

Puppet, , , . DSL — Ruby. Hiera — , . , . Hiera , , .

Puppet — , , . . , : , Hiera, . - , , .

Puppet Ruby , . , pet project. , , — . : , , , .

— . , pull-. , , . , , pull push. Bolt, Puppetlabs , , PuppetDB, .

, , :

, .
.
.
.
CI .
, workflow .
External Node Classifier .
.
.

Infrastruktur als Code in Avito: Lehren, die wir gezogen haben

External Node Classifier

Development kit

code style

—

Canary

Puppet

More articles: