이 글은 CloudNet@ 팀 gasida님의 스터디 A101 1기 내용 및 실습으로 작성된 글입니다.
모니터링 자동화
# 팩트를 이용한 시스템 모니터링
- 팩트는 관리 노드에서 시스템과 관련된 정보들(ex.호스트 이름, 커널 버전, 네트워크 인터페이스 이름 등) 을 찾아 변수로 제공한다.
- 시스템 모니터링을 만들 프로젝트 디렉터리와 ansible.cfg, inventory를 생성한다.
- 프로젝트 디렉터리 생성 및 ansible.cfg, inventory 파일 작성
- ~/my-ansible/chapter_12.1/..
#
mkdir ~/my-ansible/chapter_12.1
cd ~/my-ansible/chapter_12.1
# ansible.cfg, inventory 파일 작성
cat <<EOT> ansible.cfg
[defaults]
inventory = ./inventory
remote_user = ubuntu
ask_pass = false
[privilege_escalation]
become = true
become_method = sudo
become_user = root
become_ask_pass = false
EOT
cat <<EOT> inventory
[tnode]
tnode1
tnode2
tnode3
EOT- 플레이북 파일 작성
- ~/my-ansible/chapter_12.1/monitoring_facts.yml
---
- hosts: tnode
vars:
log_directory: /var/log/daily_check
tasks:
- name: Print system info
ansible.builtin.debug:
msg:
- "################ Start #####################"
- "Date: {{ ansible_facts.date_time.date }} {{ ansible_facts.date_time.time }}"
- "HostName: {{ ansible_facts.hostname }}"
- "OS: {{ ansible_facts.distribution }}"
- "OS Version: {{ ansible_facts.distribution_version }}"
- "OS Kernel: {{ ansible_facts.kernel }}"
- "CPU Cores: {{ ansible_facts.processor_vcpus }}"
- "Memory: {{ ansible_facts.memory_mb.real }}"
- "Interfaces: {{ ansible_facts.interfaces }}"
- "IPv4: {{ ansible_facts.all_ipv4_addresses }}"
- "Devices: {{ ansible_facts.mounts }}"
- "################# End #######################"
register: result
- name: Create log directory
ansible.builtin.file:
path: "{{ log_directory }}"
state: directory
- name: Print logs to log file
ansible.builtin.shell: |
echo "{{ item }}" >> "{{ log_directory }}"/system_info.logs
loop: "{{ result.msg }}"- 플레이북 실행
# 실행
ansible-playbook monitoring_facts.yml
# 확인
for i in {1..3}; do echo ">> tnode$i <<"; ssh tnode$i ls -l /var/log/daily_check; echo; done
ssh tnode1 sudo cat /var/log/daily_check/system_info.logs
ansible-playbook monitoring_facts.yml
ssh tnode1 sudo cat /var/log/daily_check/system_info.logs- 결과값 확인
################ Start #####################
Date: 2024-02-11 19:39:43
HostName: tnode1
OS: Ubuntu
OS Version: 22.04
OS Kernel: 6.2.0-1018-aws
CPU Cores: 2
Memory: {'total': 3847, 'used': 1005, 'free': 2842}
Interfaces: ['ens5', 'lo']
IPv4: ['10.10.1.11']
Devices: [{'mount': '/', 'device': '/dev/root', 'fstype': 'ext4', 'options': 'rw,relatime,discard,errors=remount-ro', 'size_total': 31024283648, 'size_available': 29126275072, 'block_size': 4096, 'block_total': 7574288, 'block_available': 7110907, 'block_used': 463381, 'inode_total': 3870720, 'inode_available': 3795990, 'inode_used': 74730, 'uuid': '1127d3f4-3de6-4e02-8705-713269caf74d'}, {'mount': '/snap/core18/2812', 'device': '/dev/loop1', 'fstype': 'squashfs', 'options': 'ro,nodev,relatime,errors=continue,threads=single', 'size_total': 58458112, 'size_available': 0, 'block_size': 131072, 'block_total': 446, 'block_available': 0, 'block_used': 446, 'inode_total': 10944, 'inode_available': 0, 'inode_used': 10944, 'uuid': 'N/A'}, {'mount': '/snap/core20/2105', 'device': '/dev/loop2', 'fstype': 'squashfs', 'options': 'ro,nodev,relatime,errors=continue,threads=single', 'size_total': 67108864, 'size_available': 0, 'block_size': 131072, 'block_total': 512, 'block_available': 0, 'block_used': 512, 'inode_total': 12041, 'inode_available': 0, 'inode_used': 12041, 'uuid': 'N/A'}, {'mount': '/snap/amazon-ssm-agent/7628', 'device': '/dev/loop0', 'fstype': 'squashfs', 'options': 'ro,nodev,relatime,errors=continue,threads=single', 'size_total': 26214400, 'size_available': 0, 'block_size': 131072, 'block_total': 200, 'block_available': 0, 'block_used': 200, 'inode_total': 16, 'inode_available': 0, 'inode_used': 16, 'uuid': 'N/A'}, {'mount': '/snap/lxd/26881', 'device': '/dev/loop3', 'fstype': 'squashfs', 'options': 'ro,nodev,relatime,errors=continue,threads=single', 'size_total': 91226112, 'size_available': 0, 'block_size': 131072, 'block_total': 696, 'block_available': 0, 'block_used': 696, 'inode_total': 959, 'inode_available': 0, 'inode_used': 959, 'uuid': 'N/A'}, {'mount': '/snap/snapd/20671', 'device': '/dev/loop4', 'fstype': 'squashfs', 'options': 'ro,nodev,relatime,errors=continue,threads=single', 'size_total': 42467328, 'size_available': 0, 'block_size': 131072, 'block_total': 324, 'block_available': 0, 'block_used': 324, 'inode_total': 658, 'inode_available': 0, 'inode_used': 658, 'uuid': 'N/A'}, {'mount': '/boot/efi', 'device': '/dev/nvme0n1p15', 'fstype': 'vfat', 'options': 'rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro', 'size_total': 109395456, 'size_available': 103061504, 'block_size': 512, 'block_total': 213663, 'block_available': 201292, 'block_used': 12371, 'inode_total': 0, 'inode_available': 0, 'inode_used': 0, 'uuid': '2D7D-66A4'}]
################# End #######################
################ Start #####################
Date: 2024-02-11 19:39:52
HostName: tnode1
OS: Ubuntu
OS Version: 22.04
OS Kernel: 6.2.0-1018-aws
CPU Cores: 2
Memory: {'total': 3847, 'used': 1005, 'free': 2842}
Interfaces: ['lo', 'ens5']
IPv4: ['10.10.1.11']
Devices: [{'mount': '/', 'device': '/dev/root', 'fstype': 'ext4', 'options': 'rw,relatime,discard,errors=remount-ro', 'size_total': 31024283648, 'size_available': 29126238208, 'block_size': 4096, 'block_total': 7574288, 'block_available': 7110898, 'block_used': 463390, 'inode_total': 3870720, 'inode_available': 3795988, 'inode_used': 74732, 'uuid': '1127d3f4-3de6-4e02-8705-713269caf74d'}, {'mount': '/snap/core18/2812', 'device': '/dev/loop1', 'fstype': 'squashfs', 'options': 'ro,nodev,relatime,errors=continue,threads=single', 'size_total': 58458112, 'size_available': 0, 'block_size': 131072, 'block_total': 446, 'block_available': 0, 'block_used': 446, 'inode_total': 10944, 'inode_available': 0, 'inode_used': 10944, 'uuid': 'N/A'}, {'mount': '/snap/core20/2105', 'device': '/dev/loop2', 'fstype': 'squashfs', 'options': 'ro,nodev,relatime,errors=continue,threads=single', 'size_total': 67108864, 'size_available': 0, 'block_size': 131072, 'block_total': 512, 'block_available': 0, 'block_used': 512, 'inode_total': 12041, 'inode_available': 0, 'inode_used': 12041, 'uuid': 'N/A'}, {'mount': '/snap/amazon-ssm-agent/7628', 'device': '/dev/loop0', 'fstype': 'squashfs', 'options': 'ro,nodev,relatime,errors=continue,threads=single', 'size_total': 26214400, 'size_available': 0, 'block_size': 131072, 'block_total': 200, 'block_available': 0, 'block_used': 200, 'inode_total': 16, 'inode_available': 0, 'inode_used': 16, 'uuid': 'N/A'}, {'mount': '/snap/lxd/26881', 'device': '/dev/loop3', 'fstype': 'squashfs', 'options': 'ro,nodev,relatime,errors=continue,threads=single', 'size_total': 91226112, 'size_available': 0, 'block_size': 131072, 'block_total': 696, 'block_available': 0, 'block_used': 696, 'inode_total': 959, 'inode_available': 0, 'inode_used': 959, 'uuid': 'N/A'}, {'mount': '/snap/snapd/20671', 'device': '/dev/loop4', 'fstype': 'squashfs', 'options': 'ro,nodev,relatime,errors=continue,threads=single', 'size_total': 42467328, 'size_available': 0, 'block_size': 131072, 'block_total': 324, 'block_available': 0, 'block_used': 324, 'inode_total': 658, 'inode_available': 0, 'inode_used': 658, 'uuid': 'N/A'}, {'mount': '/boot/efi', 'device': '/dev/nvme0n1p15', 'fstype': 'vfat', 'options': 'rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro', 'size_total': 109395456, 'size_available': 103061504, 'block_size': 512, 'block_total': 213663, 'block_available': 201292, 'block_used': 12371, 'inode_total': 0, 'inode_available': 0, 'inode_used': 0, 'uuid': '2D7D-66A4'}]
################# End #######################
# CPU 메모리 디스크 / 이용한 시스템 모니터링
- 팩트에서 제공되지 않은 정보를 모니터링해야되는 상황
- 자세한 CPU, 메모리, 디스크 사용률 모니터링을 위해 dstat, iostat, vmstat 명령어 사용 → 툴 설치 필요
- 각각의 명령어 실행은 ansible.builtin.shell 이용하여 실행하고, loop 키워드를 이용하여 모니터링 명령어별로 여러 옵션을 추가하여 명령 실행
- 실행된 명령어 결과는 로그 디렉터리에 저장
- 프로젝트 디렉터리 생성 및 ansible.cfg, inventory 파일 작성
- ~/my-ansible/chapter_12.2/..
#
mkdir ~/my-ansible/chapter_12.2
cd ~/my-ansible/chapter_12.2
# ansible.cfg, inventory 파일 작성
cat <<EOT> ansible.cfg
[defaults]
inventory = ./inventory
remote_user = ubuntu
ask_pass = false
[privilege_escalation]
become = true
become_method = sudo
become_user = root
become_ask_pass = false
EOT
cat <<EOT> inventory
[tnode]
tnode1
tnode2
tnode3
EOT- 변수 파일 작성 : 로그 디렉터리와 설치할 패키지
- ~/my-ansible/chapter_12.2/vars_packages.yml
---
log_directory: /home/ubuntu/logs
packages:
- dstat
- sysstat- ~/my-ansible/chapter_12.2/monitoring_system.yml
---
- hosts: tnode
vars_files: vars_packages.yml
tasks:
- name: Install packages on RedHat
ansible.builtin.dnf:
name: "{{ item }}"
state: present
loop: "{{ packages }}"
when: ansible_facts.os_family == "RedHat"
- name: Install packages on Ubuntu
ansible.builtin.apt:
name: "{{ item }}"
state: present
loop: "{{ packages }}"
when: ansible_facts.os_family == "Debian"
- name: Create log directory
ansible.builtin.file:
path: "{{ log_directory }}"
state: directory
- name: Monitoring dstat
ansible.builtin.shell: |
{{ item }} >> {{ log_directory }}/dstat.log
loop:
- dstat 2 10
- dstat -cmdlt -D vda 2 10
- name: Monitoring iostat
ansible.builtin.shell: |
{{ item }} >> {{ log_directory }}/iostat.log
loop:
- iostat
- echo "==============="
- iostat -t -c -dp vda
- echo "==============="
- name: Monitoring vmstat
ansible.builtin.shell: |
{{ item }} >> {{ log_directory }}/vmstat.log
loop:
- vmstat
- echo "==============="
- vmstat -dt
- echo "==============="
- vmstat -D
- echo "==============="
- name: Monitoring df
ansible.builtin.shell: |
df -h >> {{ log_directory }}/df.log- 플레이북 실행 / 확인
# (옵션) 툴 기본 설치 확인
vmstat
dstat
iostat
# 실행
ansible-playbook monitoring_system.yml
# 확인
for i in {1..3}; do echo ">> tnode$i <<"; ssh tnode$i ls -l ~/logs; echo; done
ssh tnode1 cat ~/logs/df.log
ssh tnode1 cat ~/logs/*.log
ansible-playbook monitoring_system.yml
ssh tnode1 cat ~/logs/df.log
ssh tnode1 cat ~/logs/*.log
# 도전과제
도전과제4 : 12.3 앤서블 갤럭시에서 ‘elasticsearch’ 롤을 검색하여 es와 kibana를 tnode1에 설치 후 접속해보자!
- 인벤토리 생성 / ansible.cfg 파일 작성
cat <<EOT > ansible.cfg
[defaults]
inventory = ./inventory
remote_user = ubuntu
ask_pass = false
[privilege_escalation]
become = true
become_method = sudo
become_user = root
become_ask_pass = false
EOT
cat <<EOT> inventory
tnode1
tnode2
tnode3
EOT- ~/my-ansible/chapter_12.1/..
mkdir ~/my-ansible/chapter_13.1
cd ~/my-ansible/chapter_13.1
# 디렉토리에 role 구조 생성
ansible-galaxy role install -p roles geerlingguy.postgresql
ansible-galaxy role install geerlingguy.java
tree -h
[4.0K] .
├── [ 175] ansible.cfg
├── [ 29] inventory
├── [ 77] main.yml
└── [4.0K] roles
├── [4.0K] geerlingguy.elasticsearch
│ ├── [1.1K] LICENSE
│ ├── [2.5K] README.md
│ ├── [4.0K] defaults
│ │ └── [ 391] main.yml
│ ├── [4.0K] handlers
│ │ └── [ 80] main.yml
│ ├── [4.0K] meta
│ │ └── [ 450] main.yml
│ ├── [4.0K] molecule
│ │ └── [4.0K] default
│ │ ├── [ 288] converge.yml
│ │ ├── [ 477] molecule.yml
│ │ └── [ 28] requirements.yml
│ ├── [4.0K] tasks
│ │ ├── [1.4K] main.yml
│ │ ├── [ 441] setup-Debian.yml
│ │ └── [ 272] setup-RedHat.yml
│ └── [4.0K] templates
│ ├── [ 298] elasticsearch.repo.j2
│ ├── [2.9K] elasticsearch.yml.j2
│ ├── [ 696] heap.options.j2
│ └── [3.9K] jvm.options.j2
└── [4.0K] geerlingguy.java
├── [1.1K] LICENSE
├── [1.7K] README.md
├── [4.0K] defaults
│ └── [ 177] main.yml
├── [4.0K] meta
│ └── [ 693] main.yml
├── [4.0K] molecule
│ └── [4.0K] default
│ ├── [ 268] converge.yml
│ └── [ 477] molecule.yml
├── [4.0K] tasks
│ ├── [1.7K] main.yml
│ ├── [ 100] setup-Amazon.yml
│ ├── [ 451] setup-Debian.yml
│ ├── [ 347] setup-FreeBSD.yml
│ └── [ 100] setup-RedHat.yml
├── [4.0K] templates
│ └── [ 33] java_home.sh.j2
└── [4.0K] vars
├── [ 190] Amazon-2.yml
├── [ 195] Amazon-2023.yml
├── [ 103] Debian-10.yml
├── [ 103] Debian-11.yml
├── [ 103] Debian-12.yml
├── [ 121] Debian-8.yml
├── [ 101] Debian-9.yml
├── [ 111] Fedora.yml
├── [ 120] FreeBSD.yml
├── [ 195] RedHat-7.yml
├── [ 145] RedHat-8.yml
├── [ 141] RedHat-9.yml
├── [ 121] Ubuntu-12.yml
├── [ 121] Ubuntu-14.yml
├── [ 121] Ubuntu-16.yml
├── [ 103] Ubuntu-18.yml
├── [ 103] Ubuntu-20.yml
└── [ 103] Ubuntu-22.yml
- ~/my-ansible/chapter_13.1/main.yml
- geerlingguy.java는 의존성 패키지이므로 미리 설치해뒀다.
- hosts: all
roles:
- geerlingguy.java
- geerlingguy.elasticsearch- 플레이북 실행
# cd ~/my-ansible/chapter_13.1
ansible-playbook main.yml- 에러 발생 시 /tmp 폴더에 chmod 777 폴더 권한 수정
W: GPG 오류: http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease: Couldn't create temporary file /tmp/apt.conf.rVavhs for passing config to apt-key
E: The repository 'http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
출처: https://hancho1111.tistory.com/200 [한초블로그:티스토리]- 다시
- 플레이북 실행
# cd ~/my-ansible/chapter_13.1
ansible-playbook main.yml
buntu@tnode2:~$ curl localhost:9200
{
"name" : "tnode2",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "A2xKF0plTueQXRDCg83DmQ",
"version" : {
"number" : "7.17.18",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "8682172c2130b9a411b1bd5ff37c9792367de6b0",
"build_date" : "2024-02-02T12:04:59.691750271Z",
"build_snapshot" : false,
"lucene_version" : "8.11.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}'DevOps > Ansible' 카테고리의 다른 글
| [A101] 7. Ansible 시스템 보안 설정 (1) | 2024.02.11 |
|---|---|
| [A101] 6. Ansible 시스템 구축 및 환경 설정 자동화 (0) | 2024.02.03 |
| [A101] 5. Ansible Kubespray 분석 (0) | 2024.01.19 |
| [A101] 4. Ansible 반복문, 조건문 (0) | 2024.01.18 |
| [A101] 3. Ansible 기본사용(2) (0) | 2024.01.08 |
