2017년 6월 28일 수요일
[Cluster] Pacemaker and HAProxy - HA구성
[Cluster] Pacemaker and HAProxy - HA구성
출처: https://m.blog.naver.com/PostView.nhn?blogId=sunsync&logNo=220802338953
-- cluster를 구성하는 모든 node에 설치 --
-- vip : 192.168.56.202
-- vip : 192.168.56.203
-- vip : 192.168.56.204
-- node1: root@cent7_m_hp1 서버 - 192.168.56.131
-- node2: root@cent7_m_hp2 서버 - 192.168.56.132 -- 서버 : CentOS7, haproxy-1.5.14-3.el7.x86_64 pacemaker-1.1.13-10.el7_2.4.x86_64-- HAProxy 설정은 목록의 'HAProxy 설정' 참고-- 모든 테스트는 PC에 Oracle VM을 이용 가상서버들을 설치하여 진행했음.
1. 시간 동기화
- 여러서버가 통신하는데 시간 동기화는 매우 중요한 요소, 여기서는 ntp를 이용 시간 동기화
# yum install ntp
2. 방화벽 설정
- corosync는 udp 5404 ~ 5406 포트를 사용. 방화벽에서 설정해 줌.
ex)
# iptables -A INPUT -i eth1 -p udp -m multiport --dports 5404,5405,5406 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
# iptables -A OUTPUT -o eth1 -p udp -m multiport --sports 5404,5405,5406 -m conntrack --ctstate ESTABLISHED -j ACCEPT
3. host name 등록(모든 node 동일)
[root@cent7_m_L1 ~]# vi /etc/hosts
127.0.0.1 localhost
::1 localhost
192.168.56.131 ha-master.exam.com ha-master
192.168.56.132 ha-slave.exam.com ha-slave
4. pacemaker 설치(모든 node 동일)
-- pacemaker를 설치하면 의존성으로 corosync도 설치가 됨
-- pcs는 pacemaker와 corosync를 config하는 tool
-- pcs를 설치하면 pcsd가 설치가 됨.
-- pcsd is openssl based daemon written in ruby, manages pcs authentication between nodes,
-- the authentication files are located in /var/lib/pcsd.
[root@cent7_m_hp1 /var/log]# yum install pacemaker pcs
[root@cent7_m_hp2 /var/log]# yum install pacemaker pcs
5. pcsd 서비스 시작(모든 node 동일)
[root@cent7_m_hp1 /var/log]# systemctl start pcsd
[root@cent7_m_hp2 /var/log]# systemctl start pcsd
6. cluster node간 인증을 위한 설정(모든 node 동일)
-- 이들 패키지 설치로 인해, 시스템에 새로운 user가 만들어짐(hacluster).
-- 1) 클러스터를 구성 할 노드 인증, node들간의 user,password를 동일하게 설정.
-- 2) 클러스터 노드를 구성하고 동기화.
-- 3) 클러스터 노드에서 클러스터 서비스를 시작,중지
[root@cent7_m_hp1 /etc/corosync]# passwd hacluster
[root@cent7_m_hp2 /etc/corosync]# passwd hacluster
7. cluster node간의 인증(master node 에서만 적용)
[root@cent7_m_hp1 /var/lib]# pcs cluster auth ha-master.exam.com ha-slave.exam.com -u hacluster -p
Username: hacluster
Password:
ha-slave.exam.com: Authorized
ha-master.exam.com: Authorized
[root@cent7_m_hp1 /var/lib]#
-- 위 실행 후 /var/lib/pcsd/tokens 파일에 토큰 정보 생성
8. 'Main_Cluster'라는 cluster를 만들고, corosync config를 node간에 동기화(master node에서)
[root@cent7_m_hp1 /var/lib]# pcs cluster setup --name Main_Cluster ha-master.exam.com ha-slave.exam.com
Shutting down pacemaker/corosync services...
Redirecting to /bin/systemctl stop pacemaker.service
Redirecting to /bin/systemctl stop corosync.service
Killing any remaining services...
Removing all cluster configuration files...
ha-master.exam.com: Succeeded
ha-slave.exam.com: Succeeded
Synchronizing pcsd certificates on nodes ha-master.exam.com, ha-slave.exam.com...
ha-slave.exam.com: Success
ha-master.exam.com: Success
Restaring pcsd on the nodes in order to reload the certificates...
ha-slave.exam.com: Success
ha-master.exam.com: Success
[root@cent7_m_hp1 /var/lib]#
9. starting cluster(master node에서)
root@cent7_m_hp1 /etc/corosync]# pcs cluster start --all
ha-master.exam.com: Starting Cluster...
ha-slave.exam.com: Starting Cluster...
[root@cent7_m_hp1 /etc/corosync]# pcs status
Cluster name: Main_Cluster
WARNING: no stonith devices and stonith-enabled is not false
Last updated: Wed Aug 31 15:57:03 2016 Last change: Wed Aug 31 15:56:51 2016 by hacluster via crmd on ha-slave.exam.com
Stack: corosync
Current DC: ha-slave.exam.com (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
2 nodes and 0 resources configured
Online: [ ha-master.exam.com ha-slave.exam.com ]
Full list of resources:
PCSD Status:
ha-master.exam.com: Online
ha-slave.exam.com: Online
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
[root@cent7_m_hp1 /etc/corosync]#
[root@cent7_m_hp1 /etc/corosync]# pcs status corosync
Membership information
----------------------
Nodeid Votes Name
1 1 ha-master.exam.com (local)
2 1 ha-slave.exam.com
[root@cent7_m_hp1 /etc/corosync]#
-- cluster를 멈출때
# pcs cluster stop [--all] [node] [...]
-- local node에서 강제로 멈출때
# pcs cluster kill
-- node를 standby mode로 바꿀때, 또는 제거할때
# pcs cluster standby node_name | --all
# pcs cluster unstandby node_name | --all
-- cluster의 모든 config 파일과 내용을 지우고 cluster를 멈출때
# pcs cluster stop
# pcs cluster destroy
10. Disabling STONITH and Ignoring Quorum
-- What is STONITH (Shoot The Other Node In The Head) ?
You will see a warning in the output of pcs status that no STONITH devices are configured
and STONITH is not disabled:
-- What is Quorum?
A cluster has quorum when more than half of the nodes are online.
Pacemaker's default behavior is to stop all resources if the cluster does not have quorum.
However, this does not make sense in a two-node cluster; the cluster will lose quorum if one node fails.
For this tutorial, we will tell Pacemaker to ignore quorum by setting the no-quorum-policy:
[root@cent7_m_hp1 /etc/corosync]# pcs property set stonith-enabled=false
# change to "ignore", it does not need for 2 nodes cluste
[root@cent7_m_hp1 /etc/corosync]# pcs property set no-quorum-policy=ignore
11. 가상IP 구성 및 관리그룹생성(한쪽 node에서만 진행, 주로 master에서)
-- 가상IP 리소스를 추가
-- 이를위해 'ocf:heartbeat:IPaddr2' (기본임) 리소스 agent를 구성,
-- 모든 리소스 agent는 2 ~ 3개의 필드로 이루어짐.
-- 첫째 - resource class, OCF (Open Cluster Framework)
둘째 - 표준에 의존
셋째 - resource agent 이름
-- 다음은 'Main_VIP' 라는 resource를 VIP:192.168.56.202, netmask-32bit, 모티너링 interval - 10초 로 생성
-- 'Sub_VIP', 'Etc_VIP' 는 테스트를 위해 생성하는 것임.
[root@cent7_m_hp1 /etc/corosync]# pcs resource create Main_VIP IPaddr2 ip=192.168.56.202 cidr_netmask=32 op monitor interval=10s
[root@cent7_m_hp1 /etc/corosync]# pcs resource create Sub_VIP IPaddr2 ip=192.168.56.203 cidr_netmask=32 op monitor interval=10s
[root@cent7_m_hp1 /etc/corosync]# pcs resource create Etc_VIP IPaddr2 ip=192.168.56.204 cidr_netmask=32 op monitor interval=10s
[root@cent7_m_hp1 /etc/corosync]# pcs status
-- 그룹 만들기
[root@cent7_m_hp1 /etc/corosync]# pcs resource group add VIP_Group Main_VIP Sub_VIP Etc_VIP
[root@cent7_m_hp1 /etc/corosync]# pcs status
Cluster name: Main_Cluster
Last updated: Thu Sep 1 15:22:36 2016 Last change: Thu Sep 1 15:19:53 2016 by root via cibadmin on ha-master.exam.com
Stack: corosync
Current DC: ha-slave.exam.com (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
2 nodes and 3 resources configured
Online: [ ha-master.exam.com ha-slave.exam.com ]
Full list of resources:
Resource Group: VIP_Group
Main_VIP (ocf::heartbeat:IPaddr2): Started ha-master.exam.com
Sub_VIP (ocf::heartbeat:IPaddr2): Started ha-master.exam.com
Etc_VIP (ocf::heartbeat:IPaddr2): Started ha-master.exam.com
PCSD Status:
ha-master.exam.com: Online
ha-slave.exam.com: Online
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
[root@cent7_m_hp1 /etc/corosync]#
12. slave node에서도 상태확인
[root@cent7_s_hp2 /etc/corosync]# pcs status
13. 부팅시 서비스 시작을 위해 등록(모든 node에 적용)
[root@cent7_m_hp1 /etc/corosync]# systemctl enable pcsd
[root@cent7_m_hp1 /etc/corosync]# systemctl enable corosync
[root@cent7_m_hp1 /etc/corosync]# systemctl enable pacemaker
14. 각 config file 확인
[root@cent7_m_hp1 /etc/corosync]# more corosync.conf
[root@cent7_m_hp1 /var/lib/pcsd]# more pcs_settings.conf
[root@cent7_m_hp1 /var/lib/pcsd]# more pcs_users.conf
[root@cent7_m_hp1 /var/lib/pcsd]# more tokens
15. 가상 IP 확인
-- 실제 interface에는 올라오지 않음.
[root@cent7_m_hp1 /var/lib/pcsd]# ip addr
...
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:ce:a2:a1 brd ff:ff:ff:ff:ff:ff
inet 192.168.56.131/24 brd 192.168.56.255 scope global enp0s8
valid_lft forever preferred_lft forever
inet 192.168.56.202/32 brd 192.168.56.255 scope global secondary enp0s8
valid_lft forever preferred_lft forever
[root@cent7_m_hp1 /var/lib/pcsd]#
[root@cent7_m_hp1 /var/lib/pcsd]# ifconfig
...
enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.56.131 netmask 255.255.255.0 broadcast 192.168.56.255
ether 08:00:27:ce:a2:a1 txqueuelen 1000 (Ethernet)
RX packets 46174 bytes 5641875 (5.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 64794 bytes 8655261 (8.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@cent7_m_hp1 /var/lib/pcsd]#
16. haproxy 로드밸런서 resource 를 만들고 그룹으로 등록
-- HA 부하분산으로 VIP그룹과 LB그룹이 서로 다른 node에서 실행됨.
-- 제약조건으로 이를 한곳에서 실행할 수있게 제한
[root@cent7_m_hp1 /etc/corosync]# pcs resource create HAProxy_LB systemd:haproxy op monitor interval=10s
[root@cent7_m_hp1 /etc/corosync]# pcs resource group add LB_Group HAProxy_LB
[root@cent7_m_hp1 /etc/corosync]# pcs status
Cluster name: Main_Cluster
Last updated: Thu Sep 1 15:30:16 2016 Last change: Thu Sep 1 15:28:04 2016 by root via cibadmin on ha-master.exam.com
Stack: corosync
Current DC: ha-slave.exam.com (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
2 nodes and 4 resources configured
Online: [ ha-master.exam.com ha-slave.exam.com ]
Full list of resources:
Resource Group: VIP_Group
Main_VIP (ocf::heartbeat:IPaddr2): Started ha-master.exam.com
Sub_VIP (ocf::heartbeat:IPaddr2): Started ha-master.exam.com
Etc_VIP (ocf::heartbeat:IPaddr2): Started ha-master.exam.com
Resource Group: LB_Group
HAProxy_LB (systemd:haproxy): Started ha-slave.exam.com
PCSD Status:
ha-master.exam.com: Online
ha-slave.exam.com: Online
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
[root@cent7_m_hp1 /etc/corosync]#
-- 리소스 그룹 만들기
-- 그룹에 속한 리소스들은 순차적으로 실행되고, 역순으로 종료한다.
1)만들기
# pcs resource group add group_name resource_id...
2)제거하기(주의, 리소스아이디가 없으면 그룹자체가 지워짐)
# pcs resource group remove group_name resource_id...
3)그룹리스트 보기
# pcs resource group list
18. 제약조건 생성
-- 필요에 의해 같은 곳에 있어야하는 리소스는 제약조건을 걸어야 함.
-- 모든 VIP 리소스가 LB리소스와 연동되는 상황
-- VIP 각 리소스에 HAProxy_LB를 매핑할 필요 없이 최초 하나의 VIP와 연결시키면 됨.
[root@cent7_m_hp1 /etc/corosync]# pcs constraint colocation add HAProxy_LB with Main_VIP score=INFINITY
-- 제약 체크 순서는 VIP -> LV 데몬으로 한다.
-- 맨 마지막 VIP와 LB를 연결해 관리
[root@cent7_m_hp1 /etc/corosync]# pcs constraint order set Main_VIP Sub_VIP Etc_VIP
[root@cent7_m_hp1 /etc/corosync]# pcs constraint order set Etc_VIP HAProxy_LB
-- 마스터IP(location)에서 주된 서비스를 하도록 설정(일부는 반대로 slave에서 하도록 할 수도 있음)
[root@cent7_m_hp1 /etc/corosync]# pcs constraint location Main_VIP prefers ha-master.exam.com=INFINITY
[root@cent7_m_hp1 /etc/corosync]# pcs constraint location Sub_VIP prefers ha-master.exam.com=INFINITY
[root@cent7_m_hp1 /etc/corosync]# pcs constraint location Etc_VIP prefers ha-master.exam.com=INFINITY
[root@cent7_m_hp1 /etc/corosync]# pcs constraint location HAProxy_LB prefers ha-master.exam.com=INFINITY
-- order 제약 삭제[root@cent7_m_hp1 /etc/corosync]# pcs constraint order remove [constraint id-- 제약 확인
[root@cent7_m_hp1 /etc/corosync]# pcs constraint --full
18. 리소스 이동
1) 한 노드에서 동작중인 모든 리소스들을 다른 노드로 이동시키려면, 해당 노드를 standby mode로 변경한다.
[root@cent7_m_hp1 /etc/haproxy]# pcs cluster standby node_name
리소스가 이동한다음 해당 cluster node를 다시 회복시킨다.
[root@cent7_m_hp1 /etc/haproxy]# pcs cluster unstandby node_name
2) 하나의 동작중인 리소스를 옮기려면, pcs resource move resource_id [destination_node] 를 이용
-- 리소스가 이동하고나면, 관련 모든 제약조건은 자동으로 삭제된다.
[root@cent7_m_hp1 /etc/haproxy]# pcs resource move Etc_VIP ha-slave.exam.com
[root@cent7_m_hp1 /etc/haproxy]# pcs resource relocate run Etc_VIP
[root@cent7_m_hp1 /etc/haproxy]# pcs resource relocate clear
[root@cent7_m_hp1 /etc/haproxy]# pcs resource relocate show
-------------- 기타 -----------------------------------
1. cluster node stop
[root@cent7_m_hp1 /var/lib/pcsd]# pcs cluster stop ha-slave.exam.com
-- stop 이 잘 안될경우
1) # pcs cluster disable 대상_노드이름 <-- disable 시킴
2) # pcs status <-- 상태 및 에러확인
3) # pcs resource cleanup <-- 에러 메시지 제거
# pcs resource cleanup HAProxy --node proxy24) # pcs status <-- 다시확인
5) # pcs cluster stop ha-slave.exam.com
6) # pcs status <-- 최종확인
2) 새로운 cluster resource를 생성
[root@cent7_m_hp1 /var/lib/pcsd]# pcs resource create Web_VIP ocf:heartbeat:IPaddr2 ip=192.168.56.203 cidr_netmask=32 op monitor interval=10s
[root@cent7_m_hp1 /var/lib/pcsd]#
[root@cent7_m_hp1 /var/lib/pcsd]# pcs status
Cluster name: MainWeb_VIP
WARNING: corosync and pacemaker node names do not match (IPs used in setup?)
Last updated: Mon Aug 29 13:15:06 2016 Last change: Mon Aug 29 13:15:04 2016 by root via cibadmin on cent7_m_hp1
Stack: corosync
Current DC: cent7_s_hp2 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
2 nodes and 2 resources configured
Online: [ cent7_m_hp1 cent7_s_hp2 ]
Full list of resources:
Cluster_VIP (ocf::heartbeat:IPaddr2): Started cent7_s_hp2
Web_VIP (ocf::heartbeat:IPaddr2): Started cent7_m_hp1
PCSD Status:
cent7_m_hp1 (192.168.56.131): Online
cent7_s_hp2 (192.168.56.132): Online
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/disabled
[root@cent7_m_hp1 /var/lib/pcsd]#
3) cluster node중 하나를 삭제
[root@cent7_m_hp1 /var/lib/pcsd]# pcs cluster node remove node_name
4) 삭제 node 다시 추가
-- node를 추가해도 관련 데몬들은 시작을 안함.
-- 추가된 노드를 시작시켜 줘야함.
[root@cent7_m_hp1 /etc/corosync]# pcs cluster node add node_name
[root@cent7_m_hp1 /etc/corosync]# pcs cluster start node_name
-- 기타 명령어 --
1. 상태확인
-- 두 서버가 online 상태인지 확인
[root@cent7_m_L1 /var/lib/pcsd]# pcs status
2. config error 체크
[root@cent7_m_L1 /etc/corosync]# crm_verify -L -V
error: unpack_resources: Resource start-up disabled since no STONITH resources have been defined
error: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option
error: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity
Errors found during check: config not valid
-- 위와 같은 에러발생시
-- no-quorum-policy=ignore 설정 안할 경우 cluster 간 failover가 부자연스러울 수 있음
[root@cent7_m_L1 /etc/corosync]# pcs property set stonith-enabled=false
[root@cent7_m_L1 /etc/corosync]# pcs property set no-quorum-policy=ignore
[root@cent7_m_L1 /etc/corosync]# crm_verify -L -V
[root@cent7_m_L1 /etc/corosync]# pcs property
Cluster Properties:
cluster-infrastructure: corosync
cluster-name: WebCluster
dc-version: 1.1.13-10.el7_2.4-44eb2dd
have-watchdog: false
no-quorum-policy: ignore
stonith-enabled: false
3. 서비스(리소스 or 가상아이피) 등록
-- master node에서만 실행
[root@cent7_m_L1 /etc/corosync]# pcs resource create MainWebVirtualIP IPaddr2 ip=192.168.56.201 cidr_netmask=32
4. 리소스 전체 확인
[root@cent7_m_hp1 /etc/corosync]# pcs resource --full
5. 리소스 수정
[root@cent7_m_hp1 /etc/corosync]# pcs resource update Cluster_VIP ip=192.168.56.202 cidr_netmask=32 op monitor interval=10s
6. 제약조건
[root@cent7_m_hp1 /etc/corosync]# pcs constraint colocation add HAProxy_Main HAProxy_Main_VIP INFINITY
[root@cent7_m_hp1 /etc/corosync]# pcs constraint order HAProxy_Main_VIP then HAProxy_Main
[root@cent7_m_hp1 /etc/corosync]# pcs constraint location HAProxy_Main_VIP prefers node_id
[root@cent7_m_hp1 /etc/corosync]# pcs constraint location HAProxy_Main prefers node_id
-- 확인
[root@cent7_m_hp1 /etc/corosync]# pcs constraint
[root@cent7_m_hp1 /etc/corosync]# pcs constraint list
[root@cent7_m_hp1 /etc/corosync]# pcs constraint list --full
7. Enable, disable resource
-- pcs resource disable resource_id [--wait[=n]]
-- pcs resource enable resource_id [--wait[=n]]
[root@cent7_m_hp1 /etc/haproxy]# pcs resource disable HAProxy_Main
[root@cent7_m_hp1 /etc/haproxy]# pcs resource enable HAProxy_Main
[root@cent7_m_hp1 /etc/haproxy]# pcs resource disable HAProxy_Main --wait=5
[root@cent7_m_hp1 /etc/haproxy]# pcs resource enable HAProxy_Main
8. pcs status - fail action 메시지 clear
-- pcs status 에 나오는 fail action 메시지 제거
[root@cent7_m_hp1 /etc/haproxy]# pcs resource cleanup
2017년 6월 27일 화요일
Clustering with Pacemaker, DRBD and GFS2 on Bare-Metal servers in SoftLayer
출처: https://icicimov.github.io/blog/high-availability/Clustering-with-Pacemaker-DRBD-and-GFS2-on-Bare-Metal-servers-in-SoftLayer/
Softlayer is IBM company providing cloud and Bare-Metal hosting services. We are going to setup a cluster of Pacemaker, DRBD and GFS2 on couple of Bare-Metal servers to host our Encompass services. This will provide high availability of the shared storage for our applications.
The services are running on two 2U Supermicro 2 x Hexa Core (6 cores per cpu = 24 cpu’s in total due to hyper threading) Intel Xeon 2650 bare-metal servers with 64GB of RAM and Ubuntu-14.04.4 server minimal install for OS and 4 x 1TB hard drives. The root file system is on one 1TB SATA drive and the other 3 x 1TB are in hardware RAID5 array via LSI controller, to be used for the shared storage.
The shared file system resides on the 2TB RAID5 SATA array and is kept in sync via DRBD (on top of LVM for easy extension) block level replication and GFS2 clustered file system. The DRBD and GFS2 are managed as resources by Pacemaker. The below ASCII chart might describe this layout better:
+----------+ +----------+ +----------+ +----------+
| Service | | Service | | Service | | Service |
+----------+ +----------+ +----------+ +----------+
|| || || ||
+------------------------+ cluster FS +------------------------+
| gfs2 |<~~~~~~~~~~~>| gfs2 |
+------------------------+ replication +------------------------+
| drbd r0 |<~~~~~~~~~~~>| drbd r0 |
+------------------------+ +------------------------+
| lv_vol | | lv_vol |
+------------------------+ +------------------------+
| volume group vg1 | | volume group vg1 |
+------------------------+ +------------------------+
| physical volume | | physical volume |
+------------------------+ +------------------------+
| sdb1 | | sdb1 |
+------------------------+ +------------------------+
server01 server02
SoftLayer gives you one public and one private VLAN to connect your server for which you can opt for 0.1, 1 or 10 Gbps throughput. Each server has bond of 2 interfaces connected to each VLAN for HA and fail-over plus one IPMI/KVM BCM interface connected to the private VLAN.
Disk Setup
We have 3 x 1TB SATA3 disks in RAID5 =~ 2TB usable space. I have created the following partitions on the RAID5 block device
/dev/sdb (using GPT partition table since it’s 2TB disk):root@server01:~# gdisk -l /dev/sdb
GPT fdisk (gdisk) version 0.8.8
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR; using GPT.
Disk /dev/sdb: 3904897024 sectors, 1.8 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 18E19822-8B06-460E-B2C4-A98E63C284FD
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 3904896990
Partitions will be aligned on 2048-sector boundaries
Total free space is 2604662717 sectors (1.2 TiB)
Number Start (sector) End (sector) Size Code Name
1 2048 524290047 250.0 GiB 8300 Linux filesystem
2 524290048 1048578047 250.0 GiB 8300 Linux filesystem
3 1048578048 1300236287 120.0 GiB 8300 Linux filesystem
For optimal performance we need to find the Strip size of the RAID5 array of the volume we will create the file system on:
root@server01:~# storcli /c0/v1 show all | grep Strip
Strip Size = 256 KB
So the strip size is 256KB and we have 2 data disks in RAID5 so we ned to take this into account when creating the LVM and the file system.
For the shared file system I used the first partition to create LVM of size 200GB leaving around 20% for snapshots:
[ALL]:~# pvcreate --dataalignment 512K /dev/sdb1
Physical volume "/dev/sdb1" successfully created
where
dataalignment is calculated as Strip size * No. Data disks. To check the data alignment we can run:root@server01:~# pvs -o +pe_start /dev/sdb1
Next we create the VG and LV:
[ALL]:~# vgcreate -A y vg_drbd0 /dev/sdb1
Volume group "vg_drbd0" successfully created
[ALL]:~# lvcreate --name lv_drbd0 -L 200G vg_drbd0
Logical volume "lv_drbd0" created
At the end we need to tell LVM where to look for logical volumes and which devices to skip:
[ALL]:~# vi /etc/lvm/lvm.conf
...
filter = [ "r|^/dev/drbd.*$|", "a|^/dev/sda.*$|", "a|^/dev/sdb.*$|", "r/.*/" ]
write_cache_state = 0
...
and we also turn off the LVM write cache to avoid another caching level. Then we need to update the
ramdisk in order to synchronize the initramfs’s copy of lvm.conf with the main system one:[ALL]:~# # update-initramfs -u
update-initramfs: Generating /boot/initrd.img-3.13.0-86-generic
otherwise devices might go missing upon reboot.
Services Setup
We start by updating the kernel and the packages and installing the needed software:
[ALL]:~# aptitude update && aptitude safe-upgrade -y && shutdown -r now
[ALL]:~# aptitude install -y heartbeat pacemaker corosync fence-agents openais cluster-glue resource-agents xfsprogs lvm2 gfs2-utils dlm
[ALL]:~# aptitude install -y linux-headers build-essential module-assistant flex debconf-utils docbook-xml docbook-xsl dpatch xsltproc autoconf2.13 autoconf debhelper git
I also setup DNS names for the private VLAN ip’s in the
/etc/hosts file:...
10.10.10.91 sl01.private
10.10.10.26 sl02.private
Now we can go on and configure our services.
Clustering Components
For this to work properly we must set passwordless access for the root user on the private VLAN. We generate SSH keys on both servers:
[ALL]:~# ssh-keygen -t rsa -b 2048 -f ~/.ssh/id_rsa -N ''
and copy-paste the public key into the others server
/root/.ssh/authorized_keys file or use ssh-copy-id for that purpose.Corosync
We start by generating private key on one of the servers and copying it over to the other:
root@server01:~# corosync-keygen -l
root@server01:~# scp /etc/corosync/authkey server02.private:/etc/corosync/authkey
In this way, for added security, only a server that has this key can join the cluster communication. Next is the config file
/etc/corosync/corosync.conf:totem {
version: 2
# How long before declaring a token lost (ms)
token: 3000
# How many token retransmits before forming a new configuration
token_retransmits_before_loss_const: 10
# How long to wait for join messages in the membership protocol (ms)
join: 60
# How long to wait for consensus to be achieved before starting a new round of membership configuration (ms)
consensus: 3600
# Turn off the virtual synchrony filter
vsftype: none
# Number of messages that may be sent by one processor on receipt of the token
max_messages: 20
# Limit generated nodeids to 31-bits (positive signed integers)
clear_node_high_bit: yes
# Disable encryption
secauth: off
# How many threads to use for encryption/decryption
threads: 0
# Optionally assign a fixed node id (integer)
# nodeid: 1234
# CLuster name, needed for GFS2 and DLM or DLM wouldn't start
cluster_name: slcluster
# This specifies the mode of redundant ring, which may be none, active, or passive.
rrp_mode: none
interface {
# The following values need to be set based on your environment
ringnumber: 0
bindnetaddr: 10.10.10.91
mcastaddr: 226.94.1.1
mcastport: 5405
}
transport: udpu
}
nodelist {
node {
ring0_addr: 10.10.10.91
nodeid: 1
}
node {
ring0_addr: 10.10.10.26
nodeid: 2
}
}
amf {
mode: disabled
}
quorum {
# Quorum for the Pacemaker Cluster Resource Manager
provider: corosync_votequorum
expected_votes: 1
two_node: 1
wait_for_all: 1
last_man_standing: 1
auto_tie_breaker: 0
}
aisexec {
user: root
group: root
}
logging {
fileline: off
to_stderr: yes
to_logfile: no
to_syslog: yes
syslog_facility: daemon
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
tags: enter|leave|trace1|trace2|trace3|trace4|trace6
}
}
On the other node we replace
bindnetaddr to read bindnetaddr: 10.10.10.26. Then we enable the service on both servers in /etc/default/corosync file:# start corosync at boot [yes|no]
START=yes
and start it up:
[ALL]:~# service corosync start
Confirm all is ok:
root@server02:~# corosync-cfgtool -s
Printing ring status.
Local node ID 2
RING ID 0
id = 10.10.10.26
status = ring 0 active with no faults
root@server02:~# corosync-quorumtool
Quorum information
------------------
Date: Mon May 23 01:46:03 2016
Quorum provider: corosync_votequorum
Nodes: 2
Node ID: 2
Ring ID: 24
Quorate: Yes
Votequorum information
----------------------
Expected votes: 2
Highest expected: 2
Total votes: 2
Quorum: 2
Flags: Quorate
Membership information
----------------------
Nodeid Votes Name
2 1 10.10.10.26 (local)
1 1 10.10.10.91
For the end, we make sure to open
UDP port 5405 in the firewall on the private VLAN interface and make sure the service is enabled on startup:[ALL]# update-rc.d corosync enable
Pacemaker
Since we already installed it all we need to do is start it up:
[ALL]:~# service pacemaker start
then set “no-quorum-policy
to ignore` since this is a 2-node cluster and we want to continue running when one of them crushes (meaning we’ve lost quorum) and disable fencing for now.:root@server01:~# crm configure property stonith-enabled=false
root@server01:~# crm configure property no-quorum-policy=ignore
and then we should see both nodes online if we check the status:
root@server01:~# crm status
Last updated: Mon May 23 01:42:02 2016
Last change: Mon May 23 01:08:41 2016 via cibadmin on server02
Stack: corosync
Current DC: server01 (1) - partition with quorum
Version: 1.1.10-42f2063
2 Nodes configured
2 Resources configured
Online: [ server01 server02 ]
Last, we enable the Pacemaker service on startup and make sure it starts after Corosync:
[ALL]# update-rc.d -f pacemaker remove
[ALL]# update-rc.d pacemaker start 50 1 2 3 4 5 . stop 01 0 6 .
[ALL]# update-rc.d pacemaker enable
Fencing
To make sure the cluster functions properly we need to configure some kind of fencing. This is to prevent
split-brain situation in case of partitioned cluster. In Pacemaker terms this is called STONITH (Shoot The Other Node In The Head) and we’ll be using the IPMI-over-lan device we saw configured above. On one node only we do:root@server01:~# crm configure
crm(live)configure# primitive p_fence_server01 stonith:fence_ipmilan \
pcmk_host_list="server01" ipaddr="10.10.10.52" \
action="reboot" login="<my-admin-user>" passwd="<my-admin-password>" delay=15 \
op monitor interval="60s"
crm(live)configure# primitive p_fence_server02 stonith:fence_ipmilan \
params pcmk_host_list="server02" ipaddr="10.10.10.71" \
action="reboot" login="<my-admin-user>" passwd="<my-admin-password>" delay=5 \
op monitor interval=60s
crm(live)configure# location l_fence_server01 p_fence_server01 -inf: server01
crm(live)configure# location l_fence_server02 p_fence_server02 -inf: server02
crm(live)configure# property stonith-enabled="true"
crm(live)configure# commit
crm(live)configure# exit
root@server01:~#
Now if we check the cluster state we can see our new fencing resources configured:
root@server01:~# crm status
Last updated: Mon May 23 01:42:02 2016
Last change: Mon May 23 01:08:41 2016 via cibadmin on server02
Stack: corosync
Current DC: server01 (1) - partition with quorum
Version: 1.1.10-42f2063
2 Nodes configured
2 Resources configured
Online: [ server01 server02 ]
p_fence_server01 (stonith:fence_ipmilan): Started server02
p_fence_server02 (stonith:fence_ipmilan): Started server01
DRBD
I built DRBD kernel module and the utilities for the current running kernel
3.13.0-86-generic from the current git repository. For DRBD utils:[ALL]:~# git clone --recursive git://git.drbd.org/drbd-utils.git
[ALL]:~# cd drbd-utils/
[ALL]:~/drbd-utils# ./autogen.sh
[ALL]:~/drbd-utils# ./configure --prefix=/usr --localstatedir=/var --sysconfdir=/etc \
--with-pacemaker=yes --with-heartbeat=yes --with-rgmanager=yes \
--with-xen=yes --with-bashcompletion=yes
[ALL]:~/drbd-utils# make
[ALL]:~/drbd-utils# debuild -i -us -uc -b
And for the kernel driver:
[ALL]:~# git clone --recursive git://git.drbd.org/drbd-8.4.git
[ALL]:~# cd drbd-8.4
[ALL]:~/drbd-8.4# git checkout drbd-8.4.7
[ALL]:~/drbd-8.4# make && make clean
[ALL]:~/drbd-8.4# debuild -i -us -uc -b
This has created
.deb packages in the parent directory of the current working directory. All is left is to install them:[ALL]:~/drbd-8.4# dpkg -i ../drbd-dkms_8.4.1-1_all.deb ../drbd-utils_8.9.6-1_amd64.deb
At the end we pin the kernel so we don’t accidentally run upgrade:
[ALL]:~/drbd-8.4# vi /etc/apt/preferences.d/kernel
Package: linux-generic linux-headers-generic linux-image-generic linux-restricted-modules-generic
Pin: version 3.13.0-86
Pin-Priority: 1001
To confirm the installation we run:
root@server01:~# modinfo drbd
filename: /lib/modules/3.13.0-86-generic/updates/drbd.ko
alias: block-major-147-*
license: GPL
version: 8.4.7-2
description: drbd - Distributed Replicated Block Device v8.4.7-2
author: Philipp Reisner <phil@linbit.com>, Lars Ellenberg <lars@linbit.com>
srcversion: 74731AD693E4C2E56E1C448
depends: libcrc32c
vermagic: 3.13.0-86-generic SMP mod_unload modversions
parm: minor_count:Approximate number of drbd devices (1-255) (uint)
parm: disable_sendpage:bool
parm: allow_oos:DONT USE! (bool)
parm: proc_details:int
parm: enable_faults:int
parm: fault_rate:int
parm: fault_count:int
parm: fault_devs:int
parm: usermode_helper:string
root@server01:~# drbdadm --version
DRBDADM_BUILDTAG=GIT-hash:\ c6e62702d5e4fb2cf6b3fa27e67cb0d4b399a30b\ build\ by\ ubuntu@server01\,\ 2016-05-23\ 05:30:41
DRBDADM_API_VERSION=1
DRBD_KERNEL_VERSION_CODE=0x080407
DRBDADM_VERSION_CODE=0x080906
DRBDADM_VERSION=8.9.6
Now we can start with the configuration, first is the common config file
/etc/drbd.d/global_common.conf on one server only:global {
usage-count no;
# minor-count dialog-refresh disable-ip-verification
}
common {
handlers {
# These are EXAMPLE handlers only.
# They may have severe implications,
# like hard resetting the node under certain circumstances.
# Be careful when chosing your poison.
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
# Hook into Pacemaker's fencing
fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";
# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
}
startup {
# wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
wfc-timeout 300;
degr-wfc-timeout 120;
outdated-wfc-timeout 120;
}
options {
# cpu-mask on-no-data-accessible
on-no-data-accessible io-error;
#on-no-data-accessible suspend-io;
}
disk {
# size max-bio-bvecs on-io-error fencing disk-barrier disk-flushes
# disk-drain md-flushes resync-rate resync-after al-extents
# c-plan-ahead c-delay-target c-fill-target c-max-rate
# c-min-rate disk-timeout
fencing resource-and-stonith;
# Setup syncer rate, start with 30% and let the dynamic planer do the job by
# letting it know our network parameters (1Gbps), and c-fill-target which is
# calucated as BDP x 2 (twice the Bandwith Delay Product)
# used http://www.speedguide.net/bdp.php to find the BDP
resync-rate 33M;
c-max-rate 110M;
c-min-rate 10M;
c-fill-target 16M;
}
net {
# protocol timeout max-epoch-size max-buffers unplug-watermark
# connect-int ping-int sndbuf-size rcvbuf-size ko-count
# allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
# after-sb-1pri after-sb-2pri always-asbp rr-conflict
# ping-timeout data-integrity-alg tcp-cork on-congestion
# congestion-fill congestion-extents csums-alg verify-alg
# use-rle
# Protocol "C" tells DRBD not to tell the operating system that
# the write is complete until the data has reach persistent
# storage on both nodes. This is the slowest option, but it is
# also the only one that guarantees consistency between the
# nodes. It is also required for dual-primary, which we will
# be using.
protocol C;
# Tell DRBD to allow dual-primary. This is needed to enable
# live-migration of our servers.
allow-two-primaries yes;
# This tells DRBD what to do in the case of a split-brain when
# neither node was primary, when one node was primary and when
# both nodes are primary. In our case, we'll be running
# dual-primary, so we can not safely recover automatically. The
# only safe option is for the nodes to disconnect from one
# another and let a human decide which node to invalidate.
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
}
}
then we create a resource config file
/etc/drbd.d/r0.res where we utilize previously created LVM:resource r0 {
startup {
# This tells DRBD to promote both nodes to 'primary' when this
# resource starts. However, we will let pacemaker control this
# so we comment it out, which tells DRBD to leave both nodes
# as secondary when drbd starts.
#become-primary-on both;
}
net {
# This tells DRBD how to do a block-by-block verification of
# the data stored on the backing devices. Any verification
# failures will result in the effected block being marked
# out-of-sync.
verify-alg md5;
# This tells DRBD to generate a checksum for each transmitted
# packet. If the data received data doesn't generate the same
# sum, a retransmit request is generated. This protects against
# otherwise-undetected errors in transmission, like
# bit-flipping. See:
# http://www.drbd.org/users-guide/s-integrity-check.html
data-integrity-alg md5;
# Increase send buffer since we are on 1Gbs bonded network
sndbuf-size 512k;
# Improve write performance of the replicated data on the
# receiving node
max-buffers 8000;
max-epoch-size 8000;
}
disk {
# This tells DRBD not to bypass the write-back caching on the
# RAID controller. Normally, DRBD forces the data to be flushed
# to disk, rather than allowing the write-back cachine to
# handle it. Normally this is dangerous, but with BBU-backed
# caching, it is safe. The first option disables disk flushing
# and the second disabled metadata flushes.
disk-flushes no;
md-flushes no;
disk-barrier no;
# In case of error DRBD will operate in diskless mode, and carries
# all subsequent I/O operations, read and write, on the peer node
on-io-error detach;
# Increase metadata activity log to reduce disk writing and
# improve performance
al-extents 3389;
}
volume 0 {
device /dev/drbd0;
disk /dev/mapper/vg_drbd0-lv_drbd0;
meta-disk internal;
}
on server01 {
address 10.10.10.91:7788;
}
on server02 {
address 10.10.10.26:7788;
}
}
To note here is we disable the disk flushes and disk barriers to improve performance since our disk controller has BBU backed volatile cache:
root@server01:~# /opt/MegaRAID/storcli/storcli64 /c0 show all | grep BBU
BBU Status = 0
BBU = Yes
BBU = Present
Cache When BBU Bad = Off
root@server01:~# /opt/MegaRAID/storcli/storcli64 -LDInfo -L1 -aALL -NoLog | grep 'Current Cache Policy'
Current Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache if Bad BBU
Since everything needs to be identical on the second server we simply copy over the files:
root@server01:~# rsync -r /etc/drbd.d/ server02:/etc/drbd.d/
Then on both servers we load the kernel module, create the resource and its meta data and bring the resource up:
[ALL]:~# modprobe drbd
[ALL]:~# drbdadm create-md r0
[ALL]:~# drbdadm up r0
By default both resources will come up as
Secondary so on one node only we make the resource Primary which will trigger the initial disk synchronization:root@server01:~# drbdadm primary --force r0
This can take lots of time depending on the disk size so to speedup the initial sync, on the sync target we run:
root@server02:~# drbdadm disk-options --c-plan-ahead=0 --resync-rate=110M r0
to let it take as much as possible of the 1Gb bandwidth we have. After the initial sync has completed we can make the second node
Primary too:root@server02:~# drbdadm primary r0
and check the final status of the resource:
root@server01:~# cat /proc/drbd
version: 8.4.7-2 (api:1/proto:86-101)
GIT-hash: e0fc2176f53dda5aa32a59e6466af9d9dc6493be build by root@server01, 2016-05-23 02:14:03
0: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
ns:209989680 nr:0 dw:280916 dr:209974404 al:858 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
And to get back to the configured re-sync speed we run on the sync target node:
root@server02:~# drbdadm adjust r0
At the end some settings to reduce latency. Enabling the deadline scheduler as recommended by LinBit:
[ALL]:~# echo deadline > /sys/block/sdb/queue/scheduler
Reduce read I/O deadline to 150 milliseconds (the default is 500ms):
[ALL]:~# echo 150 > /sys/block/sdb/queue/iosched/read_expire
Reduce write I/O deadline to 1500 milliseconds (the default is 3000ms):
[ALL]:~# echo 1500 > /sys/block/sdb/queue/iosched/write_expire
and we also put them in the
/etc/sysctl.conf to make them permanent.GFS2
On one node only, we create the file system:
root@server01:~# mkfs.gfs2 -p lock_dlm -j 2 -t slcluster:slgfs2 /dev/drbd0
This will destroy any data on /dev/drbd0
Are you sure you want to proceed? [y/n]y
Device: /dev/drbd0
Block size: 4096
Device size: 199.99 GB (52427191 blocks)
Filesystem size: 199.99 GB (52427189 blocks)
Journals: 2
Resource groups: 800
Locking protocol: "lock_dlm"
Lock table: "slcluster:slgfs2"
UUID: 701d9bfe-b220-d58a-2734-ad10efc2afdc
where
slcluster is the cluster name we setup in corosync previously:root@server02:~# grep cluster /etc/corosync/corosync.conf
cluster_name: slcluster
and
slgfs2 is an unique file system name. On each node, make the file system mount point and configure it in /etc/fstab for GFS2 daemon to find it on startup: ...
# GFS2/DRBD mount point
UUID=701d9bfe-b220-d58a-2734-ad10efc2afdc /data gfs2 defaults,noauto,noatime,nodiratime,nobootwait 0 0
Finishing off the Cluster Configuration
Now that we have DRBD and DLM configured we can add them to Pacemaker for management. We also add some constraints and ordering so the resources start and stop in proper order and dependencies. When finished with the configuration and all changes are committed Pacemaker will automatically start the services, mount file systems etc. The final Pacemaker config looks like this:
root@server01:~# crm configure show | cat
node $id="1" server01
node $id="2" server02
primitive p_controld ocf:pacemaker:controld \
op monitor interval="60" timeout="60" \
op start interval="0" timeout="90" \
op stop interval="0" timeout="100" \
params daemon="dlm_controld" \
meta target-role="Started"
primitive p_drbd_r0 ocf:linbit:drbd \
params drbd_resource="r0" \
op monitor interval="10" role="Master" \
op monitor interval="20" role="Slave" \
op start interval="0" timeout="240" \
op stop interval="0" timeout="100"
primitive p_fence_server01 stonith:fence_ipmilan \
params pcmk_host_list="server01" ipaddr="10.10.10.52" action="reboot" login="<my-admin-user>" passwd="<my-admin-password>" delay="15" \
op monitor interval="60s"
primitive p_fence_server02 stonith:fence_ipmilan \
params pcmk_host_list="server02" ipaddr="10.10.10.71" action="reboot" login="<my-admin-user>" passwd="<my-admin-password>" delay="5" \
op monitor interval="60s"
primitive p_fs_gfs2 ocf:heartbeat:Filesystem \
params device="/dev/drbd0" directory="/data" fstype="gfs2" options="_netdev,noatime,rw,acl" \
op monitor interval="20" timeout="40" \
op start interval="0" timeout="60" \
op stop interval="0" timeout="60" \
meta is-managed="true"
ms ms_drbd p_drbd_r0 \
meta master-max="2" master-node-max="1" clone-max="2" clone-node-max="1" notify="true" interleave="true"
clone cl_dlm p_controld \
meta globally-unique="false" interleave="true" target-role="Started"
clone cl_fs_gfs2 p_fs_gfs2 \
meta globally-unique="false" interleave="true" ordered="true" target-role="Started"
location l_fence_server01 p_fence_server01 -inf: server01
location l_fence_server02 p_fence_server02 -inf: server02
colocation cl_fs_gfs2_dlm inf: cl_fs_gfs2 cl_dlm
colocation co_drbd_dlm inf: cl_dlm ms_drbd:Master
order o_dlm_fs_gfs2 inf: cl_dlm:start cl_fs_gfs2:start
order o_drbd_dlm_fs_gfs2 inf: ms_drbd:promote cl_dlm:start cl_fs_gfs2:start
property $id="cib-bootstrap-options" \
dc-version="1.1.10-42f2063" \
cluster-infrastructure="corosync" \
no-quorum-policy="ignore" \
stonith-enabled="true" \
last-lrm-refresh="1464141632"
rsc_defaults $id="rsc-options" \
resource-stickiness="100" \
migration-threshold="3"
Now we can disable the drbd service from autostart since Pacemaker will take care of that for us:
[ALL]# update-rc.d drbd disable
Some useful commands we can run to check and confirm the status of all resources in Pacemaker:
root@server02:~# crm_mon -Qrf1
Stack: corosync
Current DC: server01 (1) - partition with quorum
Version: 1.1.10-42f2063
2 Nodes configured
8 Resources configured
Online: [ server01 server02 ]
Full list of resources:
p_fence_server01 (stonith:fence_ipmilan): Started server02
p_fence_server02 (stonith:fence_ipmilan): Started server01
Master/Slave Set: ms_drbd [p_drbd_r0]
Masters: [ server01 server02 ]
Clone Set: cl_dlm [p_controld]
Started: [ server01 server02 ]
Clone Set: cl_fs_gfs2 [p_fs_gfs2]
Started: [ server01 server02 ]
Migration summary:
* Node server02:
* Node server01:
The DLM lock manager has its own tool as well:
root@server02:~# dlm_tool status
cluster nodeid 2 quorate 1 ring seq 24 24
daemon now 262695 fence_pid 0
node 1 M add 262497 rem 0 fail 0 fence 0 at 0 0
node 2 M add 262497 rem 0 fail 0 fence 0 at 0 0
root@server02:~# dlm_tool ls
dlm lockspaces
name slgfs2
id 0x966db418
flags 0x00000000
change member 2 joined 1 remove 0 failed 0 seq 1,1
members 1 2
Simple check if the GFS2 file system is mounted:
root@server02:~# cat /proc/mounts | grep /data
/dev/drbd0 /data gfs2 rw,noatime,acl 0 0
And maybe GFS2 overview using one of the GFS2 own tools
gfs2_edit:root@server01:~# gfs2_edit -p sb master /dev/drbd0
Block #16 (0x10) of 52427191 (0x31ff9b7) (superblock)
Superblock:
mh_magic 0x01161970(hex)
mh_type 1 0x1
mh_format 100 0x64
sb_fs_format 1801 0x709
sb_multihost_format 1900 0x76c
sb_bsize 4096 0x1000
sb_bsize_shift 12 0xc
master dir: 2 0x2
addr: 134 0x86
root dir : 1 0x1
addr: 133 0x85
sb_lockproto lock_dlm
sb_locktable slcluster:slgfs2
sb_uuid 701d9bfe-b220-d58a-2734-ad10efc2afdc
The superblock has 2 directories
1/1 [00000000] 1/133 (0x1/0x85): Dir root
2/2 [00000000] 2/134 (0x2/0x86): Dir master
------------------------------------------------------
Block #134 (0x86) of 52427191 (0x31ff9b7) (disk inode)
-------------- Master directory -----------------
Dinode:
mh_magic 0x01161970(hex)
mh_type 4 0x4
mh_format 400 0x190
no_formal_ino 2 0x2
no_addr 134 0x86
di_mode 040755(decimal)
di_uid 0 0x0
di_gid 0 0x0
di_nlink 4 0x4
di_size 3864 0xf18
di_blocks 1 0x1
di_atime 1463999842 0x5742dd62
di_mtime 1463999842 0x5742dd62
di_ctime 1463999842 0x5742dd62
di_major 0 0x0
di_minor 0 0x0
di_goal_meta 134 0x86
di_goal_data 134 0x86
di_flags 0x00000201(hex)
di_payload_format 1200 0x4b0
di_height 0 0x0
di_depth 0 0x0
di_entries 8 0x8
di_eattr 0 0x0
Directory block: lf_depth:0, lf_entries:0,fmt:0 next=0x0 (8 dirents).
1/1 [0ed4e242] 2/134 (0x2/0x86): Dir .
2/2 [9608161c] 2/134 (0x2/0x86): Dir ..
3/3 [5efc1d83] 3/135 (0x3/0x87): Dir jindex
4/4 [486eee32] 6/65812 (0x6/0x10114): Dir per_node
5/5 [446811e9] 13/66331 (0xd/0x1031b): File inum
6/6 [1aef248e] 14/66332 (0xe/0x1031c): File statfs
7/7 [b1799d75] 15/66333 (0xf/0x1031d): File rindex
8/8 [6c1c0fed] 16/66353 (0x10/0x10331): File quota
------------------------------------------------------
Cluster testing
Hang the first node and monitor how the second node initiates fencing:
root@server01:~# echo c > /proc/sysrq-trigger
Monitor the logs on the second node:
root@server02:~# tail -f /var/log/syslog
...
May 23 07:21:26 server02 pengine[4342]: warning: process_pe_message: Calculated Transition 17: /var/lib/pacemaker/pengine/pe-warn-3.bz2
May 23 07:21:26 server02 crmd[4343]: notice: te_fence_node: Executing reboot fencing operation (56) on server01 (timeout=60000)
May 23 07:21:26 server02 crmd[4343]: notice: te_rsc_command: Initiating action 69: notify p_drbd_r0_pre_notify_demote_0 on server02 (local)
May 23 07:21:26 server02 stonith-ng[4339]: notice: handle_request: Client crmd.4343.6f0f4fdc wants to fence (reboot) 'server01' with device '(any)'
May 23 07:21:26 server02 stonith-ng[4339]: notice: initiate_remote_stonith_op: Initiating remote operation reboot for server01: c2fb8a55-7d37-479b-a913-42dc30b61e70 (0)
We can see fencing in action and the stalled node being rebooted. We check the cluster state:
root@server02:~# crm status
Last updated: Mon May 23 07:24:21 2016
Last change: Mon May 23 07:21:52 2016 via cibadmin on server02
Stack: corosync
Current DC: server02 (2) - partition WITHOUT quorum
Version: 1.1.10-42f2063
2 Nodes configured
8 Resources configured
Online: [ server02 ]
OFFLINE: [ server01 ]
p_fence_server01 (stonith:fence_ipmilan): Started server02
Master/Slave Set: ms_drbd [p_drbd_r0]
Masters: [ server02 ]
Stopped: [ server01 ]
Clone Set: cl_dlm [p_controld]
Started: [ server02 ]
Stopped: [ server01 ]
Clone Set: cl_fs_gfs2 [p_fs_gfs2]
Started: [ server02 ]
Stopped: [ server01 ]
and can see all is still running on the surviving node.
Cluster Monitoring
We can use the
crm_mon cluster tool for this purpose started in daemon mode on both nodes and managed by Supervisord. We create our /etc/supervisor/conf.d/local.conf file:[program:crm_mon]
command=crm_mon --daemonize --timing-details --watch-fencing --mail-to igorc@encompasscorporation.com --mail-host smtp.mydomain.com --mail-prefix "Pacemaker cluster alert"
process_name=%(program_name)s
autostart=true
autorestart=true
startsecs=0
stopsignal=QUIT
user=root
stdout_logfile=/var/log/crm_mon.log
stdout_logfile_maxbytes=1MB
stdout_logfile_backups=3
stderr_logfile=/var/log/crm_mon.log
stderr_logfile_maxbytes=1MB
stderr_logfile_backups=3
Then we reload
Supervisord and start the process:root@server02:~# supervisorctl reread
crm_mon: available
http-server: changed
root@server02:~# supervisorctl reload
Restarted supervisord
root@server02:~# supervisorctl status
crm_mon RUNNING pid 18259, uptime 0:00:00
The daemon will now send me emails every time the cluster state changes. It can also create a web page if used with
--as-html=/path/to/page parameter for monitoring the state using browser.
피드 구독하기:
글 (Atom)