Linux > General Linux
How to Install Replicated File System using GlusterFS on CentsOS 7
(1/1)
akhilu:
In one of our previous post, we have shown how to how to Install Distributed File System using "GlusterFS". You can refer the below link to read more:
https://admin-ahead.com/forum/general-linux/how-to-install-distributed-file-system-'glusterfs'-and-setup/msg1516/#msg1516
In this post, we will see how to set up replicated storage volume using glusterfs. on CentOS 7.
Replicated Glusterfs Volume is like a RAID 1, and volume maintains exact copies of the data on all bricks. You can decide the number of replicas while creating the volume, so you would need to have atleast two bricks to create a volume with two replicas or three bricks to create a volume of 3 replicas.
Terminologies:
Brick: Brick is basic storage (directory) on a server in the trusted storage pool.
Volume: Volume is a logical collection of bricks
Replicated File System: A file system in which data is spread across multiple storages nodes and allows clients to access it over the network.
Server: is a machine where the actual file system is hosted in which the data will be stored.
Client: is a machine which mounts the volume.
glusterd: glusterd is a daemon that runs on all servers in the trusted storage pool.
Requirements:
1) For this demo, I am using 5 CentOS 7 servers all 64 bit. Two of these will act as servers and will maintain two replicas of the volume on each server.
--- Code: ---68.232.175.206 server1
45.77.110.210 server2
144.202.11.51 client1
149.28.45.206 client2
149.28.41.112 client3
--- End code ---
2) Make sure that all the servers in the cluster have a free disk attached to it to create the storage volumes.
--- Code: ---In our case both server1 and server2 has 10 GB free disk attached to it
Disk /dev/vdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
--- End code ---
3) GlusterFS components use DNS for name resolutions. Since I do not have a DNS on my environment I am using below entries in /etc/hosts file on all server
--- Code: ---68.232.175.206 server1
45.77.110.210 server2
144.202.11.51 client1
149.28.45.206 client2
--- End code ---
Install GlusterFS server on server1 and server2
1) First, we need to make sure that epel repository is enabled on both server1 and server2 by running below command:
--- Code: ---yum install epel-release -y
--- End code ---
If epel is already present then the above command will give you a message saying it already exist or else it will install epel repo
2) Create Gluster repository on server1 and server2
--- Code: ---vi /etc/yum.repos.d/Gluster.repo
Add below code in the above file.
[gluster38]
name=Gluster 3.8
baseurl=http://mirror.centos.org/centos/7/storage/$basearch/gluster-3.8/
gpgcheck=0
enabled=1
--- End code ---
Once done run the below command to see if the epel and gluster repository is enabled
--- Code: ---yum repolist
--- End code ---
Output:
--- Code: ---repo id repo name status
base/7/x86_64 CentOS-7 - Base 9,911
epel/x86_64 Extra Packages for Enterprise Linux 7 - x86_64 12,718
extras/7/x86_64 CentOS-7 - Extras 434
glusterfs Glusterfs5 40
updates/7/x86_64 CentOS-7 - Updates 1,614
repolist: 24,717
--- End code ---
3) Run below command on server1 and server2 to install gluster
--- Code: ---yum install glusterfs-server -y
--- End code ---
4) Run the below command to start glusterd and enable it on boot
--- Code: ---systemctl enable glusterd
systemctl start glusterd
--- End code ---
Creating LVM on server1 and server2
We have a 10 GB disk attached on both server1 and server2 which we want to convert to storage brick.
Run below commands on server1 and server2
1) check the disks attached to server1 and server2
--- Code: ---[root@server1 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
vda 253:0 0 25G 0 disk
└─vda1 253:1 0 25G 0 part /
vdb 253:16 0 10G 0 disk
--- End code ---
You can see that we have two disk attached to the server vda and vdb. vdb is totally free right now.
2) Now we need to create a new partition in /dev/vdb on server1 and server2
--- Code: ---[root@server1 ~]# fdisk /dev/vdb
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0xb3065797.
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-20971519, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-20971519, default 20971519):
Using default value 20971519
Partition 1 of type Linux and of size 10 GiB is set
Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
--- End code ---
3) run partprobe command on server1 and server2 to update kernel about new partitions
--- Code: ---[root@server1 ~]# partprobe
[root@server1 ~]#
--- End code ---
4) Create LVM with 10 GB space on both server1 and server2. You can allocate as per your needs
--- Code: ---[root@server1 ~]# pvcreate /dev/vdb1
Physical volume "/dev/vdb1" successfully created.
[root@server1 ~]# vgcreate vg1 /dev/vdb1
Volume group "vg1" successfully created
[root@server1 ~]# lvcreate -l 100%FREE -n lv1 vg1
Logical volume "lv1" created.
--- End code ---
Mounting LVM on server1 and server2 onto the /bricks/brick1 folder
1) Creating brick directory on server1 and server2
--- Code: ---[root@server1 ~]# mkdir -p /bricks/brick1
--- End code ---
2) Setting xfs file system for the LVM created
--- Code: ---[root@server1 ~]# mkfs.xfs /dev/mapper/vg1-lv1
meta-data=/dev/mapper/vg1-lv1 isize=512 agcount=4, agsize=655104 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=2620416, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
--- End code ---
3) Define the mount point in the fstab file
--- Code: ---[root@server1 ~]# vi /etc/fstab
/dev/mapper/vg1-lv1 /bricks/brick1 xfs defaults 0 0
--- End code ---
4) Mount the LVM to the brick folder
--- Code: ---[root@server2 ~]# mount -a
[root@server2 ~]#
--- End code ---
5) Check if the volume is properly mounted
--- Code: ---[root@server2 ~]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/vda1 ext4 25G 1.3G 23G 6% /
devtmpfs devtmpfs 486M 0 486M 0% /dev
tmpfs tmpfs 496M 0 496M 0% /dev/shm
tmpfs tmpfs 496M 13M 483M 3% /run
tmpfs tmpfs 496M 0 496M 0% /sys/fs/cgroup
tmpfs tmpfs 100M 0 100M 0% /run/user/0
/dev/mapper/vg1-lv1 xfs 10G 33M 10G 1% /bricks/brick1
--- End code ---
Create the trusted storage pool in glusterfs
Configure Firewall:
You would need to either disable the firewall or configure the firewall to allow all connections within a cluster.
By default, glusterd will listen on tcp/24007 but opening that port is not enough on the gluster nodes. Each time you will add a brick , it will open a new port (that you’ll be able to see with “gluster volumes status”)
--- Code: ---# Disable FirewallD
systemctl stop firewalld
systemctl disable firewalld
OR
# Run below command on a node in which you want to accept all traffics comming from the source ip
firewall-cmd --zone=public --add-rich-rule='rule family="ipv4" source address="<ipaddress>" accept'
firewall-cmd --reload
--- End code ---
1) Here I will run all GlusterFS commands in server1 node.
--- Code: ---[root@server1 ~]# gluster peer probe server2
peer probe: success.
--- End code ---
2) Verify the status of the trusted storage pool.
--- Code: ---[root@server1 ~]# gluster peer status
Number of Peers: 1
Hostname: server2
Uuid: 4040a194-a30b-43f2-9e8a-a896cd92c37d
State: Peer in Cluster (Connected)
--- End code ---
3) List the storage pool.
--- Code: ---[root@server1 ~]# gluster pool list
UUID Hostname State
4040a194-a30b-43f2-9e8a-a896cd92c37d server2 Connected
22479d4a-d585-4f0c-ad35-cbd5444d3bbd localhost Connected
--- End code ---
Setup gluster volumes
1) Create brick directory on the mounted file systems on both server1 and server2. In my case I am creating directory vol1
--- Code: ---[root@server2 ~]# mkdir /bricks/brick1/vol1
[root@server2 ~]#
--- End code ---
2) Since we are going to use replicated volume, so create the volume named “vol1” with two replicas on server1
--- Code: ---[root@server1 ~]# gluster volume create vol1 replica 2 server1:/bricks/brick1/vol1 server2:/bricks/brick1/vol1
volume create: vol1: success: please start the volume to access data
--- End code ---
3) Start the volume.
--- Code: ---[root@server1 ~]# gluster volume start vol1
volume start: vol1: success
--- End code ---
4) Check the status of the created volume on server1 and server2
--- Code: ---[root@server1 ~]# gluster volume info vol1
Volume Name: vol1
Type: Replicate
Volume ID: 9b7939f5-01a8-487a-8c3e-9dcd734553d5
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: server1:/bricks/brick1/vol1
Brick2: server2:/bricks/brick1/vol1
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
--- End code ---
Setup GlusterFS Client
1) On all client server client1, client2 and client3 run below command to install glusterfs client package to support mounting of glusterfs filesystems.
--- Code: ---yum install -y glusterfs-client
--- End code ---
2) Create a directory to mount the GlusterFS filesystem
--- Code: ---mkdir -p /mnt/glusterfs
--- End code ---
3) Now mount the gluster file system to the above directory on all client servers by adding below entry in /etc/fstab
--- Code: ---[root@client2 ~]# vi /etc/fstab
server1:/vol1 /mnt/glusterfs glusterfs defaults,_netdev 0 0
--- End code ---
4) Run below command on all client-server to mount the gluster volumes
--- Code: ---[root@client1 ~]# mount -a
[root@client1 ~]#
--- End code ---
5) Check if the mount point has been properly mounted.
--- Code: ---[root@client1 ~]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/vda1 ext4 25G 1.3G 23G 6% /
devtmpfs devtmpfs 486M 0 486M 0% /dev
tmpfs tmpfs 496M 0 496M 0% /dev/shm
tmpfs tmpfs 496M 13M 483M 3% /run
tmpfs tmpfs 496M 0 496M 0% /sys/fs/cgroup
tmpfs tmpfs 100M 0 100M 0% /run/user/0
server1:/vol1 fuse.glusterfs 10G 135M 9.9G 2% /mnt/glusterfs
--- End code ---
Testing if the data replication is working accross client and server
1) Since we have mounted the gluster volumes on /mnt/glusterfs folder on both client system we need to move into /mnt/glusterfs and create some test files.
--- Code: ---[root@client1 glusterfs]# touch file{1..5}
[root@client1 glusterfs]# ll
total 0
-rw-r--r-- 1 root root 0 Nov 25 06:40 file1
-rw-r--r-- 1 root root 0 Nov 25 06:40 file2
-rw-r--r-- 1 root root 0 Nov 25 06:40 file3
-rw-r--r-- 1 root root 0 Nov 25 06:40 file4
-rw-r--r-- 1 root root 0 Nov 25 06:40 file5
--- End code ---
2) Now you need to go to the server nodes and see if these files are there in the brick volume that we created /bricks/brick1/vol1/
--- Code: ---[root@server1 vol1]# ll
total 0
-rw-r--r-- 2 root root 0 Nov 25 06:40 file1
-rw-r--r-- 2 root root 0 Nov 25 06:40 file2
-rw-r--r-- 2 root root 0 Nov 25 06:40 file3
-rw-r--r-- 2 root root 0 Nov 25 06:40 file4
-rw-r--r-- 2 root root 0 Nov 25 06:40 file5
--- End code ---
3) It should be also there in server2 and the client 2 and client3 servers.
--- Code: ---[root@server2 vol1]# ll
total 0
-rw-r--r-- 2 root root 0 Nov 25 06:40 file1
-rw-r--r-- 2 root root 0 Nov 25 06:40 file2
-rw-r--r-- 2 root root 0 Nov 25 06:40 file3
-rw-r--r-- 2 root root 0 Nov 25 06:40 file4
-rw-r--r-- 2 root root 0 Nov 25 06:40 file5
--- End code ---
--- Code: ---[root@client2 glusterfs]# ll
total 0
-rw-r--r-- 1 root root 0 Nov 25 06:40 file1
-rw-r--r-- 1 root root 0 Nov 25 06:40 file2
-rw-r--r-- 1 root root 0 Nov 25 06:40 file3
-rw-r--r-- 1 root root 0 Nov 25 06:40 file4
-rw-r--r-- 1 root root 0 Nov 25 06:40 file5
--- End code ---
This type of setup is important for data redundancy. This setup is also efficient while using website with load balancer setup.
I hope you find this information useful :) Thank you for reading.
Navigation
[0] Message Index
Go to full version