I want to play with stuff that allows you to build High Available systems. Today I'll start with DRBD. If you are reading this, probably you came here googling something related to DRBD. If not, go to pornotube or something like that and you will find more entertaining stuff. I'm not going to explain what DRBD is just copy-paste for you the definition from its webpage:

DRBD^®^ refers to block devices designed as a building block to form high availability (HA) clusters. This is done by mirroring a whole block device via an assigned network. DRBD can be understood as network based raid-1.

I'll do this small exercise with two virtual machines with the following names/IP's:

  • Host #1. Name = l11. IP Address: 192.168.0.15
  • Host #2. Name = l12. IP Address: 192.168.0.17

It is very important (required!!) to be able to do a ping from one machine to another using its hostname (l11 and l12). This can be achieved by executing this commands in both hosts. Note: from now on ALL the commands executed in this guide are executed as ROOT user. Caution! Note that I'm using ">>" and not ">". Respect it.

root@l11:~# echo "192.168.0.17 l12" >> /etc/hosts

root@l12:~# echo "192.168.0.15 l11" >> /etc/hosts

Both virtual machines are running Debian/Stable (wheezy). This is not coincidence. Right now (2013-09-21) the kernel version of DRBD and the version of the DRBD package on Debian/Testing are different, and you will be unable to configure the whole thing.

[Installing packages]

Let's get started. What about install drbd in both nodes?

aptitude install drbd8-utils

[Warming up - creating a block device]

For the sake of the exercise, lets not use a dedicated storage device (such as a disk or a LVM partition). If you have an spare block device in both machines with the same size, you can use them. Instead we will create a "fake" block device using "dd" and "losetup". Again, in a "real" case you must use a hard drive, but because we are just trying to learn how to use drbd this is not important and we will simulate it.

First of all, create a file of 20MB filled with zeros:

$ dd if=/dev/zero of=/root/drbd_fs bs=1024 count=20000  
20000+0 records in  
20000+0 records out  
20480000 bytes (20 MB) copied, 0.0754812 s, 271 MB/s

We will use the drbd_fs as a block device and we will set up a loopback divece executing:

losetup /dev/loop0 drbd_fs

Now de /dev/loop0 simulates another HDD that could be in /dev/sdb0, for example.

[Configuration file]

This is my configuration file. I haven't spend the time to know wich are the optimal values and I'll do a little bit of research on a future post. Right now I'm just using the default ones trusting that they will be "good enough".

#include "drbd.d/global_common.conf";  
#include "drbd.d/*.res";  
global {  
usage-count no;  
}

common {  
# Speed of syncronization. Speed of the dedicated link:  
syncer {rate 100M;}  
}

resource r1 {  
  # transfer protocol to use.  
  # C: write IO is reported as completed, if we know it has  
  #    reached _both_ local and remote DISK.  
  #    * for critical transactional data.  
  # B: write IO is reported as completed, if it has reached  
  #    local DISK and remote buffer cache.  
  #    * for most cases.  
  # A: write IO is reported as completed, if it has reached  
  #    local DISK and local tcp send buffer. (see also sndbuf-size)  
  #    * for high latency networks  
protocol C;  
startup {  
wfc-timeout 15;  
degr-wfc-timeout 60;  
}

net {  
cram-hmac-alg sha1;  
shared-secret "secret";  
#allow-two-primaries;  
}

on l11 {  
device /dev/drbd0;  
disk /dev/loop0;  
address 192.168.0.15:7788;  
meta-disk internal;  
}

on l12 {  
device /dev/drbd0;  
disk /dev/loop0;  
address 192.168.0.17:7788;  
meta-disk internal;  
}  
}

This file must be EXACTLY THE SAME in both hosts. I recomend you to copy it with SCP:

scp /etc/drbd.conf l12:/etc/drbd.conf

[Initializing everything]

We create (on boths hosts) the device metadata. This must be executed just once.

$ drbdadm create-md r1  
Writing meta data...  
initializing activity log  
NOT initialized bitmap  
New drbd meta data block successfully created.

Load the drbd kernel module (execute on both hosts)

$ modprobe drbd

And enable the resource. This associates the two machines, set the replication parameters, etc... (execute on both hosts):

drbdadm up r1

After that we can check if everything is OK. You should be able to see Secondary/Secondary

$ cat /proc/drbd  
version: 8.3.11 (api:88/proto:86-96)  
srcversion: F937DCB2E5D83C6CCE4A6C9  
0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C
r-----  
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:19960

[Promoting a master]

Now the documentation says the following:

Select an initial sync source. If you are dealing with newly-initialized, empty disk, this choice is entirely arbitrary. If one of your nodes already has valuable data that you need to preserve, however, it is of crucial importance that you select that node as your synchronization source. If you do initial device synchronization in the wrong direction, you will lose that data. Exercise caution.

If I execute the command I get the following error:

drbdadm primary --force r1  
0: State change failed: (-2) Need access to UpToDate data  
Command 'drbdsetup 0 primary' terminated with exit code 17

It seems that neither server can tell if it has the correct data, so we need to invalidate one of them, so the other one thinks it is up-to-date. I'll invalidate the l12 and then promote de l11 as Master/Primary.

root@l12:~# drbdadm invalidate r1

root@l11:~# drbdadm primary --force r1

Now if I issue this command I get that l11 is the primary! Yey!

root@l11:~# cat /proc/drbd  
version: 8.3.11 (api:88/proto:86-96)  
srcversion: F937DCB2E5D83C6CCE4A6C9  
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----  
ns:19960 nr:0 dw:0 dr:20112 al:0 bm:2 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f
oos:0

If we issue the same command in l12 we will get "Secondary/Primary UpToDate/UpToDate".

[Creating the FileSystem]

Right now we have a block device in /dev/drbd0 but this device as no format, yet. In the master host issue this command:

$ mkfs.ext3 /dev/drbd0  
$ mount /dev/drbd0 /mnt  
$ touch /mnt/IT_WORKS

[Basic Manual Fail-over]

This will be the definitive test to prove that we have finally configured DRBD successfully. On the current primary execute the following commants to demote it to secondary:

$ umount /mnt  
$ drbdadm secondary r1

On the node you wish to make primary execute:

$ drbdadm primary r1  
$ mount /dev/drbd0 /mnt  
$ ls -l /mnt  
-rw-r--r--  1 root root    0 Sep 21 18:40 IT_WORKS

YEEEEYY!! It works!!!

[Final thoughts]

Lets be clear. This is not really exciting. You must umount and demote the primary and promote and mount the other host in order to make the failover. This sucks and is slow. We will see in the next posts how to use it with Hearbeat and have a Master-Slave Apache or MySQL. Stay tuned.

Documentation: http://www.drbd.org/users-guide/ch-admin.html