
$Id: raid5.HOWTO,v 1.6 1998/11/02 22:55:59 jakob Exp $

The RAID5 mini HOWTO

Updated the 2nd of November 1998


This howto is written by Jakob Østergaard (jakob@ostenfeld.dk) based
on a large number of emails between the author and Ingo Molnar
(mingo@chiara.csoma.elte.hu) -- one of the RAID developers --, the
linux-raid mailing list (linux-raid@vger.rutgers.edu) and various
other people.

Some of the information in this HOWTO may seem trivial, if you know
RAID all ready. Just skip those parts.

This HOWTO assumes you are using the 2.0.35 kernel with the
raid19980510-C patch, and the 0.90 version of the raidtools. Both can
be found at  ftp://ftp.kernel.org/pub/linux/daemons/raid/alpha.


The mandatory disclaimer:
**********************************************************************
Although RAID seems stable for me, and stable for many other people,
it may not work for you.  If you loose all your data, it's not my
fault, nor the developers'.  Be very aware, that you use the RAID
software and this information on your own risk!  There is no guarantee
whatsoever, that any of the software, or this information, is in
anyway correct, nor suited for any use whatsoever. Back up all your
data before experimenting with this.  Better safe than sorry!
**********************************************************************


Contents:

* What, how, and why RAID-5 ?
* How should I set up the RAID ?
* Running the RAID
* How do I recover from a disk crash ?
* Credits

---------------------------------------
* What, how, and why RAID-5 ?

As you probably allready know, RAID-5 is a way of combining some
number of disks into one large virtual disk, while maintaining some
redundant information, to hopefully let you recover from a disk crash
with all your information intact.

The setup I base this HOWTO on, is the following:

Two NCR53c8xx Fast-SCSI controllers.
Six Quantum Fireball ST 6GB disks (three on each controller)

You can run RAID on IDE disks too, but IDE is not nearly as safe as
SCSI. There is no parity information in the IDE bus, and you can
basically cut several wires in your IDE cable, thus destroying data,
with the IDE controller (and therefore your operating system) never
ever finding out (not before you need the data again, at least).

The disks are quite low-end, but enough of them (on enough
controllers) will run at a fair speed anyway.

The main difference between RAID-{0} and RAID-{1,4,5} is, that these
latter configurations maintain redundant information. This will
degrade performance, since for every write, the RAID subsystem needs
to calculate some parity information, and write this to the disks as
well. If you only want performance, not reliability, you want RAID-0,
not RAID-5.

The main difference between RAID-4 and RAID-5 is, that the latter
configuration distributes the parity information among the disks in
the RAID, while RAID-4 keeps one disk as the ``parity disk''. This
parity disk will then be accessed _every_ time any of the other disks
are written to. Thus, if the parity disk is not very much faster than
the other disks, it will be a bottleneck. Data security is the same in
RAID-4 and RAID-5. RAID-1 is mirroring. This will make one disk a copy
of the other disk. To mirror more disks you can create a RAID-1 over
two RAID-0 or Linear disk sets.

>From here on, I shall only discuss RAID-5. The word ``RAID'' is to be
read as ``RAID-5''.


---------------------------------------
* How should I set up the RAID ?


In theory you could set up the RAID using several partitions on one
drive. This is meaningless however, since it will only give you bad
performance and no redundancy anyway, if a disk holding several
partitions of the RAID crashes.

The partitions do not have to be of the same size (which can also be
impossible, since even disks of the same model can have different
geometries). However, the smallest partition encountered in the RAID
will determine how much space is used on the other partitions. Eg. a
RAID using two 3GB partitions and four 4GB partitions will hold
2x3+4x3-3 = 15 GB.  The space of one disk is used for parity
information, that's why we subtract 3GB.


First of all, you should partition the disks. Just mark the partitions
as ``Linux Native'' for now.


Also, you should have compiled and installed the raidtools, and have a
kernel running with at least the RAID-5 personality compiled in. If
you look in the /proc filesystem, you should have a file called
``mdstat'' and that file should tell you that the raid5 personality is
registered. If the file doesn't exist, or the personality is not
there, you haven't configured your kernel correctly.


Next, we create the raidtab RAID configuration file in
/etc/raidtab. The raidtab file holds information about how your RAID
is set up, eg. what partitions you want to use, what raid-level,
etc. For the RAID I set up, the file looks like this:

raiddev /dev/md0
	raid-level	5
	nr-raid-disks	6
	nr-spare-disks	0
	parity-algorithm	left-symmetric
	chunk-size	32
	device 		/dev/sdb1
	raid-disk	0
	device 		/dev/sdc1
	raid-disk	1
	device 		/dev/sdd1
	raid-disk	2
	device 		/dev/sde1
	raid-disk	3
	device 		/dev/sdf1
	raid-disk	4
	device 		/dev/sdg1
	raid-disk	5

+ First we say what device we're talking about. This is /dev/md0.
+ Then, we state that it should be a RAID-5 array
+ We have 6 disks in the array. You can use from 3 up to ``many''
  disks for RAID-5.
+ I have no spare disks. The spare disks can be used for hot
  reconstruction, eg. when a device fails, the RAID subsystem can
  automatically begin rebuilding the lost (but redundant) data on some
  spare-disk. I have no experience using spare-disks, and I will not
  discuss it further here.
+ The parity-algorithm determines how the parity disk is selected. For
  every write to any disk in the array, some parity information must be
  maintained. I haven't experimented with this, but there are four
  choices available; left-symmetric, right-symmetric, left-asymmetric
  and right-asymmetric. If you don't know what you are doing, go with
  the default, that worked at least for me.
+ The chunk-size determines how much data should be kept contiguous on
  a single disk. This is somewhat analogous to the block-size used in
  the ext2fs filesystem. 32 kilobytes is a fairly small value. If you
  will primarily have large files on your system, you could go for 256
  instead.
+ Finally, all the partitions to be used in the array are
  listed. Every partition has a device name, and a number which is used
  for internal RAID book-keeping.
  You can at all times change the order in which the devices are
  listed, but it is imperative that you do not change the raid-disk
  number associated with each device!


Use ``mkraid /dev/md0'' to begin constructing the RAID.  Hopefully
this completes successfully within a few seconds. You can look in
/proc/mdstat to see that the RAID update process in running, and how
long time it believes it will take for it to complete.

This raid reconstruction runs completely transparently, using idle I/O
bandwidth. The RAID is running and functional. It just doesn't yet
have the redundancy information updated, thus, if you loose a disk
now, you will not be able to reconstruct data, but since the array is
empty, no harm is done.

A very good way to trouble-shoot is to look in the syslog. Go ahead
and see what the kernel tells you.  Even if everything works, it's
great fun to see what the MD (multiple-disks) driver has to say.  Or,
perhaps it's just me...   ;)


With the RAID up running, we should do a format (just like we would do
with any other disk before first use). There is a little performance
trick involved here though.

Assuming you will be using 4KB block size on the array, and that it is
configured with a 32KB chunk-size as described earlier, we realize
that there will be 8 blocks in every chunk (8x4=32). In order to let
mke2fs place the backup super-blocks and possibly other information (I
don't know), we use the -R switch:
 mke2fs -b 4096 -R stride=8 /dev/md0

The drive is now formatted, and when complete, should be ready for use.


Try mounting the array, even though it hasn't finished updating
yet. Eg:
 mkdir /mnt/raid
 mount /dev/md0 /mnt/raid


Now, if everything succeeded, you should now be happily running with a
RAID-5 disk array on your system.  Lucky you.

If you did not succeed, try the following:
1) Check the syslog for useful information
2) Have a look in /proc/mdstat
3) Ask the linux-raid mailing list


---------------------------------------
* Running the RAID


You now have a RAID running. To stop it again, do the following:
Unmount it:  umount /mnt/raid
Stop the RAID subsystem:  raidstop /dev/md0

The RAID can be started again using  raidstart /dev/md0, which will
also cause any pending reconstruction to re-start.


You may want to have your RAID starting at boot-time. This can be
accomplished in a very elegant way. Instead of setting up init-scripts
etc. we simply let the kernel ``autostart'' the RAID when it finds the
partitions involved.

To make this happen, all partitions participating in the RAID should
be marked with type 0xFD, instead of ``Linux Native''.  Use fdisk to
do this. Note that you should type ``fd'' as the partition type, not
0xFD. This is just a fdisk oddity. Furthermore, fdisk shows the
partition type as ``Unknown'' which is perfectly normal.

A sample output from fdisk could look like:

Disk /dev/sdb: 200 heads, 62 sectors, 1020 cylinders
Units = cylinders of 12400 * 512 bytes

   Device Boot   Begin    Start      End   Blocks   Id  System
/dev/sdb1            1        1     1000  6199969   fd  Unknown
/dev/sdb2         1001     1001     1020   124000   fd  Unknown

Here I have two autostart partitions. One which is part of a RAID-5
and one which is part of a small RAID-0. Although you cannot tell
which RAID level the partitions are part of from the above output. The
RAID level and configuration information is stored in the RAID
superblocks initialized with the mkraid command. This is why the
kernel can start your RAIDs before the root filesystem is mounted.

Furthermore, you may have to enable the Autostart option in your
kernel configuration. This is not necessary in all kernels, but see if
you can find the option in your kernel.

When all partitions are marked as type 0xFD, you should be able to put
/dev/md0 in your fstab as you would with any other disk.


Reboot, and see what happens. The syslog on my system says something
like:
...
Oct 22 00:51:59 malthe kernel: SCSI device sdg: hdwr sector= 512
bytes. Sectors= 12657717 [6180 MB] [6.2 GB]
Oct 22 00:51:59 malthe kernel: Partition check:
Oct 22 00:51:59 malthe kernel:  sda: sda1 sda2 sda3 sda4
Oct 22 00:51:59 malthe kernel:  sdb: sdb1 sdb2
Oct 22 00:51:59 malthe kernel:  sdc: sdc1 sdc2
Oct 22 00:51:59 malthe kernel:  sdd: sdd1 sdd2
Oct 22 00:51:59 malthe kernel:  sde: sde1 sde2
Oct 22 00:51:59 malthe kernel:  sdf: sdf1 sdf2
Oct 22 00:51:59 malthe kernel:  sdg: sdg1 sdg2
Oct 22 00:51:59 malthe kernel: autodetecting RAID arrays
Oct 22 00:51:59 malthe kernel: (read) sdb1's sb offset: 6199872
Oct 22 00:51:59 malthe kernel: bind<sdb1,1>
Oct 22 00:51:59 malthe kernel: (read) sdc1's sb offset: 6199872
Oct 22 00:51:59 malthe kernel: bind<sdc1,2>
Oct 22 00:51:59 malthe kernel: (read) sdd1's sb offset: 6199872
Oct 22 00:51:59 malthe kernel: bind<sdd1,3>
Oct 22 00:51:59 malthe kernel: (read) sde1's sb offset: 6199872
Oct 22 00:51:59 malthe kernel: bind<sde1,4>
Oct 22 00:51:59 malthe kernel: (read) sdf1's sb offset: 6205376
Oct 22 00:51:59 malthe kernel: bind<sdf1,5>
Oct 22 00:51:59 malthe kernel: (read) sdg1's sb offset: 6205376
Oct 22 00:51:59 malthe kernel: bind<sdg1,6>
Oct 22 00:51:59 malthe kernel: autorunning md0
Oct 22 00:51:59 malthe kernel: running: <sdg1><sdf1><sde1><sdd1><sdc1><sdb1>
Oct 22 00:51:59 malthe kernel: now!
Oct 22 00:51:59 malthe kernel: md: md0: raid array is not clean --
starting background reconstruction 
...
The very neat thing is, that the file-system layer never sees any of
this. It just sees the (large) disk /dev/md0, mounts it, and carries
on with other things. The RAID reconstruction will run if neede, but
this happens without the users of the RAID knowing it. Again, check
out /proc/mdstat for status information.


As a side note:  You should NEVER EVER thange the partitions of any
disks currently being a part of a running RAID. NOT EVEN partition
types of partitions not being used. First, stop the RAID, then, change
what you have to change.


Although it is possible to eg. build a RAID-1 over two RAID-0 arrays,
it is not yet possible to have the ``top-level'' RAID autostart. This
will be added in a later version of the RAIDtools.


---------------------------------------
* How do I recover from a disk crash ?

If a disk physically crashes, what precisely happens will depend on
how exactly the disk behaves, how your controller sees it, how the
SCSI driver handles the broken device, and how the SCSI layer handles
the whole thing. Chances are, you end up with a locked up system.

If the RAID layer gets to see that a device is broken, it marks it as
broken, and will begin hot reconstruction if any spare disks are
available.

When I deliberately unplugged the SCSI cable from one of the disks
(while running two concurrent disk-benchmarks) the syslog said:
...
Oct 21 20:49:24 malthe kernel: scsi1 : target 2 lun 0 unexpected disconnect
Oct 21 20:49:24 malthe kernel: scsi1 : DCMD|DBC=0x80003c0,
DNAD=0x89dcc40 (virt 0x089dcc40) 
...
Oct 21 20:49:26 malthe kernel: SCSI disk error : host 1 channel 0 id 2
lun 0 return code = 2504 0000
Oct 21 20:49:26 malthe kernel: scsidisk I/O error: dev 08:21, sector
1826592, absolute sector 1 826654
Oct 21 20:49:26 malthe kernel: raid5: Disk failure on sdc1, disabling
device. Operation continuing on 5 devices
...
Oct 21 20:49:26 malthe kernel: md: updating md0 RAID superblock on device
Oct 21 20:49:26 malthe kernel: sdg1(write) sdg1's sb offset: 6205376
Oct 21 20:49:26 malthe kernel: md: recovery thread got woken up ...
Oct 21 20:49:26 malthe kernel: md0: no spare disk to reconstruct
array! -- continuing in degraded mode 
Oct 21 20:49:26 malthe kernel: md: recovery thread finished ...
...
However, the SCSI layer was not able to handle this whole thing, so
the system locked up.

After a reboot, my /mnt/raid filesystem was fully operational, running
on five disks instead of six.

This should be similar to removing a disk that has crashed.


When you plug in a new disk (or in my case, re-connect the
disconnected disk after writing a lot more information to the RAID),
you should see the RAID subsystem automatically beginning to
reconstruct the information that was erased. 

Of course, you should partition the new disk to resemble the one that
went away, before this can happen. Once again, look in /proc/mdstat to
see what is going on!


If the reconstruction will not begin, that is, the raidstart command
fails with an ``/dev/md0: invalid argument'', this may be because of
the order in which the devices are listed in the raidtab file.  You
can try re-ordering the devices. My raidtab file needed this
re-ordering, and ended up looking like:
raiddev /dev/md0
	raid-level	5
	nr-raid-disks	6
	nr-spare-disks	0
	parity-algorithm	left-symmetric
	chunk-size	32
	device 		/dev/sdc1
	raid-disk	1
	device 		/dev/sdb1
	raid-disk	0
	device 		/dev/sdd1
	raid-disk	2
	device 		/dev/sde1
	raid-disk	3
	device 		/dev/sdf1
	raid-disk	4
	device 		/dev/sdg1
	raid-disk	5

REMEMBER! The disks MUST keep their raid-disk numbers from before!


Also, the partition on the new disk should be of at least the size of
the smalles partition in your array (as it was before the disk
crash). There is no way you can magically shrink the array, and have
the filesystem on it resized along with it.

---------------------------------------
* Credits

The following peopled contributed to the creation of this
documentation:

* Ingo Molnar
* Jim Warren
* Louis Mandelstam
* The Linux-RAID mailing list
* The ones I forgot,  sorry   :)
