1      Install RAID Disk

RAID (Redundant Array of Independent Disks) is a data storage virtualization technology that combines multiple physical disk partitions into a single logical unit for the purposes of data redundancy (reliability), performance improvements, or both.

This script creates a RAID disk (type 1) from two equal size disk partitions and installs / configures email notifications providing RAID status whenever anything (such as disk errors / failure) changes. Combined with regular backups, RAID guarantees that no data will be lost, or, if so, it is easily replaced.

An overview of RAID is available at WikiPedia.

RAID is desirable for mission critical applications where a disk failure does not result in loss of data / services and time is available to replace a failed disk resulting in no loss of data.

RAID disks are intended for extra bulk data storage such as websites, shared files, media (such as ZoneMinder security camera footage), virtual machines and docker containers. SecureOffice boot and rootfs partitions are not part of any RAID array since they are easily replaced from backups in the event of a system disk failure.

1.1                Prerequisites

• SecureOffice installed and configured for a single disk (default installation).
• Subscription to SecureOffice custom package repository for access to the RAID installation script.
• Credentials for an email server to send RAID status notifications.
• Do not use USB, SD or TF disks or any removable media for RAID, performance penalty.
• Choose disk size for RAID disk. 256GB leaves lots of space to grow, 128GB should be adequate for two virtual machines or docker containers. 64GB is enough for one. Any extra storage requirements for /home/data (default bulk storage location) should be factored in.

1.2    Backup Target Disk Data

(Optional) If either of the RAID target partitions / disks contains important data, it needs to be backed up to another location and restored to the RAID array once RAID installation is complete. If multiple partitions need backup / restore, repeat this procedure for each partition.

1.2.1      Backup Procedure

• Assumes source partition is mounted
• Using a SecureOffice SSH session change to the root directory of the mounted target partition. Example: "cd /home/data/".
• Enter "tar -zcvf <backup file directory>/home_data.tar.gz".
• "<backup file directory>" is some other storage location such as a USB disk or mounted network location.
• The "home_data.tar.gz" filename is a matter of preference.

1.2.2      Restore Procedure

• Assumes RAID partition is mounted at "/home/data".
• After the RAID disk is created and mounted, from a SecureOffice SSH session change to the root directory of the mounted RAID disk. Example: "cd /home/data/".
• Enter "tar -zxvf <backup file directory>/home_data.tar.gz" to restore your data to the RAID partition.

1.3                Common System and Data Disks

In default configuration, SecureOffice boot and system partitions are on the same disk as bulk data storage (for virtual machines, websites, file server data, etc) using partition 4 of the system disk. Partition 4 of the system disk will be configured as a RAID disk component.

Requirements:

• Additional SATA hard or SSD disk.
• One free SATA port on motherboard.
• Room and enough power for extra disk in PC enclosure.
• No valuable data on system disk data partition (/dev/sda4), else, backup data to another disk, to be restored to RAID later.
• Size of SecureOffice boot disk data partition (/dev/sda4) is large enough for the desired RAID disk size. If not, replace the system disk with a larger one.
• Size of the additional disk is at least the size of the desired RAID (data) partition (/dev/sda4) plus 16GB (if backing up SecureOffice system partitions - boot, rootfs, swap) else plus 8G (swap). The additional disk can be larger than this, resulting in extra space that can be partitioned for other purposes.
• It is recommended that the size and contents of partitions 1 to 3 (boot, rootfs, swap) of both disks be made identical. This provides a failsafe boot option (select as primary boot disk in BIOS settings) should the primary boot disk fail. The RAID install script will prompt to backup partitions 1 to 3 (from boot disk) to partitions 1 to 3 of the additional disk.
• The size of the RAID disk will be equal to the size of the SecureOffice system disk data partition (/dev/sda4).
• Recommendation: Use equal sized SecureOffice system and additional disk.

Skip to Section 1.5.

1.4                Separate System and Data Disks

In this configuration, SecureOffice boot and system partitions are on a separate device such as mSATA. Bulk data storage (for virtual machines, websites, file servers, data, etc) uses two additional SATA disks configured for RAID. Each additional disk will have two partitions: swap (partition 3) and RAID (partition 4). The size of the RAID partition (using both disks) will be the size of the smaller disk less the size of the swap partition (default: 8GB).

Requirements:

• Two additional SATA hard or SSD disks, sized to meet desired RAID array size plus 8G (swap partition size).
• Two free SATA ports on motherboard.
• Space and enough power for two extra disks in PC enclosure.
• With the SecureOffice PC unpowered, physically install two identical SATA hard or SSD disks.
• Boot SecureOffice to detect the new disks.

1.5                Configure and Install RAID

In subsequent instructions, DRIVE1 and DRIVE2 will be the disks containing the RAID disk components / partitions. They need to be identified for use by the RAID installation script.

If using common system and data disks (you added one disk for RAID), <DRIVE1> will be the existing system disk and <DRIVE2> will be the new disk. Assuming your boot / system disk is "/dev/sda" and your new disk is "/dev/sdb", <DRIVE1> will be "/dev/sda" and <DRIVE2> will be "/dev/sdb".

If two new SATA disks were added for RAID, <DRIVE1> will be the lower (b less than c) new disk. <DRIVE2> will be the higher (c greater than b) new disk. Assuming the two new disks were detected as "/dev/sdb" and "/dev/sdc":

• New swap partitions will be created on "/dev/sdb3" and "/dev/sdc3".
• New RAID partitions will be "/dev/sdb4" and "/dev/sdc4".

Using the data partition (4) of the boot / system disk as a RAID member provides the option (RAID script will prompt) of backing up the system partitions (1 to 3) to the additional disk. This provides a failsafe boot option (select as primary boot disk in BIOS setup) should the system disk fail. To use this option, <DRIVE1> value (below) must be the boot / system disk. This will result in partitions 1 to 3 (boot, rootfs, swap) of both disks being identical, with partition 4 of each disk being RAID members. Note that the system partitions of the additional disk will not be updated (configuration, packages) as the system partitions of the primary disk change. This is the recommended configuration.

• Partitions 1 to 3 of both disks will be identical.
• Partition 4 of boot / system disk will be first RAID member.
• Partition 4 of additional disk will be second RAID member.

Choosing to not backup partitions 1 to 3 of the boot / system disk to the secondary disk will result in following partition layout:

• Partition 4 of boot / system disk will be first RAID member.
• Partition 3 of additional disk will be swap.
• Partition 4 of additional disk will be second RAID member.

The disks (DRIVE1, DRIVE2) corresponding to the two disks to be used for RAID need to be identified, by following these identify disk instructions.

SecureOffice must be connected to the internet from your registered domain. To test this, enter "ping <www.your_domain>". The result should be the WAN IP address of SecureOffice (WAN Topology) or main router (LAN Topology) IP address. If not, DDNS is not working and needs to be configured or checked.

Enter the following (one line) at a SecureOffice command prompt to download the RAID installation script:

"cd /tmp; sget Files/do_raid.sh; chmod +x do_raid.sh" ("sget" is not a typo, it is a script that wraps wget by providing path prefix and download credentials)

The following values at script beginning must be changed to adapt to your installation (nano /tmp/do_raid.sh).

• DRIVE1="The first RAID disk identified above. Eg: /dev/sda". It must be the system disk if using a common disk for system and RAID partition.
• DRIVE2="The second RAID disk identified above. Eg: /dev/sdb"
• RAID_MOUNT="The mount point for RAID disk. Default: /home/data"
• EMAIL_HOST="Your send email server IP or URL"
• EMAIL_PORT="Your send email server port"
• EMAIL_USER="Your user ID for email server login"
• EMAIL_PASSWORD="Your password for email server login"
• EMAIL_FROM="Email address where emails are alleged to be from. Does not have to be real. Example: SecureOffice@yourdomain.com"
• EMAIL_TO="Email address to send RAID notifications to. Example: you@gmail.com"

Enter the following command to install and configure RAID:

"./do_raid.sh"

Answer prompts / questions as follows:

Common system disk and first RAID component / partition:

• "Have you set the configuration values at the beginning of this script? [Y/n]" y
• "Disk partition DRIVE1(4) and entire disk DRIVE2 will be erased. Are you certain? [y/N]" y
• "Is your SecureOffice installation on DRIVE1? [Y/n]" y (to backup DRIVE1 partitions 1 to 3 to DRIVE2)
• OR: "Is your SecureOffice installation on DRIVE1? [Y/n]" n (no system backup)
• If "y" to previous question: "Do you want to backup your SecureOffice installation from DRIVE1 to DRIVE2? May take some time. [Y/n]" y (or "n"). SecureOffice can, if system disk failure, be booted from the backup disk by selecting the backup disk as primary disk using BIOS settings at boot. This will allow recovery if the primary system disk fails.
• If message "mdadm: Note: this array has metadata at the start, etc" Continue creating array? (y/n). Answer "y"

Separate system disk and RAID components / partitions:

• "Have you set the configuration values at the beginning of this script? [Y/n]" y
• "Disk partition DRIVE1(4) and entire disk DRIVE2 will be erased. Are you certain? [y/N]" y
• "Is your SecureOffice installation on DRIVE1? [Y/n]" n
• If message "mdadm: Note: this array has metadata at the start, etc" Continue creating array? (y/n). Answer "y"

Follow the prompts and fix any errors (such as RAID device already exists) until installation is successful. At the end of RAID installation, result is:

• File "/etc/config/fstab" (what to mount at boot) has been replaced (old one saved as "/etc/config/fstab_pre_raid") to use swap from <DRIVE2> and mount the RAID disk at <RAID_MOUNT>. If there are other devices to be mounted, merge additional mounts / configuration from "/etc/config/fstab_pre_raid" to "/etc/config/fstab".
• "/etc/config/mdadm" altered for email notifications and RAID disk UUID.
• RAID disk (/dev/md0), formatted as ext4 should be mounted (at <RAID_MOUNT>) and configured to automatically start and mount at boot.
• Two RAID status emails should be received.

Any errors can be diagnosed from the script output and log (/var/log/do_raid.log).

1.6                Verify RAID Status

Enter the following commands:

"mount" You should see "/dev/md0 on"the value specified for "RAID_MOUNT" above.

If "/dev/md0" is not mounted, enter "logread | grep md" for diagnostic information and fix any problems.

After your next reboot, repeat the "mount" command above to confirm that the RAID disk auto mounts at startup and the swap partition from DRIVE2 is active (Enter "swapon -s"). If not, repeat the "logread" command for diagnostic information and fix any problems.

1.7                Verify Email Notifications

After RAID installation, you should have received two status emails (may be delayed by servers). If not, your send email server may need additional configuration.

Send a test email with debug enabled by entering the following command (omitting outer quotes):

'echo -e "From: <EMAIL_FROM>\nSubject: Raid Status\n\n Raid Installed!" | msmtp --debug --timeout=10 -t <EMAIL_TO>'

Replacing <EMAIL_FROM> and <EMAIL_TO> with the RAID script configuration values entered above. Debug output from the above command should provide clues regarding any issues.

If the email is not received, do an internet search "msmtp your_email_provider" to determine any additional configuration required.

Once required configuration additions / changes are identified, edit ("nano /etc/msmtprc") to enter any changes required.

Further information regarding msmtp email configuration / debugging is available at the Arch Wiki.

1.8                Controlling RAID

SecureOffice / OpenWrt RAID installation uses a standard initscript ("/etc/init.d/mdadm") for controlling the RAID array. The RAID UUID and device ("/dev/md0") is specified in "/etc/config/mdadm". The RAID mount point (default: "/home/data") for the RAID disk (/dev/md0) is specified in "/etc/config/fstab".

After RAID is installed, the RAID disk is active and mounted at (default: /home/data, or script value RAID_MOUNT) and configured to auto mount at boot.

• To enable RAID start at boot: enter "/etc/init.d/mdadm enable" (already done by script)
• To disable RAID start at boot: enter "/etc/init.d/mdadm disable"
• To stop and unmount the RAID array: enter "/etc/init.d/mdadm stop"
• To start and mount the RAID array: enter "/etc/init.d/mdadm start"
• To change the RAID array mount point, create the mount point directory and edit "/etc/config/fstab" to specify the new mount point.
• If changes to email notifications are desired, consult the msmtp manual and edit configuration file "/etc/msmtprc".

1.9                RAID Backup / Restore

During system backup (GUI: "Backup / Flash Firmware -> Backup"), all RAID configuration is saved. During system restore (GUI: "Backup / Flash Firmware -> Restore"), all packages required for RAID are re-installed and configuration restored. Script "/etc/backup/raid" was installed by the RAID script and is used by system backup / restore.

What is not restored is the RAID disk contents (bulk data) which must be backed up / restored by an alternate method. The backup / restore method used by docker-nextcloud can serve as an example for backing up / restoring bulk data such as virtual machines.

1.10            Replace Failed RAID Disk

Every time a significant RAID event such as SecureOffice boot (RAID mounted) or RAID disk failure occurs, an email will be received containing details.

When a RAID disk member fails, the RAID array will continue to run using the redundant (good) RAID disk member. This allows time to replace the faulty disk without losing data (the whole point of RAID).

To identify a failed RAID disk:

Enter "cat /proc/mdstat". If both RAID members (disks) are active, you will see output of the form below. The "[UU]" result indicates both disks are good.

If one of the RAID members has failed, you will see output of the form below: "(F)" will appear next to the failed disk and "[_U]" or "[U_]" will be reported. In this example, "/dev/sdb4" has failed and needs to be replaced.

The failed disk must be removed from the RAID array. Enter "mdadm --manage /dev/md0 --remove /dev/sdX4" where "X" is the identity (a, b, etc) of the failed disk reported above.

Power down SecureOffice (Enter "shutdown") and replace the failed disk with one of equal or greater size.

If the failed disk is the boot / system disk (Common system and data disks), SecureOffice must be re-installed on the replacement disk: Replace System Disk, including restore from backup.

Automatically Restore Failed RAID Disk

The easiest way to restore a failed RAID disk is to re-install RAID by re-running the RAID installation script. To do so:

• Backup the contents of the RAID disk to another device or mounted network location: "cd /home/data; tar -zcvf <backup location>/raid.tar.gz ./".
• Re-run the RAID installation script (Configure and Install RAID).
• Restore RAID contents: "cd /home/data; tar -zxvf <backup location>/raid.tar.gz"

Manually Restore Failed RAID Disk

A harder way to restore a failed RAID disk is:

If the failed disk was the system disk, partition 4 (RAID) on the replacement system disk must be created with the same size as partition 4 of the good RAID disk.

• After replacing the system disk (partitions 1 (boot), 2 (rootfs), 3 (swap) were created), it is necessary to create a RAID partition (4) of the required size on the system disk.
• The existing partition 4 on the replacement system disk must be deleted before resizing. Enter 'echo -e "d\n4\n\nw" | fdisk /dev/sdX' where "X" is the system disk identity.
• Create RAID partition (next section).

If the failed disk was not the system disk and using a common disk for system and RAID partitions and it is not required to backup the system / swap partitions to the replacement disk.

• Delete all partitions from the replacement disk. Enter 'umount /dev/sdY*; dd if=/dev/zero of=/dev/sdY bs=512 count=1; echo -e "w" | fdisk /dev/sdY' where "Y" is the replacement disk identity.
• Create swap partition on replacement disk. Enter 'echo -e "n\np\n3\n\n+8192M\nt\n82\nw" | fdisk /dev/sdY; mkswap /dev/sdY3' where "Y" is the replacement disk identity.
• Create RAID partition (next section).

Else, if it is desired to backup the system partitions to the replacement disk, copy partitions 1 and 2 (boot, rootfs) from the system disk to the replacement disk:

• Identify the start sector of the system disk swap sector: Enter "fdisk -l /dev/SdX", where "X" is the system disk (normally sda). You are looking for an output line of the form: "/dev/sdX3 <start sector> <end sector> <sector size> 8G 82 Linux swap / Solaris".
• Enter "dd if=/dev/sdX of=/dev/sdY bs=512 count=<start sector>" where <start sector> is the sector number of the swap partition, "X" is the system disk and "Y" is the replacement disk.
• Delete the swap partition (created by dd) from the partition table of replacement disk. Enter "'echo -e "d\n3\n\nw" | fdisk /dev/sdY', where "Y" is the replacement disk.
• Delete the data partition (created by dd) from the partition table of replacement disk. Enter "'echo -e "d\n4\n\nw" | fdisk /dev/sdY', where "Y" is the replacement disk.
• Create swap partition on replacement disk. Enter 'echo -e "n\np\n3\n\n+8192M\nt\n82\nw" | fdisk /dev/sdY; mkswap /dev/sdY3' where "Y" is the replacement disk identity.
• Create RAID partition (next section).

1.10.1  Create RAID Partition on Replacement Disk

RAID requires the partition size of member disks to be identical. When replacing a RAID disk, the required partition size can be determined from the good RAID partition. Assuming the good partition is "/dev/sdX" Enter "fdisk -l /dev/sdX" where "X" is "a, b, c, etc" to determine partition layout. You will see output as below:

The entry in the "/dev/sdX4" row and "Sectors" column is the required size of the RAID partition (<RAID SECTORS>).

Enter "fdisk -l /dev/sdY" where "Y" is the replacement disk. If the replacement disk is the system disk or system partitions were backed up to the replacement disk, you should see entries for partitions 1, 2, 3. If the replacement disk is not the system disk and system partitions were not backed up, you should see a single swap partition (created previously).

Enter 'echo -e "n\np\n4\n\n+<RAID SECTORS>\nw" | fdisk /dev/sdxY' where "< RAID SECTORS>" was determined above and "Y" is the replacement disk.

As a final check, Enter "fdisk -l /dev/sdX" and "fdisk -l /dev/sdY". The size of the swap (3) and data (4) partitions should be identical for both disks.

1.10.2  Add Replacement Disk to RAID

Enter "mdadm --manage /dev/md0 --add /dev/sdY4" where "Y" is the replacement disk ID.

Enter "cat /proc/mdstat". You should see output as below, indicating the RAID array is rebuilding:

After recovery is complete, you will see output as below:

 

If data was backed up from either disk (such as /home/data) prior to installing RAID (Section 1.2), or was backed up prior to replacing a failed disk, restore it to the RAID disk if required.

Reboot to verify that the RAID array starts correctly and all configuration / files have been restored.

Additional instructions are available to replace the failed disk and restore it to the RAID array. Should the previous link change, google "Linux replace failed RAID disk".