Raspberry Pi RAID File Server
A little recipe to build a networked RAID storage array that is shared via Apple File Sharing using a Raspberry Pi and some old USB hard disks.
Starting Point
Install a clean Raspberry Pi. This article is based on it being Raspbian Stretch Lite.
Be sure to update Raspbian to the latest version:
$ sudo apt-get update && apt-get dist-upgrade
Add some needed packages:
$ sudo apt-get install screen $ sudo apt-get install mdadm
screen: I tend to SSH into the box and leave commands that need a long time run in the background while I disconnect (C-d to detach and later screen -r to resume)
mdadm: the package used to manage RAID volumes
Prepare the First Disk
Find the USB drive with dmesg:
$ dmesg [ 1389.493714] usb 1-1.3: new high-speed USB device number 5 using dwc_otg [ 1389.625266] usb 1-1.3: New USB device found, idVendor=059f, idProduct=0651 [ 1389.625297] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 1389.625310] usb 1-1.3: Product: LaCie Hard Drive USB [ 1389.625321] usb 1-1.3: Manufacturer: LaCie [ 1389.625330] usb 1-1.3: SerialNumber: 10000E0003C3AC29 [ 1389.632736] usb-storage 1-1.3:1.0: USB Mass Storage device detected [ 1389.653577] usb-storage 1-1.3:1.0: Quirks match for vid 059f pid 0651: 200 [ 1389.654822] scsi host0: usb-storage 1-1.3:1.0 [ 1390.685192] scsi 0:0:0:0: Direct-Access WDC WD1600BB-55GUC0 08.0 PQ: 0 ANSI: 2 [ 1390.687446] sd 0:0:0:0: [sda] 312581808 512-byte logical blocks: (160 GB/149 GiB) [ 1390.687476] sd 0:0:0:0: [sda] Assuming Write Enabled [ 1390.687492] sd 0:0:0:0: [sda] Assuming drive cache: write through [ 1390.773317] sd 0:0:0:0: Attached scsi generic sg0 type 0 [ 1390.806677] sda: sda1 sda2 [ 1390.810289] sd 0:0:0:0: [sda] Attached SCSI disk
Use lsblk to see what is available:
$ lsblk -l NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 149,1G 0 disk sda1 8:1 0 200M 0 part sda2 8:2 0 148,7G 0 part mmcblk0 179:0 0 3,7G 0 disk mmcblk0p1 179:1 0 41,5M 0 part /boot mmcblk0p2 179:2 0 3,7G 0 part /
-l: to print a white space list
Use dd to wipe the current partition table:
$ sudo dd if=/dev/zero of=/dev/sda bs=512 count=1 1+0 Datensätze ein 1+0 Datensätze aus 512 Bytes kopiert, 0,0185235 s, 27,6 kB/s
Use fdisk to create a new partition:
$ sudo fdisk /dev/sda
Command (m for help): g
Created a new GPT disklabel (GUID: 0C5D8867-F662-4156-B385-039E6F0DF2EA).
Command (m for help): n
Partition number (1-128, default 1): 1
First sector (2048-312581774, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-312581774, default 312581774):
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
Use lsblk to check there is now one partition available:
$ lsblk -l NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 149,1G 0 disk sda1 8:1 0 149,1G 0 part mmcblk0 179:0 0 3,7G 0 disk mmcblk0p1 179:1 0 41,5M 0 part /boot mmcblk0p2 179:2 0 3,7G 0 part /
Format the drive with mkfs.ext4:
$ sudo mkfs.ext4 -c -L "LaCie 160GB HDD" /dev/sda1
-c: check for bad blocks
-L: set the partition label
This will take a while because we are checking for bad blocks with mkfs.ext4.
Prepare the Second Disk
Find the USB drive with dmesg:
$ dmesg [ 2964.665263] usb 1-1.2: new high-speed USB device number 6 using dwc_otg [ 2964.797810] usb 1-1.2: New USB device found, idVendor=059b, idProduct=0370 [ 2964.797840] usb 1-1.2: New USB device strings: Mfr=10, Product=11, SerialNumber=5 [ 2964.797853] usb 1-1.2: Product: External HD [ 2964.797863] usb 1-1.2: Manufacturer: Iomega [ 2964.797872] usb 1-1.2: SerialNumber: 995EFFFFFFFF [ 2964.816603] usb-storage 1-1.2:1.0: USB Mass Storage device detected [ 2964.827924] scsi host1: usb-storage 1-1.2:1.0 [ 2965.857226] scsi 1:0:0:0: Direct-Access ST350082 0AS PQ: 0 ANSI: 2 CCS [ 2965.862824] sd 1:0:0:0: [sdb] 976773168 512-byte logical blocks: (500 GB/466 GiB) [ 2965.863902] sd 1:0:0:0: [sdb] Write Protect is off [ 2965.863933] sd 1:0:0:0: [sdb] Mode Sense: 34 00 00 00 [ 2965.864953] sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA [ 2965.885140] sd 1:0:0:0: Attached scsi generic sg1 type 0 [ 2966.010811] sdb: sdb1 sdb2 [ 2966.034340] sd 1:0:0:0: [sdb] Attached SCSI disk
Use lsblk to see what is available:
$ lsblk -l NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 149,1G 0 disk sda1 8:1 0 149,1G 0 part sdb 8:16 0 465,8G 0 disk sdb1 8:17 0 200M 0 part sdb2 8:18 0 465,5G 0 part mmcblk0 179:0 0 3,7G 0 disk mmcblk0p1 179:1 0 41,5M 0 part /boot mmcblk0p2 179:2 0 3,7G 0 part /
-l: to print a white space list
Use dd to wipe the current partition table:
$ sudo dd if=/dev/zero of=/dev/sdb bs=512 count=1 1+0 Datensätze ein 1+0 Datensätze aus 512 Bytes kopiert, 0,0421397 s, 12,2 kB/s
Use fdisk to create a new partition:
$ sudo fdisk /dev/sdb
Command (m for help): g
Created a new GPT disklabel (GUID: 5B0986EA-8135-4C07-9A82-65F79831AA8A).
Command (m for help): n
Partition number (1-128, default 1): 1
First sector (2048-976773134, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-312581774, default 312581774): +150G
Created a new partition 1 of type 'Linux filesystem' and of size 150 GiB.
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
Use lsblk to check there is now one partition available:
$ lsblk -l NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 149,1G 0 disk sda1 8:1 0 149,1G 0 part sdb 8:16 0 465,8G 0 disk sdb1 8:17 0 150G 0 part mmcblk0 179:0 0 3,7G 0 disk mmcblk0p1 179:1 0 41,5M 0 part /boot mmcblk0p2 179:2 0 3,7G 0 part /
-l: to print a white space list
Format the drive with mkfs.ext4:
$ sudo mkfs.ext4 -c -L "LaCie 500GB HDD" /dev/sdb1
mke2fs 1.43.4 (31-Jan-2017)
Ein Dateisystems mit 39321600 (4k) Blöcken und 9830400 Inodes wird erzeugt.
UUID des Dateisystems: c1c86603-bcaa-4019-b570-3de66c7a4eeb
Superblock-Sicherungskopien gespeichert in den Blöcken:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Es wird nach defekten Blöcken gesucht (Nur-Lesen-Modus):erledigt
beim Anfordern von Speicher für die Gruppentabellen: erledigt
Inode-Tabellen werden geschrieben: erledigt
Das Journal (262144 Blöcke) wird angelegt: erledigt
Die Superblöcke und die Informationen über die Dateisystemnutzung werden
geschrieben: erledigt
-c: check for bad blocks
-L: set the partition label
Create the RAID Array
$ sudo mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1
mdadm: /dev/sda1 appears to contain an ext2fs file system
size=157286400K mtime=Thu Jan 1 01:00:00 1970
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device. If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
mdadm: /dev/sdb1 appears to contain an ext2fs file system
size=156289860K mtime=Thu Jan 1 01:00:00 1970
mdadm: size set to 156158784K
mdadm: automatically enabling write-intent bitmap on large array
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
--verbose: ensures that all mdadm information will be presented during the process
--level=1: refers to the RAID level being created
Wait for the RAID array to finish assembling (this may take some time). You can check the progress periodically using the command reading /proc/mdstat:
$ cat /proc/mdstat
You’d get this:
Personalities : [raid1]
md0 : active raid1 sdb1[0] sda1[1]
156158784 blocks super 1.2 [2/2] [UU]
[=====>...............] resync = 25.4% (39666240/156158784) finish=122.9min speed=15795K/sec
bitmap: 2/2 pages [8KB], 65536KB chunk
unused devices:
If you check, and it says “PENDING”
$ cat /proc/mdstat
Personalities : [raid1]
md0 : active (auto-read-only) raid1 sdb1[0] sda1[1]
156158784 blocks super 1.2 [2/2] [UU]
resync=PENDING
unused devices:
Then you can trigger the sync to continue with:
$ mdadm --readwrite /dev/md0
When it’s done, you’ll see:
$ cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[0] sda1[1]
156158784 blocks super 1.2 [2/2] [UU]
bitmap: 0/2 pages [0KB], 65536KB chunk
unused devices:
Now add the array configuration to /etc/mdadm/mdadm.conf. Edit the file and add:
DEVICE /dev/sda1 /dev/sdb1 ARRAY /dev/md0 devices=/dev/sda1,/dev/sdb1
The array should come up ready, but you can assemble the RAID disks together:
$ sudo mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1
Check to make sure that everything looks good with the new RAID device:
$ sudo mdadm --detail /dev/md0
Format the new RAID device:
$ sudo mkfs.ext4 -L "150 GB USB RAID" /dev/md0
mke2fs 1.43.4 (31-Jan-2017)
Ein Dateisystems mit 39039696 (4k) Blöcken und 9764864 Inodes wird erzeugt.
UUID des Dateisystems: 149a2788-7f1c-4904-ace4-cb37a3e20e7a
Superblock-Sicherungskopien gespeichert in den Blöcken:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
beim Anfordern von Speicher für die Gruppentabellen: erledigt
Inode-Tabellen werden geschrieben: erledigt
Das Journal (262144 Blöcke) wird angelegt: erledigt
Die Superblöcke und die Informationen über die Dateisystemnutzung werden
geschrieben: erledigt
Create a directory for the mount point where ever you like for your RAID device:
$ sudo mkdir /media/150_GB_USB_RAID
Find the UUID for your RAID array device:
$ lsblk --fs | grep md0 └─md0 ext4 150 GB USB RAID 149a2788-7f1c-4904-ace4-cb37a3e20e7a └─md0 ext4 150 GB USB RAID 149a2788-7f1c-4904-ace4-cb37a3e20e7a
Add an entry to the end of /etc/fstab to auto-mount the RAID array device at the UUID and mount point.
UUID=d409ca76-9eee-40ea-a306-4838a7b813c7 /media/150_GB_USB_RAID ext4 defaults 0 2
Test your set up by mounting the RAID array device:
$ sudo mount -a
Now the storage array is up and ready.
Configure File Sharing over Apple Filing Protocol
Create a directory where shared files will be:
$ sudo mkdir /media/150_GB_USB_RAID/shared $ sudo chmod 777 /media/150_GB_USB_RAID/shared
Install netatalk which is the package that provides Apple Filing Protocol.
$ sudo apt-get install netatalk
Edit the netatalk configuration file /etc/netatalk/afpd.conf:
$ sudo vi /etc/netatalk/afpd.conf
On the last line remove the # and remove uams_dhx.so, to set the last line to:
- -tcp -noddp -uamlist uams_dhx2.so
- apply these settings as the default
-tcp listen on tcp
-noddp do not listen on DDP/AppleTalk
-uamlist uams_dhx2.so use Diffie-Hellman eXchange 2 as authentication method
Edit the netatalk configuration file /etc/netatalk/AppleVolumes.default to define shares:
$ sudo vi /etc/netatalk/AppleVolumes.default
Remove the home directory being shared by default by adding a #:
# ~/ "Home Directory"
Add a line to make the share available:
/media/150_GB_USB_RAID/shared "Mirrored File Share" allow:pi
Enable netatalk at boot and (re)start it now:
$ sudo systemctl enable netatalk netatalk.service is not a native service, redirecting to systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable netatalk $ sudo systemctl restart netatalk
Done!
Possibly Good to Know
You can stop an array if neededww.:
$ mdadm --stop /dev/md0
Start an existing array:
mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1