This morning one of my harddisks ada5p2 in the tank pool decided to become unavailable. Even though I store critical data on this pool, I have nothing really to worry about because this ZFS pool is configured as raidz2 - a disk pool that can tolerate two simultaneous disk failures.

# zpool status -v tank
	pool: tank
	state: DEGRADED
	status: One or more devices could not be opened.  Sufficient replicas exist for the pool to continue functioning in a degraded state.
	action: Attach the missing device and online it using 'zpool online'. 
	see: http://www.sun.com/msg/ZFS-8000-2Q
	scrub: scrub in progress for 0h0m, 0.00% done, 73h11m to go
	config:
 
		NAME        STATE     READ WRITE CKSUM
		tank        DEGRADED     0     0     0
		  raidz2    DEGRADED     0     0     0
			ada1p2  ONLINE       0     0     0
			ada2p2  ONLINE       0     0     0
			ada3p2  ONLINE       0     0     0
			ada4p2  ONLINE       0     0     0
			ada5p2  UNAVAIL      3 3.69K     0  cannot open

	errors: No known data errors

Without shutting down my storage system, I just yanked the SATA cable from that broken harddisk and hot replaced it with another of similar size. Now ZFS would resilver that replaced drive on its own in the next couple hours but I was essentially done without any downtime and without any data errors. ZFS is nice indeed.

A cron job periodically scrubbing the zpools helps. ZFS has a built in scrub function that checks for errors and corrects them when possible. Running this task is pretty essential to prevent more errors that aren’t correctable. By default, ZFS doesn’t run this periodically, you have to tell it when to scrub. The easiest way to set up periodic scrubbing is to use crontab, a feature present in all UNIX systems for scheduling background tasks. Start the editing of root user’s crontab by issuing the command crontab -e as root. The crontab is set up by a simple set of commands:

* * * * * command to run
- - - - -
| | | | |
| | | | +----- day of week (0-6) (Sunday is 0)
| | | +------- month (1-12)
| | +--------- day of month (1-31)
| +----------- hour (0-23)
+------------- min (0-59)

For example, I want my system to scrub my tank zpool on Sundays at 04:00 and my twoteebee zpool on Thursdays at 04:00. The specific commands that I put in my crontab are:

0 4 * * 0 /sbin/zpool scrub tank
0 4 * * 4 /sbin/zpool scrub twoteebee