Example of upgrading my FreeNAS server running ZFS from two 750Gig & two 1TB drives to one 750Gig & three 1 TB drives. I upgraded my Drobo with a 2 TB drive. I removed a 1TB drive in it to make room for a new 2TB drive. Then I took that 1TB drive and used it in my FreeNAS ZFS server (in Drobo like config).
In the end I will have added 250Gigs more to my ZFS pool. In reality my protected space will increase by 114 Gigs. Because of the way I’ve configured my ZFS pool and partitioned my drives I can increase the ZFS Zpool without having to copy all the data off the raid arrays within the pool and then destroy the Zpool and start again with the larger drives. This config allows you to expand the drives one at a time.
This is how I did it:
freenas02:~# zpool export -f tank0
freenas02:~# gpt show ad4
start size index contents
0 1 PMBR
1 1 Pri GPT header
2 32 Pri GPT table
34 1465149101 1 GPT part – FreeBSD ZFS
1465149135 32 Sec GPT table
1465149167 1 Sec GPT header
freenas02:~# gpt show ad6
start size index contents
0 1 PMBR
1 1 Pri GPT header
2 32 Pri GPT table
34 1465149101 1 GPT part – FreeBSD ZFS
1465149135 32 Sec GPT table
1465149167 1 Sec GPT header
freenas02:~# gpt show ad8
start size index contents
0 1 PMBR
1 1 Pri GPT header
2 32 Pri GPT table
34 1465149101 1 GPT part – FreeBSD ZFS
1465149135 122094000 2 GPT part – FreeBSD ZFS
1587243135 122094000 3 GPT part – FreeBSD ZFS
1709337135 122094000 4 GPT part – FreeBSD ZFS
1831431135 122094000 5 GPT part – FreeBSD ZFS
1953525135 32 Sec GPT table
1953525167 1 Sec GPT header
freenas02:~# gpt show ad14
start size index contents
0 1 PMBR
1 1 Pri GPT header
2 32 Pri GPT table
34 1465149101 1 GPT part – FreeBSD ZFS
1465149135 122094000 2 GPT part – FreeBSD ZFS
1587243135 122094000 3 GPT part – FreeBSD ZFS
1709337135 122094000 4 GPT part – FreeBSD ZFS
1831431135 122094000 5 GPT part – FreeBSD ZFS
1953525135 32 Sec GPT table
1953525167 1 Sec GPT header
Reboot with ad6 750Gig drive replaced with ad6 1000G drive
freenas02:~# zpool status
pool: tank0
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using ‘zpool online’.
see: http://www.sun.com/msg/ZFS-8000-D3
scrub: resilver completed with 0 errors on Wed Sep 23 23:08:57 2009
config:
NAME STATE READ WRITE CKSUM
tank0 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
ad4p1 ONLINE 0 0 0
ad6p1 UNAVAIL 0 0 0 cannot open
ad8p1 ONLINE 0 0 0
ad14p1 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
ad8p2 ONLINE 0 0 0
ad8p3 ONLINE 0 0 0
ad14p2 ONLINE 0 0 0
ad14p3 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
ad8p4 ONLINE 0 0 0
ad8p5 ONLINE 0 0 0
ad14p4 ONLINE 0 0 0
ad14p5 ONLINE 0 0 0
errors: No known data errors
freenas02:~# zpool export -f tank0
freenas02:~# zpool status tank0
cannot open ‘tank0’: no such pool
freenas02:~# zpool status
no pools available
freenas02:~# gpt create -f ad6
freenas02:~# gpt show ad6
start size index contents
0 1 PMBR
1 1 Pri GPT header
2 32 Pri GPT table
34 1953525101
1953525135 32 Sec GPT table
1953525167 1 Sec GPT header
freenas02:~# gpt add -s 1465149101 ad6
freenas02:~# gpt show ad6
start size index contents
0 1 PMBR
1 1 Pri GPT header
2 32 Pri GPT table
34 1465149101 1 GPT part – FreeBSD UFS/UFS2
1465149135 488376000
1953525135 32 Sec GPT table
1953525167 1 Sec GPT header
freenas02:~# gpt add -s 244188000 ad6
freenas02:~# gpt show ad6
start size index contents
0 1 PMBR
1 1 Pri GPT header
2 32 Pri GPT table
34 1465149101 1 GPT part – FreeBSD UFS/UFS2
1465149135 244188000 2 GPT part – FreeBSD UFS/UFS2
1709337135 244188000
1953525135 32 Sec GPT table
1953525167 1 Sec GPT header
freenas02:~# gpt add -s 244188000 ad6
freenas02:~# gpt show ad6
start size index contents
0 1 PMBR
1 1 Pri GPT header
2 32 Pri GPT table
34 1465149101 1 GPT part – FreeBSD UFS/UFS2
1465149135 244188000 2 GPT part – FreeBSD UFS/UFS2
1709337135 244188000 3 GPT part – FreeBSD UFS/UFS2
1953525135 32 Sec GPT table
1953525167 1 Sec GPT header
freenas02:~# zpool import tank0
freenas02:~# zpool status
pool: tank0
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using ‘zpool online’.
see: http://www.sun.com/msg/ZFS-8000-D3
scrub: resilver completed with 0 errors on Wed Sep 23 23:29:51 2009
config:
NAME STATE READ WRITE CKSUM
tank0 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
ad4p1 ONLINE 0 0 0
ad6p1 UNAVAIL 0 0 0 cannot open
ad8p1 ONLINE 0 0 0
ad14p1 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
ad8p2 ONLINE 0 0 0
ad8p3 ONLINE 0 0 0
ad14p2 ONLINE 0 0 0
ad14p3 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
ad8p4 ONLINE 0 0 0
ad8p5 ONLINE 0 0 0
ad14p4 ONLINE 0 0 0
ad14p5 ONLINE 0 0 0
errors: No known data errors
freenas02:~# zpool replace tank0 ad6p1
freenas02:~# zpool status
pool: tank0
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress, 0.00% done, 161h0m to go
config:
NAME STATE READ WRITE CKSUM
tank0 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
ad4p1 ONLINE 0 0 0
replacing DEGRADED 0 0 0
ad6p1/old UNAVAIL 0 0 0 cannot open
ad6p1 ONLINE 0 0 0
ad8p1 ONLINE 0 0 0
ad14p1 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
ad8p2 ONLINE 0 0 0
ad8p3 ONLINE 0 0 0
ad14p2 ONLINE 0 0 0
ad14p3 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
ad8p4 ONLINE 0 0 0
ad8p5 ONLINE 0 0 0
ad14p4 ONLINE 0 0 0
ad14p5 ONLINE 0 0 0
errors: No known data errors
freenas02:~# zpool replace tank0 ad8p2 ad6p2
freenas02:~# zpool replace tank0 ad14p2 ad6p3
freenas02:~# zpool status
pool: tank0
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress, 0.87% done, 9h25m to go
config:
NAME STATE READ WRITE CKSUM
tank0 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
ad4p1 ONLINE 0 0 0
replacing DEGRADED 0 0 0
ad6p1/old UNAVAIL 0 0 0 cannot open
ad6p1 ONLINE 0 0 0
ad8p1 ONLINE 0 0 0
ad14p1 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
replacing ONLINE 0 0 0
ad8p2 ONLINE 0 0 0
ad6p2 ONLINE 0 0 0
ad8p3 ONLINE 0 0 0
replacing ONLINE 0 0 0
ad14p2 ONLINE 0 0 0
ad6p3 ONLINE 0 0 0
ad14p3 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
ad8p4 ONLINE 0 0 0
ad8p5 ONLINE 0 0 0
ad14p4 ONLINE 0 0 0
ad14p5 ONLINE 0 0 0
errors: No known data errors
freenas02:~# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/md0 88M 76M 12M 86% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/da0a 38M 35M 3.8M 90% /cf
/dev/md1 31M 4.0K 28M 0% /var
tank0 2.2T 1.9T 339G 85% /mnt/tank0
freenas02:~#
freenas02:~# zpool status
pool: tank0
state: ONLINE
scrub: resilver completed with 0 errors on Thu Sep 24 05:05:00 2009
config:
NAME STATE READ WRITE CKSUM
tank0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad4p1 ONLINE 0 0 0
ad6p1 ONLINE 0 0 0
ad8p1 ONLINE 0 0 0
ad14p1 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
ad6p2 ONLINE 0 0 0
ad8p3 ONLINE 0 0 0
ad6p3 ONLINE 0 0 0
ad14p3 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
ad8p4 ONLINE 0 0 0
ad8p5 ONLINE 0 0 0
ad14p4 ONLINE 0 0 0
ad14p5 ONLINE 0 0 0
errors: No known data errors
freenas02:~# zpool export -f tank0
freenas02:~# gpt show ad8
start size index contents
0 1 PMBR
1 1 Pri GPT header
2 32 Pri GPT table
34 1465149101 1 GPT part – FreeBSD ZFS
1465149135 122094000 2 GPT part – FreeBSD ZFS
1587243135 122094000 3 GPT part – FreeBSD ZFS
1709337135 122094000 4 GPT part – FreeBSD ZFS
1831431135 122094000 5 GPT part – FreeBSD ZFS
1953525135 32 Sec GPT table
1953525167 1 Sec GPT header
freenas02:~# gpt remove -i 2 ad8
ad8p2 removed
freenas02:~# gpt remove -i 3 ad8
ad8p3 removed
freenas02:~# gpt show ad8
start size index contents
0 1 PMBR
1 1 Pri GPT header
2 32 Pri GPT table
34 1465149101 1 GPT part – FreeBSD ZFS
1465149135 244188000
1709337135 122094000 4 GPT part – FreeBSD ZFS
1831431135 122094000 5 GPT part – FreeBSD ZFS
1953525135 32 Sec GPT table
1953525167 1 Sec GPT header
freenas02:~# gpt add -i 2 -s 244188000 ad8
freenas02:~# gpt show ad8
start size index contents
0 1 PMBR
1 1 Pri GPT header
2 32 Pri GPT table
34 1465149101 1 GPT part – FreeBSD ZFS
1465149135 244188000 2 GPT part – FreeBSD UFS/UFS2
1709337135 122094000 4 GPT part – FreeBSD ZFS
1831431135 122094000 5 GPT part – FreeBSD ZFS
1953525135 32 Sec GPT table
1953525167 1 Sec GPT header
freenas02:~# zpool import tank0
freenas02:~# zpool status tank0
pool: tank0
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using ‘zpool replace’.
see: http://www.sun.com/msg/ZFS-8000-4J
scrub: resilver completed with 0 errors on Thu Sep 24 14:54:35 2009
config:
NAME STATE READ WRITE CKSUM
tank0 DEGRADED 0 0 0
raidz1 ONLINE 0 0 0
ad4p1 ONLINE 0 0 0
ad6p1 ONLINE 0 0 0
ad8p1 ONLINE 0 0 0
ad14p1 ONLINE 0 0 0
raidz2 DEGRADED 0 0 0
ad6p2 ONLINE 0 0 0
6422553418725777030 FAULTED 0 0 0 was /dev/ad8p5
ad6p3 ONLINE 0 0 0
ad14p3 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
ad8p4 ONLINE 0 0 0
ad8p5 ONLINE 0 0 0
ad14p4 ONLINE 0 0 0
ad14p5 ONLINE 0 0 0
errors: No known data errors
freenas02:~# zpool replace tank0 6422553418725777030 ad8p2
freenas02:~# zpool status tank0
pool: tank0
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress, 0.53% done, 5h25m to go
config:
NAME STATE READ WRITE CKSUM
tank0 DEGRADED 0 0 0
raidz1 ONLINE 0 0 0
ad4p1 ONLINE 0 0 0
ad6p1 ONLINE 0 0 0
ad8p1 ONLINE 0 0 0
ad14p1 ONLINE 0 0 0
raidz2 DEGRADED 0 0 0
ad6p2 ONLINE 0 0 0
replacing DEGRADED 0 0 0
6422553418725777030 FAULTED 0 0 0 was /dev/ad8p5
ad8p2 ONLINE 0 0 0
ad6p3 ONLINE 0 0 0
ad14p3 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
ad8p4 ONLINE 0 0 0
ad8p5 ONLINE 0 0 0
ad14p4 ONLINE 0 0 0
ad14p5 ONLINE 0 0 0
errors: No known data errors
freenas02:~# zpool status tank0
pool: tank0
state: ONLINE
scrub: resilver completed with 0 errors on Thu Sep 24 16:34:09 2009
config:
NAME STATE READ WRITE CKSUM
tank0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad4p1 ONLINE 0 0 0
ad6p1 ONLINE 0 0 0
ad8p1 ONLINE 0 0 0
ad14p1 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
ad6p2 ONLINE 0 0 0
ad8p2 ONLINE 0 0 0
ad6p3 ONLINE 0 0 0
ad14p3 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
ad8p4 ONLINE 0 0 0
ad8p5 ONLINE 0 0 0
ad14p4 ONLINE 0 0 0
ad14p5 ONLINE 0 0 0
errors: No known data errors
freenas02:~# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/md0 88M 76M 12M 86% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/da0a 38M 35M 3.8M 90% /cf
/dev/md1 31M 4.0K 28M 0% /var
tank0 2.2T 1.9T 339G 85% /mnt/tank0
freenas02:~# zpool export -f tank0
freenas02:~# gpt show ad14
start size index contents
0 1 PMBR
1 1 Pri GPT header
2 32 Pri GPT table
34 1465149101 1 GPT part – FreeBSD ZFS
1465149135 122094000 2 GPT part – FreeBSD ZFS
1587243135 122094000 3 GPT part – FreeBSD ZFS
1709337135 122094000 4 GPT part – FreeBSD ZFS
1831431135 122094000 5 GPT part – FreeBSD ZFS
1953525135 32 Sec GPT table
1953525167 1 Sec GPT header
freenas02:~# gpt remove -i 2 ad14
ad14p2 removed
freenas02:~# gpt remove -i 3 ad14
ad14p3 removed
freenas02:~# gpt show ad14
start size index contents
0 1 PMBR
1 1 Pri GPT header
2 32 Pri GPT table
34 1465149101 1 GPT part – FreeBSD ZFS
1465149135 244188000
1709337135 122094000 4 GPT part – FreeBSD ZFS
1831431135 122094000 5 GPT part – FreeBSD ZFS
1953525135 32 Sec GPT table
1953525167 1 Sec GPT header
freenas02:~# gpt add -i 2 -s 244188000 ad14
freenas02:~# gpt show ad14
start size index contents
0 1 PMBR
1 1 Pri GPT header
2 32 Pri GPT table
34 1465149101 1 GPT part – FreeBSD ZFS
1465149135 244188000 2 GPT part – FreeBSD UFS/UFS2
1709337135 122094000 4 GPT part – FreeBSD ZFS
1831431135 122094000 5 GPT part – FreeBSD ZFS
1953525135 32 Sec GPT table
1953525167 1 Sec GPT header
freenas02:~# zpool import tank0
freenas02:~# zpool status tank0
pool: tank0
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using ‘zpool replace’.
see: http://www.sun.com/msg/ZFS-8000-4J
scrub: resilver completed with 0 errors on Thu Sep 24 16:40:20 2009
config:
NAME STATE READ WRITE CKSUM
tank0 DEGRADED 0 0 0
raidz1 ONLINE 0 0 0
ad4p1 ONLINE 0 0 0
ad6p1 ONLINE 0 0 0
ad8p1 ONLINE 0 0 0
ad14p1 ONLINE 0 0 0
raidz2 DEGRADED 0 0 0
ad6p2 ONLINE 0 0 0
ad8p2 ONLINE 0 0 0
ad6p3 ONLINE 0 0 0
3452430189445945816 FAULTED 0 0 0 was /dev/ad14p5
raidz2 ONLINE 0 0 0
ad8p4 ONLINE 0 0 0
ad8p5 ONLINE 0 0 0
ad14p4 ONLINE 0 0 0
ad14p5 ONLINE 0 0 0
errors: No known data errors
freenas02:~# zpool replace tank0 3452430189445945816 ad14p2
freenas02:~# zpool status tank0
pool: tank0
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress, 2.53% done, 4h33m to go
config:
NAME STATE READ WRITE CKSUM
tank0 DEGRADED 0 0 0
raidz1 ONLINE 0 0 0
ad4p1 ONLINE 0 0 0
ad6p1 ONLINE 0 0 0
ad8p1 ONLINE 0 0 0
ad14p1 ONLINE 0 0 0
raidz2 DEGRADED 0 0 0
ad6p2 ONLINE 0 0 0
ad8p2 ONLINE 0 0 0
ad6p3 ONLINE 0 0 0
replacing DEGRADED 0 0 0
3452430189445945816 FAULTED 0 0 0 was /dev/ad14p5
ad14p2 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
ad8p4 ONLINE 0 0 0
ad8p5 ONLINE 0 0 0
ad14p4 ONLINE 0 0 0
ad14p5 ONLINE 0 0 0
errors: No known data errors
freenas02:~# zpool status tank0
pool: tank0
state: ONLINE
scrub: resilver completed with 0 errors on Thu Sep 24 18:20:52 2009
config:
NAME STATE READ WRITE CKSUM
tank0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad4p1 ONLINE 0 0 0
ad6p1 ONLINE 0 0 0
ad8p1 ONLINE 0 0 0
ad14p1 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
ad6p2 ONLINE 0 0 0
ad8p2 ONLINE 0 0 0
ad6p3 ONLINE 0 0 0
ad14p2 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
ad8p4 ONLINE 0 0 0
ad8p5 ONLINE 0 0 0
ad14p4 ONLINE 0 0 0
ad14p5 ONLINE 0 0 0
errors: No known data errors
freenas02:~# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/md0 88M 76M 12M 86% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/da0a 38M 35M 3.8M 90% /cf
/dev/md1 31M 4.0K 28M 0% /var
tank0 2.2T 1.9T 339G 85% /mnt/tank0
freenas02:~# zpool export -f tank0
freenas02:~# shutdown -r now
Reboot
Check disk size
freenas02:~# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/md0 88M 76M 12M 86% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/da0a 38M 35M 3.8M 90% /cf
tank0 2.3T 1.9T 453G 81% /mnt/tank0
/dev/md1 31M 4.0K 28M 0% /var
Glen's Weblog 
That post is exactly why I have a Drobo at home. If I had more free time, I still wouldn’t want to go through that nightmare.
Jimbo
Glen,
Thanks for taking the time to document this. I am many others appreciate it.
Cheers,
Gregg
Aside from the GPT insanity, this looks (more or less) like the normal procedure used for replacing a disk in a zpool. I’m not sure why user “Jimbo” thinks this is a nightmare; try looking at Linux LVM if you want a nightmare. 🙂
Hello there,
I have found your great articles about allowing for ZFS hard driver replacement and initial setup. What I am still trying to fathom is why you need to setup 5 different RAIDZ sets before making the pool. Couldn’t it be done just as easily with only 3 RAIDZ sets?
Well Jimbo’s saying it’s a “nightmare” compared to an actual Drobo where you just swap out the drives and it’s done. None of this shutting down, zpool / gpt business.
Speaking of which, this post is over a year old, but I’ve yet to see anyone come up with an AUTOMATED way of doing this similar to the drobo. How hard could it really be?