Replacing a 750Gig with a 1TB drive in my FreeNAS ZFS (Drobo like) server config

Example of upgrading my FreeNAS server running ZFS from two 750Gig & two 1TB drives to one 750Gig & three 1 TB drives.  I upgraded my Drobo with a 2 TB drive.  I removed a 1TB drive in it to make room for a new 2TB drive.  Then I took that 1TB drive and used it in my FreeNAS ZFS server (in Drobo like config).

In the end I will have added 250Gigs more to my ZFS pool.  In reality my protected space will increase by 114 Gigs.  Because of the way I’ve configured my ZFS pool and partitioned my drives I can increase the ZFS Zpool without having to copy all the data off the raid arrays within the pool and then destroy the Zpool and start again with the larger drives.  This config allows you to expand the drives one at a time.

This is how I did it:
freenas02:~# zpool export -f tank0
freenas02:~# gpt show ad4
start        size  index  contents
0           1         PMBR
1           1         Pri GPT header
2          32         Pri GPT table
34  1465149101      1  GPT part – FreeBSD ZFS
1465149135          32         Sec GPT table
1465149167           1         Sec GPT header
freenas02:~# gpt show ad6
start        size  index  contents
0           1         PMBR
1           1         Pri GPT header
2          32         Pri GPT table
34  1465149101      1  GPT part – FreeBSD ZFS
1465149135          32         Sec GPT table
1465149167           1         Sec GPT header
freenas02:~# gpt show ad8
start        size  index  contents
0           1         PMBR
1           1         Pri GPT header
2          32         Pri GPT table
34  1465149101      1  GPT part – FreeBSD ZFS
1465149135   122094000      2  GPT part – FreeBSD ZFS
1587243135   122094000      3  GPT part – FreeBSD ZFS
1709337135   122094000      4  GPT part – FreeBSD ZFS
1831431135   122094000      5  GPT part – FreeBSD ZFS
1953525135          32         Sec GPT table
1953525167           1         Sec GPT header

freenas02:~# gpt show ad14
start        size  index  contents
0           1         PMBR
1           1         Pri GPT header
2          32         Pri GPT table
34  1465149101      1  GPT part – FreeBSD ZFS
1465149135   122094000      2  GPT part – FreeBSD ZFS
1587243135   122094000      3  GPT part – FreeBSD ZFS
1709337135   122094000      4  GPT part – FreeBSD ZFS
1831431135   122094000      5  GPT part – FreeBSD ZFS
1953525135          32         Sec GPT table
1953525167           1         Sec GPT header

Reboot with ad6 750Gig drive replaced with ad6 1000G drive

freenas02:~# zpool status
pool: tank0
state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using ‘zpool online’.
see: http://www.sun.com/msg/ZFS-8000-D3
scrub: resilver completed with 0 errors on Wed Sep 23 23:08:57 2009
config:

NAME        STATE     READ WRITE CKSUM
tank0       DEGRADED     0     0     0
raidz1    DEGRADED     0     0     0
ad4p1   ONLINE       0     0     0
ad6p1   UNAVAIL      0     0     0  cannot open
ad8p1   ONLINE       0     0     0
ad14p1  ONLINE       0     0     0
raidz2    ONLINE       0     0     0
ad8p2   ONLINE       0     0     0
ad8p3   ONLINE       0     0     0
ad14p2  ONLINE       0     0     0
ad14p3  ONLINE       0     0     0
raidz2    ONLINE       0     0     0
ad8p4   ONLINE       0     0     0
ad8p5   ONLINE       0     0     0
ad14p4  ONLINE       0     0     0
ad14p5  ONLINE       0     0     0

errors: No known data errors
freenas02:~# zpool export -f tank0

freenas02:~# zpool status tank0
cannot open ‘tank0’: no such pool
freenas02:~# zpool status
no pools available

freenas02:~# gpt create -f ad6
freenas02:~# gpt show ad6
start        size  index  contents
0           1         PMBR
1           1         Pri GPT header
2          32         Pri GPT table
34  1953525101
1953525135          32         Sec GPT table
1953525167           1         Sec GPT header

freenas02:~# gpt add -s 1465149101 ad6
freenas02:~# gpt show ad6
start        size  index  contents
0           1         PMBR
1           1         Pri GPT header
2          32         Pri GPT table
34  1465149101      1  GPT part – FreeBSD UFS/UFS2
1465149135   488376000
1953525135          32         Sec GPT table
1953525167           1         Sec GPT header

freenas02:~# gpt add -s 244188000 ad6
freenas02:~# gpt show ad6
start        size  index  contents
0           1         PMBR
1           1         Pri GPT header
2          32         Pri GPT table
34  1465149101      1  GPT part – FreeBSD UFS/UFS2
1465149135   244188000      2  GPT part – FreeBSD UFS/UFS2
1709337135   244188000
1953525135          32         Sec GPT table
1953525167           1         Sec GPT header

freenas02:~# gpt add -s 244188000 ad6
freenas02:~# gpt show ad6
start        size  index  contents
0           1         PMBR
1           1         Pri GPT header
2          32         Pri GPT table
34  1465149101      1  GPT part – FreeBSD UFS/UFS2
1465149135   244188000      2  GPT part – FreeBSD UFS/UFS2
1709337135   244188000      3  GPT part – FreeBSD UFS/UFS2
1953525135          32         Sec GPT table
1953525167           1         Sec GPT header

freenas02:~# zpool import tank0
freenas02:~# zpool status
pool: tank0
state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using ‘zpool online’.
see: http://www.sun.com/msg/ZFS-8000-D3
scrub: resilver completed with 0 errors on Wed Sep 23 23:29:51 2009
config:

NAME        STATE     READ WRITE CKSUM
tank0       DEGRADED     0     0     0
raidz1    DEGRADED     0     0     0
ad4p1   ONLINE       0     0     0
ad6p1   UNAVAIL      0     0     0  cannot open
ad8p1   ONLINE       0     0     0
ad14p1  ONLINE       0     0     0
raidz2    ONLINE       0     0     0
ad8p2   ONLINE       0     0     0
ad8p3   ONLINE       0     0     0
ad14p2  ONLINE       0     0     0
ad14p3  ONLINE       0     0     0
raidz2    ONLINE       0     0     0
ad8p4   ONLINE       0     0     0
ad8p5   ONLINE       0     0     0
ad14p4  ONLINE       0     0     0
ad14p5  ONLINE       0     0     0

errors: No known data errors

freenas02:~# zpool replace tank0 ad6p1
freenas02:~# zpool status
pool: tank0
state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress, 0.00% done, 161h0m to go
config:

NAME             STATE     READ WRITE CKSUM
tank0            DEGRADED     0     0     0
raidz1         DEGRADED     0     0     0
ad4p1        ONLINE       0     0     0
replacing    DEGRADED     0     0     0
ad6p1/old  UNAVAIL      0     0     0  cannot open
ad6p1      ONLINE       0     0     0
ad8p1        ONLINE       0     0     0
ad14p1       ONLINE       0     0     0
raidz2         ONLINE       0     0     0
ad8p2        ONLINE       0     0     0
ad8p3        ONLINE       0     0     0
ad14p2       ONLINE       0     0     0
ad14p3       ONLINE       0     0     0
raidz2         ONLINE       0     0     0
ad8p4        ONLINE       0     0     0
ad8p5        ONLINE       0     0     0
ad14p4       ONLINE       0     0     0
ad14p5       ONLINE       0     0     0

errors: No known data errors

freenas02:~# zpool replace tank0 ad8p2 ad6p2
freenas02:~# zpool replace tank0 ad14p2 ad6p3
freenas02:~# zpool status
pool: tank0
state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress, 0.87% done, 9h25m to go
config:

NAME             STATE     READ WRITE CKSUM
tank0            DEGRADED     0     0     0
raidz1         DEGRADED     0     0     0
ad4p1        ONLINE       0     0     0
replacing    DEGRADED     0     0     0
ad6p1/old  UNAVAIL      0     0     0  cannot open
ad6p1      ONLINE       0     0     0
ad8p1        ONLINE       0     0     0
ad14p1       ONLINE       0     0     0
raidz2         ONLINE       0     0     0
replacing    ONLINE       0     0     0
ad8p2      ONLINE       0     0     0
ad6p2      ONLINE       0     0     0
ad8p3        ONLINE       0     0     0
replacing    ONLINE       0     0     0
ad14p2     ONLINE       0     0     0
ad6p3      ONLINE       0     0     0
ad14p3       ONLINE       0     0     0
raidz2         ONLINE       0     0     0
ad8p4        ONLINE       0     0     0
ad8p5        ONLINE       0     0     0
ad14p4       ONLINE       0     0     0
ad14p5       ONLINE       0     0     0

errors: No known data errors

freenas02:~# df -h
Filesystem    Size    Used   Avail Capacity  Mounted on
/dev/md0       88M     76M     12M    86%    /
devfs         1.0K    1.0K      0B   100%    /dev
/dev/da0a      38M     35M    3.8M    90%    /cf
/dev/md1       31M    4.0K     28M     0%    /var
tank0         2.2T    1.9T    339G    85%    /mnt/tank0
freenas02:~#

freenas02:~# zpool status
pool: tank0
state: ONLINE
scrub: resilver completed with 0 errors on Thu Sep 24 05:05:00 2009
config:

NAME        STATE     READ WRITE CKSUM
tank0       ONLINE       0     0     0
raidz1    ONLINE       0     0     0
ad4p1   ONLINE       0     0     0
ad6p1   ONLINE       0     0     0
ad8p1   ONLINE       0     0     0
ad14p1  ONLINE       0     0     0
raidz2    ONLINE       0     0     0
ad6p2   ONLINE       0     0     0
ad8p3   ONLINE       0     0     0
ad6p3   ONLINE       0     0     0
ad14p3  ONLINE       0     0     0
raidz2    ONLINE       0     0     0
ad8p4   ONLINE       0     0     0
ad8p5   ONLINE       0     0     0
ad14p4  ONLINE       0     0     0
ad14p5  ONLINE       0     0     0

errors: No known data errors
freenas02:~# zpool export -f tank0

freenas02:~# gpt show ad8
start        size  index  contents
0           1         PMBR
1           1         Pri GPT header
2          32         Pri GPT table
34  1465149101      1  GPT part – FreeBSD ZFS
1465149135   122094000      2  GPT part – FreeBSD ZFS
1587243135   122094000      3  GPT part – FreeBSD ZFS
1709337135   122094000      4  GPT part – FreeBSD ZFS
1831431135   122094000      5  GPT part – FreeBSD ZFS
1953525135          32         Sec GPT table
1953525167           1         Sec GPT header
freenas02:~# gpt remove -i 2 ad8
ad8p2 removed
freenas02:~# gpt remove -i 3 ad8
ad8p3 removed

freenas02:~# gpt show ad8
start        size  index  contents
0           1         PMBR
1           1         Pri GPT header
2          32         Pri GPT table
34  1465149101      1  GPT part – FreeBSD ZFS
1465149135   244188000
1709337135   122094000      4  GPT part – FreeBSD ZFS
1831431135   122094000      5  GPT part – FreeBSD ZFS
1953525135          32         Sec GPT table
1953525167           1         Sec GPT header
freenas02:~# gpt add -i 2 -s 244188000 ad8
freenas02:~# gpt show ad8
start        size  index  contents
0           1         PMBR
1           1         Pri GPT header
2          32         Pri GPT table
34  1465149101      1  GPT part – FreeBSD ZFS
1465149135   244188000      2  GPT part – FreeBSD UFS/UFS2
1709337135   122094000      4  GPT part – FreeBSD ZFS
1831431135   122094000      5  GPT part – FreeBSD ZFS
1953525135          32         Sec GPT table
1953525167           1         Sec GPT header

freenas02:~# zpool import tank0
freenas02:~# zpool status tank0
pool: tank0
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid.  Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using ‘zpool replace’.
see: http://www.sun.com/msg/ZFS-8000-4J
scrub: resilver completed with 0 errors on Thu Sep 24 14:54:35 2009
config:

NAME                     STATE     READ WRITE CKSUM
tank0                    DEGRADED     0     0     0
raidz1                 ONLINE       0     0     0
ad4p1                ONLINE       0     0     0
ad6p1                ONLINE       0     0     0
ad8p1                ONLINE       0     0     0
ad14p1               ONLINE       0     0     0
raidz2                 DEGRADED     0     0     0
ad6p2                ONLINE       0     0     0
6422553418725777030  FAULTED      0     0     0  was /dev/ad8p5
ad6p3                ONLINE       0     0     0
ad14p3               ONLINE       0     0     0
raidz2                 ONLINE       0     0     0
ad8p4                ONLINE       0     0     0
ad8p5                ONLINE       0     0     0
ad14p4               ONLINE       0     0     0
ad14p5               ONLINE       0     0     0

errors: No known data errors
freenas02:~# zpool replace tank0 6422553418725777030 ad8p2
freenas02:~# zpool status tank0
pool: tank0
state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress, 0.53% done, 5h25m to go
config:

NAME                       STATE     READ WRITE CKSUM
tank0                      DEGRADED     0     0     0
raidz1                   ONLINE       0     0     0
ad4p1                  ONLINE       0     0     0
ad6p1                  ONLINE       0     0     0
ad8p1                  ONLINE       0     0     0
ad14p1                 ONLINE       0     0     0
raidz2                   DEGRADED     0     0     0
ad6p2                  ONLINE       0     0     0
replacing              DEGRADED     0     0     0
6422553418725777030  FAULTED      0     0     0  was /dev/ad8p5
ad8p2                ONLINE       0     0     0
ad6p3                  ONLINE       0     0     0
ad14p3                 ONLINE       0     0     0
raidz2                   ONLINE       0     0     0
ad8p4                  ONLINE       0     0     0
ad8p5                  ONLINE       0     0     0
ad14p4                 ONLINE       0     0     0
ad14p5                 ONLINE       0     0     0

errors: No known data errors

freenas02:~# zpool status tank0
pool: tank0
state: ONLINE
scrub: resilver completed with 0 errors on Thu Sep 24 16:34:09 2009
config:

NAME        STATE     READ WRITE CKSUM
tank0       ONLINE       0     0     0
raidz1    ONLINE       0     0     0
ad4p1   ONLINE       0     0     0
ad6p1   ONLINE       0     0     0
ad8p1   ONLINE       0     0     0
ad14p1  ONLINE       0     0     0
raidz2    ONLINE       0     0     0
ad6p2   ONLINE       0     0     0
ad8p2   ONLINE       0     0     0
ad6p3   ONLINE       0     0     0
ad14p3  ONLINE       0     0     0
raidz2    ONLINE       0     0     0
ad8p4   ONLINE       0     0     0
ad8p5   ONLINE       0     0     0
ad14p4  ONLINE       0     0     0
ad14p5  ONLINE       0     0     0

errors: No known data errors
freenas02:~# df -h
Filesystem    Size    Used   Avail Capacity  Mounted on
/dev/md0       88M     76M     12M    86%    /
devfs         1.0K    1.0K      0B   100%    /dev
/dev/da0a      38M     35M    3.8M    90%    /cf
/dev/md1       31M    4.0K     28M     0%    /var
tank0         2.2T    1.9T    339G    85%    /mnt/tank0

freenas02:~# zpool export -f tank0
freenas02:~# gpt show ad14
start        size  index  contents
0           1         PMBR
1           1         Pri GPT header
2          32         Pri GPT table
34  1465149101      1  GPT part – FreeBSD ZFS
1465149135   122094000      2  GPT part – FreeBSD ZFS
1587243135   122094000      3  GPT part – FreeBSD ZFS
1709337135   122094000      4  GPT part – FreeBSD ZFS
1831431135   122094000      5  GPT part – FreeBSD ZFS
1953525135          32         Sec GPT table
1953525167           1         Sec GPT header
freenas02:~# gpt remove -i 2 ad14
ad14p2 removed
freenas02:~# gpt remove -i 3 ad14
ad14p3 removed
freenas02:~# gpt show ad14
start        size  index  contents
0           1         PMBR
1           1         Pri GPT header
2          32         Pri GPT table
34  1465149101      1  GPT part – FreeBSD ZFS
1465149135   244188000
1709337135   122094000      4  GPT part – FreeBSD ZFS
1831431135   122094000      5  GPT part – FreeBSD ZFS
1953525135          32         Sec GPT table
1953525167           1         Sec GPT header
freenas02:~# gpt add  -i 2 -s 244188000 ad14
freenas02:~# gpt show ad14
start        size  index  contents
0           1         PMBR
1           1         Pri GPT header
2          32         Pri GPT table
34  1465149101      1  GPT part – FreeBSD ZFS
1465149135   244188000      2  GPT part – FreeBSD UFS/UFS2
1709337135   122094000      4  GPT part – FreeBSD ZFS
1831431135   122094000      5  GPT part – FreeBSD ZFS
1953525135          32         Sec GPT table
1953525167           1         Sec GPT header

freenas02:~# zpool import tank0
freenas02:~# zpool status tank0
pool: tank0
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid.  Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using ‘zpool replace’.
see: http://www.sun.com/msg/ZFS-8000-4J
scrub: resilver completed with 0 errors on Thu Sep 24 16:40:20 2009
config:

NAME                     STATE     READ WRITE CKSUM
tank0                    DEGRADED     0     0     0
raidz1                 ONLINE       0     0     0
ad4p1                ONLINE       0     0     0
ad6p1                ONLINE       0     0     0
ad8p1                ONLINE       0     0     0
ad14p1               ONLINE       0     0     0
raidz2                 DEGRADED     0     0     0
ad6p2                ONLINE       0     0     0
ad8p2                ONLINE       0     0     0
ad6p3                ONLINE       0     0     0
3452430189445945816  FAULTED      0     0     0  was /dev/ad14p5
raidz2                 ONLINE       0     0     0
ad8p4                ONLINE       0     0     0
ad8p5                ONLINE       0     0     0
ad14p4               ONLINE       0     0     0
ad14p5               ONLINE       0     0     0

errors: No known data errors
freenas02:~# zpool replace tank0 3452430189445945816 ad14p2
freenas02:~# zpool status tank0
pool: tank0
state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress, 2.53% done, 4h33m to go
config:

NAME                       STATE     READ WRITE CKSUM
tank0                      DEGRADED     0     0     0
raidz1                   ONLINE       0     0     0
ad4p1                  ONLINE       0     0     0
ad6p1                  ONLINE       0     0     0
ad8p1                  ONLINE       0     0     0
ad14p1                 ONLINE       0     0     0
raidz2                   DEGRADED     0     0     0
ad6p2                  ONLINE       0     0     0
ad8p2                  ONLINE       0     0     0
ad6p3                  ONLINE       0     0     0
replacing              DEGRADED     0     0     0
3452430189445945816  FAULTED      0     0     0  was /dev/ad14p5
ad14p2               ONLINE       0     0     0
raidz2                   ONLINE       0     0     0
ad8p4                  ONLINE       0     0     0
ad8p5                  ONLINE       0     0     0
ad14p4                 ONLINE       0     0     0
ad14p5                 ONLINE       0     0     0

errors: No known data errors

freenas02:~# zpool status tank0
pool: tank0
state: ONLINE
scrub: resilver completed with 0 errors on Thu Sep 24 18:20:52 2009
config:

NAME        STATE     READ WRITE CKSUM
tank0       ONLINE       0     0     0
raidz1    ONLINE       0     0     0
ad4p1   ONLINE       0     0     0
ad6p1   ONLINE       0     0     0
ad8p1   ONLINE       0     0     0
ad14p1  ONLINE       0     0     0
raidz2    ONLINE       0     0     0
ad6p2   ONLINE       0     0     0
ad8p2   ONLINE       0     0     0
ad6p3   ONLINE       0     0     0
ad14p2  ONLINE       0     0     0
raidz2    ONLINE       0     0     0
ad8p4   ONLINE       0     0     0
ad8p5   ONLINE       0     0     0
ad14p4  ONLINE       0     0     0
ad14p5  ONLINE       0     0     0

errors: No known data errors
freenas02:~# df -h
Filesystem    Size    Used   Avail Capacity  Mounted on
/dev/md0       88M     76M     12M    86%    /
devfs         1.0K    1.0K      0B   100%    /dev
/dev/da0a      38M     35M    3.8M    90%    /cf
/dev/md1       31M    4.0K     28M     0%    /var
tank0         2.2T    1.9T    339G    85%    /mnt/tank0
freenas02:~# zpool export -f tank0
freenas02:~# shutdown -r now

Reboot

Check disk size

freenas02:~# df -h
Filesystem    Size    Used   Avail Capacity  Mounted on
/dev/md0       88M     76M     12M    86%    /
devfs         1.0K    1.0K      0B   100%    /dev
/dev/da0a      38M     35M    3.8M    90%    /cf
tank0         2.3T    1.9T    453G    81%    /mnt/tank0
/dev/md1       31M    4.0K     28M     0%    /var

Advertisements
This entry was posted in ZFS and tagged , , . Bookmark the permalink.

5 Responses to Replacing a 750Gig with a 1TB drive in my FreeNAS ZFS (Drobo like) server config

  1. Jimbo says:

    That post is exactly why I have a Drobo at home. If I had more free time, I still wouldn’t want to go through that nightmare.

    Jimbo

  2. Gregg says:

    Glen,

    Thanks for taking the time to document this. I am many others appreciate it.

    Cheers,

    Gregg

  3. koitsu says:

    Aside from the GPT insanity, this looks (more or less) like the normal procedure used for replacing a disk in a zpool. I’m not sure why user “Jimbo” thinks this is a nightmare; try looking at Linux LVM if you want a nightmare. 🙂

  4. Jon says:

    Hello there,

    I have found your great articles about allowing for ZFS hard driver replacement and initial setup. What I am still trying to fathom is why you need to setup 5 different RAIDZ sets before making the pool. Couldn’t it be done just as easily with only 3 RAIDZ sets?

  5. ajones says:

    Well Jimbo’s saying it’s a “nightmare” compared to an actual Drobo where you just swap out the drives and it’s done. None of this shutting down, zpool / gpt business.

    Speaking of which, this post is over a year old, but I’ve yet to see anyone come up with an AUTOMATED way of doing this similar to the drobo. How hard could it really be?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s