RAIDZ Expansion feature discussion

[justinclift]

This is carrying on from PR #15022, so we don't crowd the PR out with general discussion traffic. 😄

[yorickdowne]

So to summarize that: Afaik, a copy command will rewrite the data with the new width after expansion. I think I saw a script sloshing around that basically does that for an entire array, cleaning up after itself. Does it file by file for space reasons. Takes a minute, but, you know. Only have to do it once.

From my perspective that’s plenty good enough. Raid expansion is a niche feature, if much beloved by that niche. Adjusting width is then a niche within the niche. Handling that with a community script sounds like an entirely reasonable thing to do. Permissions and paths stay as is, filenames stay, last accessed and creation date changes and width changes.

To explain the niche comments: I am part of that niche. Home user but without the funds to start at full width. I did a poor man’s raid expansion eventually, from 5 to 8. So the niche are people who start at a smaller width than they know they’ll need, and will only run one vdev. By definition home users, and then only a fraction of those.

The niche within the niche are users where rewriting archival data makes an appreciable difference, or that care even though it doesn’t.

That need not be handled in the file system. Handling it with good UX outside seems great. Setups that use ZFS like TrueNAS and Proxmox could even integrate such a script into their UI.

[IvanVolosyuk]

@TurkeyMan
The 'assumed parity' is explained here by Matthew Ahrens after this #12225 (comment)

[Dulanic]

I'm part of that niche crowd, and I'm cool with it. If I decide to expand, I'm fine with keeping the old setup until it gets a rewrite. Storage is pretty cheap these days, which is why I'm looking to expand. That being said, I think my use case is decent. Right now, I've got two vdevs, but one of them has a bunch of older, 8TB drives. It just doesn't make sense to swap an 8TB with another 8TB when a 18TB is way cheaper per TB. As such, it just makes much more sense to expand my current vdev with 18TB drives and end up removing my old vdevof 8TB drives.

So, I much follow the path that you did.

With that said, when I remove my 2nd vdev, my understanding is zfs will shift my data over to the 1 remaining vdev. If this code/process exists, couldn't that process be called, albeit maybe modified, to balance it out again when expanding? Maybe not since its just moving the data from vdev to vdev vs adding a drive to expand which I guess would be a very different flow. This is where I may be a bit lost.

[IvanVolosyuk]

You can swap individually each 8TB with 18TB drives and once all of them are 18TB you can expand them to occupy entire space of the new drives.

[nonlinearsugar]

This script, with caveats (read the readme), helps automate rewriting your data by copying/pasting/deleting/moving at the filesystem level.

https://github.com/markusressel/zfs-inplace-rebalancing

[Dulanic]

FYI, Link is bad - goes to https://github.com/openzfs/zfs/discussions/url

[nonlinearsugar]

Fixed. Thanks.

[the8472]

The script doesn't use direct IO which means the data might end up sitting in caches and you're only hashing stuff from the page cache, not reading back what makes it to disk. I think this leaves a small window for data corruption, especially on systems without ECC memory.

[ahrens]

This is cool, and I think solves the problem well if you don't have snapshots.

If you wanted to preserve space sharing between snapshots and clones, you could use zfs send to copy entire "clone families" (all filesystems and snapshots related by cloning and snapshotting). The downside would be that you'd need enough free space for another copy of your largest "clone family", rather than just your largest file. The zfs send approach would also work with zvols.

[yorickdowne]

@Dulanic you can’t removed a raidz vdev from a pool. Mirror vdevs only (single disk vdev is special case of mirror) can be removed.

You may be able to create a new pool with your 18TB drives, copy data over, then expand that. This really depends on how much you’re using already though. This is why a lot of home users stick with one vdev, lest they are “stuck” with two to manage.

I have one 8x8 raidz2 which will likely last me forever. But in case it ever doesn’t, I intend to replace drives as they fail with larger-capacity ones. Once the last drive has been replaced, I’ll have more space.

[Dulanic]

Oh no! Lol I am still learning! That's a problem lol I have 80TB already! If I have 2 vdevs can I expand 1 if them? I wasn't sure if I needed to match the width.

[TurkeyMan]

My current take on this, the stripe-width expansion seems less of an urgent issue than the fact it only adds one drive at a time.
As advertised, at 50mb/s, it will take 4 days to expand my vdev. I can handle it taking 4 days I suppose, but the real problem is it can only expand one drive at a time. So, expanding 4-wide raidz2 to 8-wide, that'll take 2 or 3 weeks... and it's not clear how much the standard operating performance may be degraded during that time? Is there any additional risk from drive failure during that time?
It's a very long timeframe to expand a vdev.
It seems that this feature really needs to allow adding multiple drives at once...

[yorickdowne]

Unsurprisingly people have been making wider raidz2 vdevs for years, even without official expansion support. The “use sparse files for parity and offline them” trick has gotten good mileage :).

[ahrens]

FYI: Time to expand is limited by CPU usage. Improving it by 10x would be a relatively straightforward follow-on project. Also note that the 50MB/s that I measured was with 512-byte sector size. With 4K sector size it would be around 8x that (~400MB/s).

[justinclift]

Improving it by 10x would be a relatively straightforward follow-on project.

Is that something likely to be done in the near-ish future? 😄

[darkbasic]

Improving it by 10x would be a relatively straightforward follow-on project

I think that would be make raidz expansion much more viable.

[ahrens]

Is that something likely to be done in the near-ish future? 😄

Not by me, but PR's are welcome 😄

[GrayXu]

Is RAIDZ "contraction" support included in this PR? 🤔

[GrayXu]

Reasonable.
But since not all data in a zpool requires the same level of protection, I think the ultimate solution should allow different datasets to have different levels of parity protection. Like in the existing RAIDZ expansion, not all data has the same number of parities for protection.

[yorickdowne]

I’d really rather not see any such thing. ZFS is wondrously complex as is. Per-dataset parity adds another level of complexity for no gain that I can see.

In this expansion PR, all data have the same parity in the expanded vdev. It’s the stripe width that changes, not the parity.

If I recall right there has been discussion of a follow-on PR that’d expand (but not contract) parity, such as from raidz1 to raidz2 or raidz3. That’d be separate work, and whether it will ever be tackled is a good question.

The existing mechanisms for changing pool layouts - make a new pool, and/or add vdevs - work for the use cases I can see. Raidz expansion is a lovely hobbyist use case and I am stoked it’s getting there; raidz parity expansion would be another hobbyist use case but even more niche.

[amotin]

The way expansion is implemented now the contraction is simply impossible. Even in unlikely case that end of the vdev is absolutely empty to be dropped, contraction would not produce sufficient level of redundancy to sustain disk failures.

[GrayXu]

@yorickdowne @amotin thanks for reply!

1. Regarding the issue of data reliability. The number of data dev and parity dev are both related to reliability, and expanding stripe width will also reduce system reliability. Similar topics are widely researched with different coding parameters in erasure coded systems.
2. Regarding different datasets having different levels of protection. Obviously, we can achieve different levels of protection through manual partitioning to different zpools, but this cannot achieve the effect of sharing the same storage space. I think it is reasonable to have different types of files requiring different levels of protection, such as both 3-way replicas and tape replicas in data centers. So I am particularly curious about the engineering challenges of this on ZFS.

[justinclift]

@GrayXu While some of that sounds like a potentially interesting discussion topic, it's probably best to create a new discussion for it. It's not very on-topic for this one. 😄

[yorickdowne]

To @farkeytron: Discussion is here, not in the PR. IxSystems is paying for this work, so it’ll definitely come into TrueNAS. SCALE first likely, since that is where canary features land, then Core in due time.

[kmoore134]

Lol, very appropriate meme :)

This feature will involve more middleware and UI changes to support properly, which means it will be targeted at SCALE only. CORE is mostly feature frozen, so we aren't making big sweeping changes to features / backend / front-end. SCALE is where active development resides.

[MihaMarkic]

"next major update to our ZFS base" What timeframe is this? Next major build after Cobia? Sorry if I've missed this info.

[kmoore134]

24.10 (Electric Eel - October release) is the next major TrueNAS SCALE release that will include ZFS 2.3.

[CBattershillDev]

Reading this makes me want to upgrade my core server to scale but migrating things like nextcloud configurations and data makes me concerned. Is it a certainty that this feature wouldn't make its way to core? (even if it was just for cli access)

[kmoore134]

That feature won't make it into 13.3, so odds are low it will land in CORE in the next 12-24 months. If you need a hand with migrating nextcloud we've had plenty of other folks do that. Might be worth asking questions here:

https://forums.truenas.com/

[KungFuJesus]

we have this issue with ZFS encryption support after Datto stopped contributing to it.

@bghira Is there more info on that around?

Asking because I'm using ZFS encryption on some systems, and hadn't heard mention of potential upcoming problems with it. Would like to know before problems potentially bite. 😄

Hah, I take you don't follow the issue tracking on github too carefully, then. The amount of issues I see with the encryption support (and not just performance problems, but like, swallowed my data or corrupted it on a replication ones) has made me pretty hesitant to recommend ZFS's native encryption support for anyone.

[justinclift]

Hah, I take you don't follow the issue tracking on github too carefully, then.

Yeah, not at all. I think this discussion and it's matching PR are the only things I'm subscribed to in the repo.

swallowed my data or corrupted it on a replication ones

Damn. Sounds like we'll need to investigate that, and potentially find an alternative to ZFS then. Or at least an alternative to native ZFS encryption. 😦

Can't be having unreliable storage backend for production systems. 🤦

[darkbasic]

I use encryption in production: it's not worse than many other features like zvol (#11407).
zfs is not perfect, but what's the alternative? dm-crypt + grub2 patched for argon2? I find it extremely irritating that without out of tree patches your system is as secure as one without encryption. If I have to sign my own bootloader (because I have to build grub from source) I'd rather sign a zfs module instead.

[justinclift]

@don-brady @ahrens Thank you SO much for getting the ZFS expansion PR across the line.

It'll likely be a key capability of ZFS from here on (in the next release onwards of course). 😄

[Sparklingx]

is the expansion option for all setups including stripe?

[mumbleskates]

it's for raidz vdevs specifically, which have 1-3 devices worth of parity.

[KaeTuuN]

This Feature is not for switching from stripe to raidz1-3!

[IvanVolosyuk]

Quick question: if I update to zfs master and enable corresponding feature flag for raidz expansion, will it be active or enabled after expansion completes? Thus, can I downgrade to stable branch after that or there will be some incompatibilities in on disk layout and I'm limited to master branch after that until the next stable release.

[amotin]

On-disk format after expansion is different from one supported before this feature. So I bet the feature stays active forever.

[ahrens]

That's correct. After expansion the feature will be "active", and you can't downgrade the software to an earlier version that doesn't support RAIDZ Expansion.

[emaste]

A couple of days ago @mmatuska updated OpenZFS in FreeBSD main, including RAIDZ Expansion. You can try this out with the next set of 15-CURRENT snapshot images.

freebsd/freebsd-src@e716630

[badmannen]

I tried to read through the DOCS here but still not sure if I can extend my 2-disk VDEV ( only option to create was a mirror ) Can this be expanded with a 3rd disk as RAIDZ-1 ?

[yorickdowne]

No. A mirror vdev is not raidz-1. But this is what you can do:

Break the mirror, remove one drive. Take that drive, a second fresh one, and a sparse file, and make your raidz1 in a new pool, not the same pool! Offline the sparse file.

You now have two pools with one vdev each, and no redundancy. Hope you have a good backup.

zsend/zrecv from the old pool to the new one

verify everything is there, twice. Because the next step means you can’t go back.

destroy the old pool, replace the offlined sparse file in the raidz1 with the now free drive. Let the raidz1 resilver.

This would also work with a raidz2 and two new drives. Which is a bit more future proof. Raidz1 with larger drives has a tendency to break during resilver.

[badmannen]

With my N00b level of understanding you did a great job of explaining. With that said I think I lean towards Backup-copy My 2 disk mirrored Vdev, then create a brand new with my now 3 disks and then copy back the data into the new vdev? ( more failsafe and faster ? or am I missing something , got 14 TB disks)

[yorickdowne]

With 14TB disks raidz-1 is risky when rebuilding. I recommend you do a 4-wide raidz2, which you can then expand later if desired.

[badmannen]

Sounds like a good solution ... just need to squeeze my wallet upsidedown and hope some monetas will fall out from it =) ( but yes If I don´t I will probably regret it later on )

[yorickdowne]

“Buy nice or buy twice” :)

[DurvalMenezes]

This is arriving at a really good moment for me, I was about to expand a 4-disk raidz2 into a 6-disk by the old method (create an entirely new pool then zfs send | recv from the old one into it), so if this is implemented and working, it's going to save me a ton of time and effort.

So I must ask:

1. IIUC what I read here and at the PR, it's implemented on zfs-master, right?
2. Is there an estimate for when this will be available in a main release?
3. Has anyone used it already? How did it go?

TIA!

[DurvalMenezes]

Thanks @yorickdowne and @darkbasic for your responses.

And I just read a little more about the limitations, and it seems the data-to-parity ratio isn't recalculated either, so a ton of space would end up being lost if the expansion is done using this new functionality.

Seriously, we've been waiting since forever for... this? :-( don't want to sound ungrateful but it's really disappointing to see that ZFS remains ITR so far behind what other folks (eg Linux md) have been doing for literally decades.

Anyway, back to the ancient zfs-send-recv-into-new-pool I go...

EDIT: the situation is much less worse than I was led to believe, see my next comment below,

[darkbasic]

it seems the data-to-parity ratio isn't recalculated either

That is just until you rewrite the existing data. If your data is going to change over time (or if you don't care about snapshots and simply rewrite it) you will regain the lost space.

[ahrens]

I might very well be wrong but I think you won't be able to expand existing arrays, only new ones created after the feature has been enabled.

That is incorrect. You can expand existing RAIDZ groups.

[darkbasic]

That is incorrect. You can expand existing RAIDZ groups.

That is awesome, I've never been so happy to be proven wrong :)

[DurvalMenezes]

Thanks @ahrens and @darkbasic for the additional corrections/clarifications. So the situation is much better than I was led to believe.

The major (only?) conses seem to be the (a) inability of the upgraded array to be handled by previous versions of the software not supporting this feature, and (b) desirability of rewriting all data in order to "rebalance" already-existing data (which I understand means both to redistribute its data on the new array including the added vdevs, and recalculate its parity blocks for the new scheme).

(a) is not a major problem (just test the new version well enough to be damn sure no version downgrading will be necessary) and (b) is just an inconvenience for my use case (I presume that, once I have the necessary disk space, I can just shuttle the data inside it from the previous datasets to new ones to force the rewriting with zfs send|recv, and then destroying the pre-rebalance datasets and renaming the new ones to the old names).

I will now mark this on my original comment above.

[jakesteele]

How does expand address fragmentation, if my cap is under 50% does the end result remove most/all fragmentation? What if my capacity is over 50%, does the end result in more fragmentation then before?

[ahrens]

The newly-available space will be one big free chunk, but expansion doesn't change fragmentation of the existing data.

[jakesteele]

Gotcha, I figured expansion when re-writing between disks would fix fragmentation if the free space was greater than 50%.

[GottemHams]

I'm considering upgrading the disks in my system with something faster, and add an extra disk in the process. So let's say I first add the extra disk and let ZFS expand the pool, then swap out the pre-existing disks one by one (or perhaps connect the faster one alongside) and let it resilver each time. Would the resulting pool have the updated data-parity ratio? I would assume so since eventually all disks would have newly written data, but because it happened in response to a resilver it may not take the earlier expansion (fully) into account? As in: disk X is pretty much copied bit-for-bit to replacement disk X, resulting in the same original ratio (only that disk has to be rebuilt so there's no (re)balancing of data-parity). I don't think there's enough room to connect all the new disks alongside the original ones and just send it over. :DD

[Cosine5000]

You realize this functionality is still a year or two away, yes?

[GottemHams]

Ever heard of planning? If the scenario I described wouldn't work then it's probably better to find a way to make room for a new pool and send over the data after all, which I could do right this moment. Otherwise, having to wait for 1-2 years with the current setup is perfectly acceptable.

[Quadrubo]

I'm planning to use ZFS for my homeserver with proxmox. Would it be possible to setup a 2 Drive RAIDZ1 right now and add a third drive later using this method when it releases?

[justinclift]

Yeah, that's the kind of thing this is for. 😄

[Quadrubo]

Awesome thank you!
Wasn't even sure if RAIDZ1 with just 2 drives is a valid configuration.

[justinclift]

Oh, that's a good point. I think the smallest # of drives I've tried with a RAIDZ1 is actually 3. I'm not actually sure about 2. Err... oops? 🤦

[albertz]

Sure, RAIDZ1 with 2 drives is perfectly valid. I'm using that right now.

[IvanVolosyuk]

2 drive raidz is just a mirror with crippled performance. You better of creating real mirror now and one you have all 3 drive - split the mirror and create degraded raidz from 2 of 3 drives and zfs send all the data from the 3rd. After that you can attach 3rd to make it redundant again. You will sacrifice redundancy during migration though.

…

On Thu, Feb 29, 2024, 02:55 Quadrubo ***@***.***> wrote: Awesome thank you! Wasn't even sure if RAIDZ1 with just 2 drives is a valid configuration. — Reply to this email directly, view it on GitHub <#15232 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXQ6HJZEA2NGDSHANUI3LDYV5HP7AVCNFSM6AAAAAA4HOSFKSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DMMRQGM3DM> . You are receiving this because you commented.Message ID: ***@***.***>

[enekux]

Hi when is this feature going to be available? From what I know it was said that will be added with "OpenZFS 2.3" version but do we have some ETA? Thanks

[DieSteinhose]

It is supposed to be added to TrueNAS Scale 24.10 Electric Eel. So probably before October.
ref.: (NAS-123548).

[Xyz00777]

so TrueNAS Scale 24.10 got released 6 days ago and its in the rchangelog 🎉
BUT when does it get released for everyone else? ZFS2.2.6 got released yesterday and at least i dont find it in the change log. so is it still version 2.3 when it get released and when will these be? Like still these year, in Q1 next year?

[justinclift]

@Xyz00777 Cool. Yeah, that looks interesting: https://www.truenas.com/docs/scale/gettingstarted/scalereleasenotes/

Seems like there's even documentation in place for it too:

https://www.truenas.com/docs/scale/scaletutorials/storage/managepoolsscale/#extending-a-raidz-vdev-

Note that TrueNAS Scale 24.10 release is still a beta though, not the full release. So the software is clearly still "in development".

Hopefully that gets a bunch of people testing it out, and if there's bugs then they get fixed before it becomes widely available for the rest of us (ZFS 2.3?).

[dany-nonstop]

ixSystems just posted this release 24.10 version notes a week ago. It has a clear answer to your question (and many other peoples as well). Put on a big grin on your face and be ready for this Thanksgiving or Christmas gift, depending on how adventurous you are. Mark the date on your calendar, and make your hard drive shopping list accordingly. 😄 highlight by me for those with bad eyes like me.

This wait had been like years. Kudos to @don-brady and all who contributed.
TrueNAS Scale 24.10.0 Release Note

[kmoore134]

Yes, special thank you to everybody on the OpenZFS side who helped get this one over the finish line. We've got this in our TrueNAS 24.10 BETA1 release already for testing, anticipating updating to OpenZFS 2.3 as soon as it is tagged.

[djjaeger82]

IMO "rewriting the data" should be part of the expansion process and part of this feature before release. I suppose making it optional at the time of expansion would be fine, but not too happy its up to users to manage/delete snapshops and copy/delete data to be able to take advantage of the expanded free space properly

[Xyz00777]

So do I understand it correct?
When data are not in a snapshot than they will get rewritten, but when they are than not?

[darkbasic]

That's not what I said. If zfs didn't support snapshots it would have been possible (even advisable) to automatically rewrite data after an expansion in order to regain space. Since zfs does support snapshots such task cannot be automated because it requires deleting the existing snapshots first.

[Xyz00777]

oh okay, but wouldnt it be an idea to configure it to ask an person if he have no snapshots if he want to rewrite the data and if he have snapshots to ask the person if he want to delete these to do a rewrite? So the person can choose what he want to do... like im planing to add 4 drives to my 4 drives z2 setup and than i would do an rewrite just after i added the last one... or is there any way else to trigger an rewrite without moving the files from the drive to an other and back? because is ne point in not asking to rewrite the data

[amotin]

@Xyz00777 There is just no code implementing the rewrite. People used some user-level scripts to rebalance vdevs, which is quite a similar problem, but it is unsafe to do under load. I am thinking about implementing that on ZFS level, but so far only thinking.

[diego351]

@amotin Yes please!

[RampantAndroid]

I was wondering if someone can comment on the state of this feature? The PR is complete, but it's not in OpenZFS 2.2 and it looks like it's slated for 2.3.

iX is however shipping this feature in their next release, which is tentatively end of October 2024. Is iX shipping a feature not yet ready for general use? Or is OpenZFS simply holding this back for version reasons waiting on on other changes slated for 2.3?

[ndoo]

https://github.com/openzfs/zfs/releases/tag/zfs-2.3.0-rc1

[robn]

It's not in 2.2, and will be in 2.3.

You should probably ask specific vendors what they're shipping, if it's different to what's here. My understanding is that iX are shipping a 2.3 prerelease, but I don't know if they've made other changes.

[aaomidi]

Not exactly related to raidz expansion, but more so changing the form of a ZFS pool.

What are the blockers on allowing users to switch their pool from a mirrored setup, to a radiz setup (either offline, or online). It seems like it should be possible to go from mirror -> raidz, but obviously not in reverse.

[amotin]

While 2-disk mirror and 2-disk RAIDZ might look pretty similar (if not identical) on the physical level, they have completely different block pointers and space accounting, that makes such conversion pretty difficult.

[Overwatching]

@amotin
I had to re-read @aaomidi post because I initially thought you misunderstood their question.

I thought they were asking about starting with a 2 disk mirror, adding a 3rd blank disk, then converting to a z1 pool using the 3rd disk as parity.

This initially seemed rather 'easy'.

1. Compute parity for blocks 1 and 2. Write parity to Disk3.
2. Confirm parity.
3. Copy block 1 to Drive1 and block 2 to Drive2. (So they are in the same 'stripe'. Or maybe just delete block 1/2 from Drive 2/1.)
4. Repeat with each set of 2x blocks.

But if all the bookkeeping is that different between the VDEV types, that now seems WILDLY more difficult.
Something conceptually like:

1. Shrink Mirror VDEV to 'free up' space for the new VDEV type.
2. Create new Z1 VDEV in the remaining space on Drive1, Drive2, and Drive3 (Same size and place as 1&2. leave the other part empty here)
3. Copy Block(n) & Block(n+1) to Z1 VDEV.
4. Delete Block Block(n) & Block(n+1) from Mirror VDEV.
5. Shrink Mirror VDEV.
6. Grow Z1.
7. Repeat Steps 3-6 until Mirror VDEV is empty.
8. Delete Mirror VDEV.
9. Grow Z1 VDEV to fill entire drive.

Assuming you could repeatedly shrink/grow the VDEVs, this is actually kind of a simple concept.
It would be pretty insane in practice though. That's so much bookkeeping.
Ideally you wouldn't do 2 blocks at a time. You would want to do 2-5% of the drive size per cycle. Or more.

Then again, if it was really that easy, we could have used the same process to add a drive to a raidz array, so I'm assuming that the process simply wouldn't work or that there is some major problem with it that I can't see.

[amotin]

As I have told, 2-disk mirror and 2-disk RAIDZ effectively have the same on-disk data since parity for one block is the block itself. But the problem is that when block pointer is created mirror vdevs are equivalent to single disk, just measuring offsets from the beginning of the disk to its end. RAIDZ same time measure offsets through all the disks in the vdev. The same is with space accounting: mirror capacity is equal to one disk, while RAIDZ -- a sum of all disks. That is why straight conversion would be problematic.

But if you just need to convert from 2-disk mirror to 3-disk RAIDZ, you could take one of the disks from mirror, then create new 2-disk RAIDZ pool, copy all the data from the old pool and then add use RAIDZ expansion to get from 2-disks RAIDZ to 3-disk. Two downsides of this scheme is a loss of redundancy during the process and low space efficiency as result of conversion. To fix the last it would be nice if it would be possible to create 3-disk RAIDZ with one disk missing, but it is not supported now, even though was discussed several times and should not be hard. Or it may be possible (I haven't tried) to do the trick via a temporary file instead of one of the disks and delete it just after successful pool creation.

[m-sundman]

I've upgraded raidz pools many times by first scrubbing the old pool (don't forget the scrub!!!), then taking all redundant disks offline (so the pool is degraded, but can still be used), them used those disks while creating a new raidz including temporary (sparse) files as placeholders for new redundant disks, then taking the sparse files offline (so the new pool is degraded but can still be used), then copying the data from the old pool to the new pool, then deleting the old pool, and using the disks from there to replace the offline disks in the new pool, then resilver.

[aaomidi]

Yeah in my case I have a 4 disk mirror I wanted to convert to a 4 disk RAIDZ (1 parity, 3 data).

Alright, well it seems consensus for this is that I'm probably going to need to just get more disks and just move the data over.

[Bommel24]

Would it be possible to use a similar approach to implement RAIDZ Shrinking. So if less space is used we can remove a disk permanently without losing parity?

[IvanVolosyuk]

No. Imagine the current version keeps all block numbers when reshuffling all the data. Conceptually, in shrinking the range of numbers decrease and there is no guarantee that all numbers currently in use will be available in the new layout even if you do the process in reverse order.

[tavern2782]

My understanding is that after an expansion, old data remains on the old logical stripe width and only new data gets the benefit of larger logical stripes.

Is that still correct with this feature now being released? If yes, is there a way to convert old data to use the new, larger stripe?

[amotin]

Yes, that is correct. The only way to convert it is overwrite the data one way or another. For example, replicate the datasets locally and delete the original, if you have enough free space. I am dreaming about a way to rewrite data in place, but haven't got to implementing it yet.

[Xyz00777]

@amotin could the issue #15335 be used for that?
I think it would also solve issue #15679

[amotin]

@Xyz00777 Scrub is not designed to modify anything. It just checks and recovers what is already there. #15335 is only a dream. In practice we've already had some "pointer rewrite" code as part of device removal, but it was dropped AFAIK because of inability to properly lock simultaneous accesses from mounted file systems. The design I am dreaming about assumes operation in open context on requests from user-space, alike to read()/write() system calls.