Anyraid phase 2; raidz, rebalance, and contraction

[pcd1193182]

Note: only the final 4 commits are part of this review, all previous commits are part of #17567 , which is a prerequisite for this PR.

Sponsored by: Eshtek, creators of HexOS
Sponsored by: Klara, Inc.

Motivation and Context

AnyRAID Phase 1 introduced the AnyRAID architecture and added support for AnyRAID vdevs with mirror-style parity. Additional work, however, is required to support raidz-style parity as well as other features that the AnyRAID architecture can support, like rebalancing tiles between disks and removing disks from the AnyRAID vdev.

Description

This PR has a few sets of changes.

The first is a set of changes that fix small issues in the AnyRAID code and generally prepare for later patches.

The second is the raidz-style parity option for AnyRAID. This is implemented as a separate set of vdev ops that can be selected with the appropriate command line arguments. anyraidz has an additional parameter to anymirror; in addition to a parity count, it also has a data width. anyraidzX:Y will store data in a similar layout to a raidzX vdev with Y child disks; any additional disks in the AnyRAID vdev will be use to balance the distribution of tiles and provide improved space utilization. The anyraidz IO code mostly leverages the existing raidz code to construct the raidz map and dispatch the child IOs, but it does have an additional step where the offsets and disk IDs are modified to take into account the indirection of the tile map. In addition to the core changes to enable raidz parity, there are some other changes to the AnyRAID code to take into account that stripe widths are no longer just nparity + 1, and to more easily check if a vdev is an AnyRAID vdev of either type elsewhere in the kernel. Finally, there are additional unit tests and ztest support for raidz-style parity.

The next commit adds the rebalance functionality to AnyRAID. When a new device is added to an AnyRAID vdev, the new device has no allocated tiles. If there are sufficient free tiles left in the existing vdevs to take advantage of all the new space, that's not a problem (mostly; we still aren't getting to use the IOps/bandwidth of the new device until tiles are allocated on it). If, however, the vdev was very close to full, then the new device will not be able to be fully utilized. Rebalance solves this problem by selecting tiles from existing vdevs and moving them to the newly empty device.

Rebalance borrows some of its architecture from the raidz expansion feature. First, a plan is generated that moves tiles from fuller disks to emptier disks. We repeatedly consider the fullest disk, select an allocated tile on that disk, and find a new place to move it. If there is nowhere to move it that is less full than the source disk, we must have finished the balancing process. Note that there is some additional logic here to prevent and handle attempting to move tiles such that multiple tiles from the same stripe would be on the save physiacl device. Once the plan is generated, it is persisted into a MOS object. Then, the relocate thread is started. The relocate thread is much like the raidz expansion thread; it runs in the background, issuing IO and completing tasks. Each TXG its progress is synced out. Once the tasks have all been completed, the relocation is marked as complete. At that point a scrub is started, to check that all of the data was moved correctly. Until that scrub completes, we don't actually free the tiles that were previously in use, so that we can attempt data recovery if something went badly wrong.

The rebalance commit also includes some tests and ztest support.

The final commit is the contraction feature. Contraction solves the problem of wanting to be able to remove devices from an anyraid vdev. In cases where storage capacity is no longer needed or a device is beginning to show signs of age but replacements are not available, the ability to fully remove a child device from the AnyRAID vdev is valuable.

Contraction has a similar overall architecture to rebalance, with a few caveats. The first is that if a plan cannot be generated to move all of the tiles off of the device being removed, we have to fail the contraction. This is not expected to be a major issue in practice unless the vdev is nearly full. There can be cases where this happens using the implemented algorithm but a different algorithm would be able to find a valid solution; for now, the greedy algorithm does a good enough job that this isn't necessary to fix, but this could be an area of future work. Second, because contraction actually reduces the allocatable size of a vdev, there are restrictions around that. The vdev cannot be contracted if there is data allocated in the tail area of the vdev that would eventually not be present after contraction would complete, since that data needs to be moved first. In the future, a mini-device-removal process could allow that data to be moved if sufficient free space exists in earlier tiles. Finally, because contraction requires an actual change to the vdev configuration, it cannot be performed if there is a checkpoint.

The contraction patch contains the plan generation logic for contraction, as well as some other changes around starting and managing a contraction. It also contains changes to the vdev logic necessary to allow for shrinking a top-level vdev, something that was not previously possible. Finally, zfs-test and ztest support are added for contraction.

How Has This Been Tested?

There are a variety of new zfs-test suite tests that were written for this project. In addition, ztest support was added for many of the new features. Both were run repeatedly and extensively to find and resolve issues.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Quality assurance (non-breaking change which makes the code more robust against bugs)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.