Fix two issues with dynamic gang headers

[pcd1193182]

Sponsored by: [Klara, Inc.; Wasabi Technology, Inc.]

Motivation and Context

When running some tests on a heavily fragmented system using raidz, I noticed a large number of checksum errors. I investigated, and determined that these were occurring when gang blocks were created. I modified the gang block tests to use raidz instead of plain vdevs, and the issue occurred in the tests immediately.

Description

First, there are test changes to better test more configurations. Most of the test changes are just indentation, but the calls to check_not_gang_dva have been replaced with calls to the new functions check_gang_bp and check_not_gang_bp respectively, which is what should actually have been tested there.

Next, there is a fix in zio_checksum.c for the case where there are more intermediate ZIOs between the one being checksummed and the gang header. This occurred in mirror devices in my testing, but would also occur if you had raidz devices being replaced, or possibly indirect vdevs.

Finally, there is a fix in zio.c for the raidz code. The problem we were running into in that case is that we were using the asize of the whole allocation used by the gang header as the psize of the IO, which resulted in an even larger read/write being done. We need to convert back down from the gang_header_asize to the actual psize that write provides.

How Has This Been Tested?

Ran the updated gang block tests, verified that on the original system where I saw the issue this prevented checksum errors from occurring.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Quality assurance (non-breaking change which makes the code more robust against bugs)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[behlendorf]

That's subtle, but the fix makes good sense. As does extending the test case to cover raidz and draid.