Cannot receive snapshots on Ventura / 2.1.6

[sper23]

Hi,
Since upgrading to Ventura & installing OpenZFS 2.1.6 I have been unable to receive snapshots— certainly incremental streams, and I think complete snapshots as well. The error is "cannot receive incremental stream: invalid backup stream."

Here's a demo. Machine 1 is running MacOS 12.6.3 and OpenZFS 2.1.0. Machine 2 is running MacOS 13.4 and OpenZFS 2.1.6. You will see that the receive operation concludes successfully on Machine 1 but fails on Machine 2.

Thanks.

Code: Select all

machine1:~ root# uname -a
machine1.local 21.6.0 Darwin Kernel Version 21.6.0: Mon Apr 24 21:11:35 PDT 2023; root:xnu-8020.240.18.701.5~1/RELEASE_ARM64_T8101 arm64
machine1:~ root# zfs -V
zfs-macOS-2.1.0-1
zfs-kmod-2.1.0-1
machine1:root root# dd if=/dev/zero of=afile bs=1024k count=64
64+0 records in
64+0 records out
67108864 bytes transferred in 0.026766 secs (2507242920 bytes/sec)
machine1:root root# zpool create testpool /var/root/afile
machine1:root root# zfs create testpool/fs
machine1:root root# zfs snapshot testpool/fs@asnap
machine1:root root# zfs snapshot testpool/fs@bsnap
machine1:root root# zfs send -i testpool/fs@asnap testpool/fs@bsnap > bsnap_incr
machine1:root root# zfs destroy testpool/fs@bsnap
machine1:root root# zfs receive -Fdv testpool < bsnap_incr
receiving incremental stream of testpool/fs@bsnap into testpool/fs@bsnap
received 312B stream in 1 seconds (312B/sec)
machine1:root root#

---------------------------------------------

machine2:~ root# uname -a
machine2.local 22.5.0 Darwin Kernel Version 22.5.0: Mon Apr 24 20:52:24 PDT 2023; root:xnu-8796.121.2~5/RELEASE_ARM64_T6000 arm64
machine2:~ root# zfs -V
zfs-macOS-2.1.6-1
zfs-kmod-2.1.6-1
machine2:~ root# dd if=/dev/zero of=afile bs=1024k count=64
64+0 records in
64+0 records out
67108864 bytes transferred in 0.018230 secs (3681232255 bytes/sec)
machine2:~ root# zpool create testpool /var/root/afile
machine2:~ root# zfs create testpool/fs
machine2:~ root# zfs snapshot testpool/fs@asnap
machine2:~ root# zfs snapshot testpool/fs@bsnap
machine2:~ root# zfs send -i testpool/fs@asnap testpool/fs@bsnap > bsnap_incr
machine2:~ root# zfs destroy testpool/fs@bsnap
machine2:~ root# zfs receive -Fdv testpool < bsnap_incr
receiving incremental stream of testpool/fs@bsnap into testpool/fs@bsnap
cannot receive incremental stream: invalid backup stream
machine2:~ root#



[sper23]

I also have this error, do you have a solution?
Geometry Dash retro bowl college

Unfortunately, I do not have a solution to this problem. But I'm glad to hear that I'm not the only one. I'm just hoping that the next OpenZFS release fixes things.

I didn't specify this in my original post, but I'll now add that this problem affects receiving into pools housed on physical disks in addition to file-based pools.

[sper23]

I just tried this with OpenZFS On OSX 2.2.99 arm64/Ventura, same problem.

[sper23]

Great news— I think I found a workaround.

First, I confirmed that sending a snapshot directly through a pipe (e.g. zfs send | zfs receive) worked fine. Taking things a step further, I tried sending the snapshot file through a pipe using cat(1). Success!

So in summary, this fails:

Code: Select all

# zfs receive -F testpool2/fs2 < dsnap_file
cannot receive incremental stream: invalid backup stream

And this succeeds:

Code: Select all

# cat dsnap_file | zfs receive -F testpool2/fs2

So, although both methods are delivering the stream to STDIN, only using a pipe works. Whereas, delivering the stream by input direction with '<' used to work in OpenZFS-On-OSX prior to 2.1.6, but now does not. I think this might qualify as a bug?

@kalyl Try this workaround and please post back to this thread if it holds up.

[sper23]

The "checksum mismatch" error during the receiving of an incremental backup suggests a data integrity issue. Remember to back up any critical data before attempting any recovery operations to avoid furthergeometry dash data loss.

Thanks, that's good to keep in mind. The checksum mismatch error only occurs when directing input to the 'zfs receive' command using the '<' operator, and in that case the command fails and no data is actually received. The checksum is correct and the 'receive' command succeeds when piped in with the '|' operator, so I think the data received that way is fine. Both ways used to work equally, and I don't know if the bug is in the shell or in OpenZFS.

[lundman]

Yeah in the XNU kernel, they will let us to IO to files, but not pipes. So we do a bunch of code to work around that,
under the hood, I create a temporary named-pipe (pipe with a filename) in /tmp/.zfs.XXXXX, then open it has a file
(since pipes aren't allowed) to do IO on.

Jumping through lots of hoops just because Apple decided pipes are something we don't need in kernel development.

But sounds like something has gone wrong with that extra code, perhaps, the non-hacked code (since files fail, pipes work).