Reboots when using encryption and zfs send/recv

All your general support questions for OpenZFS on OS X.

Reboots when using encryption and zfs send/recv

Postby mtx » Sun Nov 05, 2017 1:58 pm

Thank you for making openzfs available on macOS!

on macOS 10.13/openzfs 1.7.0

zfs send/recv using the old method of Core Storage encryption and passing the logical volume to zpool create works as expected.

However when a volume is created using at rest encryption, random spontaneous reboots occur.
zpool create is used including the following options:
Code: Select all
-o ashift=12 -O compression=lz4 -O casesensitivity=insensitive -O atime=off -O normalization=formD -O encryption=on -O keyformat=passphrase -O keylocation=prompt


On using the --raw commands, random spontaneous reboots occur.
Notably during:
Code: Select all
zfs send -Rcv --raw pool1@backup | zfs recv -F pool2/pool1

or in the event that the raw stream is successfully sent, reboots may occur with
Code: Select all
zpool import -l pool2
which contains the raw stream. This occurs instantly after passwords are entered for all datasets
mtx
 
Posts: 3
Joined: Sun Nov 05, 2017 1:45 pm

Re: Reboots when using encryption and zfs send/recv

Postby lundman » Sun Nov 05, 2017 4:26 pm

Should it even let you do what you are doing?

Code: Select all
bash-3.2# ./cmd.sh zpool create -f  -o ashift=12 -O compression=lz4 -O casesensitivity=insensitive -O atime=off -O normalization=formD -O encryption=on -O keyformat=passphrase -O keylocation=prompt BOOM disk2
Enter passphrase:
Re-enter passphrase:

bash-3.2# ./cmd.sh zpool create -f tank disk3

bash-3.2# rsync -ar . /Volumes/tank/
bash-3.2# ./cmd.sh zfs snapshot tank@send

bash-3.2# ./cmd.sh zfs send -Rcv --raw tank@send | ./cmd.sh zfs recv -F BOOM/tank
full send of tank@send estimated size is 1.80G
total estimated size is 1.80G
TIME        SENT   SNAPSHOT
cannot receive new filesystem stream: parent 'BOOM' must not be encrypted to receive unenecrypted property
warning: cannot send 'tank@send': signal received



However, if I remove the raw, and let it receive normally, so it gets encrypted, it dies:

Code: Select all
zio_crypt.c zio_do_crypt_data 1798 : ASSERTION(puio != NULL) failed
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Reboots when using encryption and zfs send/recv

Postby lundman » Sun Nov 05, 2017 4:49 pm

User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Reboots when using encryption and zfs send/recv

Postby mtx » Sun Nov 05, 2017 5:24 pm

lundman wrote:Should it even let you do what you are doing?

Code: Select all
bash-3.2# ./cmd.sh zfs send -Rcv --raw tank@send | ./cmd.sh zfs recv -F BOOM/tank


That scenario appears to be sending an unencrypted dataset to an encrypted dataset.

My scenario is backup of an encrypted pool1to another encrypted pool2, hence the need to enter passphrases with zpool import -l pool2.

Should I not be sending encrypted raws this way? > I will check with zfsonlinux
I.e. Encrypted pool1 > send --raw > recv > Encrypted pool2
mtx
 
Posts: 3
Joined: Sun Nov 05, 2017 1:45 pm

Re: Reboots when using encryption and zfs send/recv

Postby lundman » Sun Nov 05, 2017 5:55 pm

Oh heh, I may have reversed the pools in my example, and assumed you were sending a plain into an encrypted. One second.

Code: Select all
# ./cmd.sh zfs send -Rcv --raw pool1@send | ./cmd.sh zfs recv -F pool2/pool1
full send of pool1@send estimated size is 2.61M
total estimated size is 2.61M
TIME        SENT   SNAPSHOT
libzfs_sendrecv.c:2723: recv_fix_encryption_heirarchy: Assertion `0 == nvlist_lookup_string(props, zfs_prop_to_name(ZFS_PROP_KEYLOCATION), &stream_keylocation)` failed.
Abort trap: 6


Let me dig deeper
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Reboots when using encryption and zfs send/recv

Postby lundman » Sun Nov 05, 2017 6:37 pm

Dropping the -R in zfs send makes it work here (with the above fix). You'd have to send the lower filesystems by hand of course. Can you confirm the fix today stops it from hanging at least? (If you can't compile, I can produce a kext for you).

Meanwhile, I am discussing the -R assert problem with Caputi.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Reboots when using encryption and zfs send/recv

Postby lundman » Sun Nov 05, 2017 7:36 pm

Ok, the property problem is resolved in:

https://github.com/openzfsonosx/zfs/com ... 594b3b02f0

So in theory, your problem should be handled now :)
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Reboots when using encryption and zfs send/recv

Postby mtx » Mon Nov 06, 2017 3:37 am

Many thanks.

Using the latest master, I can confirm that things are much improved.

It appears the problem stems from sending a raw encrypted dataset to an online, decrypted zpool (which may be unintended usage and may be a user error on my part).
The reboots do not occur if the target zpool is not decrypted.
The reboots also do not occur if the target zpool is completely unencrypted.


SCENARIO 1: decrypted pool1 to decrypted pool2 - random reboots or Malformed Mach-o file
Backup encrypted pool1 to decrypted pool2 using --raw initially works:
Code: Select all
zfs send -Rcv --raw

However when exporting and importing pool2 containing a raw encrypted dataset, random reboots occur during the process of exporting and importing. These random reboots are much reduced since updating to latest master.
If reboot does not occur, it reports: cannot mount 'pool2/pool1-backup': Malformed Mach-o file.
At this point I am unable to export pool2: 'pool is busy'.


SCENARIO 2: decrypted pool1 to encrypted pool2 - ok
Backup decrypted pool1 to encrypted pool2 using --raw works:
Code: Select all
zfs send -Rcv --raw

Restore decrypted pool2 to decrypted pool1 works:
Code: Select all
zfs load-key -r pool2
zfs send -cv pool2/pool1-backup@backup | zfs recv pool1/backup

Note: do not use -R or -p as we do not want to send properties; an encrypted dataset may not be sent with properties without the raw flag; also I want the dataset to inherit properties from pool1 on restore.
The datasets are immediately mounted and available.


SCENARIO 3: decrypted pool1 to unencrypted pool2 - ok
Backup decrypted pool1 to unencrypted pool2 works:
Code: Select all
zfs send -Rcv --raw

The child datasets on pool2 are encrypted; checked via:
Code: Select all
zfs get -r encryptionroot

Restore unencrypted pool2 to decrypted pool1 with --raw partially works; but I do not recommend it:
The datasets are not automatically mounted on completion.
Whilst no decryption key for pool2 is required to do the restore, passphrase for each child dataset will be required at each mount.
Code: Select all
zfs send -cv --raw

Restore unencrypted pool2 to decrypted pool1 without --raw works:
N.B. If original backup was using -Rcv, only 1x passphrase required; otherwise I suspect each child dataset would require its own passphrase upon mount.
Code: Select all
zfs send -cv
warning: cannot send: source key must be loaded
zfs load-key -r pool2
zfs send -cv



SUMMARY
zfs send -Rcv --raw can backup to either encrypted (but not decrypted) targets or unencrypted targets (where it will make encrypted child datasets).
zfs send -Rcv --raw to decrypted targets results in reboots or Malformed Mach-o file
There is no point to zfs send --raw on restores.

Thank you once again for your great work :D
mtx
 
Posts: 3
Joined: Sun Nov 05, 2017 1:45 pm

Re: Reboots when using encryption and zfs send/recv

Postby lundman » Mon Nov 06, 2017 3:52 pm

zfs send -Rcv --raw to decrypted targets results in reboots or Malformed Mach-o file


Can you give more details on how to create a test for this? As in, the actual commands to create pools.
(Malformed Mach-o is errno 88, which we use for ECKSUM - ie bad authenticated checksum)
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Reboots when using encryption and zfs send/recv

Postby lundman » Mon Nov 06, 2017 4:57 pm

Ok, I can trigger this amusing situation:

Code: Select all
bash-3.2# ./cmd.sh zfs send -Rcv --raw pool1@send | ./cmd.sh zfs recv pool2/r1
bash-3.2# ./cmd.sh zfs send -Rcv --raw pool1@send | ./cmd.sh zfs recv pool2/r2
bash-3.2# ./cmd.sh zfs send -Rcv --raw pool1@send | ./cmd.sh zfs recv pool2/r3
bash-3.2# ./cmd.sh zfs send -Rcv --raw pool1@send | ./cmd.sh zfs recv pool2/r4
bash-3.2# ./cmd.sh zfs send -Rcv --raw pool1@send | ./cmd.sh zfs recv pool2/r5
bash-3.2# ./cmd.sh zfs send -Rcv --raw pool1@send | ./cmd.sh zfs recv pool2/r6
bash-3.2# ./cmd.sh zfs send -Rcv --raw pool1@send | ./cmd.sh zfs recv pool2/r7
bash-3.2# ./cmd.sh zfs send -Rcv --raw pool1@send | ./cmd.sh zfs recv pool2/r8
bash-3.2# ./cmd.sh zfs send -Rcv --raw pool1@send | ./cmd.sh zfs recv pool2/r9
bash-3.2# yes "testtest" | ./cmd.sh zfs mount -al
cannot mount 'pool2/r1': Input/output error
cannot mount 'pool2/r9': Input/output error


Amusingly, it can be avoided by using "zfs recv $dataset ; zfs load-key $dataset ; zfs mount $dataset"

Which when dtracing, turns out to be dmu_objset_own() returning EBUSY, ie

Code: Select all
  1  66338            dmu_objset_own:return 16

              zfs`dmu_objset_own+0xb2
              zfs`zvol_create_minor_impl+0x103
              zfs`zvol_create_minors_cb+0x66
              zfs`dmu_objset_find_impl+0x4fb
              zfs`dmu_objset_find+0x45
              zfs`zvol_task_cb+0x265
              spl`taskq_thread+0x1f6
              kernel.development`call_continuation+0x17


Which turns out to be mount racing with zvol_create_minors to be first to dmu_objset_own. If zvol_create_minor_impl is stubbed out, it never fails to mount. A good question here is why would zvol_create_minor_impl even wake up for a non-volume creation. I will look at this.

Caputi mentioned there is a panic-fix in the works, hoping to have it out this week.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Next

Return to General Help

Who is online

Users browsing this forum: No registered users and 25 guests