Utility bash script- create encrypted datasets(all ciphers)

Here you can discuss every aspect of OpenZFS on OS X. Note: not for support requests!

Utility bash script- create encrypted datasets(all ciphers)

Postby kingneutron » Wed May 27, 2020 9:15 am

--Available for testing: OSX bash script to create separate encrypted datasets for all currently-known / supported ZFS encryption formats.
Useful for testing I/O speed for each encryption cipher.

https://pastebin.com/GHAXLJW6

--OSX version 2 of the script: Tested on OSX 10.13 High Sierra and relies on (macports or brew) gnu "coreutils" and bash v5
-- Needs testing on Mojave / Catalina

--It is HIGHLY recommended to review/edit the script before running it, as there are a few EDITME's scattered in it that make some assumptions. (And don't forget to chmod +x it.)


--Features:

o Creates RAMdisk if you have at least 6GB RAM, copies ISO file to ramdisk

o Runs openssl benchmark tests and checks if VeraCrypt is installed

o Reads the man page for zfs and creates separate encrypted datasets for each known cipher (aes-128-ccm / gcm, etc)
( NOTE - this relies on a up-to-date man page )

o Generates a helpful log file

o Also cleans up after itself (needs to be called explicitly) by deleting the test datasets and ejecting the ramdisk.

--Known bug: logfile does not capture initial openssl tests:

#Doing aes-128 cbc for 3s on 16 size blocks: 14957278 aes-128 cbc's in 2.87s

--OSX version was adapted from the initial Linux version that I wrote. Once the encrypted datasets are created, you can e.g.
' time cp -v $isofile /Volumes/zpoolname/Test-aes-128-ccm ' and run fio and whatever other tests you like on each dataset.

--The script was a lot of fun to adapt for OSX after completing the Linux version. It's pretty heavily commented and contains a lot of tips for getting OSX information about the running environment that is normally provided by /proc in Linux. :D

--Log excerpt:

Code: Select all
# bash ./zfs-test-encryption-speeds--osx.sh
/var/root/zfs-test-encryption-speeds.log -> /var/root/zfs-test-encryption-speeds.log-old
Wed May 27 11:45:24 CDT 2020 - imac513.local - BEGIN encryption speed tests
o Kernel: 17.7.0
o Zpool version: zfs-1.9.4-0
zfs-kmod-1.9.4-0
ZFS kext module version: net.lundman.zfs(1.9.4)
o CPU detected:       Processor Name: Intel Core i5
      Processor Speed: 2.7 GHz
      Number of Processors: 1
      Total Number of Cores: 4
o CPU supports AES acceleration (blank=NO): hw.optional.aes: 1
machdep.cpu.features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE SSE3 PCLMULQDQ DTES64 MON DSCPL VMX SMX EST TM2 SSSE3 CX16 TPR PDCM SSE4.1 SSE4.2 x2APIC POPCNT AES PCID XSAVE OSXSAVE TSCTMR AVX1.0
kstat.zfs.darwin.tunable.icp_aes_impl: cycle [fastest] generic x86_64 aesni
o Check for openssl / run benchmarks
/usr/bin/openssl
Doing aes-128 cbc for 3s on 16 size blocks: 15026856 aes-128 cbc's in 2.88s
Doing aes-128 cbc for 3s on 64 size blocks: 3939291 aes-128 cbc's in 2.84s
.
. ( Fast forward, fast forward )
.
LibreSSL 2.2.7
built on: date not available
options:bn(64,64) rc4(ptr,int) des(idx,cisc,16,int) aes(partial) blowfish(idx)
compiler: information not available
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128 cbc      83482.53k    88772.75k   101248.00k    91318.16k    90517.10k
aes-192 cbc      80441.12k    77077.44k    78188.11k    84693.78k    79960.18k
.
. ( FF )
.
o PREP - Copy ~1GB iso file to ramdisk if not already there
/Volumes/ramdisk/bionic-desktop-amd64.iso not overwritten
real    0m0.005s

o Supported ZFS encryption types (per ' man zfs ')
aes-128-ccm,aes-192-ccm,aes-256-ccm,aes-128-gcm,aes-192-gcm,aes-256-gcm

o Creating zfilepool dataset for aes-128-ccm if it doesnt already exist
NAME                        PROPERTY     VALUE                                  SOURCE
zfilepool/Test-aes-128-ccm  encryption   aes-128-ccm                            -
zfilepool/Test-aes-128-ccm  keylocation  file:///var/root/zek-testencr-zfs.key  local
o Creating zfilepool dataset for aes-192-ccm if it doesnt already exist
NAME                        PROPERTY     VALUE                                  SOURCE
zfilepool/Test-aes-192-ccm  encryption   aes-192-ccm                            -
zfilepool/Test-aes-192-ccm  keylocation  file:///var/root/zek-testencr-zfs.key  local
o Creating zfilepool dataset for aes-256-ccm if it doesnt already exist
NAME                        PROPERTY     VALUE                                  SOURCE
zfilepool/Test-aes-256-ccm  encryption   aes-256-ccm                            -
zfilepool/Test-aes-256-ccm  keylocation  file:///var/root/zek-testencr-zfs.key  local
o Creating zfilepool dataset for aes-128-gcm if it doesnt already exist
NAME                        PROPERTY     VALUE                                  SOURCE
zfilepool/Test-aes-128-gcm  encryption   aes-128-gcm                            -
zfilepool/Test-aes-128-gcm  keylocation  file:///var/root/zek-testencr-zfs.key  local
o Creating zfilepool dataset for aes-192-gcm if it doesnt already exist
NAME                        PROPERTY     VALUE                                  SOURCE
zfilepool/Test-aes-192-gcm  encryption   aes-192-gcm                            -
zfilepool/Test-aes-192-gcm  keylocation  file:///var/root/zek-testencr-zfs.key  local
o Creating zfilepool dataset for aes-256-gcm if it doesnt already exist
NAME                        PROPERTY     VALUE                                  SOURCE
zfilepool/Test-aes-256-gcm  encryption   aes-256-gcm                            -
zfilepool/Test-aes-256-gcm  keylocation  file:///var/root/zek-testencr-zfs.key  local
Filesystem                                          Type  Size  Used Avail Use% Mounted on
/dev/disk8s1                                        zfs   7.3G  460K  7.3G   1% /Volumes/zfilepool
/dev/disk9s1                                        hfs   1.1G  1.1G   30M  98% /Volumes/ramdisk
zfilepool/Test-aes-128-ccm                          zfs   7.3G  868K  7.3G   1% /Volumes/zfilepool/Test-aes-128-ccm
zfilepool/Test-aes-192-ccm                          zfs   7.3G  876K  7.3G   1% /Volumes/zfilepool/Test-aes-192-ccm
zfilepool/Test-aes-256-ccm                          zfs   7.3G  844K  7.3G   1% /Volumes/zfilepool/Test-aes-256-ccm
zfilepool/Test-aes-128-gcm                          zfs   7.3G  844K  7.3G   1% /Volumes/zfilepool/Test-aes-128-gcm
zfilepool/Test-aes-192-gcm                          zfs   7.3G  812K  7.3G   1% /Volumes/zfilepool/Test-aes-192-gcm
zfilepool/Test-aes-256-gcm                          zfs   7.3G  876K  7.3G   1% /Volumes/zfilepool/Test-aes-256-gcm
-rw-r-xr--  1 _unknown  _unknown   1.0G May 27 11:14 /Volumes/ramdisk/bionic-desktop-amd64.iso
Wed May 27 11:46:11 CDT 2020 ./zfs-test-encryption-speeds--osx.sh - imac513.local - Ready for testing
o ./zfs-test-encryption-speeds--osx.sh Logfile is: /var/root/zfs-test-encryption-speeds.log
kingneutron
 
Posts: 13
Joined: Sat Mar 16, 2019 4:37 pm

Re: Utility bash script- create encrypted datasets(all ciphe

Postby lundman » Wed Jun 03, 2020 9:51 pm

very interesting, thanks for doing the extra work for osx.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Utility bash script- create encrypted datasets(all ciphe

Postby JasonBelec » Thu Jun 04, 2020 7:58 am

Actually kinda cool.
JasonBelec
 
Posts: 32
Joined: Mon Oct 26, 2015 1:07 pm


Return to General Discussions

Who is online

Users browsing this forum: No registered users and 13 guests

cron