dataset or volume?
Posted:
Tue Aug 16, 2016 12:04 am
by mbunal
hi
which should i choose dataset or volume? why? i wa started to copy my archive to dataset. but i wonder what is dataset and volume?
Re: dataset or volume?
Posted:
Tue Aug 16, 2016 6:38 pm
by Sharko
Well, I can probably address half of your question, since I've become familiar with the limitations and benefits of ZFS volumes.
ZFS volumes, good points to not so good points:
+ offer absolute compatibility with the host operating system, since the OS X host sees a true HFS+ filesystem
+ all data is checksummed so you can be sure that the data has not been altered or corrupted
+ give you the option of building a self-healing system with redundancy through either mirroring (faster, but less efficient) or RAIDZ (slower, possibly more redundant)
+ allows you to utilize compression (recommended) or even deduplication (not recommended) to save storage space
+/- offer you the ability to snapshot the volume (good), but with the apparent limitation that the volume must be unmounted to snapshot it (not so good)
- requires you to choose a fixed size for the volume that cannot be changed later - you have to copy the data to a larger volume if you outgrow it (just like HFS+ requires)
- you can't go into snapshots and pull out files - all you can seemingly do with snapshots is rollback or destroy or clone to new volume
- possibly a little slower than native ZFS filesystem, since there are more layers in the software stack
- I believe you lose the benefits of copy-on-write, since the ZFS is just acting as block storage, and knows nothing about the atomicity of file system writes (since the writes are managed by HFS+)
ZFS native filesystems (datasets) have ALL the goodies: no fixed size, copy on write, checksums, redundancy, compression, deduplication, ability to snapshot on the fly, ability to access files in snapshots, ability to use scripts to rotate snapshots on live filesystems, etc.
Probably the only concern about native ZFS filesystems is compatibility with certain applications and/or portions of the operating system. I believe that ZFS emulates all the essential features of HFS (permissions, and I think even ACLs), but there may be some esoteric features of HFS+ that are not in ZFS. I have an open question regarding compatibility in this same section of the forum; no one has responded to it yet, which I suspect means that for most uses you won't run into a problem. I have seen some threads concerning Time Machine and Spotlight that make me think that these might not work out of the box on ZFS volume - or maybe they do in the latest version. Anyway, I've copied my user data to a test dataset, and I'm going to dive in and see what I encounter for compatibility first-hand.
Kurt
(post edited to reflect my better understanding of copy on write applicability to volumes)
Re: dataset or volume?
Posted:
Tue Aug 16, 2016 11:22 pm
by mbunal
thank you very muck Kurt.
Sharko wrote:I've copied my user data to a test dataset, and I'm going to dive in and see what I encounter for compatibility first-hand.
Kurt
me too ((-;<
Re: dataset or volume?
Posted:
Mon Sep 05, 2016 12:29 am
by mbunal
just one question too. it was very slowly copy my archive hdd's to new 8tb zfs dataset. there was lots of files less then 10kb. it took 4 days. anyway, i am good now.
if i use volume and hfs+ format, how was change the copy speed? (i just try myself but i am out for a while...)
Re: dataset or volume?
Posted:
Mon Sep 05, 2016 10:18 am
by haer22
mbunal wrote:just one question too. it was very slowly copy my archive hdd's to new 8tb zfs dataset. there was lots of files less then 10kb. it took 4 days. anyway, i am good now. if i use volume and hfs+ format, how was change the copy speed? (i just try myself but i am out for a while...)
Your pool, how is it set up? mirrors? raid? single? Do you have a cache device? Do you have a log device? How much memory do you have?
Small files are really painful speed-wise, especially if you do not have a spare log-device on an SSD.
Re: dataset or volume?
Posted:
Tue Sep 06, 2016 11:15 pm
by mbunal
haer22 wrote:Your pool, how is it set up? mirrors? raid? single? Do you have a cache device? Do you have a log device? How much memory do you have?
creation comand: sudo zpool create -f -o ashift=12 -O compression=lz4 -O casesensitivity=insensitive -O atime=off -O normalization=formD zfsArchive disk2
no mirror, no raid, only single 8 tb nas hdd. i dont have any cache device. i dont have any log device. i5 gen6 and 2x16 gb ddr4 ram (it is hackintosh).
haer22 wrote:Small files are really painful speed-wise, especially if you do not have a spare log-device on an SSD
should i use spare log-device? how much size?
Re: dataset or volume?
Posted:
Wed Sep 07, 2016 1:30 am
by Brendon
Perhaps you should do some basic benchmarking - copy a large video file onto that drive while running "zpool iostat 1" Post results.
Sounds like the drive is an internal SATA drive, I'd be amazed if you couldnt push 100-180MB/sec onto the drive.
- Brendon
Re: dataset or volume?
Posted:
Tue Sep 20, 2016 4:36 am
by mbunal
Brendon wrote:Perhaps you should do some basic benchmarking - copy a large video file onto that drive while running "zpool iostat 1" Post results.
Sounds like the drive is an internal SATA drive, I'd be amazed if you couldnt push 100-180MB/sec onto the drive.
- Brendon
1.05 gb mp4 copied from zfsroot/videos to zfsroot:
- Code: Select all
~$ zpool iostat 1
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
zfs8tb 3,14T 4,11T 0 0 10,1K 43,9K
zfs8tb 3,14T 4,11T 960 63 119M 8,00M
zfs8tb 3,14T 4,11T 1,02K 69 129M 8,75M
zfs8tb 3,14T 4,11T 600 39 74,5M 4,98M
zfs8tb 3,14T 4,11T 407 26 50,5M 3,37M
zfs8tb 3,14T 4,11T 45 3 5,75M 512K
zfs8tb 3,14T 4,11T 43 1 5,48M 255K
zfs8tb 3,14T 4,11T 165 11 20,4M 1,49M
zfs8tb 3,14T 4,11T 140 8 17,5M 1,12M
zfs8tb 3,14T 4,11T 293 19 36,5M 2,50M
zfs8tb 3,14T 4,11T 74 4 9,21M 637K
zfs8tb 3,14T 4,11T 400 25 49,7M 3,25M
zfs8tb 3,14T 4,11T 16 0 2,12M 128K
zfs8tb 3,14T 4,11T 219 79 27,3M 10,0M
zfs8tb 3,14T 4,11T 23 100 2,99M 11,5M
zfs8tb 3,14T 4,11T 25 64 3,24M 7,55M
zfs8tb 3,14T 4,11T 38 77 4,74M 8,94M
zfs8tb 3,14T 4,11T 34 86 4,36M 10,1M
zfs8tb 3,14T 4,11T 16 30 2,12M 3,75M
zfs8tb 3,14T 4,11T 16 20 2,12M 2,61M
zfs8tb 3,14T 4,11T 27 48 3,37M 6,11M
zfs8tb 3,14T 4,11T 25 39 3,24M 4,98M
zfs8tb 3,14T 4,11T 14 29 1,87M 3,73M
zfs8tb 3,14T 4,11T 44 79 5,62M 9,99M
zfs8tb 3,14T 4,11T 11 9 1,49M 1,24M
zfs8tb 3,14T 4,11T 28 49 3,62M 6,24M
zfs8tb 3,14T 4,11T 36 60 4,49M 7,59M
zfs8tb 3,14T 4,11T 12 28 1,62M 3,61M
zfs8tb 3,14T 4,11T 28 49 3,62M 6,23M
zfs8tb 3,14T 4,11T 23 29 2,99M 3,73M
zfs8tb 3,14T 4,11T 27 49 3,37M 6,23M
zfs8tb 3,14T 4,11T 42 69 5,35M 8,72M
zfs8tb 3,14T 4,11T 12 19 1,62M 2,49M
zfs8tb 3,14T 4,11T 47 79 5,99M 9,98M
zfs8tb 3,14T 4,11T 17 29 2,24M 3,74M
zfs8tb 3,14T 4,11T 12 29 1,50M 3,73M
zfs8tb 3,14T 4,11T 46 70 5,86M 8,85M
zfs8tb 3,14T 4,11T 17 38 2,24M 4,86M
zfs8tb 3,14T 4,11T 23 39 2,99M 4,98M
zfs8tb 3,14T 4,11T 36 59 4,49M 7,47M
zfs8tb 3,14T 4,11T 11 19 1,50M 2,50M
zfs8tb 3,14T 4,11T 46 69 5,85M 8,71M
zfs8tb 3,14T 4,11T 29 49 3,74M 6,24M
zfs8tb 3,14T 4,11T 11 29 1,50M 3,74M
zfs8tb 3,14T 4,11T 35 59 4,38M 7,49M
zfs8tb 3,14T 4,11T 35 59 4,49M 7,48M
zfs8tb 3,14T 4,11T 35 50 4,48M 6,35M
zfs8tb 3,14T 4,11T 62 108 7,72M 13,6M
zfs8tb 3,14T 4,11T 26 39 3,37M 5,00M
zfs8tb 3,14T 4,11T 5 9 764K 1,24M
zfs8tb 3,14T 4,11T 47 80 5,99M 10,1M
zfs8tb 3,14T 4,11T 29 49 3,73M 6,22M
zfs8tb 3,14T 4,11T 64 108 7,98M 13,6M
zfs8tb 3,14T 4,11T 31 59 3,99M 7,48M
zfs8tb 3,14T 4,11T 46 79 5,73M 9,95M
zfs8tb 3,14T 4,11T 54 89 6,86M 11,2M
zfs8tb 3,14T 4,11T 40 59 5,10M 7,46M
zfs8tb 3,14T 4,11T 29 60 3,63M 7,61M
zfs8tb 3,14T 4,11T 34 59 4,37M 7,49M
zfs8tb 3,14T 4,11T 47 69 5,97M 8,71M
zfs8tb 3,14T 4,11T 78 139 9,72M 17,4M
zfs8tb 3,14T 4,11T 59 99 7,46M 12,4M
zfs8tb 3,14T 4,11T 52 89 6,48M 11,2M
zfs8tb 3,14T 4,11T 95 152 11,9M 19,0M
zfs8tb 3,14T 4,11T 26 46 3,24M 5,85M
zfs8tb 3,14T 4,11T 86 143 10,8M 17,9M
zfs8tb 3,14T 4,11T 18 35 2,37M 4,49M
zfs8tb 3,14T 4,11T 86 139 10,8M 17,4M
zfs8tb 3,14T 4,11T 73 129 9,22M 16,2M
zfs8tb 3,14T 4,11T 59 89 7,47M 11,2M
zfs8tb 3,14T 4,11T 44 79 5,62M 10,0M
zfs8tb 3,14T 4,11T 62 107 7,86M 13,5M
zfs8tb 3,14T 4,11T 48 81 6,11M 10,2M
zfs8tb 3,14T 4,11T 55 89 6,98M 11,2M
zfs8tb 3,14T 4,11T 33 59 4,17M 7,47M
zfs8tb 3,14T 4,11T 23 39 2,99M 4,99M
zfs8tb 3,14T 4,11T 3 1,03K 511K 132M
zfs8tb 3,14T 4,11T 0 1,24K 0 159M
zfs8tb 3,14T 4,11T 0 1,04K 0 120M
zfs8tb 3,14T 4,11T 0 1,20K 0 151M
zfs8tb 3,14T 4,11T 0 760 0 93,8M
zfs8tb 3,14T 4,11T 0 312 0 35,3M
zfs8tb 3,14T 4,11T 0 1,00K 0 129M
zfs8tb 3,14T 4,11T 0 918 0 114M
zfs8tb 3,14T 4,11T 0 30 0 699K
zfs8tb 3,14T 4,11T 0 0 0 0