Page 1 of 1

Performance tweaking for dedup?

PostPosted: Tue Apr 03, 2018 2:27 pm
by DanielSmedegaardBuus
Hey :)

I've created a zpool on an external drive with gzip-9 compression and dedup enabled because I'm trying to sort out about a decade worth of backups-with-backups-in-em-that-got-backups-in-em-and-there-was-something-i-added-but-i-cant-remember-what. So, basically multi-terabytes of data with lots of duplication, so I thought, "a 2TB ZFS with dedup oughta do!" And I'm pretty sure it will :)

Thing is, it's really really slow when writing. It's on a 3.5" USB3 external drive. Some Lacie thing with triple-digit-MB/s performance when just using something standard like HFS+. I first started copying a 250GB set of data to the empty pool using my MacBook Air 2015 with an i7 and 8GB of RAM. After a couple of hours, it estimated 9 more days to complete the copy, so I thought I might try my 2017 MBP with 16 GB of RAM instead. Here, it completed in less than two days. Still awfully slow, though :/

I'm wondering if there's something I need to tweak to get better write performance. 16 GB of RAM is *plenty* for a 2TB dedupped pool, and considering the performance increase from going from my Air to my MBP with twice the RAM, I'm thinking it might very well be memory related?

But then I'm looking at zpool get all, which says, "ashift 0", and I'm wondering if I either forgot to create it with ashift=12, or if "ashift 0" means "default", which is "12" nowadays, or what? Perhaps this is just me being retarded and now stuck with a pool that is suffering from me having forgotten all that old stuff about transitioning to modern drives some years back?

Any ideas greatly appreciated :)

Thanks!

Re: Performance tweaking for dedup?

PostPosted: Tue Apr 03, 2018 8:32 pm
by lundman
zdb $pool | grep ashift

should at least show the ashift it is really using. But that does sound slower than it should be

Re: Performance tweaking for dedup?

PostPosted: Tue Apr 03, 2018 9:20 pm
by DanielSmedegaardBuus
Damn, that sucks :/

Code: Select all
$ zdb -U /etc/zfs/zpool.cache.tmp
lacie:
    version: 5000
    name: 'lacie'
    state: 0
    txg: 643869
    pool_guid: 2959251066757047319
    errata: 0
    hostid: 4259933762
    hostname: ''
    com.delphix:has_per_vdev_zaps
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 2959251066757047319
        children[0]:
            type: 'disk'
            id: 0
            guid: 12938823168595308534
            path: '/private/var/run/disk/by-id/media-3C78A53A-BD53-47FA-A379-5726225D4C44'
            whole_disk: 0
            metaslab_array: 34
            metaslab_shift: 32
            ashift: 9
            asize: 1990681165824
            is_log: 0
            DTL: 155
            create_txg: 4
            com.delphix:vdev_zap_leaf: 188
            com.delphix:vdev_zap_top: 189
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data


I could've sworn I read somewhere that ashift=12 was the default nowadays :/

Is there a quick way of transferring the fs to a new pool with ashift 12 without having to do dedup again?

Re: Performance tweaking for dedup?

PostPosted: Wed Apr 04, 2018 4:12 pm
by lundman
There was talk about changing default, not sure if anyone did. ZOL?

You'll have to create a new pool with ashift=12, but I think zfs send -D sends dedup information. I think it was -D.