Performance tweaking for dedup?

All your general support questions for OpenZFS on OS X.

Performance tweaking for dedup?

Postby DanielSmedegaardBuus » Tue Apr 03, 2018 2:27 pm

Hey :)

I've created a zpool on an external drive with gzip-9 compression and dedup enabled because I'm trying to sort out about a decade worth of backups-with-backups-in-em-that-got-backups-in-em-and-there-was-something-i-added-but-i-cant-remember-what. So, basically multi-terabytes of data with lots of duplication, so I thought, "a 2TB ZFS with dedup oughta do!" And I'm pretty sure it will :)

Thing is, it's really really slow when writing. It's on a 3.5" USB3 external drive. Some Lacie thing with triple-digit-MB/s performance when just using something standard like HFS+. I first started copying a 250GB set of data to the empty pool using my MacBook Air 2015 with an i7 and 8GB of RAM. After a couple of hours, it estimated 9 more days to complete the copy, so I thought I might try my 2017 MBP with 16 GB of RAM instead. Here, it completed in less than two days. Still awfully slow, though :/

I'm wondering if there's something I need to tweak to get better write performance. 16 GB of RAM is *plenty* for a 2TB dedupped pool, and considering the performance increase from going from my Air to my MBP with twice the RAM, I'm thinking it might very well be memory related?

But then I'm looking at zpool get all, which says, "ashift 0", and I'm wondering if I either forgot to create it with ashift=12, or if "ashift 0" means "default", which is "12" nowadays, or what? Perhaps this is just me being retarded and now stuck with a pool that is suffering from me having forgotten all that old stuff about transitioning to modern drives some years back?

Any ideas greatly appreciated :)

Thanks!
DanielSmedegaardBuus
 
Posts: 11
Joined: Thu Aug 28, 2014 11:00 pm

Re: Performance tweaking for dedup?

Postby lundman » Tue Apr 03, 2018 8:32 pm

zdb $pool | grep ashift

should at least show the ashift it is really using. But that does sound slower than it should be
User avatar
lundman
 
Posts: 402
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: Performance tweaking for dedup?

Postby DanielSmedegaardBuus » Tue Apr 03, 2018 9:20 pm

Damn, that sucks :/

Code: Select all
$ zdb -U /etc/zfs/zpool.cache.tmp
lacie:
    version: 5000
    name: 'lacie'
    state: 0
    txg: 643869
    pool_guid: 2959251066757047319
    errata: 0
    hostid: 4259933762
    hostname: ''
    com.delphix:has_per_vdev_zaps
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 2959251066757047319
        children[0]:
            type: 'disk'
            id: 0
            guid: 12938823168595308534
            path: '/private/var/run/disk/by-id/media-3C78A53A-BD53-47FA-A379-5726225D4C44'
            whole_disk: 0
            metaslab_array: 34
            metaslab_shift: 32
            ashift: 9
            asize: 1990681165824
            is_log: 0
            DTL: 155
            create_txg: 4
            com.delphix:vdev_zap_leaf: 188
            com.delphix:vdev_zap_top: 189
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data


I could've sworn I read somewhere that ashift=12 was the default nowadays :/

Is there a quick way of transferring the fs to a new pool with ashift 12 without having to do dedup again?
DanielSmedegaardBuus
 
Posts: 11
Joined: Thu Aug 28, 2014 11:00 pm

Re: Performance tweaking for dedup?

Postby lundman » Wed Apr 04, 2018 4:12 pm

There was talk about changing default, not sure if anyone did. ZOL?

You'll have to create a new pool with ashift=12, but I think zfs send -D sends dedup information. I think it was -D.
User avatar
lundman
 
Posts: 402
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan


Return to General Help

Who is online

Users browsing this forum: No registered users and 1 guest