Remove file marked as permanent error

Here you can discuss every aspect of OpenZFS on OS X. Note: not for support requests!

Remove file marked as permanent error

Postby AlexWasserman » Sat Oct 25, 2014 8:37 pm

Hey guys,

One of my files on a pool has been marked as having a permanent error. I deleted it. However, I can't get ZFS to forget about it. I've rebooted, rescrubbed, etc.
zpool clear will clear the cksum numbers, until a scrub finds the error again.

Originally the file was a video file: Shaun.The.Sheep.S02E02_Hiccups.avi

I don't much care about the kids cartoons, I have the DVD anyway.

So, what have I done?

Normal status check to find the file:

Code: Select all
alex@smiley:~|⇒  zpool status -v
  pool: ZFS
 state: ONLINE
status: One or more devices has experienced an error resulting in data
   corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
   entire pool from backup.
   see: http://zfsonlinux.org/msg/ZFS-8000-8A
  scan: scrub repaired 0 in 10h4m with 1 errors on Tue Oct 21 10:36:17 2014
config:

   NAME        STATE     READ WRITE CKSUM
   ZFS         ONLINE       0     0     1
     mirror-0  ONLINE       0     0     0
       disk6   ONLINE       0     0     0
       disk5   ONLINE       0     0     0
     mirror-1  ONLINE       0     0     2
       disk7   ONLINE       0     0     2
       disk4   ONLINE       0     0     2
     mirror-2  ONLINE       0     0     2
       disk2   ONLINE       0     0     2
       disk3   ONLINE       0     0     2

errors: Permanent errors have been detected in the following files:

        ZFS/Media:<0x113bf3>


ok, that's normal for a deleted file.

Note, this is freshly scrubbed. This error happened months ago, now it's bugging me so I'm trying to fix it. I scrub weekly.

We can look up the inode information with zdb:

Code: Select all
alex@smiley:~|⇒  zdb -ddddd ZFS/Media 0x113bf3
Dataset ZFS/Media [ZPL], ID 62, cr_txg 83, 2.28T, 979515 objects, rootbp DVA[0]=<1:e830d20000:1000> DVA[1]=<2:2543466f000:1000> [L0 DMU objset] fletcher4 lzjb LE contiguous unique double size=800L/200P birth=4102431L/4102431P fill=979515 cksum=19bc47bf6b:7cca3c9411e:14b5b20d91238:278b4b4674310c

    Object  lvl   iblk   dblk  dsize  lsize   %full  type
   1129459    1    16K    512     8K    512  100.00  ZFS directory
                                        168   bonus  System attributes
   dnode flags: USED_BYTES USERUSED_ACCOUNTED
   dnode maxblkid: 0
   path   ???<object#1129459>
   uid     502
   gid     20
   atime   Mon Mar 10 12:13:52 2014
   mtime   Mon Mar 10 12:13:52 2014
   ctime   Sat Mar 15 17:10:29 2014
   crtime   Mon Mar 10 12:13:52 2014
   gen   51827
   mode   41777
   size   3
   parent   1129458
   links   0
   pflags   40800000145

Indirect blocks:
               0 L0 1:ec7c409000:1000 200L/200P F=1 B=51827/51827

      segment [0000000000000000, 0000000000000200) size   512


ok, so, this is a ZFS directory, that might come in handy. We also know it's location on disk. We can pull the actual file information with ZDB:

Code: Select all
alex@smiley:~|⇒  sudo zdb -R ZFS/Media 1:ec7c409000:1000:r
Found vdev type: mirror
���ݰ�;�com.apple.quarantine%


So, this is from a mirror, yes, it is, all three vdevs are mirrors.

There are no snapshots on this filesystem.

The file is blank expect for this com.apple.quarantine, which is a special flag (extended attribute) on the file apple sets when a file is downloaded. It gives that annoying "This file was downloaded, are you sure you want to open it" prompt.

This explains why it's a ZFS directory file. I believe that extended attributes were originally implemented in ZFS using ZFS directories to represent the file containing the file and the attributes. Could that be the case here?

Could it be that when the file was corrupted that the file was deleted by me, somehow leaving the corrected attribute behind?

I've tried to recreate the file and delete it:

sudo zdb -R ZFS/Media 1:ec7c409000:1000:r > Shaun.The.Sheep.S02E02_Hiccups.avi


But, that doesn't do anything. I've had the recreated file hanging around for several scrubs. Possibly because this isn't a regular file.

Anyway, I don't care about the file, I just want to remove all record of it from ZFS so it'll be happy again.
AlexWasserman
 
Posts: 11
Joined: Sat Mar 29, 2014 3:40 pm

Re: Remove file marked as permanent error

Postby ilovezfs » Sat Oct 25, 2014 10:19 pm

Did you run zpool clear?
ilovezfs
 
Posts: 232
Joined: Thu Mar 06, 2014 7:58 am

Re: Remove file marked as permanent error

Postby AlexWasserman » Sun Oct 26, 2014 5:08 am

Yes, it clears the CHKSUM errors, not the file error.

Then, on the next scrub it finds the error again and the CHKSUM numbers come back.
AlexWasserman
 
Posts: 11
Joined: Sat Mar 29, 2014 3:40 pm

Re: Remove file marked as permanent error

Postby ilovezfs » Sun Oct 26, 2014 9:34 pm

zfs list -t all
ilovezfs
 
Posts: 232
Joined: Thu Mar 06, 2014 7:58 am

Re: Remove file marked as permanent error

Postby AlexWasserman » Mon Oct 27, 2014 5:31 pm

Code: Select all
⇒  zfs list -t all
NAME                         USED  AVAIL  REFER  MOUNTPOINT
ZFS                         5.01T  2.12T   676K  none
ZFS/Backups                 1.23T  2.12T  1.23T  /Backups
ZFS/Documents                411G  2.12T   276G  none
ZFS/Documents@UsersSnap_31   576K      -   407G  none
ZFS/Documents@UsersSnap_30   400K      -   407G  none
ZFS/Documents@UsersSnap_29   512K      -   407G  none
ZFS/Documents@UsersSnap_28   380K      -   407G  none
ZFS/Documents@UsersSnap_27   544K      -   407G  none
ZFS/Documents@UsersSnap_26   404K      -   407G  none
ZFS/Documents@UsersSnap_25   524K      -   407G  none
ZFS/Documents@UsersSnap_24  1.36M      -   407G  none
ZFS/Documents@UsersSnap_23  1.71M      -   407G  none
ZFS/Documents@UsersSnap_22  1.36M      -   408G  none
ZFS/Documents@UsersSnap_21   119M      -   409G  none
ZFS/Documents@UsersSnap_20   508K      -   408G  none
ZFS/Documents@UsersSnap_19   524K      -   408G  none
ZFS/Documents@UsersSnap_18   464K      -   408G  none
ZFS/Documents@UsersSnap_17   688K      -   408G  none
ZFS/Documents@UsersSnap_16  1.92M      -   409G  none
ZFS/Documents@UsersSnap_15   302M      -   410G  none
ZFS/Documents@UsersSnap_14   476K      -   410G  none
ZFS/Documents@UsersSnap_13   536K      -   410G  none
ZFS/Documents@UsersSnap_12   460K      -   410G  none
ZFS/Documents@UsersSnap_11   556K      -   276G  none
ZFS/Documents@UsersSnap_10   380K      -   276G  none
ZFS/Documents@UsersSnap_9    500K      -   276G  none
ZFS/Documents@UsersSnap_8    356K      -   276G  none
ZFS/Documents@UsersSnap_7    576K      -   276G  none
ZFS/Documents@UsersSnap_6    416K      -   276G  none
ZFS/Documents@UsersSnap_5    536K      -   276G  none
ZFS/Documents@UsersSnap_4   1.09M      -   276G  none
ZFS/Documents@UsersSnap_3    508K      -   276G  none
ZFS/Documents@UsersSnap_2    372K      -   276G  none
ZFS/Documents@UsersSnap_1    668K      -   276G  none
ZFS/Documents@UsersSnap_0   4.57M      -   277G  none
ZFS/Games                    159G  2.12T   159G  /Applications/Games
ZFS/Media                   2.28T  2.12T  2.28T  /Media
ZFS/TimeMachine              106G  2.13T   101G  -
ZFS/Users                    516G  2.44T   191G  -
ZFS/iLife                    340G  2.12T   340G  -


TimeMachine, Users, and iLife are all volumes, formatted as HFS+ to work around issues with OS X apps working better - iPhoto, iTunes, etc all seem to work better on HFS+, so I've migrated away from pure ZFS to HFS on ZFS (on OS X :) )

Games, Media, Documents, and Backups are all regular filesystems.

I have a script which creates nightly snaps of Documents, incrementing the number each night, and taking in a custom variable for how many to keep, so _0 is always last night, _1 is day before, etc. Documents is unmounted as I'm currently moving everything to HFS+. Long story - started as my /Users mount, became for documents only, etc.
AlexWasserman
 
Posts: 11
Joined: Sat Mar 29, 2014 3:40 pm

Re: Remove file marked as permanent error

Postby ilovezfs » Mon Oct 27, 2014 6:35 pm

It seems to me the most straightforward solution is to destroy and re-create the dataset.
ilovezfs
 
Posts: 232
Joined: Thu Mar 06, 2014 7:58 am

Re: Remove file marked as permanent error

Postby AlexWasserman » Mon Oct 27, 2014 6:38 pm

Yes, that would be theoretically easiest, but I don't have a place to reliably back-up 5TB of data, hence asking for a way to fix this.
AlexWasserman
 
Posts: 11
Joined: Sat Mar 29, 2014 3:40 pm

Re: Remove file marked as permanent error

Postby ilovezfs » Mon Oct 27, 2014 6:50 pm

It's 2.28T not 5T, and you can move the data from one dataset to another within the same pool, which will free and re-use the space as you move the files.

You could also try the runat command on illumos/Solaris to see if it can still find and access the extended attribute even though you deleted the file first.
ilovezfs
 
Posts: 232
Joined: Thu Mar 06, 2014 7:58 am


Return to General Discussions

Who is online

Users browsing this forum: No registered users and 5 guests