Page 1 of 1

How to fix: errors: Permanent errors have been detected?

PostPosted: Wed Jun 09, 2021 7:25 pm
by novice
Hi everyone,

Recently I was facing error:

mirror0: checksum error 20
disk1 checksum error 40
disk2 checksum error 40
mirror1
disk3
disk4
mirror2
disk5
disk6
errors: Permanent errors have been detected in the following files:
/foo/bar
/foo/baz

What I did was replaced disk2 with a new one and resilvered. And got:

mirror0: checksum error 2
disk1 checksum error 4
disk2_new checksum error 4
mirror1
disk3
disk4
mirror2
disk5
disk6
errors: Permanent errors have been detected in the following files:
/foo/bar
/foo/baz

I ignored the checksum error (reduced from 20 to 2, 40 to 4) and did a scrub.

After the scrub finished, still got:
mirror0: checksum error 2
disk1 checksum error 4
disk2_new checksum error 4
mirror1
disk3
disk4
mirror2
disk5
disk6
errors: Permanent errors have been detected in the following files:
/foo/bar
/foo/baz

When I read these two files, I got io error.

I was wondering:
1. is what I did with resilver correct? Could I do things better?
2. is there anyway to read these files with error anyway (ignoring the error)? I tried to disable checksum, but still got io error when read it.
3. how to clear the error message "errors: Permanent errors have been detected in the following files:" after I recovered the two files?

Thanks!

Re: How to fix: errors: Permanent errors have been detected?

PostPosted: Mon Jun 21, 2021 4:45 pm
by lundman
I think you should "rm" the two files, but you also need to make sure any snapshot/bookmark that contain them have been removed, then you run a "zpool clear" command.

Re: How to fix: errors: Permanent errors have been detected?

PostPosted: Fri Jul 02, 2021 8:23 pm
by novice
Thanks! I replaced disk and removed these two files, the errors are gone now.
After using ZFS I begin to see file corruption very occasionally, I guess before using ZFS, I simply did not know data was corrupted :)

Re: How to fix: errors: Permanent errors have been detected?

PostPosted: Wed Jul 07, 2021 6:08 am
by JasonBelec
Might I suggested you test the hardware your ZFS is on. We usually see these issues with hardware, like bad cables, bad back planes, bad power, very rarely is it a drive issue per say.

Re: How to fix: errors: Permanent errors have been detected?

PostPosted: Sat Jul 10, 2021 9:57 am
by novice
Thanks all. Seems it's bad cable.