MAJOR CATASTROPHE! LOST EVERYTHING! [edit - FALSE ALARM]

Moderators: jhartley, MSR734, nola

MAJOR CATASTROPHE! LOST EVERYTHING! [edit - FALSE ALARM]

Post by markpat3rson » Thu Sep 20, 2012 2:43 am

oh my god. i think i've just lost 3TB of irreplaceable personal and work files. :cry:

let me back up a bit and explain. I have a Mac Pro, 32GB.

I created a 3TB x 3TB x 3TB Raidz called Ocean, with a Dataset called Home.

NAME USED AVAIL REFER MOUNTPOINT
Ocean 3.09Ti 2.28Ti 51.8Mi /Volumes/Ocean
Ocean/Home 3.09Ti 2.28Ti 3.09Ti /Volumes/Ocean/Home

I rsync'd my OS X home folder contents to 'Home', changed my advanced user settings, and rebooted. Everything worked fine. My home was safely on my ZFS dataset. I copied over the rest of my files to it using rsync, took about 2 days.

Everything seemed fine until I needed to restart my computer. When it booted up, and i logged in, i basically got the OS X setup wizard. uh oh. My home dataset was gone.

actually, it's still inside Ocean, but every folder (except Library, which I assume was created anew) is now empty. Basically like a fresh OS X installation. :cry:

I headed over to terminal and did zpool status and got this…

---
NAME STATE READ WRITE CKSUM
Ocean ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
GPTE_4623BE4C-B931-42E3-B086-1AB4B7C25AD8 ONLINE 0 0 0 at disk2s2
GPTE_E438C301-7879-49CB-8E09-22B734793326 ONLINE 0 0 0 at disk1s2
GPTE_608CE13A-017D-4EE6-803F-18429C27904A ONLINE 0 0 0 at disk0s2

errors: No known data errors

OK, THAT SEEMS NORMAL.
---

Then I did zfs list and got this…

Ocean 3.09Ti 2.28Ti 51.9Mi /Volumes/Ocean
Ocean/Home 3.09Ti 2.28Ti 3.09Ti /Volumes/Ocean/Home

OK THAT SEEMS NORMAL TOO. HMM.

---

I also have a snapshot from right before I restarted, that is still listed

NAME USED AVAIL REFER MOUNTPOINT
Ocean/Home@2012-09-20 10.7Ki - 3.09Ti -

YEP, LOOKS NORMAL. SO YOU'D THINK FROM THESE 3 POSITIVE SIGNS, THAT I COULD SALVAGE THIS RIGHT?

I tried rolling back to this snapshot, and restarting, but it did nothing. I can't even see a .zfs folder inside Home. I toggled the visibility to on, but it did nothing.

THE CRAZY THING IS, inside the Ocean disk, the 'Home' icon is now just a folder. It is no longer showing up as a ZFS volume via Get Info. I created a test dataset, Ocean/Test, and that mounted on the desktop just fine, and it sits inside the Ocean volume as a nested volume, just like Home USED to do. 'Test' also has a .zfs folder.

I thought that if I have raidz redundancy, I can rebuild from a fault like this. PLEASE tell me this is the case. And if so, how? I am currently scrubbing to see if that helps.

I DO NOT have a backup. I created a new backup pool from my *original drives* that I copied the data from initially, and was just about to send / receive everything back to them when this happened.

ARGGGHHHHHHH.

If this is any help, my mac updated me to 10.8.2 earlier today. zfs seemed to be working fine on that, although I guess it could be a bug.

Just please tell me my data is still there. If I just can't see it for the moment, I can live with that.
Last edited by markpat3rson on Thu Sep 20, 2012 7:51 am, edited 1 time in total.
markpat3rson Offline


 
Posts: 12
Joined: Sun Sep 16, 2012 8:57 pm

Re: MAJOR CATASTROPHE! I'VE LOST EVERYTHING!

Post by markpat3rson » Thu Sep 20, 2012 3:18 am

ok, i think this is what happened. I think I was actually DOING the send / recv (it was only going for about 1 min), but i needed to cancel it and reboot, so i did control-c. Then I rebooted. I think the send/recv possibly unmounted my 'home' and somehow replaced it with a folder inside Ocean?

I just tried doing

zfs send Ocean/Home@2012-09-20 | zfs receive -v -F Offsite_Backup/Home_Backup

to try and copy the good snapshot to my 6TB backup drive.

And I noticed that if i go inside the Offsite_Backup volume while the send/recv is copying, the Home_Backup is turned from a ZFS Volume into a standard folder. Then when I cancel the send/recieve, and mount the Home_Backup, it turns back to a ZFS volume.

Now, when I try to mount the Home folder, i get this error…

cannot mount '/Volumes/Ocean/Home': directory is not empty

help????

Im guessing if the 'Home' volume wasn't mounted at reboot, then that's why OS X created a new one in place of it.
markpat3rson Offline


 
Posts: 12
Joined: Sun Sep 16, 2012 8:57 pm

Re: MAJOR CATASTROPHE! I'VE LOST EVERYTHING!

Post by markpat3rson » Thu Sep 20, 2012 3:33 am

SOLVED!!!

I took a leap of faith. I renamed the 'Home' folder inside Ocean, then attempted to mount the drive…

zfs mount Ocean/Home

SUCCESS! It mounted. All my files are there!!!!! WHAT A ROLLER COASTER OF EMOTIONS!

I'm guessing that my theory was correct, that OS X created a new 'Home' folder in lieu of the 'Home' dataset, which was unmounted at reboot because of a bad shutdown. The new Home folder prevented the Dataset from mounting, because they had the same name.

Does this theory sound correct?

Ok, i'm gonna scrub, and then backup like there's no tomorrow!
markpat3rson Offline


 
Posts: 12
Joined: Sun Sep 16, 2012 8:57 pm

Re: MAJOR CATASTROPHE! LOST EVERYTHING! [edit - FALSE ALARM]

Post by lucid » Thu Sep 20, 2012 2:35 pm

Most likely the problem was that your zfs filesystem wasn't mounted at all or wasn't mounted in the place where you expected it to be.
To check this just enter 'mount' or 'zfs mount' in a terminal to get a list of all mounted filesystems.

Of course it's of no use to restore snapshots etc in that situation, because not only won't you see any changes, you may also trash new data from after the snapshot.
Actually it seems there was not a single zfs fault, so no need to salvage anything using zfs redundancy.

It's not really clear from your description how/if at all you tried to mount your zfs dataset into your /Users directory.
Usually home directories sit in /Users/ on Mac OSX, so if you copy your home directory to /Volumes/Ocean/Home and then delete your home directory from /Users, what else should OSX do than to recreate an empty new home dir in /Users?

There are at least 3 ways to fix this:
1. mount the dataset directly into /Users
2. change your home directory setting to /Volumes/Ocean/Home using the OSX account preference pane
3. make a soft link in /Users to your dataset mounted in /Volumes/Ocean

I don't know which you chose, but usually a preexisting directory doesn't prevent a fs from beeing mounted there, whether it is empty or not.
lucid Offline


 
Posts: 1
Joined: Sat Sep 15, 2012 8:04 pm


Return to General Discussion

Who is online

Users browsing this forum: ilovezfs and 1 guest

cron