Send/Receive Issue

Moderators: jhartley, MSR734, nola

Send/Receive Issue

Post by shuman » Thu Oct 03, 2013 8:06 pm

I'm using the jollyjinx/ZFS-TimeMachine on github to do my send and receives. I had an error yesterday while performing the send and receive:

Code: Select all
WARNING: could not send zdata/Users/chris/Music@1969-12-31-180000: does not exist
cannot receive: failed to read from stream
Can't execute zfs send -v "zdata/Users/chris/Music@1969-12-31-180000" at ./zfstimemachinebackup.perl line 245.
Can't execute zfs receive -v -F "zdataTM/zdata/Users/chris/Music" at ./zfstimemachinebackup.perl line 253.


Notice the year of the time stamp is 1969. The error is in fact correct; there isn't a snapshot of Music@1969-12-31-180000. So the question is, where did the error come from and what can I do to fix it?

Snapshot list:
Code: Select all
zdata/Users/chris/Music@FullCatalogue2013-01-14  12.9Gi       -   157Gi  -
zdata/Users/chris/Music@2013-07-12-105453        8.51Mi       -   310Gi  -
zdata/Users/chris/Music@2013-07-12-133254        8.51Mi       -   310Gi  -
zdata/Users/chris/Music@2013-07-13-130740             0       -   310Gi  -
zdata/Users/chris/Music@2013-07-13-154305             0       -   310Gi  -
zdata/Users/chris/Music@2013-07-13-221436             0       -   310Gi  -
zdata/Users/chris/Music@2013-07-13-224413             0       -   310Gi  -
zdata/Users/chris/Music@2013-07-14-065412             0       -   310Gi  -
zdata/Users/chris/Music@2013-07-14-164035             0       -   310Gi  -
zdata/Users/chris/Music@2013-07-15-231053             0       -   310Gi  -
zdata/Users/chris/Music@2013-07-16-233250             0       -   310Gi  -
zdata/Users/chris/Music@2013-07-17-213014        32.3Mi       -   310Gi  -
zdata/Users/chris/Music@2013-07-17-222302        52.9Mi       -   310Gi  -
zdata/Users/chris/Music@2013-07-19-105136         120Ki       -   308Gi  -
zdata/Users/chris/Music@2013-07-19-171100             0       -   308Gi  -
zdata/Users/chris/Music@2013-07-19-171219             0       -   308Gi  -
zdata/Users/chris/Music@2013-07-19-171314             0       -   308Gi  -
zdata/Users/chris/Music@2013-07-19-171352             0       -   308Gi  -
zdata/Users/chris/Music@2013-07-19-214943           8Ki       -   308Gi  -
zdata/Users/chris/Music@2013-07-20-174830             0       -   308Gi  -
zdata/Users/chris/Music@2013-07-21-064926             0       -   308Gi  -
zdata/Users/chris/Music@2013-07-21-213131             0       -   308Gi  -
zdata/Users/chris/Music@2013-07-21-215949             0       -   308Gi  -
zdata/Users/chris/Music@2013-07-21-222242           8Ki       -   308Gi  -
zdata/Users/chris/Music@2013-07-22-093223        32.1Mi       -   308Gi  -
zdata/Users/chris/Music@2013-07-22-110454        32.1Mi       -   308Gi  -
zdata/Users/chris/Music@2013-07-26-152750             0       -   308Gi  -
zdata/Users/chris/Music@2013-07-26-153638             0       -   308Gi  -
zdata/Users/chris/Music@2013-07-26-155043             0       -   308Gi  -
zdata/Users/chris/Music@2013-07-26-172814        32.1Mi       -   308Gi  -
zdata/Users/chris/Music@2013-07-27-212433        32.1Mi       -   308Gi  -
zdata/Users/chris/Music@2013-07-28-221046        38.2Mi       -   308Gi  -
zdata/Users/chris/Music@2013-07-29-180837             0       -   308Gi  -
zdata/Users/chris/Music@2013-07-29-215519             0       -   308Gi  -
zdata/Users/chris/Music@2013-07-30-202420         104Ki       -   308Gi  -
zdata/Users/chris/Music@2013-07-31-215840         104Ki       -   308Gi  -
zdata/Users/chris/Music@2013-08-01-054448           8Ki       -   311Gi  -
zdata/Users/chris/Music@2013-08-01-180911           8Ki       -   311Gi  -
zdata/Users/chris/Music@2013-08-01-193800             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-01-195231             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-01-201758             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-01-203044             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-02-142609             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-02-150035             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-02-150056             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-02-152824             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-03-125035             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-03-223802             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-03-225450             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-04-160832             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-04-162857             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-04-221820             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-04-222438             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-06-222245             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-06-232852             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-08-205758             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-08-214154             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-10-093138             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-10-095300             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-10-105329             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-10-105344             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-13-192542             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-13-200710           8Ki       -   308Gi  -
zdata/Users/chris/Music@2013-08-13-214313             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-13-214412             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-13-222243             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-13-225114        8.61Mi       -   308Gi  -
zdata/Users/chris/Music@2013-08-13-225200         128Ki       -   308Gi  -
zdata/Users/chris/Music@2013-08-14-185833             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-14-190556             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-14-190646             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-14-190836             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-14-214118         104Ki       -   308Gi  -
zdata/Users/chris/Music@2013-08-16-221334             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-16-224507             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-17-083155             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-17-224406             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-19-060956        8.51Mi       -   308Gi  -
zdata/Users/chris/Music@2013-08-19-213105             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-20-130041             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-20-130125             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-22-211656             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-22-214854             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-24-100923             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-24-162258             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-25-223801             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-27-204742             0       -   308Gi  -
zdata/Users/chris/Music@2013-08-27-211649         104Ki       -   308Gi  -
zdata/Users/chris/Music@2013-08-27-223926        49.4Mi       -   308Gi  -
zdata/Users/chris/Music@2013-08-31-194313             0       -   309Gi  -
zdata/Users/chris/Music@2013-09-01-064900             0       -   309Gi  -
zdata/Users/chris/Music@2013-09-02-001359             0       -   309Gi  -
zdata/Users/chris/Music@2013-09-03-220306             0       -   309Gi  -
zdata/Users/chris/Music@2013-09-05-095144             0       -   309Gi  -
zdata/Users/chris/Music@2013-09-06-195642             0       -   309Gi  -
zdata/Users/chris/Music@2013-09-06-200404             0       -   309Gi  -
zdata/Users/chris/Music@2013-09-07-161942             0       -   309Gi  -
zdata/Users/chris/Music@2013-09-12-184253             0       -   309Gi  -
zdata/Users/chris/Music@2013-09-17-205604             0       -   309Gi  -
zdata/Users/chris/Music@2013-09-18-205831             0       -   309Gi  -
zdata/Users/chris/Music@2013-09-18-213551        23.7Mi       -   309Gi  -
zdata/Users/chris/Music@2013-09-19-172642           8Ki       -   309Gi  -
zdata/Users/chris/Music@2013-09-19-180112             0       -   309Gi  -
zdata/Users/chris/Music@2013-09-19-180224             0       -   309Gi  -
zdata/Users/chris/Music@2013-09-20-064528             0       -   309Gi  -
zdata/Users/chris/Music@2013-09-20-064851             0       -   309Gi  -
zdata/Users/chris/Music@2013-09-21-170555             0       -   310Gi  -
zdata/Users/chris/Music@2013-09-21-171502             0       -   310Gi  -
zdata/Users/chris/Music@2013-09-21-171905         112Ki       -   310Gi  -
zdata/Users/chris/Music@2013-09-21-174206             0       -   310Gi  -
zdata/Users/chris/Music@2013-09-21-174428             0       -   310Gi  -
zdata/Users/chris/Music@2013-09-21-194909         104Ki       -   310Gi  -
zdata/Users/chris/Music@2013-09-24-195636        31.4Mi       -   310Gi  -
zdata/Users/chris/Music@2013-09-28-184921           8Ki       -   310Gi  -
zdata/Users/chris/Music@2013-09-28-194718             0       -   310Gi  -
zdata/Users/chris/Music@2013-09-28-195237             0       -   310Gi  -
zdata/Users/chris/Music@2013-09-28-200708             0       -   310Gi  -
zdata/Users/chris/Music@2013-09-28-222114             0       -   310Gi  -
zdata/Users/chris/Music@2013-09-28-232043           8Ki       -   310Gi  -
zdata/Users/chris/Music@2013-09-29-070646           8Ki       -   310Gi  -
zdata/Users/chris/Music@2013-10-01-221034        31.3Mi       -   310Gi  -
zdata/Users/chris/Music@2013-10-02-211332        7.76Mi       -   310Gi  -
- Mac Mini (Late 2012), 10.8.5, 16GB memory, pool - 2 Mirrored 3TB USB 3.0 External Drives
shuman Offline

User avatar
 
Posts: 96
Joined: Mon Sep 17, 2012 8:15 am

Re: Send/Receive Issue

Post by shuman » Thu Oct 03, 2013 8:11 pm

Oh, here is the command I run:
Code: Select all
sh-3.2# cd /admin/TMzfs/ && sudo ./zfstimemachinebackup.perl  --sourcepool=zdata --destinationpool=zdataTM/ --snapshotstokeeponsource=100 --createsnapshotonsource --recursive --verbose
- Mac Mini (Late 2012), 10.8.5, 16GB memory, pool - 2 Mirrored 3TB USB 3.0 External Drives
shuman Offline

User avatar
 
Posts: 96
Joined: Mon Sep 17, 2012 8:15 am

1969 (predating UNIX time) – jollyjinx send and receive

Post by grahamperrin » Thu Oct 03, 2013 8:31 pm

If you restart the Mac, then is the error reproducible?

Was the date on the Mac wrong at any time? (Check its battery.)

When was the pool last scrubbed?

Has it been used with anything other than ZEVO?

How are the snapshots automated?

Please paste, as code, output from zpool status – thanks.

Maybe relevant, from one of the four [zfs-discuss] lists:

thr3ads.net - zfs discuss - ZFS Predates Unix Time? [Apr 2011]
Last edited by grahamperrin on Thu Oct 03, 2013 8:44 pm, edited 4 times in total.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: Send/Receive Issue

Post by shuman » Thu Oct 03, 2013 8:37 pm

The machine has been restarted and the date on the computer is correct right now (automated snapshots record the correct date/time). Either way, I don't have a snapshot with that time stamp so I'm not sure where the script is even getting it from.
- Mac Mini (Late 2012), 10.8.5, 16GB memory, pool - 2 Mirrored 3TB USB 3.0 External Drives
shuman Offline

User avatar
 
Posts: 96
Joined: Mon Sep 17, 2012 8:15 am

Re: Send/Receive Issue

Post by grahamperrin » Thu Oct 03, 2013 8:41 pm

Sorry, I edited my post whilst you were drafting yours. Some more questions and an observation above.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: Send/Receive Issue

Post by shuman » Thu Oct 03, 2013 9:04 pm

When was the pool last scrubbed?

Code: Select all
scan: scrub repaired 0 in 14h36m with 0 errors on Fri Sep 20 09:06:57 2013

Has it been used with anything other than ZEVO?

No
How are the snapshots automated?

They aren't automated. I have a small program that I created with Automater that I double click every now and then. I then kick off the perl script manually in the terminal with the command from my previous post.
Please paste, as code, output from zpool status – thanks.

Code: Select all
h-3.2# zpool status
  pool: zdata
 state: ONLINE
 scan: scrub repaired 0 in 14h36m with 0 errors on Fri Sep 20 09:06:57 2013
config:

   NAME                                           STATE     READ WRITE CKSUM
   zdata                                          ONLINE       0     0     0
     mirror-0                                     ONLINE       0     0     0
       GPTE_59D9A15C-8290-4B54-9BAC-C9D8306FD760  ONLINE       0     0     0  at disk6s2
       GPTE_1A52F1BF-D18F-4310-B304-1AF7F74CC944  ONLINE       0     0     0  at disk4s2
   cache
     GPTE_EA83CC80-7D3D-473F-BA31-05A061BC78C5    ONLINE       0     0     0  at disk0s4

errors: No known data errors

  pool: zdataTM
 state: ONLINE
 scan: none requested
config:

   NAME                                         STATE     READ WRITE CKSUM
   zdataTM                                      ONLINE       0     0     0
     GPTE_531AD9F1-1BAA-4C25-B79D-D50A18B675C5  ONLINE       0     0     0  at disk2s2

errors: No known data errors
- Mac Mini (Late 2012), 10.8.5, 16GB memory, pool - 2 Mirrored 3TB USB 3.0 External Drives
shuman Offline

User avatar
 
Posts: 96
Joined: Mon Sep 17, 2012 8:15 am

Wondering about caches with the jolllyjinx script, and so on

Post by grahamperrin » Fri Oct 04, 2013 12:46 am

Thanks. Patrick's attention is drawn to this topic.

Do you use the version of the script that uses per-dataset caches?

If so, I wonder whether either of the following actions will temporarily or permanently work around the issue:

  1. a run of Patrick's script without using caches (if that's possible)
  2. a non-scripted send and receive.

Additionally: as a 1969 symptom was noted in 2011 with a different implementation of ZFS, it might be worth drawing attention in irc://chat.freenode.net/#zfs
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: Send/Receive Issue

Post by jollyjinx » Fri Oct 04, 2013 2:09 am

To me it looks like the timestamp is incorrectly scanned assuming that the first Snapshot taken manually is incorrectly read then. Have a look if renaming the first snapshot in your list to the YYYY-MM-DD-hh:mm:ss format will fix this.

Patrick aka Jolly
jollyjinx Offline


 
Posts: 60
Joined: Sun Sep 16, 2012 12:40 pm
Location: Munich - Germany

Re: Send/Receive Issue

Post by shuman » Fri Oct 04, 2013 5:25 pm

I specifically named it like that to keep it from accidentally getting deleted by the automation. In the past the script would delightfully skip and give the message about the format not matching YYYY-MM-DD-hh:mm:ss. Should I clone it before I change the name? I don't want to lose that specific snapshot.
- Mac Mini (Late 2012), 10.8.5, 16GB memory, pool - 2 Mirrored 3TB USB 3.0 External Drives
shuman Offline

User avatar
 
Posts: 96
Joined: Mon Sep 17, 2012 8:15 am

zfs hold

Post by grahamperrin » Fri Oct 04, 2013 7:46 pm

shuman wrote:… to keep it from accidentally getting deleted …


… you can simply command:

zfs hold "keep this for such-and-such" zdata/Users/chris/Music@FullCatalogue2013-01-14

Then if ever you want to reveal the tag that you gave to the held snapshot:

zfs holds zdata/Users/chris/Music@FullCatalogue2013-01-14
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Next

Return to General Discussion

Who is online

Users browsing this forum: ilovezfs and 0 guests

cron