internal error: operation not supported on socket

All your general support questions for OpenZFS on OS X.

Re: internal error: operation not supported on socket

Postby lundman » Tue Feb 25, 2020 3:39 pm

We are looking for Err#45, ENOTSUP.. actually, we could just look for that specifically:

dtrace -n 'zfs*:return { if (arg1 == 45) {stack();} }'

Which will probe all zfs commands returning 45. But might also need to change "zfs*" to "dsl*" possibly. Just "*" will probably match too many functions.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: internal error: operation not supported on socket

Postby roemer » Sun Mar 01, 2020 3:10 am

lundman wrote:We are looking for Err#45, ENOTSUP.. actually, we could just look for that specifically:
dtrace -n 'zfs*:return { if (arg1 == 45) {stack();} }'


I tried this, both for the 'zfs*' as well as the 'dal*' pattern - to no avail. Simply no output.
If I use '*', then indeed a lot of dtrace messages are shown, too much to list here though...
Note that I disabled SIP for dtrace only with "csrutil enable –without dtrace" in recovery mode, in case this makes a difference.

I also tried running the zfs command (self-compiled from source) with lldb:
Code: Select all
bash$ lldb cmd/zfs/.libs/zfs
lldb cmd/zfs/.libs/zfs
(lldb) run create zdata/test
Process 32567 launched: 'cmd/zfs/.libs/zfs' (x86_64)
internal error: Operation not supported on socket
Process 32567 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
    frame #0: 0x00007fff686667fa libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill:
->  0x7fff686667fa <+10>: jae    0x7fff68666804            ; <+20>
    0x7fff686667fc <+12>: movq   %rax, %rdi
    0x7fff686667ff <+15>: jmp    0x7fff68660a89            ; cerror_nocancel
    0x7fff68666804 <+20>: retq   
Target 0: (zfs) stopped.

(lldb) thread list
Process 32567 stopped
* thread #1: tid = 0x273c1d, 0x00007fff686667fa libsystem_kernel.dylib`__pthread_kill + 10, queue = 'com.apple.main-thread', stop reason = signal SIGABRT

(lldb) thread backtrace
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x00007fff686667fa libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff68723bc1 libsystem_pthread.dylib`pthread_kill + 432
    frame #2: 0x00007fff685eda1c libsystem_c.dylib`abort + 120
    frame #3: 0x000000010057bd75 libzfs.2.dylib`zfs_verror + 174
    frame #4: 0x000000010057c006 libzfs.2.dylib`zfs_standard_error_fmt + 619
    frame #5: 0x000000010057bd94 libzfs.2.dylib`zfs_standard_error + 21
    frame #6: 0x000000010055f1a6 libzfs.2.dylib`zfs_create + 1235
    frame #7: 0x00000001000036c5 zfs`zfs_do_create(argc=<unavailable>, argv=0x00007ffeefbff918) at zfs_main.c:979:6 [opt]
    frame #8: 0x0000000100002750 zfs`main(argc=<unavailable>, argv=0x00007ffeefbff910) at zfs_main.c:8014:10 [opt]
    frame #9: 0x00007fff6851f7fd libdyld.dylib`start + 1


Does this help you?
Sorry, I openly admit that I am not really familiar with lldb or dtrace...

PS: I am not sure whether libzfs.2.dylib (probably accessed in /usr/local/lib/) matches the executed version of zfs... Installed is zfs-1.9.3-3-ge94a41240
roemer
 
Posts: 73
Joined: Sat Mar 15, 2014 2:32 pm

Re: internal error: operation not supported on socket

Postby lundman » Mon Mar 02, 2020 10:18 pm

You can tell lldb to also use the compiled libraries;

If you look at "cmd.sh", near the end it has:

exec ${topdir}/cmd/$cmd/.libs/$cmd "$@"

change that to say

exec lldb ${topdir}/cmd/$cmd/.libs/$cmd "$@"

and run it "./cmd.sh zfs" to start lldb, using the compiled libs (all the script does is set DYLD_LIBRARY_PATH for all the libs before running the command.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: internal error: operation not supported on socket

Postby roemer » Fri Mar 06, 2020 7:30 am

lundman wrote:You can tell lldb to also use the compiled libraries; ...

Hmmm, this did not work as intended; still picked up the libraries from /usr/local/lib ...
In the end, I temporarily symlinked the compiled libraries to /usr/local/lib/, and then lldb finally used them.
Very strange...

Anyway, with that out of the way, I get the error to this call here in libzfs_core.c line 178:
Code: Select all
* thread #1, queue = 'com.apple.main-thread', stop reason = step in
    frame #0: 0x00000001005a71fc libzfs_core.1.dylib`zioctl(fildes=<unavailable>, ioc=ZFS_IOC_CREATE, zc=0x00007ffeefbfca40) at libzfs_core.c:178:20 [opt]
   175    zioctl(int fildes, zfs_ioc_t ioc, zfs_cmd_t *zc)
   176    {
   177       int ioctl_err;
-> 178       ioctl_err = ioctl(g_fd, ioc, zc);
   179    
   180       // OS call failed (ZFS not reached)
   181       if (ioctl_err != 0) return ioctl_err;

This ioctl() call returns ioctl_err 2 and zc.zc_ioc_error = 102.
Call parameter are ioctl(4, ZFS_IOC_CREATE, ...)

Does this help?
Last edited by roemer on Sat Mar 07, 2020 1:56 am, edited 1 time in total.
roemer
 
Posts: 73
Joined: Sat Mar 15, 2014 2:32 pm

Re: internal error: operation not supported on socket

Postby lundman » Fri Mar 06, 2020 5:12 pm

ioctl_err = ioctl = 2

Hmm that is an error from XNU, and it doesn't even call ZFS. This shouldn't happen if the userland binaries and kernel are the same version? Can we double check the versions?

which zfs
md5sum `which zfs`
?
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: internal error: operation not supported on socket

Postby roemer » Fri Mar 06, 2020 8:05 pm

Sorry, I made a mistake yesterday night (too late, too tired):
The ioctl() call indeed returns 0, but it sets the zc_ioc_error value of zc.zc_ioc_error to 102
Code: Select all
(lldb) s
Process 1396 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step in
    frame #0: 0x00000001005a7210 libzfs_core.1.dylib`zioctl(fildes=<unavailable>, ioc=<unavailable>, zc=0x00007ffeefbfca70) at libzfs_core.c:181:16 [opt]
   178       ioctl_err = ioctl(g_fd, ioc, zc);
   179    
   180       // OS call failed (ZFS not reached)
-> 181       if (ioctl_err != 0) return ioctl_err;
   182    
   183       if (zc->zc_ioc_error == 0)
   184          return 0;
Target 0: (zfs) stopped.
(lldb) p ioctl_err
(int) $6 = 0
(lldb) p zc->zc_ioc_error
(int) $9 = 102


This part is called from here:
Code: Select all
(lldb) s
Process 1349 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step in
    frame #0: 0x00000001005a6da9 libzfs_core.1.dylib`lzc_create(fsname="zdata/test", type=<unavailable>, props=<unavailable>, wkeydata=0x0000000000000000, wkeylen=<unavailable>) at libzfs_core.c:295:2 [opt]
   292       }
   293    
   294       error = lzc_ioctl(ZFS_IOC_CREATE, fsname, args, NULL);
-> 295       nvlist_free(hidden_args);
   296       nvlist_free(args);
   297       return (error);
   298    }
Target 0: (zfs) stopped.
(lldb) p error
(int) $3 = 102
(lldb) p *args
(nvlist_t) $5 = (nvl_version = 0, nvl_nvflag = 1, nvl_priv = 4301261632, nvl_flag = 0, nvl_pad = 0)


I execute a self-compiled zfs version directly from the build directory, with two dynamic link libraries (libzfs.2.dylib and libzfs_core.1.dylib) sym-linked to /usr/local/lib
Code: Select all
bash$ md5sum cmd/zfs/.libs/zfs
43a24f96cd8acf4e420d0d618d63a97f  cmd/zfs/.libs/zfs
bash$ cmd/zfs/.libs/zfs --version
zfs-1.9.4-0
zfs-kmod-1.9.3-3-ge94a41240
roemer
 
Posts: 73
Joined: Sat Mar 15, 2014 2:32 pm

Re: internal error: operation not supported on socket

Postby lundman » Fri Mar 06, 2020 11:18 pm

Fantastic ok, so 0 from ioctl() is sane, and 102 error comes from EOPNOTSUPP

There are two such return values in dmu_objset_create_crypt_check(), could we confirm that the error code is from that?

dtrace -n 'dmu_objset_create_crypt_check:return { printf("%d", arg1); } '

Or, you could run the "check all zfs for error" but change 45 to error 102, since that is the one we
are looking for.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: internal error: operation not supported on socket

Postby roemer » Sat Mar 07, 2020 12:37 am

Indeed it seems so. Output of dtrace while I was executing zfs create:
Code: Select all
bash$ sudo dtrace -n 'dmu_objset_create_crypt_check:return { printf("%d", arg1); } '
Password:
dtrace: description 'dmu_objset_create_crypt_check:return ' matched 1 probe
CPU     ID                    FUNCTION:NAME
  0 279395 dmu_objset_create_crypt_check:return 102

And yes, the pool onto which I try to create this new dataset is an encrypted pool.
roemer
 
Posts: 73
Joined: Sat Mar 15, 2014 2:32 pm

Re: internal error: operation not supported on socket

Postby lundman » Sat Mar 07, 2020 1:27 am

Ok, two reasons

Code: Select all
        /*
         * We will now definitely be encrypting. Check the feature flag. When
         * creating the pool the caller will check this for us since we won't
         * technically have the feature activated yet.
         */
        if (parentdd != NULL &&
            !spa_feature_is_enabled(parentdd->dd_pool->dp_spa,
            SPA_FEATURE_ENCRYPTION)) {
                return (SET_ERROR(EOPNOTSUPP));
        }

        /* Check for errata #4 (encryption enabled, bookmark_v2 disabled) */
        if (parentdd != NULL &&
            !spa_feature_is_enabled(parentdd->dd_pool->dp_spa,
            SPA_FEATURE_BOOKMARK_V2)) {
                return (SET_ERROR(EOPNOTSUPP));
        }


First reason seems to be if the feature flag is not enabled, but since you already have an encrypted dataset, that is probably not it.

Second part is if you don't have bookmarks_v2 enabled. I guess we should check the the feature flags.

zpool get all - should list all features, you can also run
zpool upgrade -v - but don't put a pool name, or it will upgrade and not list.
User avatar
lundman
 
Posts: 1335
Joined: Thu Mar 06, 2014 2:05 pm
Location: Tokyo, Japan

Re: internal error: operation not supported on socket

Postby roemer » Sat Mar 07, 2020 1:32 am

Great, we seem to get closer!
Code: Select all
$ zpool get all zdata
NAME   PROPERTY                       VALUE                          SOURCE
zdata  size                           928G                           -
zdata  capacity                       89%                            -
zdata  altroot                        -                              default
zdata  health                         ONLINE                         -
zdata  guid                           17887915123981585236           default
zdata  version                        -                              default
zdata  bootfs                         -                              default
zdata  delegation                     on                             default
zdata  autoreplace                    off                            default
zdata  cachefile                      -                              default
zdata  failmode                       wait                           default
zdata  listsnapshots                  off                            default
zdata  autoexpand                     off                            default
zdata  dedupditto                     0                              default
zdata  dedupratio                     1.00x                          -
zdata  free                           101G                           -
zdata  allocated                      827G                           -
zdata  readonly                       off                            -
zdata  ashift                         13                             local
zdata  comment                        -                              default
zdata  expandsize                     -                              -
zdata  freeing                        0                              default
zdata  fragmentation                  38%                            -
zdata  leaked                         0                              default
zdata  checkpoint                     -                              -
zdata  multihost                      off                            default
zdata  autotrim                       off                            default
zdata  feature@async_destroy          enabled                        local
zdata  feature@empty_bpobj            active                         local
zdata  feature@lz4_compress           active                         local
zdata  feature@multi_vdev_crash_dump  enabled                        local
zdata  feature@spacemap_histogram     active                         local
zdata  feature@enabled_txg            active                         local
zdata  feature@hole_birth             active                         local
zdata  feature@extensible_dataset     active                         local
zdata  feature@embedded_data          active                         local
zdata  feature@bookmarks              enabled                        local
zdata  feature@filesystem_limits      enabled                        local
zdata  feature@large_blocks           enabled                        local
zdata  feature@large_dnode            disabled                       local
zdata  feature@sha512                 enabled                        local
zdata  feature@skein                  enabled                        local
zdata  feature@edonr                  active                         local
zdata  feature@encryption             active                         local
zdata  feature@device_removal         enabled                        local
zdata  feature@obsolete_counts        enabled                        local
zdata  feature@zpool_checkpoint       enabled                        local
zdata  feature@spacemap_v2            disabled                       local
zdata  feature@allocation_classes     disabled                       local
zdata  feature@bookmark_v2            disabled                       local
zdata  feature@resilver_defer         disabled                       local


Indeed, feature@bookmark_v2 is disabled. Probably because this pool was created with an older version of zfs.
What does feature@bookmark_v2 do?

PS: How much do I limit myself by upgrading the pool to include this bookmark_v2 feature or even all new features? E.g. is send/receive still possible between the upgraded pool and a pool without those flags?
Last edited by roemer on Sat Mar 07, 2020 3:45 am, edited 2 times in total.
roemer
 
Posts: 73
Joined: Sat Mar 15, 2014 2:32 pm

PreviousNext

Return to General Help

Who is online

Users browsing this forum: No registered users and 26 guests