The current implementation of support for rootless sessions on macOS, i.e.  custom sessions running a Linux application in a floating window, requires X11 to be installed on the local system.

When creating a custom session in rootless mode, the remote NoMachine service must connect to a local X server offering the window manager integration to the windows created by the applications running on the remote side. On macOS such X server is implemented by an application called XQuartz, officially supported by Apple for many years.

Given that the implementation of XQuartz is stable and reliable, at the moment we don't have in plan to replace it with our own implementation.

To run NoMachine Linux custom sessions in floating window mode on your macOS, download the XQuartz package from the official web site:  https://www.xquartz.org and install it on your mac.