On RHEL 8, enabling NoMachine VirtualGL support when the GDM display manager is in use arises the following error in /var/log/messages:

No protocol specified

[VGL] ERROR: Could not open display :0.

This problem is solved in the newer version of VirtualGL (v. 2.6.2) which will be included in the next release of NoMachine packages (the current packages v. 6.7 provide VirtualGL-2.5.2).

As a temporary workaround, it's possible to use for example LightDM as display manager. This display manager is not present in the official RHEL 8 repositories but can be installed manually by downloading it from the Fedora 28 repositories.

Retrieve and install these packages:

desktop-backgrounds-basic-28.0.0-1.fc28.noarch.rpm
desktop-backgrounds-compat-28.0.0-1.fc28.noarch.rpm
f28-backgrounds-base-28.1.3-1.fc28.noarch.rpm
lightdm-1.25.2-1.fc28.x86_64.rpm
lightdm-gobject-1.25.2-1.fc28.x86_64.rpm
lightdm-gtk-2.0.5-1.fc28.x86_64.rpm

Then run the following commands:

 systemctl stop gdm
 systemctl disable gdm
 systemctl enable lightdm
 systemctl start lightdm

Configure the system for VirtualGL:

 /etc/NX/nxserver --virtualgl-install

Editing the DefaultDesktopCommand key in /usr/NX/etc/node.cfg to start the GNOME desktop using the following command:

DefaultDesktopCommand  "/etc/gdm/Xsession 'env __GLX_VENDOR_LIBRARY_NAME=nvidia GNOME_SHELL_SESSION_MODE=classic XDG_SESSION_TYPE=x11 /usr/NX/scripts/vgl/vglrun gnome-session --session=gnome-classic --disable-acceleration-check' "