Hardware encoding by AMD video card is supported on Linux hosts since NoMachine version 8.0.168. NoMachine will be able to use the GPU-accelerated H.264 encoding encoding provided by AMD's Video Coding Engine (VCE) that uses AMF (Advanced Media Framework) SDK API on the Linux machines having a graphics card supporting it.

Step 1: Download the drivers
Download the AMD GPU drivers from the official web site at: https://www.amd.com/en/support/download/linux-drivers.html. Make sure to choose the correct version for your system.

Step 2: Install the drivers
Open a terminal and navigate to the directory where you downloaded the drivers package. Then install the drivers by running this command:

sudo apt-get install ./amdgpu-install_5.4.50401-1_all.deb

Replace the filename above with the name of the file you downloaded. If it has been downloaded in a different directory, provide the full path to the file.

Step 3: Install the AMF Use Case
Configure the drivers to use the AMF encoder. To do this run the following command:

sudo amdgpu-install --usecase=amf

This will configure the drivers to use the AMF encoder for video encoding.

TROUBLESHOOTING

If the installed version of drivers doesn't work properly for NoMachine sessions or errors are issued (you can notice them in the $HOME/.nx/node/C-..../session directory on the server host), downgrading drivers version and/or trying other versions could help.

Step 1: Uninstall the current drivers
sudo amdgpu-uninstall
sudo apt purge amdgpu-install
sudo reboot

Step 2: Download and install a drivers version
Go to https://repo.radeon.com/amdgpu/ and download a drivers version.

For example we downloaded and installed drivers with version 22.20.5. Run the following commands in a terminal:
wget https://repo.radeon.com/amdgpu/22.20.5/ubuntu/pool/main/a/amdgpu-install/amdgpu-install_22.20.50205-1511377~20.04_all.deb

sudo apt install ./amdgpu-install_22.20.50205-1511377~20.04_all.deb

Step 3: Install the AMF Use Case
Uncomment line from /etc/apt/sources.list.d/amdgpu-proprietary.list:
deb https://repo.radeon.com/amdgpu/22.20.5/ubuntu focal proprietary

and configure the drivers to use the AMF encoder:
sudo amdgpu-install --usecase=amf
sudo reboot

Some remarks
Not all the drivers version have the amf-amdgpu-pro package, therefore it's possible that the drivers installation issues an error like:
"Unable to locate package amf-amdgpu-pro"

Drivers that have the amf-amdgpu-pro package have two directories under for example /amdgpu/DRIVER_VERSION/ubuntu/pool:
-  main (which should have the amdgpu-install package)
-  proprietary (which should contain amf-amdgpu-pro package)

We suggest to try to use drivers from 22.x and 21.x directories. Note that the error about being unable to locate amf-amdgpu-pro will be still issued when running amdgpu-install unless the repository won't be commented out
in /etc/apt/sources.list.d/amdgpu-proprietary.list.

We cannot say a priori which driver version can work with the specific GPU, so it is advisable to try different versions of available drivers. Ensure also to download the correct drivers for your specific Linux distribution and version.