Self-Host RustDesk on a VPS: Docker, Ports and Setup

Helping someone with their computer should not begin with a tour of your router settings. A self-hosted RustDesk server gives your devices a stable place to register and a relay when they cannot connect directly. That makes it a useful project for a home lab, a small support team or a business that wants control of its remote-access infrastructure.

The trade-off is responsibility. You operate the server, keep the clients updated and decide who can access each computer. Installing two containers is the easy part; preserving the keys and testing the complete connection are what make the setup dependable.

What does a self-hosted RustDesk server actually do?

RustDesk first attempts a direct connection between devices; hosting your own server does not automatically put every session through it. The official architecture overview explains the rendezvous and relay roles.

OSS or Pro? The free OSS backend is suitable when you are happy to configure clients yourself. Pro adds central administration features such as a web console and identity-provider integration. Compare the current licence options if you need those controls. A VPS subscription does not include a RustDesk Pro licence.

1. Prepare a Linux VPS and a DNS-only hostname

Use a fresh Linux VPS with root or sudo access and a public IPv4 address for this walkthrough. A small server with at least 1 GB RAM is a sensible starting point for evaluating a few devices, not a promise of a particular session capacity. Watch real CPU, memory and transfer use before sizing a larger deployment.

• Operating system: a maintained Ubuntu LTS installation with Docker Engine and the Compose plugin. Follow our Docker installation guide or Docker's instructions for your exact OS release.
• Hostname: create an A record such as rustdesk.example.com pointing to the VPS IPv4 address. Replace that example everywhere below.
• Access: keep an SSH session open and confirm that you can reach the provider console before changing the firewall.
• Existing services: do not run a second RustDesk installation on the same ports. A clean VM makes ownership of the data and listeners much easier to establish.

On a minimal Ubuntu image, install the utilities used below with sudo apt update && sudo apt install ufw nano. If another firewall manager is already in use, configure that instead of layering UFW over it.

If you use Cloudflare DNS, select DNS only for this hostname. Its normal HTTP proxy does not carry the native RustDesk ports; see Cloudflare's supported proxy ports. Add an AAAA record only after checking IPv6 routing and firewall access too, rather than advertising an address that clients cannot reach.

getent ahostsv4 rustdesk.example.com
sudo docker version
sudo docker compose version

The resolved address should be your VPS. This tutorial is for Docker Engine on Linux, not a Docker Desktop installation on the computer you want to control.

2. Open only the RustDesk ports you need

These roles follow the official OSS installation reference. Do not open the entire port range just because another tutorial does.

On a fresh Ubuntu server using UFW, the following keeps incoming connections denied except for SSH and the native-client ports. Change the SSH rule first if you use a port other than 22. Preserve any other legitimate rules on an existing server; do not reset its firewall.

sudo ufw allow 22/tcp
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 21115:21117/tcp
sudo ufw allow 21116/udp
sudo ufw enable
sudo ufw status verbose

Open a second SSH session now to confirm that access still works before closing the original one.

Check any separate provider firewall as well. For a fixed set of office IPs, consider source-restricted rules instead of allowing all sources. Ubuntu's firewall guide explains UFW rule management. If IPv6 is configured, verify equivalent filtering rather than assuming the IPv4 rules cover it.

3. Install RustDesk Server with Docker Compose

Create a dedicated project directory and a root-only data directory. Keep both server processes on the same persistent storage path.

sudo install -d -m 755 /opt/rustdesk
sudo install -d -m 700 /opt/rustdesk/data
cd /opt/rustdesk
sudo nano compose.yaml

Paste the following, replacing rustdesk.example.com with your DNS name:

services:
  hbbs:
    image: rustdesk/rustdesk-server:1.1.16
    command: ["hbbs", "-r", "rustdesk.example.com:21117", "-k", "_"]
    network_mode: host
    volumes:
      - ./data:/root
    restart: unless-stopped
    logging:
      driver: local

  hbbr:
    image: rustdesk/rustdesk-server:1.1.16
    command: ["hbbr", "-k", "_"]
    network_mode: host
    volumes:
      - ./data:/root
    restart: unless-stopped
    logging:
      driver: local

The version is deliberately pinned. The 1.1.16 release notes include a fix for UDP reflection/amplification abuse, among other fixes. Check for newer security releases when deploying; do not copy an old tag indefinitely.

network_mode: host shares the Linux host's network namespace, so there is no ports: section. Host firewall rules control the listeners. This is different from publishing ports from a bridged container; Docker describes the behaviour in its host-network documentation.

Both services mount ./data at /root. The -k _ option loads or creates the key material; specifying it on hbbr enables relay key validation instead of its empty-key default. See the versioned relay implementation. It is not a replacement for permission controls on each remote computer.

Validate the file, fetch the image and start only the ID server first so the key pair exists before the relay starts:

sudo docker compose config --quiet
sudo docker compose pull
sudo docker compose up -d hbbs
sudo docker compose logs --tail=50 hbbs
sudo ls -l data/id_ed25519 data/id_ed25519.pub

Wait for both files to exist. If they do not appear, inspect the logs and directory permissions before continuing. Then protect the private key, read the public key and start the relay:

sudo chmod 600 data/id_ed25519
sudo cat data/id_ed25519.pub
sudo docker compose up -d hbbr
sudo docker compose ps
sudo docker compose logs --tail=50 hbbr

Copy only the value printed from id_ed25519.pub. Never paste id_ed25519, the private file, into a client or support ticket. The classic container image has no shell or cat; read the file on the host as shown, rather than trying docker exec ... cat. This follows the published image build.

4. Point both RustDesk clients at your server

Install the native client from the official RustDesk website on the controlling computer and the computer being accessed. In each client's network settings, unlock editing if prompted and set:

• ID Server: rustdesk.example.com.
• Key: the complete contents of data/id_ed25519.pub, including any trailing =.
• Relay Server: leave blank for this same-host setup; the ID server advertises the relay address configured above.
• API Server: leave blank for OSS.

The official client configuration guide shows the settings. Apply them on both sides; configuring only the laptop you connect from is not enough.

Connect using the remote device's RustDesk ID and accept the request on that device, or use its deliberately configured unattended-access credentials. Test from two different networks. Then test relay operation as well: RustDesk documents appending /r to the remote ID to force a relay connection in its relay troubleshooting FAQ. A successful direct session alone does not prove that TCP 21117 is reachable.

Confirm the remote user can revoke access. Disable clipboard or file-transfer permissions when the task does not need them, and avoid a shared unattended-access password across all computers.

RustDesk key mismatch, offline and relay errors

Start with evidence from the affected connection, not a reinstall. These commands inspect the deployment without replacing its keys:

cd /opt/rustdesk
sudo docker compose ps
sudo docker compose logs --since=15m --tail=100 hbbs hbbr
sudo ss -lntup | grep -E '2111[5-9]'
sudo cat data/id_ed25519.pub

The project's troubleshooting FAQ documents duplicate server processes and key mismatch cases. Do not delete the data folder to “refresh” the installation: changing the server identity can strand every configured client.

From a Windows client, Test-NetConnection rustdesk.example.com -Port 21117 checks TCP reachability. It does not test UDP registration or prove that an authenticated remote session works. If one network succeeds and another fails, compare DNS answers, outbound filtering and client settings before resizing the VPS.

Keep remote access recoverable and controlled

Back up the entire data directory and Compose file, not just the image name. The private key is part of the server's identity. Losing it can turn a simple rebuild into a client-by-client reconfiguration job.

1. Schedule a brief maintenance window and stop both services with sudo docker compose stop.
2. Archive /opt/rustdesk/data and /opt/rustdesk/compose.yaml while they are stopped. Store the archive privately and encrypt any off-server copy.
3. Restart with sudo docker compose up -d and verify both a normal session and a forced-relay session.
4. Test recovery on an isolated spare VM. Restore the files before starting the services; do not accidentally put a second production identity behind the same DNS name.

A HYEHOST Storage Box can hold an encrypted off-server backup through SFTP. Our rclone backup guide covers that workflow. Storing the only copy on the relay VPS does not protect against losing that server.

For upgrades, read the release notes, take a recoverable backup, update both image tags together and recreate the services. Pulling a pinned tag does not move you to a newer version. Use our Compose update checklist to plan verification and rollback.

Protect SSH with keys, keep the OS and RustDesk clients patched, and monitor unexpected transfer or connection activity. A matching server key is not per-person authorisation. Only grant remote access to trusted people, for devices you own or are authorised to support.

Choose a VPS region around the people using the relay

A relay adds a network path between both endpoints. Choose a location based on actual routes from the devices you support, not simply the largest advertised port speed. HYEHOST offers Cloud VPS in Wolverhampton, Ashburn and Auckland, with root access for this Docker deployment.

For UK or European users, assess Wolverhampton; for US users, assess Ashburn; for New Zealand and nearby audiences, assess Auckland. Test from the actual client networks. Geography alone does not guarantee latency, and the best location for one support team may be wrong for another.

A self-hosting VPS provides the server, not an automatically managed remote-support service. Keep responsibility for access approval, maintenance and recovery explicit before using it for customer machines.

Self-hosted RustDesk FAQ