1. Using SSH with Public Key authentication

We want our master node to have SSH access to all of the compute nodes. During the default kickstart installation a root password (orca2026) is set. Any connection attempt to one of the nodes will therefore prompt us for this password:

[root@master ~]# ssh c01
root@c01's password:
Last login: Fri Jun 13 13:31:48 2025 from 192.168.16.1
[root@c01 ~]#

Password authentication is error-prone and not secure. For cluster administration it is common and recommended to disable password authentication completely and replace it with public key authentication.

For this kind of authentication each client needs to create a key pair:

  • A public key, which you share and install on SSH-enabled servers. SSH servers use this key to verify your identity.

  • A private key, which must be kept secret and that is optionally protected by a password. This key must not be shared with anyone and must be protected. It is used to authenticate yourself with any SSH server that knows your public key.

1.1. Creating a new key pair

Key pairs are created using the ssh-keygen command on your SSH client. By default, the key created will be placed in $HOME/.ssh/ and have a name based on the used algorithm. Currently the default algorithm on AlmaLinux 10 is ED25519. Other systems might use different defaults. This creates id_ed25519 for the private key and id_ed25519.pub for the public key. Create a keypair without a password.

[root@master ~]# ssh-keygen
Generating public/private ed25519 key pair.
Enter file in which to save the key (/root/.ssh/id_ed25519):
Enter passphrase for "/root/.ssh/id_ed25519" (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_ed25519
Your public key has been saved in /root/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:uxd6UYzWdBwvW9Sj01zmJ5chK/Chwt/rsIKUoJNvrc4 root@master1
The key's randomart image is:
+--[ED25519 256]--+
|             ...o|
|         . ...o=+|
|      .   +=..BoB|
|   .   o .oo++.B+|
|  o . . S.... oo.|
| +   o   oo.     |
|  o o . .o o.    |
|  .o o ...=.     |
|  oE.   o+..     |
+----[SHA256]-----+

If you look at the permissions of these two files, you will notice that the private key is only readable by root. Only the public key is world readable. SSH will refuse to use any keys that are too liberal with their permissions.

[root@master ~]# ls -lah ~/.ssh/id_ed25519*
-rw------- 1 root root 399 May 14 15:12 /root/.ssh/id_ed25519
-rw-r--r-- 1 root root  94 May 14 15:12 /root/.ssh/id_ed25519.pub

1.2. Copying a public key to an SSH server

To authenticate using your key pair you first have to copy the public key to the SSH server you want to authenticate with.

There is two possible ways of doing this:

  1. Using the ssh-copy-id command

  2. Manually appending the contents of your public key to the user’s $HOME/.ssh/authorized_keys file

When using the ssh-copy-id command, you have to specify which key you want to install. It will then try to login to the server and usually fall back to password authentication. If your public key has not been added already, it is appended to $HOME/.ssh/authorized_keys.

[root@master ~]# ssh-copy-id -i ~/.ssh/id_ed25519 c01.hpc
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_ed25519.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@c01.hpc's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'c01.hpc'"
and check to make sure that only the key(s) you wanted were added.

After this completes successfully, you will be able to connect directly.

[root@master ~]# ssh c01
Last login: Fri Jun 13 13:32:07 2025 from 192.168.16.1
[root@c01 ~]#

If you set a password for your private key, you will be asked for this password instead. Private keys can be unlocked for longer periods of time. This avoids having to retype the password multiple times, which makes it more convenient and secure than a regular password.

1.3. Disable Password Authentication

Warning

Disabling password authentication before installing your public key is an effective way to locking yourself out of your own system. :-)

Note

Do this part only on the compute nodes!

Open /etc/ssh/sshd_config in the compute nodes to modify the available authentication methods. Set PasswordAuthentication to no.

PasswordAuthentication no

Double check if you are able to authenticate with your public key. Then reload the sshd service and try again. It is recommended to have at least one session open while testing.

[root@master ~]# systemctl reload sshd

1.4. Disable Root Login

Note

Do this part only on the compute nodes!

To further lock down SSH servers root logins are often completely disabled or limited to a specific subnet. Before attempting this, please make sure to install your root public key on all compute nodes.

Edit /etc/ssh/sshd_config in the compute nodes and set PermitRootLogin to no globally. Afterwards you can modify this rule for a particular subnet by using a Match block.

# global setting
PermitRootLogin no

# locally allow root logins with public key from internal SSH subnet
# !!! this has to be at the end of the file !!!
Match Address 192.168.16.0/20
     PermitRootLogin without-password

Finally reload the configuration.

[root@c01 ~]# systemctl reload sshd

1.4.1. Moving around the cluster as root without password

One final piece of SSH configuration that is very useful in practice is being able to freely move as root between nodes. With a single SSH key pair this would require your root private key to be copied onto each compute node.

This, however, would create the situation that one compromised system could expose more systems outside of the cluster that might be reachable using that particular key. To contain the damage within the cluster, it is useful to create a separate key pair just for accessing cluster nodes internally as root.

For this create a second key pair without a password using ssh-keygen and name the file different, such as id_ed25519_internal. Each compute node should have a copy of these files in /root/.ssh/. Add the public key to the compute nodes’ authorized_keys file too.

Finally, to make the SSH client on each compute node aware of the second key, edit /etc/ssh/ssh_config and define Host rules for the .hpc subnet. Keys defined as IdentifyFile in the SSH configuration are used for authentication.

Note

The changes in /etc/ssh/ssh_config are needed both in master and all compute nodes.

Host *.hpc
  IdentityFile ~/.ssh/id_ed25519
  IdentityFile ~/.ssh/id_ed25519_internal
  StrictHostKeyChecking yes

Host 192.168.*
  IdentityFile ~/.ssh/id_ed25519
  IdentityFile ~/.ssh/id_ed25519_internal
  StrictHostKeyChecking yes

Host *
  IdentityFile ~/.ssh/id_ed25519
  IdentityFile ~/.ssh/id_ed25519_internal

1.4.2. Synchronizing known hosts

Everytime you connect to a host, SSH will check /etc/ssh/ssh_known_hosts and $HOME/.ssh/known_hosts to see if that server is to be trusted. If it can’t be found, SSH will ask if you trust this server. If you accept, an entry will be added to your current user’s known_hosts file. That means without a valid entry in one of these known hosts files, SSH wil always want to you to confirm. This might not seem like a problem now, but once you have a few hundred compute nodes and automated tools that require SSH, this can quickly turn into a nightmare.

What we are going to do is register all hosts in our master’s /etc/ssh/ssh_known_hosts and then synchronize this file to all compute nodes. This way all nodes know who to trust.

To generate this file, use ssh-keyscan to populate the /etc/ssh/ssh_known_hosts file. Note that we add both short and full host name, as well as the IP. This way no matter how we enter the host the host key is known.

# add host keys to known hosts file
[root@master ~]# ssh-keyscan -H -t ecdsa c01 >> /etc/ssh/ssh_known_hosts
[root@master ~]# ssh-keyscan -H -t ecdsa c01.hpc >> /etc/ssh/ssh_known_hosts
[root@master ~]# ssh-keyscan -H -t ecdsa 192.168.17.1 >> /etc/ssh/ssh_known_hosts

Should a key already exist or conflict for some reason, it is recommended to first remove them. This can be done with the ssh-keygen command.

# remove host keys from file
[root@master ~]# ssh-keygen -R c01 -f /etc/ssh/ssh_known_hosts
[root@master ~]# ssh-keygen -R c01.hpc -f /etc/ssh/ssh_known_hosts
[root@master ~]# ssh-keygen -R 192.168.17.1 -f /etc/ssh/ssh_known_hosts

1.4.3. Keeping keys consistent and synchronizing known hosts

During installation, the Rocky Linux installer generates a unique host key pair for the system. That means that after every reinstall the host key you have stored and use to validate the machine identity will have changed. The SSH client will warn you and not connect to this potentially malicious server. In other words, our /etc/ssh/ssh_known_hosts becomes invalid.

Instead of patching up our known_hosts, we instead keep a copy of all host keys of our cluster on master. If we are forced to reinstall a compute node, we simply copy the correct files back after installation.

  1. Create a script which copies all the /etc/ssh/ssh_host_* files into a directory structure on master

  2. Create a second script which copies the SSH host keys back to a newly installed system. You will need to disable StrictHostKeyChecking for this.