Many advanced Linux users have been struggling to get rid of one quite annoying situation - the interference between multiple HSM tokens connected to one system. Think about the situation when one token is only needed for SSH authentication and the other token is only needed for S/MIME and web authentication. If each of the tokens contains only one X.509v3 certificate and its private key, things should be fine and in that particular case the command line:
ssh-add -s /usr/lib64/opensc-pkcs11.so
will connect only two of the keys to the SSH agent.
But what if the HSM token hosts four certificates or more than one HSM token is services by the same PKCS#11 library? In that case, during the SSH authentication process, the SSH agent will try the available private keys one after another until the server accepts any of the keys as the proper authentication token. If you have too many keys in the HSM token, the ssh client will be involved in too many authentication attempts, and as a result, the SSH server will terminate the user's authentication. Of course, the latter also depends on the SSH server-side configuration, but the defaults are not in favour of executing multiple key tests.
How to solve this issue?
First, let's start with brining forward the bad news - there exists no definitive resolution in all instances. A workaround is only available. The workaround is not something compex - simply attach the HSM token that contains the key used for SSH authentication before the other HSM tokens, and execute:
ssh-add -s /usr/lib64/opensc-pkcs11.so
Afterwards, you may add as many HSM tokens as you wish to the Linux system. But only the first hinted HSM token will be used to operate with the SSH agent.
Now some details and more information about the process of using HSM tokens for authentication/signing. Read the information below if you are interested in those details.
The X.509v3 certificates get installed inside the HSM protected memory slots along with their private keys. To help the application with the access to that slots, a dedicated PKCS#11 module (library) is needed. In most cases, we employ the PKCS#11 module as a low-level backend in both direction. First, we transfer each X.509v3 certificate along with its private key to the protected memory inside the HMS token, reading that information from PKCS#12 files. Once the keys are stored inside the protected memory of the HSM token, we employ the PKCS#11 module to point the applications to the location of the installed private keys. Note that the applications never obtain direct access to the private keys. Instead, they send some computed hash to the crypto processor inside the HSM token, asking it to sign (encrypt) that hash with a private key with a certain ID (PKCS#12 module communicates those IDs to the application in advance).
Storing multiple keys inside HSM tokens always depends on the token model itself. Some tokens allow the use of multiple slots for signing and authentication, while some do not. The link below shows a good example of tokens that can store multiple X.509v3 certificates in their protected memory:
https://www.cardomatic.de/c/smartcard-hsm
Those brand of tokens can be programmed by using Smart Card Shell 3.
Even though the Yubico PIV tokens are popular, they cannot store multiple X.509v3 certificates and their corresponding private keys. While it is true that those PIV devices come with four slots, each slot has a different purpose, and only one of them can be employed for authentication. Which does not mean Yubico PIV tokens are not as good as the others. They just cannot store more than one certificate/key per dedicated slot. You may try to examine that by running the YubiKey Manager (Ykman) - it shows the four slots and reveals their purpose.
Regardless of the HSM management application, installation of X.509v3 certificates into the HSM tokens does not bring therein the corresponding certificate chains (even if those chains are stored in the original PKCS#12 blocks). The S/MIME client or browser that uses HSM tokens for signing or authentication is responsible for bringing the missing chain certificates into the signing or authentication process. Certificates included in chains are usually installed into special local files accessible and manageable by processing software cryptographic storage. Different programs rely on different file backends for saving and accessing the certificates needed to finish the chains when asked. For instance, Firefox, regardless of the OS environment, always employs Mozilla's NSS Shared DB files for that purpose. While running on Linux, Brave and Chrome for Linux also rely on Mozilla's NSS Shared DB. However, in the event that they are running on Windows, the Windows Certificate Store is utilized as a means of searching for certificates to complete the chain.
If you possess an HSM token that contains more than three X.509v3 certificates installed and accessible through PKCS#11 library, it is not recommended to use that token for SSH authentication. That advice is based on the way the ssh client utilizes the private keys during the authentication process.
What is the good practice to follow when preparing an HSM token, which will be used for SSH authentication only? Store only one certificate, which should be self-signed one. That self-signed X.509v3 certificate must be based on the originally generated SSH key pair stored under the ~/.ssh folder, and should be created and then inserted into the HSM token based on a PKCS#12 file. To create the self-signed certificate, simply run the command line:
openssl req -sha384 -x509 -key ~/.ssh/id_ecdsa -out ~/vesso@ssh-ecdsa.crt -subj "/CN=vesso@ssh-ecdsa" -days 3650
The self-signed certificate will be stored in ~/vesso@ssh-ecdsa.crt. Then compiose the PKCS#12 file:
openssl pkcs12 -export -legacy -key ~/.ssh/id_ecdsa -in ~/vesso@ssh-ecdsa.crt -name "vesso@ssh-ecdsa" -out vesso@ssh-ecdsa.p12
If you want to store that certificate and its private key onto your HSM key, use the appropriate device manager to import the content of the PKCS#12 file into the protected memory of the HSM token. If that is one of the HSM tokens mentioned above, adopt Smart Card Shell 3 for the import. In case you would like to use Yubico PIV token, then rely on the Yukman.
