This note describes what a DKEK is, how DKEK shares work for key management on SmartCard-HSM class devices, and how that relates to the Smart Card Shell (scsh) distribution. It is aimed at operators and integrators; it does not replace manufacturer documentation or a formal security analysis. The SmartCard-HSM site is https://www.smartcard-hsm.com/. Smart Card Shell 3 is distributed from https://www.openscdp.org/scsh3/download.html.
Terminology
On SmartCard-HSM devices, DKEK usually denotes Device Key Encryption Key: a 256-bit symmetric key held inside the device that acts as a key-encryption key (KEK) for wrapping sensitive key material when using backup, restore, or migration features.
The same three-letter acronym can appear elsewhere in cryptography with different meanings (for example some standards use KEK hierarchies without this exact name). In this document, DKEK refers to the SmartCard-HSM mechanism unless stated otherwise.
Why a DKEK exists
Many HSMs and smart-card HSMs never release private keys in plaintext. To move a key between devices, or to keep an offline backup, the implementation encrypts the key under a KEK that exists on the card. On SmartCard-HSM, that KEK is the DKEK.
Typical properties:
Keys generated on the card can be exported only if a DKEK key domain was configured during initialisation (or equivalent setup), depending on device options and policy.
If the device is re-initialised and the DKEK is wiped, encrypted backups that depended on the old DKEK cannot be recovered.
The DKEK is not intended to be held as a single secret by one person alone; the design allows splitting the material that builds the DKEK into shares.
What a DKEK share is
A DKEK share is a random 256-bit (32-byte) value. During setup the device is configured to expect a chosen number of shares. Each share is generated inside the device (using the card RNG), then passed to a custodian who stores it, usually as a password-encrypted file or using other formats such as a printable encoding.
The final DKEK is assembled by combining the shares with exclusive-or (XOR): each share is XORed into a running 256-bit value until all shares are imported. Published technical descriptions for SmartCard-HSM state that multiple shares are XORed into a single KEK value inside the key domain.
Important distinction: this is not Shamir secret sharing. With plain XOR composition, you normally need every share to reconstruct the DKEK. Published material also describes optional threshold schemes applied to the password that protects a share file (n-of-m password fragments), which is a separate layer from the XOR combination of random shares.
Roles in key management
Key domain: a logical grouping for keys and for the KEK material used to wrap them. Newer devices support multiple key domains; older or simpler profiles may expose a single DKEK domain.
Custodians: individuals who receive and protect distinct DKEK shares. Organisational policy decides how many shares to use (common examples in public guides are one to three shares).
Key check value (KCV): a short fingerprint used to confirm that the assembled DKEK on a device matches expectations before you rely on backup or import operations.
Off-card handling of shares (files and passwords)
Shares leave the device only in forms that depend on custodian-chosen secrets:
Password-based encryption of the share (for example .pbe files in command-line workflows).
Optional password threshold splitting, where several people hold fragments of the password needed to decrypt one share.
Alternative encodings (for example human-entered formats) aimed at offline storage.
The security of an encrypted share file depends on password strength, resistance to offline guessing, and physical protection of the storage medium.
In the Key Manager sources shipped with Smart Card Shell 3 (under keymanager/ in the unpacked distribution), the Key Manager loads PKCS#12 (.p12) files with BouncyCastle (KeyStore("BC", "PKCS12", ...)). The PKCS#12 password only decrypts the container on the host; it is not the DKEK. The shell then has to move private keys into the SmartCard-HSM without sending them in plaintext over the card interface. That step uses the same DKEK-based wrap format as other tooling: the host builds a key blob with DKEK.encodeKey(privateKey, publicKeyFromCertificate), and HSMKeyStore.importRSAKey / importECCKey call unwrapKey on the device so the key material is loaded into protected storage and PKCS#15-style metadata is written. Certificates are stored separately (storeEndEntityCertificate for end-entity certs, storeCACertificate when the P12 entry has no usable private key).
The excerpts below cite concrete paths and line numbers from one unpacked tree; after installation your top-level directory name typically matches the release you downloaded (for example scsh-3.x.y), and line numbers may shift in other releases.
Two implementations exist side by side; both hinge on assembling the same 256-bit DKEK on the host as the card holds when unwrapKey runs.
Shares entered on the host (classic “Import from PKCS#12” in keymanager.js)
The handler importPKCS12 asks how many DKEK shares to apply (often one). For each share, the operator supplies material through the same paths as ordinary DKEK share import (password-protected file, n-of-m password fragments, or PaperKey). Each 32-byte share is XORed into the host DKEK instance. After all shares are combined, the tool opens the PKCS#12 file, extracts the private key and certificate, forms the wrapped blob, and calls importRSAKey or importECCKey followed by certificate storage.
Share assembly and wrapping:
KeyManager.prototype.importPKCS12 = function(node) {
var str = Dialog.prompt(KeyManager.DKEK_NO_OF_SHARES, "1");
if (str == null) {
return;
}
var shares = parseInt(str);
var dkek = new DKEK(this.crypto);
for (var cnt = 0; cnt < shares; cnt++) {
var dkekshare = this.inputDKEKShare();
dkek.importDKEKShare(dkekshare);
dkekshare.clear();
}
if (key != null) {
print("Importing key and certificate...");
var pubkey = cert.getPublicKey();
var blob = dkek.encodeKey(key, pubkey);
if (pubkey.getComponent(Key.MODULUS)) {
hkey = this.ks.importRSAKey(alias, blob, pubkey.getSize());
var signalgo = Crypto.RSA_PSS_SHA256;
} else {
hkey = this.ks.importECCKey(alias, blob, pubkey.getSize());
var signalgo = Crypto.ECDSA_SHA256;
}
this.ks.storeEndEntityCertificate(alias, cert);
// Test import
var msg = new ByteString("Hello World", ASCII);
var signature = hkey.sign(signalgo, msg);
assert(this.crypto.verify(pubkey, signalgo, msg, signature), "Signature verification of imported key failed");
print("Import completed");
} else {
print("Importing certificate...");
this.ks.storeCACertificate(alias, cert);
}
this.createOutline();
} while (aliases.length > 1 && Dialog.prompt("Import more keys ?"));
} while (Dialog.prompt("Import more PKCS#12 files ?"));
dkek.clear();
For this path to succeed, the SmartCard-HSM must already contain the same DKEK that those shares define. In practice that means the device was initialised for that key domain and all required shares were imported onto the card earlier; the Key Manager session only recombines the shares on the PC so it can encrypt the import blob consistently with the firmware’s unwrap. If the host DKEK and the card DKEK differ, unwrap fails or the import is rejected.
Transient key domain for PKCS#12 (220-importp12-plugin.js)
The plug-in “Import from PKCS#12” uses a different pattern when an empty key domain slot is available: it creates a fresh DKEK key domain that expects a single share, generates one random 32-byte share on the host, imports that share into the card, XORs it in the host DKEK object, performs the same encodeKey / importRSAKey or importECCKey / certificate store sequence, then clears the host state and calls deleteKEK on that key domain identifier.
Key-domain creation, share import, and teardown:
var kdid = -1;
do {
kdid++;
var kd = sc.queryKeyDomainStatus(kdid);
if ((kd.sw == 0x6A86) || (kd.sw == 0x6D00)) {
Dialog.prompt("No empty key domain found.");
return
}
} while (kd.sw != 0x6A88);
// Create DKEK domain with random DKEK
sc.createDKEKKeyDomain(kdid, 1);
var share = crypto.generateRandom(32);
sc.importKeyShare(kdid, share);
// Create DKEK encoder and import share
var dkek = new DKEK(crypto);
dkek.importDKEKShare(share);
} while (Dialog.prompt("Import more PKCS#12 files ?"));
dkek.clear();
sc.deleteKEK(kdid);
Here the share is created for a one-off import: the card and host agree on a temporary DKEK solely to wrap the PKCS#12 private key for unwrapKey. The plug-in then deletes the key-encryption key in that domain (deleteKEK); the imported asymmetric keys remain in the device’s normal key store, while the short-lived DKEK used for the transfer is not left in place for backup or migration unless you also use a longer-lived key-domain configuration elsewhere.
How many shares appear in practice
One share is the minimal case: a single random or custodian-held 32-byte value XORed into the DKEK.
Multiple shares mean the loop runs several times on the host (importDKEKShare for each); the card must have received the same number of shares during its own setup so the XOR result matches.
Operational distinction worth preserving
PKCS#12 password: unwraps the file on the workstation only; choose it independently of DKEK share passwords.
DKEK shares: define the KEK the card uses to accept wrapped private-key blobs; organisational controls (files, PaperKey, n-of-m on a share password) apply here, not to the PKCS#12 file’s own password.
Relationship to the Smart Card Shell installer
Smart Card Shell 3 is distributed as an IzPack-based, self-contained JAR (alongside zip archives) from OpenSCDP / CardContact. The downloadable artefact name follows the release, for example scsh-3.x.y-installer.jar. Current installers and archives are published at https://www.openscdp.org/scsh3/download.html. The JAR packs a large core payload (resources/packs/pack-Core) that contains JavaScript modules for card tooling.
Among those modules is a DKEK helper used with SmartCard-HSM:
Module path in the bundle: scsh/sc-hsm/DKEK (see require("scsh/sc-hsm/DKEK") inside the SmartCard-HSM support code).
It implements host-side operations that correspond to command-line tools for manipulating DKEK shares outside the card.
What the bundled DKEK helper does (implementation-level)
The following points summarise behaviour visible in the embedded DKEK.js source inside pack-Core. They describe the host library, not necessarily every firmware detail inside the chip.
Internal state
The helper keeps a 32-byte value initialised to zero. Importing a share XORs it into that value, matching the XOR assembly model.
Key check value
The KCV is derived as the first eight bytes of SHA-256 over the 32-byte DKEK value. This lets operators compare devices without revealing the full key.
Wrapping keys under the DKEK
From the 32-byte DKEK, two AES keys are derived with SHA-256 and distinct domain separation constants (labels 00000001 and 00000002 in hex) for encryption and integrity (CMAC-style use appears in the wrapping code paths). This matches the documented pattern that the DKEK seeds separate encryption and integrity keys for protected blobs.
Encrypting a share for storage
A random eight-byte salt is generated.
A key and IV are produced from the salt and password using a stretching routine based on repeated MD5 processing (the code attempts a single-call MD5 with a very large iteration count, with a slower fallback loop).
The plaintext share is padded and encrypted with AES-CBC.
The on-disk form begins with the ASCII prefix Salted__ followed by the salt and ciphertext, similar in spirit to common OpenSSL password-based encryption envelopes.
Card protocol surface in the same module family
The SmartCard-HSM host wrapper issues secure-message APDUs with instruction 0x52 and function variants such as creating a DKEK key domain (P1 = 0x01) and importing a share (P1 = 0x00). Wrap and unwrap operations use other implementation-specific INS codes in the same source. Exact allowable parameters depend on firmware and initialisation state.
Practical guidance (non-exhaustive)
Decide how many shares you need based on organisational control, not on convenience alone. More shares increase operational friction; fewer shares increase concentration of risk.
Treat encrypted share files as highly sensitive; offline guessing attacks are a real concern if files leak.
Record KCVs and key-domain identifiers as part of your key-management records so restores can be validated.
Distinguish DKEK key domains from XKEK-style domains on devices that support both: XKEK domains use different agreement mechanisms (documented authenticated ECDH-based setups), whereas DKEK domains are built from imported shares as described here.
References and further reading
SmartCard-HSM: technical articles and documentation on DKEK shares, key domains, and recovery — https://www.smartcard-hsm.com/
SmartCard-HSM hardware (tokens, cards, and related form factors that can be used with Smart Card Shell) is available from several suppliers; one catalogue is https://www.cardomatic.de/en/c/smartcard-hsm.
Firmware evolves; APDU layouts, maximum numbers of shares, and domain features depend on device generation and version. Always verify behaviour against the documentation for your exact device and software stack. This document does not analyse side-channel resistance, PKCS#11 policy, or organisational compliance requirements.
Random window resizing on MSI MS-158K laptops: a Rocky Linux fix
1. Problem
On MSI gaming laptops built on the MS-158K motherboard and running Rocky Linux 9 or 10, all open windows shrink briefly at random intervals and then snap back to their original size. There is no obvious trigger and the timing is irregular. The board carries two AMD GPUs: a Radeon Vega (Cezanne) integrated GPU at 0000:07:00.0 which drives the built-in screen, and a discrete Radeon RX 5500M (Navi 14) at 0000:03:00.0 which has no connected display output. The only trace left in the kernel log is:
[19809.613998] [drm] kiq ring mec 2 pipe 1 q 0
[19809.616987] amdgpu 0000:03:00.0: [drm] Cannot find any crtc or sizes
[19809.617003] amdgpu 0000:03:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
... (full ring reinitialization output)
[19809.620157] amdgpu 0000:03:00.0: [drm] Cannot find any crtc or sizes
2. Diagnosis
Step 1: Identify which GPU owns the display
The MS-158K board carries two AMD GPUs: a Radeon RX 5500M discrete GPU and a Radeon Vega integrated GPU. Check which DRM card has a connected output:
for card in /sys/class/drm/card*/; do
echo "$card: $(cat ${card}*/status 2>/dev/null | sort -u)"
done
card0 is the Radeon RX 5500M with no connected output. card1 is the Radeon Vega driving the laptop's built-in screen.
Step 2: Correlate the log pattern
Expanding the dmesg context around each occurrence of the log message reveals a consistent pattern appearing immediately before every window disruption:
sudo dmesg | grep -B 20 "kiq ring mec"
Every occurrence follows this sequence:
[ 55.515940] pci_bus 0000:03: Allocating resources ← dGPU waking from suspend
[ 55.515971] amdgpu 0000:03:00.0: amdgpu: PSP is resuming...
[ 55.753346] amdgpu 0000:03:00.0: amdgpu: SMU is resumed successfully!
[ 55.754555] [drm] kiq ring mec 2 pipe 1 q 0 ← full GPU reinit
[ 55.757627] amdgpu 0000:03:00.0: [drm] Cannot find any crtc or sizes
...
The line pci_bus 0000:03: Allocating resources appears immediately before every resume cycle, confirming that the kernel PCI bus is repeatedly reallocating resources as the dGPU wakes from the D3 (powered-off) power state.
This confirms the dGPU is under auto runtime power management, causing it to cycle between suspended and active states repeatedly.
3. Root cause
The Radeon RX 5500M at PCI address 0000:03:00.0 is repeatedly suspended and resumed by the kernel's PCI runtime power management. Each time it resumes, the kernel fires a DRM hotplug event. The Wayland compositor (GNOME Shell) reacts to that hotplug event by briefly reconfiguring the display state — which causes all windows to shrink momentarily and then restore once the compositor determines that no display change has actually taken place.
The Cannot find any crtc or sizes message is a consequence, not the cause: the Radeon RX 5500M has no connected display outputs on the MS-158K, so every time the driver queries for CRTCs after resuming, it finds none.
4. Solution
Stop the Radeon RX 5500M from entering runtime suspend by setting its power control to on.
If the window shrinking stops after running this command, the diagnosis is confirmed.
Step 2: Make the fix permanent via udev
A udev rule ensures the setting is applied automatically on every boot. Using the vendor ID (0x1002, AMD) and device ID (0x7340, Navi 14) is more reliable than matching on the PCI address alone, as the address can vary between kernels and firmware revisions:
No further output should appear once the fix is in place.
5. Notes
This issue has been observed on MSI gaming laptops based on the MS-158K motherboard running Rocky Linux 9 and 10. The board pairs a Radeon Vega (Cezanne) integrated GPU with a Radeon RX 5500M (Navi 14) discrete GPU. The discrete GPU has no connected display output and is left under automatic PCI power management by default, which triggers the problem.
The udev rule matches on the AMD vendor ID (0x1002) and the Navi 14 device ID (0x7340). This is more dependable than matching on the PCI address 0000:03:00.0, which could in principle shift between firmware updates. To find the correct IDs on another machine, run lspci -nn | grep -i vga and read the [vendor:device] values from the output.
Setting power control to on stops the Radeon RX 5500M from sleeping, which raises idle power consumption slightly and may reduce battery life a little. If this is a concern and the GPU is never used for rendering, blacklisting the amdgpu module for that device is an alternative — though this requires some care to avoid affecting the Radeon Vega.
If tlp or power-profiles-daemon is installed, it may override the udev rule after boot. Check with systemctl status tlp power-profiles-daemon. Should overriding occur, a systemd service with After=tlp.service (or the relevant daemon) can re-apply the setting, following the same approach described in the Bluetooth dongle fix post.
The symptom — all windows shrinking briefly then snapping back — is easy to mistake for a window manager fault, a compositor bug, or a monitoring tool accidentally resizing windows. The key diagnostic clue is the pci_bus: Allocating resources line in dmesg appearing immediately before each window disruption.
A practical guide to fixing Bluetooth mouse freezes caused by USB power management on Linux
1. Problem
A Bluetooth mouse connected via a USB dongle was freezing for 1-2 seconds intermittently. The cause was Linux USB power management suspending the Bluetooth dongle to save power.
2. Solution
Disable USB autosuspend for the Bluetooth dongle via a udev rule.
Step 1: Identify the Bluetooth Dongle
lsusb | grep -i bluetooth
In our case:
Bus 002 Device 006: ID 0bda:a728 Realtek Semiconductor Corp. Bluetooth Radio
Step 2: Verify udev Attribute Names
udevadm info -a /sys/bus/usb/devices/2-13/ | grep -E "idVendor|idProduct"
This confirmed the device uses ATTR (not ATTRS) for its own attributes.
for dev in /sys/bus/usb/devices/*/; do
name=$(cat "$dev/product" 2>/dev/null || echo "unknown")
control=$(cat "$dev/power/control" 2>/dev/null)
echo "$name control=$control"
done | grep -i bluetooth
Expected output:
Bluetooth Radio control=on
3. Troubleshooting: udev Rule Overridden by Power Manager
If the mouse continues to freeze after applying the udev rule, a power management service may be resetting the USB power control back to auto. Check for running power management daemons:
systemctl status tlp
systemctl status power-profiles-daemon
systemctl status tuned
In our case power-profiles-daemon was active and overriding the udev rule. The fix is to create a systemd service that enforces the setting after the daemon starts:
cat > /etc/systemd/system/usb-bluetooth-power.service << 'EOF'
[Unit]
Description=Disable USB autosuspend for Bluetooth dongle
After=power-profiles-daemon.service
[Service]
Type=oneshot
ExecStart=/bin/bash -c 'echo on > /sys/bus/usb/devices/2-13/power/control'
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
EOF
systemctl enable --now usb-bluetooth-power.service
Verify the service is active and the setting is correct:
systemctl status usb-bluetooth-power.service
cat /sys/bus/usb/devices/2-13/power/control
Expected output: service active (exited) and control=on.
4. Further Fix: Enforcing the Setting with a systemd Timer
If the mouse continues to freeze, power-profiles-daemon is periodically resetting the power control back to auto after boot. A oneshot service only runs once and cannot counter this. The solution is a systemd timer that re-enforces the setting every 30 seconds:
cat > /etc/systemd/system/usb-bluetooth-power.service << 'EOF'
[Unit]
Description=Disable USB autosuspend for Bluetooth dongle
[Service]
Type=oneshot
ExecStart=/bin/bash -c 'echo on > /sys/bus/usb/devices/2-13/power/control'
EOF
cat > /etc/systemd/system/usb-bluetooth-power.timer << 'EOF'
[Unit]
Description=Enforce USB autosuspend off for Bluetooth dongle
[Timer]
OnBootSec=10
OnUnitActiveSec=30s
[Install]
WantedBy=timers.target
EOF
systemctl daemon-reload
systemctl enable --now usb-bluetooth-power.timer
Verify the timer is running:
systemctl status usb-bluetooth-power.timer
5. Notes
Always verify the correct udev attribute style with udevadm info -a first — using ATTRS instead of ATTR is a common mistake that prevents the rule matching.
Power management daemons such as power-profiles-daemon and tlp can override udev rules — use the systemd service workaround if this occurs.
The device path /sys/bus/usb/devices/2-13/ may differ on your system — verify it with the vendor/product ID loop shown in the verification step above.
The same approach applies to any USB device suffering from autosuspend issues.
A comprehensive guide to partition-level encryption for maximum security
Abstract
This document describes the implementation of Linux Unified Key Setup (LUKS) encryption on software RAID arrays managed by Logical Volume Manager 2 (LVM2). The approach involves encrypting RAID partitions at the block device level, which provides superior security compared to encrypting logical volumes individually. By encrypting the underlying RAID partitions, the entire LVM2 structure including volume group metadata, physical extent allocation, and logical volume layout remains hidden from potential attackers. This method eliminates information leakage that could occur when encrypting only logical volumes, where LVM2 metadata and structure remain visible on unencrypted RAID arrays. The document covers implementation procedures, security considerations, performance implications, and operational best practices based on production deployment experience.
Encryption of storage systems is essential for protecting sensitive data in enterprise and high-performance computing environments. When implementing encryption on systems utilising software RAID arrays with LVM2 volume management, the choice of encryption layer significantly impacts both security posture and operational complexity.
LUKS provides a standardised on-disk format for encrypted block devices, offering multiple key slots, key derivation functions, and metadata management. The fundamental question in layered storage architectures is where to apply encryption: at the RAID partition level or at the logical volume level within LVM2.
This document demonstrates that encrypting RAID partitions before LVM2 configuration provides superior security by completely hiding the volume management structure. In contrast, encrypting logical volumes individually leaves LVM2 metadata, physical extent allocation patterns, and volume group information exposed on the underlying RAID arrays, potentially providing attack vectors through structure analysis.
This section compares three approaches to implementing LUKS encryption in systems utilising software RAID and LVM2 volume management. Each approach provides different levels of security and operational complexity.
2.1 Logical Volume-Level Encryption
When LUKS encryption is applied to individual logical volumes within an LVM2-managed system, the encryption layer sits above the LVM2 abstraction. This configuration exposes significant structural information that could aid attackers through multiple channels. Volume group metadata stored on RAID arrays remains unencrypted, allowing attackers to analyse physical extent allocation patterns, logical volume names and sizes, volume group structure and organisation, and physical volume relationships. The extent size and allocation strategies are visible, as is snapshot metadata and relationships between logical volumes.
Attackers can leverage this exposed information through several attack vectors. Structure analysis can reveal the logical volume count and approximate sizes, providing insight into the storage organisation. Allocation patterns may indicate data distribution and usage patterns, potentially revealing which volumes contain more critical or frequently accessed data. Metadata analysis could reveal volume naming conventions and organisational structure, which might expose the purpose or sensitivity of different logical volumes. Physical extent mapping provides insight into data layout across the storage system, whilst snapshot metadata exposes backup and versioning strategies, potentially revealing information about data retention policies and backup frequencies.
Encrypting RAID partitions before LVM2 configuration hides the volume management structure. Software RAID arrays created with mdadm are block devices that can be encrypted directly, ensuring that all LVM2 metadata becomes encrypted and inaccessible to attackers. Physical extent allocation becomes invisible, logical volume structure is completely obscured, and volume group organisation remains hidden. This approach eliminates structural information leakage through metadata analysis, as attackers cannot examine the internal structure without first decrypting the RAID partitions.
The security advantages of this approach are substantial. Complete structure obfuscation prevents attack vector identification, as attackers cannot determine how the storage is organised internally. No information leakage occurs about logical volume count or sizes, making it impossible to infer the storage architecture. Allocation patterns remain encrypted and unanalysable, volume naming and organisational structure remain hidden, and snapshot and backup strategies cannot be inferred from metadata analysis.
However, certain considerations apply to this approach. RAID metadata remains visible at the physical device level through the mdadm superblock, which means an attacker can identify that RAID arrays exist, but cannot determine the LVM2 structure within those arrays. The RAID array structure, including the RAID level and device count, may be inferable from physical device analysis, but the internal LVM2 organisation remains completely hidden.
Applying LUKS encryption directly to physical device partitions before creating LVM2 physical volumes provides the highest level of structure obfuscation. Since software RAID arrays are themselves partitions with metadata filesystems, we can apply the same principle to LVM2 physical devices at the partition level:
This approach provides the highest level of security advantages. There is no indication that LVM2 infrastructure exists in the partitions, as physical device analysis reveals only encrypted partitions with no visible structure. If RAID is not being used, no RAID metadata is visible either. The result is complete obfuscation of volume management structure, with no way to determine that LVM2 is in use without first decrypting the partitions.
When comparing this approach with RAID partition-level encryption, the key difference is that LUKS on RAID partitions hides LVM2 structure but RAID structure may be detectable through mdadm superblocks on physical devices. In contrast, LUKS on LVM2 physical device partitions hides both LVM2 and any underlying structure completely, providing maximum obfuscation.
The choice between these approaches depends on specific requirements. Use RAID partition-level encryption when software RAID redundancy is required, when RAID structure visibility is acceptable (since RAID metadata exists on physical devices), when you need to hide LVM2 structure from attackers, and when operational simplicity with RAID management is preferred. Use LVM2 physical device partition-level encryption when maximum structure obfuscation is required, when no indication of volume management should be visible, when RAID is not required or when RAID structure should also be hidden, and when you are willing to manage encryption on individual partitions rather than RAID arrays.
2.4 Combined Approach: LUKS on RAID Partitions with LVM2
For systems requiring both RAID redundancy and maximum security, the recommended approach is LUKS on RAID partitions:
This approach provides RAID redundancy for data protection whilst maintaining complete LVM2 structure obfuscation. It offers operational flexibility with LVM2 management capabilities and represents a balanced approach between security and functionality, making it suitable for most production environments requiring both redundancy and encryption.
3. Implementation Procedure: LUKS on Physical Device Partitions
This section describes implementing LUKS encryption directly on physical device partitions before LVM2 configuration. This approach provides maximum structure obfuscation by ensuring no indication of LVM2 infrastructure exists in the encrypted partitions.
3.1 Understanding the Approach
When applying LUKS encryption to physical device partitions before LVM2 configuration, the LVM2 physical volumes are created on the decrypted device mapper devices. This ensures that physical device analysis reveals only encrypted partitions with no visible structure. No LVM2 metadata is visible without decryption, there is no indication that volume management structure exists, and the result is complete obfuscation of storage organisation.
Important Consideration: Boot Automation
The /boot partition cannot be encrypted because the bootloader and kernel must be accessible before encryption can be unlocked. This creates a challenge for automating LVM2 activation and filesystem mounting after LUKS devices are unlocked. Automation scripts and configuration must be stored in /boot, which remains unencrypted.
3.2 Prerequisites
System Requirements:
The system must run a Linux distribution with LUKS support, such as RHEL 9/10, Rocky Linux 9/10, or equivalent distributions. A CPU with memory encryption support (Intel TME or AMD SME) is mandatory, as LUKS keys stored in memory without encryption are vulnerable to cold boot attacks. Memory encryption must be enabled in BIOS/UEFI configuration and verified in the operating system. Physical device partitions must be prepared and not yet used for LVM2 configuration. A separate /boot partition is required and cannot be encrypted, as the bootloader and kernel must be accessible before encryption can be unlocked. Sufficient disk space must be available for LUKS metadata overhead, which typically requires 16-32 MB per encrypted device.
LUKS encryption keys are stored in system memory when devices are unlocked. Without CPU-based memory encryption (Intel TME or AMD SME), LUKS keys can be extracted through cold boot attacks. Do not implement LUKS encryption on systems without CPU memory encryption support.
3.3 Step-by-Step Implementation
Step 1: Prepare Physical Device Partitions
Ensure physical device partitions are created and ready for encryption:
# Display partition information
lsblk
parted /dev/sda print
# Verify partitions are not in use
mount | grep sda
Step 2: Create LUKS Encrypted Devices on Partitions
Create LUKS encrypted devices on each physical partition. This process will format the partitions, so ensure all data is backed up:
# Create LUKS encrypted device on first partition
cryptsetup luksFormat --type luks2 --cipher aes-xts-plain64 --key-size 512 \
--hash sha512 --iter-time 5000 --pbkdf argon2id /dev/sda1
# Create LUKS encrypted device on second partition
cryptsetup luksFormat --type luks2 --cipher aes-xts-plain64 --key-size 512 \
--hash sha512 --iter-time 5000 --pbkdf argon2id /dev/sdb1
# Repeat for all partitions in the configuration
Parameter Explanation:
--type luks2: Use LUKS2 format (recommended for new installations)
--cipher aes-xts-plain64: AES encryption in XTS mode (recommended for block devices)
--key-size 512: 512-bit key size (256-bit per data and tweak key)
--hash sha512: SHA-512 hash algorithm for key derivation
--iter-time 5000: 5 seconds for key derivation (adjust based on security requirements)
--pbkdf argon2id: Argon2id key derivation function (recommended)
Step 3: Open LUKS Encrypted Devices
Open the encrypted devices to create device mapper entries:
# Open encrypted device (will prompt for passphrase)
cryptsetup open /dev/sda1 sda1_crypt
cryptsetup open /dev/sdb1 sdb1_crypt
# Verify devices are available
ls -l /dev/mapper/sda1_crypt /dev/mapper/sdb1_crypt
Step 4: Create LVM2 Physical Volumes on Encrypted Devices
Create physical volumes on the decrypted device mapper devices:
Step 8: Configure Boot Automation for LVM2 Activation
Critical Challenge: /boot Partition Cannot Be Encrypted
The /boot partition must remain unencrypted because the bootloader and kernel must be accessible before encryption can be unlocked. This means that automation scripts and configuration for LVM2 activation after LUKS unlock must be stored in /boot, which is unencrypted.
Systemd can automatically activate LVM2 volumes after LUKS devices are unlocked using systemd services:
# Enable LVM2 monitoring service
systemctl enable lvm2-monitor.service
# The service automatically scans for LVM2 volumes after encrypted devices are available
# No additional configuration needed if using standard systemd integration
Option 2: Custom systemd Service for LVM2 Activation
Create a custom systemd service that activates LVM2 after LUKS unlock:
# Create service file
cat > /etc/systemd/system/lvm2-activate.service << 'EOF'
[Unit]
Description=Activate LVM2 volumes after LUKS unlock
After=systemd-cryptsetup@.service
Requires=systemd-cryptsetup@.service
[Service]
Type=oneshot
ExecStart=/usr/sbin/vgchange -ay
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
EOF
# Enable the service
systemctl enable lvm2-activate.service
Option 3: Initramfs Hook for Early Activation
For systems requiring LVM2 activation during early boot (before root filesystem mount):
# Create initramfs hook script
cat > /etc/dracut.conf.d/90-lvm2-activate.conf << 'EOF'
# Ensure LVM2 tools are included in initramfs
add_drivers+=" dm-mod "
EOF
# Update initramfs
dracut --force
Security Implications of Boot Automation:
The unencrypted nature of the /boot partition creates security implications for boot automation. Automation scripts in /boot are unencrypted and visible to anyone with access to the partition, which means scripts may reveal LVM2 volume names and structure. To minimise information leakage, consider using generic volume names that do not reveal the purpose or content of logical volumes. Scripts should not contain sensitive information or passphrases, as they are stored in an unencrypted location. Keyfiles should be stored separately from automation scripts, ideally on encrypted devices or secure key management systems, to prevent key exposure if the /boot partition is accessed.
Minimising Information Leakage:
# Use generic volume group and logical volume names
# Instead of: vg_production_data, lv_customer_database
# Use: vg0, lv0, lv1, etc.
# Example with generic names
vgcreate vg0 /dev/mapper/sda1_crypt /dev/mapper/sdb1_crypt
lvcreate -L 100G -n lv0 vg0
lvcreate -L 50G -n lv1 vg0
Step 9: Configure Filesystem Mounting
Configure automatic filesystem mounting in /etc/fstab:
Update the initial RAM filesystem to include LUKS and LVM2 support:
# Update initramfs (RHEL/Rocky Linux)
dracut --force
# Or for systems using mkinitcpio
mkinitcpio -P
Step 11: Verify Configuration
Reboot the system and verify that encrypted devices unlock automatically and filesystems mount correctly:
# After reboot, verify encrypted devices are open
lsblk
dmsetup ls
# Verify LVM2 volumes are available
pvs
vgs
lvs
# Verify filesystems are mounted
mount | grep vg
df -h
4. Implementation Procedure: LUKS on RAID Partitions
This section describes implementing LUKS encryption on software RAID partitions before LVM2 configuration. This approach provides RAID redundancy whilst hiding the LVM2 structure.
4.1 Prerequisites
System Requirements:
Linux distribution with LUKS support (RHEL 9/10, Rocky Linux 9/10, or equivalent)
CPU with memory encryption support (Intel TME or AMD SME) - MANDATORY
Memory encryption enabled in BIOS/UEFI
Software RAID arrays created with mdadm
Sufficient disk space for LUKS metadata overhead (typically 16-32 MB per encrypted device)
LUKS encryption keys are stored in system memory when devices are unlocked. Without CPU-based memory encryption (Intel TME or AMD SME), LUKS keys can be extracted through cold boot attacks. Do not implement LUKS encryption on systems without CPU memory encryption support.
4.2 Step-by-Step Implementation
Step 1: Verify RAID Array Configuration
Before implementing encryption, verify the existing RAID array configuration:
# Display RAID array status
cat /proc/mdstat
# Display detailed RAID information
mdadm --detail /dev/md0
mdadm --detail /dev/md1
# Verify array is not in use
# Ensure no filesystems are mounted from the arrays
mount | grep md
Step 2: Create LUKS Encrypted Devices on RAID Partitions
Create LUKS encrypted devices on each RAID array partition. This process will format the devices, so ensure all data is backed up:
# Create LUKS encrypted device on first RAID array
cryptsetup luksFormat --type luks2 --cipher aes-xts-plain64 --key-size 512 \
--hash sha512 --iter-time 5000 --pbkdf argon2id /dev/md0
# Create LUKS encrypted device on second RAID array
cryptsetup luksFormat --type luks2 --cipher aes-xts-plain64 --key-size 512 \
--hash sha512 --iter-time 5000 --pbkdf argon2id /dev/md1
# Repeat for all RAID arrays in the configuration
Parameter Explanation:
--type luks2: Use LUKS2 format (recommended for new installations)
--cipher aes-xts-plain64: AES encryption in XTS mode (recommended for block devices)
--key-size 512: 512-bit key size (256-bit per data and tweak key)
--hash sha512: SHA-512 hash algorithm for key derivation
--iter-time 5000: 5 seconds for key derivation (adjust based on security requirements)
--pbkdf argon2id: Argon2id key derivation function (recommended)
Step 3: Open LUKS Encrypted Devices
Open the encrypted devices to create device mapper entries:
# Open encrypted device (will prompt for passphrase)
cryptsetup open /dev/md0 md0_crypt
cryptsetup open /dev/md1 md1_crypt
# Verify devices are available
ls -l /dev/mapper/md0_crypt /dev/mapper/md1_crypt
Step 4: Create LVM2 Physical Volumes on Encrypted Devices
Create physical volumes on the decrypted device mapper devices:
Update the initial RAM filesystem to include LUKS and LVM2 support:
# Update initramfs (RHEL/Rocky Linux)
dracut --force
# Or for systems using mkinitcpio
mkinitcpio -P
Step 10: Verify Configuration
Reboot the system and verify that encrypted devices unlock automatically and filesystems mount correctly:
# After reboot, verify encrypted devices are open
lsblk
dmsetup ls
# Verify LVM2 volumes are available
pvs
vgs
lvs
# Verify filesystems are mounted
mount | grep vg_encrypted
df -h
5. Boot Automation and `/boot` Partition Considerations
5.1 The `/boot` Partition Challenge
The /boot partition cannot be encrypted because the bootloader (GRUB2, systemd-boot, or equivalent) and kernel must be accessible before encryption can be unlocked. This creates a fundamental challenge for automating LVM2 activation and filesystem mounting after LUKS devices are unlocked.
Key Constraints:
The /boot partition presents several constraints that must be addressed. The bootloader must read the kernel and initramfs from an unencrypted /boot partition, as the bootloader cannot decrypt data before the kernel is loaded. The kernel and initramfs must contain tools to unlock LUKS devices, requiring that cryptsetup and related tools are included in the initramfs. Automation scripts for LVM2 activation must be accessible before the root filesystem is mounted, which means they must be stored in locations accessible during early boot. Configuration files such as crypttab and fstab are typically located in /etc, which may be on an encrypted root filesystem, requiring special handling during boot.
Information Leakage Risks:
The unencrypted nature of the /boot partition creates information leakage risks that must be considered. Automation scripts in /boot are unencrypted and visible to anyone with access to the partition, potentially revealing LVM2 volume names, structure, or organisation. Configuration files may expose volume group and logical volume names, providing information about the storage architecture. The boot process itself may reveal encryption setup and structure through log messages, error output, or configuration file contents that are visible during boot.
Systemd provides built-in support for automatic LVM2 activation after LUKS devices are unlocked:
# Enable LVM2 monitoring service (usually enabled by default)
systemctl enable lvm2-monitor.service
# Verify service status
systemctl status lvm2-monitor.service
# The service automatically scans for LVM2 volumes after encrypted devices are available
# No additional configuration needed if using standard systemd integration
Advantages:
No custom scripts required
Standard systemd integration
Automatic activation after LUKS unlock
Minimal information leakage (uses standard systemd services)
Solution 2: Initramfs Integration
For systems requiring LVM2 activation during early boot (before root filesystem mount), integrate LVM2 tools into initramfs:
# Ensure LVM2 tools are included in initramfs
cat > /etc/dracut.conf.d/90-lvm2.conf << 'EOF'
# Include LVM2 tools in initramfs
add_drivers+=" dm-mod "
EOF
# Update initramfs
dracut --force
# Verify LVM2 support in initramfs
lsinitrd | grep lvm
Solution 3: Custom systemd Service
Create a custom systemd service that activates LVM2 after LUKS unlock:
# Create service file
cat > /etc/systemd/system/lvm2-activate.service << 'EOF'
[Unit]
Description=Activate LVM2 volumes after LUKS unlock
After=systemd-cryptsetup@.service
Requires=systemd-cryptsetup@.service
[Service]
Type=oneshot
ExecStart=/usr/sbin/vgchange -ay
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
EOF
# Enable the service
systemctl enable lvm2-activate.service
Solution 4: Initramfs Hook Script
For maximum control, create an initramfs hook that activates LVM2 during early boot:
# Create initramfs hook
cat > /usr/lib/dracut/hooks/pre-mount/99-lvm2-activate.sh << 'EOF'
#!/bin/bash
# Activate LVM2 volumes after LUKS devices are unlocked
if command -v vgchange >/dev/null 2>&1; then
vgchange -ay
fi
EOF
chmod +x /usr/lib/dracut/hooks/pre-mount/99-lvm2-activate.sh
# Update initramfs
dracut --force
5.3 Minimising Information Leakage from `/boot`
Strategy 1: Generic Naming Conventions
Use generic, non-descriptive names for volume groups and logical volumes:
LUKS encryption keys are stored in system memory (RAM) when encrypted devices are unlocked. Without CPU-based memory encryption, these keys can be extracted through cold boot attacks, completely compromising the encryption.
Cold Boot Attack Vulnerability:
The cold boot attack vulnerability represents a critical security concern for LUKS encryption. Attackers with physical access can extract LUKS keys from memory, as memory contents persist for seconds to minutes after power loss. Keys can be extracted by rapidly rebooting the system and capturing memory contents before they decay. This vulnerability has been demonstrated through successful extraction of LUKS keys from unencrypted memory in research studies, making it a real and practical threat.
Mandatory Requirements:
To mitigate the cold boot attack vulnerability, specific hardware and configuration requirements must be met. Systems must use either Intel processors with TME (Total Memory Encryption) support or AMD processors with SME (Secure Memory Encryption) support. Memory encryption must be enabled in BIOS/UEFI configuration, and memory encryption activation must be verified in the operating system to ensure it is functioning correctly.
Verification:
# Check Intel TME status
dmesg | grep -i tme
cat /proc/cpuinfo | grep tme
# Check AMD SME status
dmesg | grep -i sme
cat /proc/cpuinfo | grep sme
# Verify memory encryption is active
dmesg | grep -i "memory encryption"
Without CPU memory encryption, LUKS encryption should not be used for sensitive data.
6.2 Key Management
Key Storage:
Keyfiles should be stored on separate encrypted devices or secure key management systems to prevent key loss if the primary storage system fails. Keyfiles must have restrictive permissions, typically 400 or 600, to prevent unauthorised access. In enterprise environments, consider using hardware security modules (HSM) for key storage, which provide tamper-resistant hardware-based key protection. For long-term deployments, implement key rotation procedures to periodically update encryption keys and maintain security over extended periods.
Key Backup:
LUKS headers should be backed up separately from encrypted data, as header corruption can prevent device unlocking even with correct keys. Header backups should be stored in secure, physically separate locations to protect against site-level disasters. Key recovery procedures must be documented in detail to ensure they can be executed correctly during recovery operations. Key recovery procedures should be tested regularly to verify their effectiveness and ensure administrators are familiar with the process.
Encrypted devices provide protection against data extraction when powered off, as the encryption layer prevents access to data without the appropriate keys. Physical access to encrypted devices does not compromise data without keys, making encryption an effective protection against theft or unauthorised physical access. However, physical security measures should still be considered for servers and storage systems to prevent tampering and ensure system integrity. Secure disposal procedures must be implemented for decommissioned devices to ensure that encryption keys are properly destroyed and devices cannot be used to access data after decommissioning.
6.4 Operational Security
Access Control:
Root access to systems with encrypted storage should be limited to authorised personnel only, as root access provides the ability to unlock encrypted devices and access all data. Role-based access control should be implemented for storage administration, ensuring that only authorised individuals can perform encryption-related operations. Access to encrypted devices and key management systems should be monitored continuously to detect unauthorised access attempts. All encryption-related operations should be logged comprehensively to provide an audit trail of who accessed encrypted storage and when.
Audit and Monitoring:
# Enable audit logging for cryptsetup operations
auditctl -w /usr/sbin/cryptsetup -p x -k cryptsetup_operations
# Monitor LUKS device status
systemctl status systemd-cryptsetup@md0_crypt.service
7. Performance Implications
7.1 Encryption Overhead
CPU Impact:
LUKS encryption adds CPU overhead for encryption and decryption operations, as all data must be processed through the encryption layer. Modern CPUs with AES-NI (Advanced Encryption Standard New Instructions) provide hardware acceleration for AES encryption operations, significantly reducing the performance impact. The performance impact typically ranges from 5-15% depending on workload characteristics, with sequential I/O operations showing lower overhead than random I/O operations due to better CPU cache utilisation and prefetching effectiveness.
I/O Performance:
Encryption adds latency to I/O operations because data must be encrypted during writes and decrypted during reads before it can be processed by the filesystem layer. Throughput may be reduced, particularly for random I/O patterns where the encryption overhead becomes more significant relative to the I/O operation size. Sequential operations benefit from CPU caching and prefetching, which can mask some of the encryption overhead. RAID arrays help distribute encryption load across multiple devices, allowing parallel encryption operations and improving overall throughput.
7.2 Performance Optimisation
Cipher Selection:
AES-XTS provides good performance with hardware acceleration through AES-NI instructions, making it the recommended choice for most deployments. Cipher performance characteristics should be considered for specific workloads, as different ciphers may perform better or worse depending on the I/O pattern and data characteristics. If performance is critical, different cipher options should be benchmarked to determine which provides the best balance between security and performance for the specific workload.
Key Derivation:
Argon2id provides strong security but higher CPU cost during unlock
Adjust --iter-time parameter based on security vs performance requirements
Longer iteration times increase security but slow device unlocking
Alignment Considerations:
Ensure proper alignment across encryption, RAID, and LVM2 layers
Misalignment can cause additional performance degradation
Follow alignment procedures from LVM2 over RAID documentation
8.1 Replacing Failed Disks with LUKS at Partition Level
Replacing a failed disk when LUKS encryption is implemented at the partition level (before LVM2) introduces significant complexity compared to unencrypted storage systems. The encryption layer must be recreated on the replacement disk, and the LVM2 physical volume must be restored or rebuilt within the encrypted device.
Complexity Factors:
Several factors contribute to the complexity of disk replacement when LUKS encryption is implemented at the partition level. The early encryption layer means that LUKS encryption occurs before the kernel discovers LVM2 structure, requiring manual intervention during boot or recovery operations. LVM2 metadata location presents another challenge, as the metadata is stored within the encrypted partition, making it inaccessible until the partition is decrypted. The volume group must recognize the replacement physical volume after decryption, which may require restoring metadata or recreating the physical volume structure. System boot dependencies add further complexity, as system boot may depend on encrypted volumes, requiring careful recovery procedures to ensure the system can start successfully. Finally, key management becomes critical, as LUKS keys must be available to unlock the replacement device, which may require secure key storage and access procedures.
Prerequisites for Disk Replacement:
Before attempting disk replacement, several prerequisites must be met. LUKS keyfile or passphrase must be available to unlock the replacement device. A backup of the LUKS header is highly recommended, as it can significantly simplify recovery if the header becomes corrupted. Administrators must have a clear understanding of which physical volume failed, which requires monitoring and diagnostic capabilities. A replacement disk of equal or larger capacity must be available and prepared. If the root filesystem is affected by the disk failure, the system must be bootable from an alternative source, such as rescue media, network boot, or an alternative boot device.
Scenario 1: Failed Disk in Non-RAID Configuration (LUKS on Physical Partitions)
Step 1: Identify the Failed Disk
# Check LVM2 physical volume status
pvs
# Example output showing failed PV:
# PV VG Fmt Attr PSize PFree
# /dev/mapper/sda1_crypt vg0 lvm2 a-- 500.00g 0
# /dev/mapper/sdb1_crypt vg0 lvm2 a-- 500.00g 0
# /dev/mapper/sdc1_crypt vg0 lvm2 a-- 500.00g missing
# Check which encrypted device corresponds to failed PV
lsblk
# Identify the underlying physical device
# If /dev/mapper/sdc1_crypt is missing, check /dev/sdc1
cryptsetup status sdc1_crypt
# Check physical device status
dmesg | grep -i sdc
smartctl -a /dev/sdc
Step 2: Prepare Replacement Disk
# Identify replacement device (e.g., /dev/sdd)
NEW_DEVICE="/dev/sdd"
# Create partition matching original configuration
# Method 1: Using sgdisk for GPT disks (recommended, most reliable)
# Backup partition table from healthy disk
sgdisk --backup=/tmp/partition_table_backup.sgdisk /dev/sda
# Restore partition table to new disk
sgdisk --load-backup=/tmp/partition_table_backup.sgdisk /dev/sdd
# Verify partition table restoration
sgdisk --verify /dev/sdd
parted /dev/sdd print
# Method 2: Using parted to dump and manually recreate
# Dump partition layout to text file for reference
parted /dev/sda unit s print > /tmp/partition_table.txt
# View the partition information
cat /tmp/partition_table.txt
# Manually recreate using parted (if sgdisk method not available)
# First, determine partition table type from original disk
parted /dev/sda print | grep "Partition Table"
# Create matching partition table type on new disk
# For GPT (most common on modern systems):
parted /dev/sdd mklabel gpt
# For MSDOS (legacy systems):
# parted /dev/sdd mklabel msdos
# Create partition matching original
# Get start and end sectors from /tmp/partition_table.txt
# Example: parted /dev/sdd unit s mkpart primary 2048s 100%FREE
# Set partition type/flag if needed
# For Linux LVM on GPT:
parted /dev/sdd set 1 lvm on
# Verify partition creation
parted /dev/sdd print
lsblk /dev/sdd
Step 3: Create LUKS Encryption on Replacement Partition
# Create LUKS device with same parameters as original
# IMPORTANT: Use the same cipher, key-size, and parameters
cryptsetup luksFormat --type luks2 --cipher aes-xts-plain64 --key-size 512 \
--hash sha512 --iter-time 5000 --pbkdf argon2id /dev/sdd1
# Add the same keyfile or passphrase used for other devices
# If using keyfile:
cryptsetup luksAddKey /dev/sdd1 /etc/luks-keys/sdd1.key
# Or use the same keyfile as other partitions (if sharing keys)
cryptsetup luksAddKey /dev/sdd1 /etc/luks-keys/sda1.key
Step 4: Open Encrypted Device
# Open the encrypted device
cryptsetup open /dev/sdd1 sdd1_crypt
# Verify device is available
ls -l /dev/mapper/sdd1_crypt
Step 5: Restore LVM2 Physical Volume
Option A: If LVM2 Metadata Exists on Other PVs (Recommended)
# Activate volume group (should detect new PV automatically)
vgchange -ay vg0
# Check if new PV is recognized
pvs
# If PV is not automatically recognized, scan for it
pvscan
# If still not recognized, restore PV metadata
# Get UUID from volume group metadata
vgcfgbackup -f /tmp/vg0_backup.txt vg0
# Restore physical volume using backup
pvcreate --restorefile /tmp/vg0_backup.txt --uuid <original-pv-uuid> /dev/mapper/sdd1_crypt
# The UUID can be found from the backup file or from previous pvs output
Option B: Recreate Physical Volume and Extend Volume Group
# Create new physical volume
pvcreate /dev/mapper/sdd1_crypt
# Extend volume group to include new PV
vgextend vg0 /dev/mapper/sdd1_crypt
# Verify volume group status
vgs
pvs
Step 6: Restore Data to Replacement Physical Volume
If Using RAID1 (Mirroring):
# If the volume group uses mirroring, data will sync automatically
# Monitor sync progress
lvdisplay vg0/lv0
# Check sync status
lvs -a -o name,copy_percent,devices vg0
If Not Using Mirroring (Data Loss Risk):
# If data was not mirrored, you may need to restore from backup
# This is why RAID or mirroring is recommended for encrypted storage
# If logical volumes span multiple PVs, you may need to:
# 1. Reduce logical volume size to exclude failed PV
# 2. Restore data from backup
# 3. Extend logical volume to include new PV
Step 7: Update System Configuration
# Update /etc/crypttab to include new device
# Get UUID of new encrypted partition
blkid /dev/sdd1
# Add entry to /etc/crypttab
cat >> /etc/crypttab << 'EOF'
sdd1_crypt UUID=<sdd1-uuid> /etc/luks-keys/sdd1.key luks
EOF
# Update initramfs
dracut --force
Step 8: Verify System Boot
# Reboot system and verify:
# 1. Encrypted device unlocks automatically
# 2. LVM2 volumes are activated
# 3. Filesystems mount correctly
# After reboot, verify:
lsblk
pvs
vgs
lvs
mount | grep vg0
Scenario 2: Failed Disk in RAID Configuration (LUKS on RAID Partitions)
When LUKS is implemented on RAID partitions, the replacement procedure involves both RAID array management and LUKS encryption:
Step 1: Identify Failed Disk in RAID Array
# Check RAID array status
cat /proc/mdstat
# Example showing failed disk:
# md0 : active raid1 sda1[0](F) sdb1[1]
# 976630528 blocks [2/1] [U_]
# Get detailed RAID information
mdadm --detail /dev/md0
# Identify which physical device failed
mdadm --examine /dev/sda1
mdadm --examine /dev/sdb1
Step 2: Remove Failed Disk from RAID Array
# Mark failed disk (if not already marked)
mdadm --manage /dev/md0 --fail /dev/sda1
# Remove failed disk from array
mdadm --manage /dev/md0 --remove /dev/sda1
# Verify removal
cat /proc/mdstat
Step 3: Replace Physical Disk
# Physically replace the disk
# Ensure new disk is detected
lsblk
# New disk should appear (e.g., /dev/sdc)
NEW_DEVICE="/dev/sdc"
Step 4: Prepare Replacement Disk for RAID
# Copy partition table from healthy RAID member
# Method 1: Using sgdisk for GPT disks (recommended, most reliable)
sgdisk --backup=/tmp/raid_partition_backup.sgdisk /dev/sdb
sgdisk --load-backup=/tmp/raid_partition_backup.sgdisk /dev/sdc
# Verify partition table restoration
sgdisk --verify /dev/sdc
parted /dev/sdc print
# Method 2: Using parted to dump and manually recreate
# Dump partition information from healthy disk for reference
parted /dev/sdb unit s print > /tmp/raid_partition_info.txt
# View partition information
cat /tmp/raid_partition_info.txt
# Determine partition table type from original
parted /dev/sdb print | grep "Partition Table"
# Create matching partition table type on new disk
# For GPT (most common on modern systems):
parted /dev/sdc mklabel gpt
# For MSDOS (legacy systems):
# parted /dev/sdc mklabel msdos
# Create partition matching original
# Get start and end sectors from /tmp/raid_partition_info.txt
# Example: parted /dev/sdc unit s mkpart primary 2048s 100%FREE
# Set partition type to Linux RAID
parted /dev/sdc set 1 raid on
# Verify partition table
parted /dev/sdc print
lsblk /dev/sdc
# Ensure partition type is Linux RAID (should show in parted output)
Step 5: Add Disk to RAID Array
# Add new disk to RAID array
mdadm --manage /dev/md0 --add /dev/sdc1
# Monitor rebuild progress
watch cat /proc/mdstat
# Or use detailed monitoring
watch 'mdadm --detail /dev/md0'
Step 6: Recreate LUKS Encryption on Rebuilt RAID
Important: After RAID rebuild completes, the RAID array contains the data, but LUKS encryption must be recreated:
# Wait for RAID rebuild to complete
# Check rebuild status
mdadm --detail /dev/md0 | grep -i "resync\|recovery"
# Once rebuild is complete, verify RAID array is healthy
cat /proc/mdstat
Critical Consideration: The LUKS encryption on the RAID partition should already be intact if the RAID array was properly rebuilt. However, if the LUKS header was corrupted or lost, you may need to restore it:
# Check LUKS header status
cryptsetup luksDump /dev/md0
# If LUKS header is intact, simply unlock:
cryptsetup open /dev/md0 md0_crypt
# If LUKS header is corrupted, restore from backup:
cryptsetup luksHeaderRestore /dev/md0 --header-backup-file /backup/md0.header
# Then unlock
cryptsetup open /dev/md0 md0_crypt
Step 7: Verify LVM2 Recognition
# Activate volume group
vgchange -ay vg0
# Verify physical volumes
pvs
# Check volume group status
vgs
lvs
If the failed disk contains the root filesystem, the procedure is more complex and may require booting from alternative media:
Step 1: Boot from Alternative Source
# Boot from:
# - Rescue media (USB, CD/DVD)
# - Network boot
# - Alternative boot device
# Ensure you have access to:
# - LUKS keyfiles or passphrases
# - System configuration files
Step 2: Identify and Replace Failed Disk
# Follow physical disk replacement procedures
# Identify failed disk
lsblk
dmesg | grep -i error
# Replace physical disk
# Prepare new disk (partition, etc.)
Step 3: Recreate Encryption and LVM2
# Follow encryption and LVM2 recreation procedures
# This may require:
# 1. Creating LUKS encryption
# 2. Restoring LVM2 physical volume
# 3. Restoring filesystem from backup
Step 4: Restore Root Filesystem
# If root filesystem was on failed disk:
# 1. Restore from backup to new encrypted device
# 2. Update bootloader configuration
# 3. Update initramfs
# 4. Verify boot configuration
8.2 Complexity Considerations and Mitigation Strategies
Challenges Introduced by Early Encryption:
Several challenges are introduced by implementing encryption at the partition level before LVM2. Boot dependency presents a significant challenge, as the system cannot boot if the root filesystem is on an encrypted device that cannot be unlocked. This can be mitigated by maintaining bootable rescue media with LUKS tools, using a separate unencrypted /boot partition, or implementing network-based key management for automatic unlocking.
LVM2 metadata access becomes more complex because LVM2 metadata is encrypted, requiring decryption before volume group operations can be performed. Mitigation strategies include maintaining LVM2 metadata backups, using RAID1 to maintain redundant copies of metadata, and documenting volume group structure and UUIDs for reference during recovery.
Key availability is critical during recovery operations, as LUKS keys must be available to unlock replacement devices. This can be addressed by storing keyfiles on separate encrypted devices, using network-based key management systems, and maintaining secure key backup procedures that ensure keys are accessible when needed but protected from unauthorised access.
Recovery time is extended when disk replacement is required, as the process takes longer due to the encryption and LVM2 layers that must be managed. This can be mitigated by using hot-swappable drives where possible, maintaining spare encrypted devices ready for use, and documenting and testing recovery procedures to ensure efficient execution when failures occur.
Best Practices for Disk Replacement:
Maintaining comprehensive documentation is essential for successful disk replacement operations. This documentation should include LUKS encryption parameters such as cipher, key-size, hash algorithm, and key derivation function settings. The LVM2 volume group structure should be documented, including physical volume UUIDs, logical volume configurations, and extent allocation strategies. Keyfile locations and access procedures must be documented to ensure keys are available when needed during recovery operations.
Regular backups form a critical component of disk replacement preparedness. LUKS headers should be backed up regularly, as header corruption can prevent device unlocking even with correct keys. LVM2 metadata should be backed up to enable volume group reconstruction if needed. Keyfiles must be backed up securely, ensuring they are protected from unauthorised access whilst remaining accessible for recovery. Backup restoration procedures should be tested regularly to ensure backups are valid and restoration processes are understood.
Comprehensive monitoring enables early detection of disk failures and provides visibility into storage system health. Disk health should be monitored using SMART (Self-Monitoring, Analysis and Reporting Technology) attributes to detect potential failures before they occur. RAID array status should be monitored continuously, with alerts configured for array degradation or device failures. LVM2 physical volume status should be monitored to detect missing or failed physical volumes. Alerting systems should be configured to notify administrators immediately when disk failures are detected, enabling rapid response and minimising data loss risk.
Thorough testing of disk replacement procedures is essential before production deployment. Disk replacement procedures should be tested in non-production environments that mirror production configurations as closely as possible. Recovery from backup should be tested regularly to ensure backup validity and restoration procedures are effective. Boot procedures with encrypted storage should be tested to verify that systems can start successfully after disk replacement. Lessons learned from tests should be documented and incorporated into procedures and documentation to continuously improve recovery capabilities.
8.4 Adding New RAID Arrays
When adding new RAID arrays to an existing encrypted LVM2 configuration:
# Create new RAID array
mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/sdc1 /dev/sdd1
# Encrypt the new array
cryptsetup luksFormat --type luks2 --cipher aes-xts-plain64 --key-size 512 \
--hash sha512 --iter-time 5000 --pbkdf argon2id /dev/md2
# Add keyfile for automatic unlocking
cryptsetup luksAddKey /dev/md2 /etc/luks-keys/md2.key
# Update crypttab
echo "md2_crypt UUID=$(blkid -s UUID -o value /dev/md2) /etc/luks-keys/md2.key luks" >> /etc/crypttab
# Open encrypted device
cryptsetup open /dev/md2 md2_crypt
# Add to volume group
pvcreate /dev/mapper/md2_crypt
vgextend vg_encrypted /dev/mapper/md2_crypt
# Extend or create logical volumes
lvextend -L +200G /dev/vg_encrypted/lv_data
8.5 Backup and Recovery
LUKS Header Backup:
# Backup header before any operations
cryptsetup luksHeaderBackup /dev/md0 --header-backup-file /backup/md0.header
# Verify header backup
cryptsetup luksDump /dev/md0 > /backup/md0.luksdump
Data Backup:
Data backup should utilise standard backup tools on mounted filesystems, ensuring that data is backed up in decrypted form for easier restoration. Snapshot-based backup strategies with LVM2 snapshots can provide efficient point-in-time backups without requiring filesystem unmounting. If backing up encrypted devices directly, backup systems must have access to decryption keys, which requires careful key management and secure key distribution to backup systems.
Recovery Procedures:
Recovery procedures must be comprehensive and well-documented. Key recovery procedures should be documented in detail, including steps for accessing keys, unlocking devices, and restoring data. Recovery procedures should be tested regularly in non-production environments to ensure they are effective and can be executed efficiently during actual failures. Secure key storage must be maintained separate from encrypted data, ensuring that key loss does not result in complete data loss, whilst also protecting keys from unauthorised access.
8.6 Monitoring
Device Status:
# Check encrypted device status
cryptsetup status md0_crypt
# Monitor LUKS device health
systemctl status systemd-cryptsetup@md0_crypt.service
# Check LVM2 status
pvs
vgs
lvs
Performance Monitoring:
# Monitor I/O performance
iostat -x 1
# Monitor encryption CPU usage
top -p $(pgrep -f cryptsetup)
9. Best Practices
9.1 Security Best Practices
Mandatory CPU Memory Encryption: Only implement LUKS on systems with Intel TME or AMD SME support
Strong Key Derivation: Use Argon2id with appropriate iteration time
Key Management: Store keyfiles securely, implement key rotation procedures
Header Backup: Regularly backup LUKS headers to secure locations
Access Control: Limit administrative access to encryption systems
Audit Logging: Enable comprehensive audit logging for encryption operations
9.2 Operational Best Practices
Documentation: Maintain detailed documentation of encryption configuration
Testing: Test all procedures in non-production environments first
Monitoring: Implement monitoring for encrypted device status and performance
Backup: Regular backup of both data and LUKS headers
Recovery Procedures: Document and test recovery procedures regularly
9.3 Performance Best Practices
Hardware Acceleration: Ensure CPU supports AES-NI for optimal performance
Alignment: Maintain proper alignment across all storage layers
Benchmarking: Benchmark performance before and after encryption implementation
Workload Analysis: Understand I/O patterns and adjust configuration accordingly
9.4 Configuration Recommendations
For High Security:
High security configurations should use Argon2id with longer iteration times, typically 10 seconds or more, to increase the computational cost of key derivation and make brute-force attacks more difficult. Multiple key slots should be implemented with different passphrases to provide redundancy and enable key rotation without data loss. Hardware security modules should be used for key storage in high-security environments, providing tamper-resistant hardware-based key protection. Comprehensive audit logging should be enabled to provide detailed records of all encryption-related operations for security monitoring and compliance purposes.
For High Performance:
High performance configurations should use AES-XTS with hardware acceleration through AES-NI instructions to minimise encryption overhead. Key derivation iteration time should be optimised to 3-5 seconds, balancing security with performance requirements. Proper alignment must be ensured across all storage layers, including RAID chunk size, LVM2 extent size, and filesystem alignment, to prevent performance degradation from misalignment. For systems with high encryption workload, dedicated CPU cores can be considered for encryption operations to isolate encryption processing from other system workloads.
10. References
LUKS (Linux Unified Key Setup) - On-Disk Format Specification: GitLab
