13. End-to-End Encryption

This chapter provides information about and instructions for managing VDURA’s end-to-end encryption implementation.

Note: End-to-end encryption works only on the platform tenant, and only in a realm that does not use multi-tenancy. It cannot be used with tenants other than the platform tenant.

13.1 Introduction and Overview

To support a more secure environment for processing data, the DirectFlow Client supports an end-to-end encryption feature. This enables data to be encrypted and decrypted at the client, protecting user data from a range of attacks. VDURA’s encryption keeps user data secure once it leaves the client and not available at any point past that.

The security of user data is a top concern for storage systems. It is important to understand the threats that these mechanisms are protecting against. VDURA’s encryption implementation protects user data from:

• Snooping on the wire.
• Theft of any components of the system (servers, storage media, JBODs).
• Hacking into the storage software or device firmware.

13.1.1 What is Encrypted

The primary goal of this feature is to encrypt user data (file contents). This means that every byte of user data that passes from a DirectFlow Client to Storage Nodes (VPODs) is encrypted. Note that the protocol data units that transport user data to the Storage Nodes are not encrypted. RPC messages between the DirectFlow Client and the realm are also unencrypted.

13.1.2 How Data is Encrypted

The DirectFlow Client uses the AES-XTS cryptographic algorithm. This algorithm is FIPS-140 approved with key size of 256 bits. Data is encrypted and decrypted within the context of the DirectFlow Client. The Storage Nodes do not have the keys and do not attempt to decrypt data.

13.1.3 Metadata Encryption

Currently, metadata is not encrypted. Since metadata must be interpreted by the system, the goal of keeping data encrypted once it leaves the client would become compromised, as more parts of the system would need keys to allow for access to and interpretation of the data.

13.2 Key Management

Clients require keys to allow them to encrypt and decrypt data. These keys are managed by an external key management system (KMS). Directors must be able to create and delete keys from the KMS, and the DirectFlow Client must be able to retrieve keys.

13.2.1 Role-Based Authentication

The KMS grants keys to DirectFlow Clients to mutually authenticate with each other. To do this, each DirectFlow Client node requires a digital certificate to present to the KMS. Creation and provision of this certificate is managed by the customer. This certificate must be installed on the DirectFlow Client node by the customer for PanFS software to use. A DirectFlow Client with a certificate has the authority to retrieve any encryption key from the KMS for any encrypted volume.

13.2.2 Encryption Domain

VDURA’s encryption implementation provides encryption at PanFS volume-level granularity. Encryption can be enabled when the volume is created and cannot be changed for the lifetime of the volume. A volume with encryption enabled will have a Volume Data Key (VDK) which will be used to encrypt and decrypt all file data in the volume. This VDK cannot be changed, as that would require re-encryption of all data in the volume, which is impractical.

VDURA system administrators can create and delete encrypted volumes through the PanCLI interface via an option on the command line when a volume is created. Since encryption is an optional feature, administrators can continue to create standard, or unencrypted, volumes.

13.2.3 KMS Integration

VDURA’s encryption implementation implements a KMIP protocol (version 1.4 or better).

13.3 DirectFlow Client Behavior

A client connected to the VDURA system requires two pieces of software for encryption. First is the PanFS Client, which is a kernel module that is loaded and runs the client side of the file system. This is the software that performs encryption and decryption of user data. The second piece is a userspace module that allows for connection to the KMIP server. This module retrieves keys on behalf of the DirectFlow Client. To do that, it requires credentials to present to the KMS.

13.4 Gateway Protocols

While VDURA offers the DirectFlow Client for a high-performance, parallel data path, other options exist to access data from other protocols, such as NFS, SMB, and S3. To enable these protocols, Directors act as gateways hosting these services. Each protocol accesses data through a PanFS Client that runs on the Director, called the Gateway Client.

The Gateway Client currently does not support encryption. Therefore, no gateway protocols are supported. Current behavior for non-DirectFlow protocols is as follows:

• NFS: If encryption is enabled, NFS support is disabled. NFS access to encrypted volumes can likely be done by re-exporting NFS over a DirectFlow Client that can decrypt data. User experience with NFS behavior will largely depend on the Linux distribution and kernel version.
• SMB: If encryption is enabled, SMB support is disabled.
• S3: If encryption is enabled, S3 support is disabled.

13.5 Configuration Considerations

Several configuration options must be set to allow a user to manage and control a system with end-to-end encryption enabled.

13.5.1 External KMS Configuration

The external KMS is configured via PanCLI. Configuration requires the following information:

• IP address or fully qualified domain name
• Port
• KMIP client credentials
• Certificates

See 13.9.1 KMIP Server Configuration below for instructions on using PanCLI commands to configure the KMIP server.

13.5.2 Key Management

Each encrypted volume has a volume Data Encryption Key (DEK). The DEK remains active in the KMS for the lifetime of the volume and its value cannot be changed. When a DirectFlow Client needs a volume’s encryption key, it requests and receives the DEK directly.

13.5.3 Encrypted Volume Configuration

When creating a volume in PanActive Manager, there is an advanced configuration item that allows you to select Encrypt Volume. This option is dependent on a KMS being configured; if there is no KMS, then this will not be visible.

13.5.4 Client Configuration

Credentials for each DirectFlow Client must be provided by the customer from a Certificate Authority. A client with proper credentials can then retrieve any encrypted volume’s key from the KMS.

Mount Options

DirectFlow Clients must be told the hostname or IP address of the KMS, as well as other relevant information to contact the KMS, via a mount option. See 13.9.4 Client Interface Configuration below for mount options to explicitly indicate that an encrypted volume is being mounted.

13.5.5 Secure Status

By querying the current state of the VDURA system via the syscontrol encryption and volume list showencryption PanCLI commands, the administrator can see what parts of the system are encrypted.

13.5.6 Mounting PanFS

When the new encryption mount option is specified, the DirectFlow Client will deny PanFS global mounts (/panfs) and realm root mounts. Attempting to do a global mount or mounting any path inside the realm root volume will result in an error returned for the mount attempt; only direct volume mounts will be allowed. Attempting to mount a volume that is encrypted without explicitly providing the new encrypted mount option will also result in a failure.

13.6 Failure Modes

Several failure modes are possible when end-to-end encryption is enabled on a VDURA system:

• KMS network connection issues.
• Lost or destroyed keys.
• Compromised keys.
• Accessing data with no key.
• Accessing data with the wrong key.

13.6.1 KMS Network Connection Issues

If the KMS is unavailable, then the data will not be available. Failure to communicate with the KMS or failure to retrieve keys will result in the DirectFlow Client returning an error to the application.

13.6.2 Lost or Destroyed Keys

If keys are lost or destroyed, then data in an encrypted volume will be permanently inaccessible.

13.6.3 Compromised Keys

No provision is made to deal with compromised keys.

13.6.4 Reading Data without a Key

If a DirectFlow Client attempts to read data without a key, an error will be returned in a manner similar to a lost or destroyed key scenario.

13.6.5 Reading Data with the Wrong Key

AES-XTS does not allow for detection of an incorrect key used for decryption. Therefore, a client using the wrong key to decrypt data will result in random data being returned to the application.

13.7 Performance Considerations

There are two main points to keep in mind when considering the implications of encryption and decryption on performance: the time it takes for the DirectFlow Client to retrieve a key relevant to the data and CPU overhead.

• Key retrieval overhead: Users should expect key retrieval overhead time will be generally minimal. Instances where it is not should be infrequent.
• CPU overhead: Various software algorithms must be run to encrypt and decrypt data. These algorithms consume CPU and memory cycles on the client systems.

13.8 Upgrades and Downgrades

Downgrading a system with encrypted volumes to a version that does not support encryption will result in volume data being permanently inaccessible.

13.9 Encryption Configuration Procedures

Administrators must use PanCLI to configure and interact with the end-to-end encryption feature; the PanActive Manager GUI does not offer any visibility into the feature.

13.9.1 KMIP Server Configuration

Use the PanCLI kmip command for KMIP server configuration.

The following information should be set with the kmip config set command:

• KMIP server host (IP address or hostname)
• KMIP server port
• CA certificate (location of root certificate PEM file)
• Client certificate (location of client certificate PEM file; used by Director Nodes)
• Client private key (location of client private key PEM file)
• KMIP protocol version to be used by Director Nodes (KMIP version number)

Below is an example series of kmip config set configuration commands. All files (such as /director.pem) should be uploaded to the Director Node by the PanFS administrator prior to running.

kmip config set host 10.70.25.173
kmip config set port 5696
kmip config set ca /RootCertificate.pem
kmip config set client /director.pem
kmip config set key /director.key
kmip config set kmip_version 1.4
kmip config show
kmip config validate
kmip config save

Display the active KMIP server configuration:

kmip server show

Delete the active KMIP server configuration:

kmip server delete

13.9.2 Realm-Wide Encryption Configuration

Use the PanCLI syscontrol encryption command to change realm-wide, default end-to-end encryption behavior when new volumes are created.

The system-wide encryption setting defines default encryption status for new volumes in the realm. By default, new volumes follow this realm-wide encryption setting; enabling encryption does not affect existing non-encrypted volumes. However, if there are encrypted volumes in the system, it is not possible to turn this setting off. Encryption cannot be enabled if either NFS, CIFS, or S3 are enabled. A valid KMIP configuration should be set as a prerequisite to enabling encryption.

Enable system-wide volume encryption for new volumes:

syscontrol encryption enable

Disable system-wide volume encryption:

syscontrol encryption disable

View the current encryption setting:

syscontrol encryption

13.9.3 Volume-Level Encryption Configuration

Create a Volume with/without Encryption

If system-wide encryption is off by default, use the PanCLI volume create command to create a volume with encryption enabled:

[pancli] volume create encryption on

If system-wide encryption is on by default, use the PanCLI volume create command to create a volume with encryption disabled:

[pancli] volume create encryption off

Show Volume Encryption Status

Use the PanCLI volume list command to show a volume’s encryption state:

[pancli] volume list showencryption

                                  E2EE   Space   Soft   Hard
Volume   StorageSet  RAID           State   Used   Quota   Quota  Status
/        Set 1     Object RAID6+  --      0 MB  524 MB  524 MB  Online
Jencvol  Set 1     Object RAID6+  On      0 MB      --      --  Online
/home    Set 1     Object RAID6+  Off     0 MB      --      --  Online
Displayed 3 out of 3 volumes in the system.

13.9.4 Client Interface Configuration

When mounting the DirectFlow Client, credentials must be set to allow KMS access. Credentials for each DirectFlow Client must be provided by the customer from a Certificate Authority. A client with proper credentials can then retrieve any encrypted volume’s key from the KMS.

Configuring a KMIP server on a DirectFlow Client assumes you have the DirectFlow Client PanFS RPMs already installed and PanFS is not mounted yet.

1. Copy the required two certificate files and one key file to a location of your choice on the DirectFlow Client. In this example, /tmp is used:

   cp <path-to-certs>/RootCertificate.pem /tmp
   cp <path-to-certs>/dfc_III.pem /tmp
   cp <path-to-certs>/dfc_III.key /tmp

2. Create a KMS config file. This file is passed to mount so the mount binary knows the KMS config. Below is an example of file contents:

   config_version = 1
   
   [global]
   primary-server = server:dev
   
   [server:dev]
   host = 10.70.25.173
   port = 5696
   version = 1.4
   
   connect-timeout = 300
   reconnect-retries = 5
   
   ca-cert-path = /tmp/RootCertificate.pem
   client-cert-path = /tmp/dfc_III.pem
   client-key-path = /tmp/dfc_III.key

3. Create a directory to mount under. You cannot use /panfs because global mounts are not allowed:

   mkdir /mnt

4. Locally mount an encrypted volume, specifying the KMS config file:

   mount -t panfs -o kmip-config-file=/tmp/kmip_config panfs://<realm_ip>/enc_vol /mnt