3. Software Configuration
This chapter describes the steps required to configure your VDURA system software once you have installed and cabled the hardware.
This chapter contains the following sections:
3.1 Initial Network Setup
You will likely see the steps below first. If the prompt in step 5 does not appear, proceed to the Out of Box Experience section.
- Power on the Directors, starting with the one at the bottom. Allow about a minute, then power on the next one above it. Repeat this process until all Directors are powered on. The system will have at least three Director Nodes.
- Physically connect to the console.
- Connect via KVM: Attach a monitor via the VGA port and a USB keyboard to the Director Node.
- Connect via admin port: The admin port is the 1 GbE port furthest to the left. It can be used for installation, but should only be used by a VDURA employee or at a VDURA employee’s direction once the realm is running. Attach an Ethernet cable from a laptop to the 1 GbE admin port 1. Configure that NIC with any IP address in 169.254.0.0/16 except 169.254.12.34.
- Open your SSH client and connect to the Director Node.
ssh admin@169.254.12.34- The system prompts you for a login and password. Log in using the default user name and password.
login: admin
Password: admin- The system prompts you to set up the high-speed data network. At the command prompt, enter Yes or press Enter to accept the default value in brackets. For example:
pancli -- Initial network setup
Would you like to setup the network now? (recommended) [yes]- The system prompts you for the IP address, netmask, and default router address.
IP Address: 10.1.32.3
Netmask: [255.255.255.0] 255.255.255.0
Default Router Address: [10.1.32.1] 10.1.32.1- IP Address is the IP address of the high-speed data ports of the first Director. The IP address should be the first in the range of IP addresses that you assign to the storage system. VDURA recommends that you assign the VDURA storage cluster its own IP subnet with at least as many available addresses as the maximum predicted number of nodes in the storage system, plus one additional IP address. You also use this IP address to access the Install Wizard.
- Netmask is the subnet mask for the network to which the VDURA system is attached.
- Default Router Address is the name or IP address of the primary routing gateway. The router must be in the same subnet as the VDURA system IP address. If the system does not need routing to reach its clients, leave this field blank.
To configure IP addresses for high-speed data ports, complete steps 5 and 6. These steps do not change the management port IP address.
- The system displays your settings and prompts you to confirm the values. To confirm the settings, enter Yes or press Enter.
Settings chosen:
IP: 10.1.32.3
Netmask: 255.255.255.0
Router: 10.1.32.1
Save settings? [yes]- Log out of the system.
exitAt this point, you can unplug the diagnostics port and terminal. However, VDURA recommends that you keep the terminal connected to the serial port while the system comes up for two reasons. First, the next step of the configuration, the Out of Box Experience (OOBE), can be run through the diagnostics/admin port or by connecting to the first Director using the address set during initial network setup. Second, even if you are using the web-based Install Wizard, you can monitor the progress of the system configuration.
After you have configured the first Director, proceed to the next section and use the Install Wizard/OOBE to perform the initial system configuration.
3.2 Out of Box Experience
This section describes the initial Out of Box Experience (OOBE) used to initially power up and configure a VDURA realm.
After the Directors and Storage Nodes are powered on (steps 1 and 2 below) and the network setup is correct, they should establish communication automatically and can be listed with the sysmap nodes command.
- The Node column contains the node identifier.
- The Type column contains the node type: Director, VDURA Data Platform, or VPOD.
Note: VDURA recommends that you run the sysmap nodes command before proceeding to ensure all nodes are ready. You may see clock sync error; while this prevents nodes from joining realm, it will go away after a few minutes. Rerun sysmap nodes to verify.
Below is example output of the sysmap nodes command:
[pancli] sysmap nodes
Node Type Serial IP Address State
VCH-1,1 Director 52f200003f4005001 10.101.1.15 Online
VCH-2,1 Director 532200003f4005001 10.101.1.64 Online
VCH-3,1 Director 52f200013f4005001 10.101.1.63 Online
VCH-4,1 VDURA Data Platform 52f200041e4029001 10.101.1.62 Online
VCH-4,2 VPOD 9ec738191d4029001 10.101.1.50 Online
VCH-4,3 VPOD 101e0dc41d4029001 10.101.1.51 Online
VCH-4,4 VPOD b1371a851d4029001 10.101.1.58 Online
VCH-4,5 VPOD bddeb9c11d4029001 10.101.1.49 Online
VCH-4,6 VPOD fc4a659b1d4029001 10.101.1.48 Online
VCH-4,7 VPOD 9400b4811d4029001 10.101.1.45 Online
VCH-4,8 VPOD 68ded6c71d4029002 10.101.1.30 Online
VCH-4,9 VPOD ab784d4a1d4029002 10.101.1.41 Online
VCH-4,10 VPOD 073406511d4029002 10.101.1.34 Online
VCH-4,11 VPOD 0d968c321d4029002 10.101.1.27 Online
VCH-4,12 VPOD 565d71601d4029002 10.101.1.28 Online
VCH-4,13 VPOD e4ed05251d4029002 10.101.1.29 Online
VCH-5,1 VDURA Data Platform 52f200021e4029001 10.101.1.60 Online
VCH-5,2 VPOD d55360781d4029001 10.101.1.42 Online
VCH-5,3 VPOD 90ea9f401d4029001 10.101.1.59 Online
VCH-5,4 VPOD 4bc7cfc71d4029001 10.101.1.46 Online
VCH-5,5 VPOD f6834dae1d4029001 10.101.1.61 Online
VCH-5,6 VPOD 6c058c231d4029001 10.101.1.54 Online
VCH-5,7 VPOD 0cc06cc71d4029001 10.101.1.53 Online
VCH-5,8 VPOD 35f41cec1d4029002 10.101.1.40 Online
VCH-5,9 VPOD 024d00771d4029002 10.101.1.33 Online
VCH-5,10 VPOD 1a5f8c5c1d4029002 10.101.1.32 Online
VCH-5,11 VPOD 991657571d4029002 10.101.1.37 Online
VCH-5,12 VPOD 39e6675a1d4029002 10.101.1.24 Online
VCH-5,13 VPOD 0a433a8e1d4029002 10.101.1.35 Online
VCH-6,1 VDURA Data Platform 52f200031e4029001 10.101.1.65 Online
VCH-6,2 VPOD 69c262fd1d4029001 10.101.1.56 Online
VCH-6,3 VPOD cfdb1fe01d4029001 10.101.1.57 Online
VCH-6,4 VPOD 0da476ff1d4029001 10.101.1.52 Online
VCH-6,5 VPOD 9e68653d1d4029001 10.101.1.55 Online
VCH-6,6 VPOD ec4f4ce21d4029001 10.101.1.44 Online
VCH-6,7 VPOD 1b5bdc331d4029001 10.101.1.47 Online
VCH-6,8 VPOD b359f4351d4029002 10.101.1.26 Online
VCH-6,9 VPOD cb3793781d4029002 10.101.1.43 Online
VCH-6,10 VPOD fdbedcba1d4029002 10.101.1.38 Online
VCH-6,11 VPOD 40f4702b1d4029002 10.101.1.39 Online
VCH-6,12 VPOD dab33c851d4029002 10.101.1.36 Online
VCH-6,13 VPOD 469f9d4f1d4029002 10.101.1.31 OnlineThe OOBE will automatically create a StorageSet named Set 1 containing / and /home directories, but will assign no VPODs to the StorageSet. You can follow the recommendation to add VPODs to the Set 1 StorageSet in step 5 and either keep the name or change it later. You can also create a new StorageSet and add VPODs to it.
Follow the steps below to complete the OOBE.
- The Directors should already be powered on from previous steps. If not, power on the Directors, starting with the one at the bottom. Allow about a minute, then power on the next one above it. Repeat this process until all Directors are powered on. The system will have at least three Director Nodes.
- If the realm includes JBODs, first power on all JBODs and wait at least 60 seconds. Then power on the Storage Nodes one enclosure at a time, starting with the enclosure at the bottom. Allow about a minute, then move to the next higher one.
- Activate all Directors to bring them into the realm:
syscontrol activate all- Use the syscontrol realmmgr add command to include the second and third Directors in the replication set (repset). You can use the sysmap nodes command to find the identifier for each Director:
syscontrol realmmgr add VCH-2,1
syscontrol realmmgr add VCH-3,1The identifiers may be different than the example above.
- Use the storageset add storage command to add VPODs to the Set 1 StorageSet created during the OOBE process. You can use the sysmap nodes command to find the identifier for each Storage Node, which will begin with VCH. Do not use the identifiers named VDURA Data Platform.
storageset addstorage -addspare 0 "Set 1" VCH-4,13.3 Configure PanFS
This section describes PanFS and how to configure it as part of the installation process.
3.3.1 PanFS Components
PanFS is composed of the following elements:
- PanFS OS
- DirectFlow
- PanFS Realm Management
- PanActive Manager
PanFS Operating System
The PanFS storage operating system (PanFS OS) is the integrated software that manages and directs VDURA systems. It dynamically distributes file activity across Storage Nodes. It also clusters Directors to coordinate file activity, balance system performance, and manage failovers. This distributed cluster-based approach eliminates performance bottlenecks and eases the administration burden for the VDURA storage network.
PanFS is an object-based, parallel file system. Files are divided into chunks that are stored in different objects and accessed in parallel for performance and redundancy. Data objects are containers for data and attributes that describe how the data fits into the overall distributed file system.
Files are striped across objects, and each file has its own map that defines the striping pattern. This allows different objects to have different striping patterns and provides different performance and redundancy characteristics. Because maps are just a small array of object identifiers and associated access capabilities, they are efficient to manage.
For example, small files and directory objects are mirrored so that they are cheap to create and cheap to update with small I/Os often associated with small files and directories. Large files are striped more widely using RAID on a per-file basis to provide redundancy with less capacity overhead than mirroring. Very large files use a two-level striping pattern that optimizes performance in very large storage systems.
The storage map for a file can change dynamically to accommodate changes in the available storage resources. For example:
- If a Storage Node fails, component objects will automatically be reconstructed on alternate VPODs.
- The addition of new storage resources will result in automatic rebalancing of capacity so that all VPODs are loaded uniformly.
- Older hardware can be retired by migrating data away from those VPODs.
All data management is automatic and performed in the background while the system is online.
DirectFlow Client
Similar to massively parallel supercomputer architectures, the DirectFlow Client software supports fully parallel reads and writes from DirectFlow Clients to Storage Nodes without encountering bottlenecks. The DirectFlow software module is a loadable kernel module that transparently adds support for the VDURA file system protocols to Linux compute nodes.
DirectFlow provides high-performance, high-fidelity POSIX semantics for large-scale distributed systems based on a cache consistency protocol with VDURA metadata managers. In addition, there are specialized interfaces that optimize the large shared file workloads found in MPI-IO applications.
PanFS Realm Management
PanFS realm management software logically clusters Directors in an active-active configuration so that all Directors are fully engaged in workload processing.
To enable scalability, realm management allows hundreds of VDURA Directors to cluster together to evenly and effectively distribute the workload. Realm management provides the following advantages:
- Optimizes the initial distribution and ongoing management of files across all hardware elements while minimizing interdependence.
- Adapts to changing client usage patterns and changes in file sizes by dynamically improving system operation.
- Automatically adapts to Director failures by providing a highly available file system.
PanActive Manager
PanActive Manager provides an HTML user interface to manage your VDURA system. This interface makes it easy to securely configure VDURA systems and to display and monitor critical system information. An SNMP interface allows integration of the storage system into enterprise data center management frameworks.
To ensure that problems are detected before they cause a failure, PanActive Manager notifies administrators of impending failures, allowing them to identify and prevent problems before they affect the system. Systems can be monitored for disk, DIMM, and fan failures, as well as temperature and other environmental threats.
For detailed information about the VDURA system and the VDURA PanFS parallel file system, see the VDURA Data Platform User Guide.
3.3.2 Configure PanActive Manager
Once VDURA hardware has been installed and cabled, the next step is to configure PanActive Manager in order to configure and manage storage. For complete information on configuring and managing PanActive Manager, refer to the VDURA Data Platform User Guide.
Prerequisites
The PanActive Manager Install Wizard will ask you for configuration information about your system. Before you begin the configuration process, make sure you have the following information at hand:
- Customer ID provided in your order confirmation. If you do not have your customer ID, call VDURA Technical Support.
- Administrator ID, password, and email address.
- Web proxy host and port, if required. The web proxy server must be configured to handle HTTPS requests if you want PanActive Link to work with your system.
- Simple Mail Transfer Protocol (SMTP) server name or IP address.
- VDURA storage cluster name and IP address.
It is recommended (but not mandatory) that the system name be the DNS name that corresponds to the IP address of the first Director. The system name appears under the /panfs mount point of the DirectFlow file system clients.
The system IP address is the address of the first Director. This address is also used to run the Install Wizard. VDURA recommends that it be the first address of the node address range that you assign to the system.
- Netmask and default router.
- Maximum Ethernet frame size, either 1500 or 9000.
- Node address range: A range of IP addresses that you assign to the VDURA storage cluster. VDURA recommends that you assign the VDURA storage cluster its own IP subnet with at least as many available addresses as the maximum predicted number of nodes in the storage system, plus one additional IP address.
- Domain Name System (DNS) domain name and IP address. If the DNS zone name is not equivalent to system DNS name plus DNS domain name, then the DNS lookups to the DNS zone name fail with NXDOMAIN.
- Network Information Service (NIS) domain name, server, and hostname resolution status.
- Network Time Protocol (NTP) server names or IP addresses. VDURA recommends setting the cluster NTP to the same NTP used by the DirectFlow Clients. A list of NTP servers is available at tf.nist.gov/tf-cgi/servers.cgi. Two or three servers are recommended for redundancy. If no NTP servers are specified, one Director is automatically chosen to operate as a stratum 0 time source for the system.
Note: When configuring NTP, be aware that NTP server time on a client and the VDURA system must be no more than ten minutes apart, and that closer is better.
- Time zone for the system.
- Network File System (NFS) for UNIX clients and Common Internet File System (CIFS) protocol for Windows clients functionality.
If you enable CIFS, also have the following available:
- NetBIOS name for the VDURA system.
- Windows domain name.
- Computer Account Organizational Unit.
- Name and password of an account with permission to join the domain. This account also should have privileges to access the System Account Management Database (SAMR).
A printable configuration checklist is available in Installation and Configuration Checklist of this guide. A PDF version is also available to download on the first page of the Install Wizard.
Configure the VDURA System
- Open a web browser.
- Enter the IP address you assigned to the first Director and press Enter:
https://10.1.32.3The first page of the Install Wizard appears.
- Follow the step-by-step instructions to complete the Install Wizard. To display online help for the Install Wizard, click the controls next to the fields.
- After you have completed the Install Wizard, click the Apply Settings button to save your settings to the system.
The system will automatically configure the first enclosure into a StorageSet and volume and will export the file system for use via NFS, DirectFlow, and (if configured) CIFS. Console messages will indicate when the file system is ready.
For detailed information about configuring and using PanActive Manager, see the VDURA Data Platform User Guide.
You can also configure your system using the VDURA command-line interface. For detailed information about connecting and using PanCLI, see the PanCLI Reference Guide.
Install DirectFlow Client
For Linux/UNIX users, you can also install the VDURA DirectFlow client software, which enables direct access to the PanFS operating system from supported hosts. Refer to DirectFlow Client Installation.
For complete information on configuring and managing the DirectFlow client, refer to the VDURA Data Platform User Guide.
3.3.3 Upgrade and Downgrade PanFS
Please contact VDURA Technical Support for current information regarding PanFS release compatibility and upgrade/downgrade paths and caveats.
Access a Release
Releases are available only from the Support Portal. The VDURA Data Platform can no longer download upgrade images itself.
Go to the Downloads tab of support.VDURA.com and download the release to a PanFS directory or a machine on your network that your VDURA system can access. Then, in PanActive Manager, navigate to Advanced Tools > Software Management > Upgrade PanFS and use the Upload PanFS Operating System Release section to load the software into your VDURA system. Once uploaded, the software will appear in the Versions Available for Upgrade table. Review the columns that indicate compatibility with your hardware. If an upgrade is not compatible with existing hardware, the column provides a link to a table summary of software releases that are compatible.
For information on upgrading system software with the command-line interface, see the PanCLI Reference Guide or PanActive Manager online help. Note that remote access via FTP or SCP requires that the remote system be running an FTP or SSH service.
Install a Release
In the PanActive Manager Advanced Tools > Software Management > Upgrade PanFS > Versions Available for Upgrade table, find the release you uploaded. The software was uploaded to a single node in the system, so the release shows a Distribute button. Click this button to distribute the upgrade package to all other nodes in the system.
If the distribution is unsuccessful, an error appears on the Upgrade PanFS page, and a new control appears in the Versions Available for Upgrade table. Use this control to redistribute the software either to all nodes or specific ones identified by IP address. Select the appropriate nodes and click Redistribute. You can also go to the Advanced Tools > Software Management > Distribution Status page to view node status and select nodes for redistribution.
Once the image has been successfully distributed, the Distribute button becomes an Install button. Click Install. The system will be taken offline during the upgrade process. When the installation is successful on all nodes, the system will reboot and all nodes will come back online. If the installation fails, the system will revert to the previously running version of the software.
Note that upgrading PanFS may also require upgrading your DirectFlow Clients. Refer to the sections below for instructions on installing and configuring DirectFlow.
Upgrade Time Frame
Upgrade and downgrade times will vary, but in general, the following holds true:
- The download and distribute phases take time, but the system is fully online during these steps. Since both download and distribution are just single-click steps, there is also practically no administrative time required.
- The time it takes to download a release depends on the method. Downloading to each node directly from the VDURA support portal takes longer than copying the file (downloaded from the support portal separately) into a volume, then downloading from that volume.
- Distribution time depends on the number of nodes in the system. VDURA uses an exponential distribution method to hasten this process-one node gets the image and sends it to two other nodes, then each of those three nodes send a copy to two additional nodes, then those nine nodes each then send to two additional nodes, etc.
- When the Install button is clicked, the system does some work to validate packages, etc., before it goes offline and reboots. At this point, clients will experience the system hanging. This step will take about ten minutes to complete before coming back online.
3.4 DirectFlow Client Installation
The DirectFlow Client software enables direct access to VDURA storage from supported Linux hosts. This section describes basic installation and configuration of DirectFlow Clients.
- Download the appropriate RPM or DEB for the client from the Downloads > DirectFlow Client for Linux page of support.VDURA.com. The VDURA Data Platform can no longer download DirectFlow clients itself.
- Copy the downloaded file to the client.
- Unmount the realm.
umount /panfsAlso, remove any NFS mountpoints that you have.
- For RPM installations with a previous version of DirectFlow installed, uninstall the old panfs module before installing the new one, where
[panfs-*]is the installed panfs module.
rpm -e [panfs-*]The rpm -U command is not supported. Using rpm -e will delete all panfs mount lines from the /etc/fstab file, but will backup the file.
- Run one of the following commands.
For RPM files, where [panfs*.rpm] is the name of the RPM file you downloaded:
rpm -hiv [panfs*.rpm]For DEB files, where [path/to/deb] is the name and location of the DEB file you downloaded:
dpkg -i /[path/to/deb]- Before configuring the DirectFlow client, make sure the client’s system is synchronized with the system (e.g., via NTP).
- If secure boot is enabled on the client, the kernel prompts you to enroll the public PanFS key to the shim security keyring. Secure boot is provided by a BIOS and kernel supporting UEFI 2.3.1 Errata C or higher specification. When the client boots, the shim bootloader must authenticate the operating system for the OS to load. The kernel then only loads kernel modules that it can authenticate using the shim security database. The Machine Owner Key utility allows the root user to manage the public keys for installed kernel modules.
Once DirectFlow is installed, register the DirectFlow key with the following command:
# mokutil --import /etc/pki/panfs/panfs-mok.derYou will be prompted to enter the client root user password twice. The command will not run without entering the root password. If you get a prompt to run mokutil and cannot run it, DirectFlow will not load. To avoid this issue, do not run config_panfs until you have the client root password.
Once the command is run, the output will indicate that the key was successfully enrolled.
Subsequent DirectFlow installations on this client should not require a public key enrollment, as the public key for any DirectFlow port is the same.
- Run config_panfs to configure the client.
/usr/local/sbin/config_panfsThe client must be set to the same NTP as the cluster. Consult the VDURA Data Platform User Guide for any errors encountered while using the script.
- You will be prompted for the DNS name or IP address of the VDURA system. Typically, you should provide the DNS name unless your site’s DNS server is not configured to resolve the system name. If you have configured your VDURA system for realm management with multiple Directors and decide to provide the IP address, enter the IP address of the president Director.
The following is a suggested starting point for a manual configuration of a four-node system:
panfs://xxx.xxx.xxx.aaa:global /panfs panfs rw,auto,_netdev,max-async-writepages=4,iscsi-sock-rsize=262144,iscsi-sock-wsize=262144,file-dirty-data-hwm=264241152,sam-resource-level=4096, num-stripes-per-write=112, rmlist=(xxx.xxx.xxx.aaa; xxx.xxx.xxx.bbb; xxx.xxx.xxx.ccc),callback-network-allow= xxx.xxx.xxx.0/24 0 0Regarding DNS/Client Time Slew
PanFS will fail to mount if the time slew between the client and the realm exceeds 10 seconds. Example error message:
client:~ # /usr/local/sbin/config_panfs -a -n 192.168.100.10
Enter the DNS/IP address of the PanFS Realm: 192.168.100.10
Started chrony.
PanFS module loaded successfully.
mount.panfs error: mount request rejected because client-realm clock skew exceeds 10 seconds
Error: mount error 1 encountered while running 'mount /panfs'. See mount man page for details.Important: Be aware that there is currently an error in the error message text above. The script error reads “time skew exceeds 10 seconds”, when it should read “time slew exceeds 10 seconds”.
Corrective Action
No corrective action is required unless clients are unable to mount DirectFlow exports. If clients are unable to mount DirectFlow exports, complete the following steps:
- Make sure the client and the cluster both use the same NTP server.
- Force a time sync on the client.
- Make sure the cluster can ping the NTP server.
- If the mount still fails, contact VDURA Technical Support before rebooting the cluster. If a reboot is needed, be aware that hosts will lose DirectFlow connectivity during the reboot period and may need to restart jobs. After reboot, the storage cluster will synchronize with the reference clock via ntpdate. Ntpdate is a PanFS utility for Directors only.
- Contact VDURA Technical Support if the steps above do not resolve the issue.
Note: The ntpdate utility cannot be run while the NTP daemon is running.
For detailed information about installing the DirectFlow client software, see the VDURA Data Platform User Guide.