9. Troubleshooting

This chapter describes some basic troubleshooting guidelines that may be useful in diagnosing problems in new and existing installations. This is not an exhaustive list and does not replace the need to involve VDURA Technical Support.

This chapter contains the following sections:

• Collect Logs
• Send Logs
• Log Analysis
• Diagnostic Utilities
• Configure the Maximum Transmission Unit

9.1 Collect Logs

9.1.1 sysstat all

The sysstat all command provides an overall VDURA system summary. Examples of output include: Director Node CPU utilization, Storage Node disk and CPU utilization, node status, Storage Set information, volume information, network configuration, gateway services, and current warnings and errors.

[pancli] sysstat all > /tmp/sysstat_all_{SR-XXXXXXX}

You can also get this data by clicking the Download System Summary button on the Status & Reports page on the PanActive Manager web interface. VDURA Technical Support may ask you to save the output as a text file and attach it to a Service Request.

9.1.2 collect_logs

The collect_logs command collects logs commonly used to diagnose issues from specified nodes. VDURA recommends to always collect logs from all repset nodes in addition to any other nodes of interest. In the event of a PanActive Manager outage, the repset option will continue to find the correct nodes to collect from. The host option followed by an IP address should also always work.

collect_logs ?[option] [option] ...?

Below is a list of all command options.

Option	Description
age [count] [units]	Do not collect /var/log/*.[0-9]+ files and corefiles that are more than count units old, where valid units are minute(s), hour(s), day(s), week(s), month(s). Default is one week. If the match option is specified, it defaults to unlimited age. If the count is 0, age is not measured.
all	Collect logs from all nodes in the system. This option is only available when collect_logs is run on a Director in the replication set.
storageset [name]	Collect logs from all nodes in the named StorageSet. You can use this option multiple times to specify multiple StorageSets. This option is available only when running collect_logs on a Director in the replication set.
container	Collect logs from all VDURA Data Platforms in the system. This option is only available when collect_logs is run on a Director in the replication set.
directors	Collect logs from all Directors in the system. This option is available only when running collect_logs on a Director in the replication set.
enclosure [name]	Collect logs from all nodes in the named enclosure. You can use this option multiple times to specify multiple enclosures. This option is available only when running collect_logs on a Director in the replication set.
enclosuremaster	Collect logs from nodes with the enclosuremaster role. One enclosuremaster node in each enclosure controls the enclosure hardware. This option is only available when the command is run on a Director in the replication set.
fileowner [PanFS file]\|[Object ID]	Collect logs from the nodes that store a portion of the given PanFS file or Object ID, as well as the primary and backup Directors for the file or object’s volume. Owners of multiple files can be specified by using this option more than once.
host [name]	Collect logs from the node with this name, IP address, or serial number. You can use this option multiple times to specify multiple nodes.
matching [file path pattern]	Collect only those files whose names match the specified glob-style pattern. Use a glob pattern that captures the full absolute path to the desired files (e.g. /var/log that have the substring http somewhere in their name. Multiple patterns can be specified by using this option more than once.
nocores	Do not collect any corefiles or executables. Stack files are always gathered.
nodbs	Do not collect any system databases.
nodiags	Do not run any diagnostic tools before collecting logs.
panfs [dir]	Copy the output file to the given PanFS directory. The directory will be created if it does not already exist.
range [ip]-[ip]	Collect logs from nodes in this range of IP addresses. You can use this option multiple times to specify multiple ranges. This option is available only when running collect_logs on a Director in the replication set.
repset	Collect logs from all Directors in the replication set. This option is available only when running collect_logs on a Director in the replication set.
scp [user] [host] [dir]	Scp the output file to the given user, host, and directory arguments. You will be prompted to enter a password. The target directory must exist on that host already.
storage	Collect logs from all Storage Nodes in the system. This option is only available when the command is run on a Director in the replication set.
storagemodules	Collect logs from all storage modules in the system. This option is only available when collect_logs is run on a Director in the replication set.
timeout [seconds]	The timeout for scp in seconds. The default is 900 seconds (15 minutes). Use 0 for no timeout.
upload [user]	Upload the output file to upload.vdura.com as the given user. You will be prompted to enter a password.
volumeowner [volume name]	Collect logs from the Director that owns the named volume. If Volume Services Failover is enabled, also collect logs from the backup Director for the given volume. You can use this option multiple times to specify multiple volume owners. This option is available only when running collect_logs on a Director in the replication set.
waitsavecore	Wait until savecore is no longer running before collecting logs.

Type help collect_logs [subcommand] to see more details for individual subcommands.

Command Examples

Gather logs for all nodes in VCH-2 and VCH-5, as well as all nodes in the replication set:

collect_logs enclosure VCH-2 enclosure VCH-5 repset

Gather logs for the enclosure master nodes in VCH-2 and VCH-5, as well as all nodes in the repset:

collect_logs enclosuremaster enclosure VCH-2 enclosure VCH-5 repset

Gather logs for all Storage Nodes in the StorageSet Set 2, as well as all nodes in the repset:

collect_logs storageset "Set 2" repset

Gather logs excluding corefiles for all Directors in VCH-4, as well as all nodes in the repset:

collect_logs nocores directors enclosure VCH-4 repset

Gather logs for both primary and backup Directors for the /home and /build volumes, as well as all Directors in the repset:

collect_logs volumeowner home volumeowner build repset

Gather logs for the Storage Nodes that cover the file /home/myFile, both the primary and backup Directors for the /home volume, and all Directors in the repset:

collect_logs fileowner /home/myFile repset

Gather all core/stack files that are less than two days old for the given node:

collect_logs host VCH-1,3 match /var/corefiles/* age 2 days

By default, collect_logs runs with a timeout value of 900 seconds, or 15 minutes. This timeout is often not adequate for logs from many hosts or for large logs with corefiles. To allow the command to run longer, use a long timeout value such as 3600 (one hour) or 7200 (two hours) for large logs collections.

You can also specify timeout 0 to specify running with no timeout. If timeout 0 is used, collect_logs will run until it either completes or the user ends it with CTRL+C. If you’re collecting logs from a host that you think may not respond, consider not using the timeout 0 parameter.

The example below collects files needed to read a corefile; if the nocores option is used, these files will not be collected:

[pancli] collect_logs timeout 3600 host {node} waitsavecore age 2 days

Note the longer value to prevent a timeout while the command is running.

The example below collects logs for a core where the core sweeper should have already had time to run:

[pancli] collect_logs timeout 3600 host {node} age 2 days

The age 2 days option instructs the command to only gather the previous two days of data. This covers the date of the crash so you get cores from that date based on error messages telling you what day the crash occurred without extra data from prior to then.

In the following examples, host {node} is used to specify a particular node or nodes by either name or IP address:

[pancli] collect_logs host {node} age 12 hours
[pancli] collect_logs host {node} host {node} age 2 days

In the following examples, the parameter directly after collect_logs is used to specify a particular group of nodes:

[pancli] collect_logs repset age 2 days
[pancli] collect_logs directors age 12 hours
[pancli] collect_logs storage age 2 days
[pancli] collect_logs storageset "Set 1" age 2 days
[pancli] collect_logs enclosure {enclosure} age 2 days

In the following examples, the parameter directly after collect_logs is used to specify volume owner, file owner, or enclosuremaster nodes.

[pancli] collect_logs volumeowner {volume_name} age 2 days
[pancli] collect_logs volumeowner {volume_name} age 12 hours
[pancli] collect_logs fileowner {/volume_name/path/to/file or object_ID} age 12 hours
[pancli] collect_logs enclosuremaster age 4 days
[pancli] collect_logs enclosuremaster enclosure {enclosure} age 4 days

Note: The fileowner option does not work on fenced objects.

For a failed or stuck reconstruction, use the age argument to go back to a time shortly before when reconstruction failed:

[pancli] collect_logs directors match /var/log/all* age 12 hours

If the volume has snapshots, also get the realm database:

[pancli] support servicemode yes
[pancli_service] diag_exec -timeout 1000 "pan_dump_realm" > /tmp/rmdb_{SR-XXXXXXX}

Use the match option to collect files that match a specified pattern:

[pancli] collect_logs all match /var/corefiles/*stack match /var/crash/*stack age 2 days
[pancli] collect_logs host {node} match /var/log/all* age 8 months
[pancli] collect_logs match /var/log/diag/driveS*.log match /var/log/all.log directors

Linux node health check only:

[pancli] collect_logs match /pan/var/log/diag/driveS*.log match /pan/var/log/all.log storagemodules

Log collections containing match report an error if the node does not have matching files; this is normal.

If logs fail to collect, ping the node. If the ping succeeds, try to ssh into the host and collect logs locally.

9.2 Send Logs

Once you have collected logs from a server, you may want to send them to VDURA for analysis. Below are scp commands to move log files from a server to your client.

Pull to client from system:

1. Log in to the client.
2. Use the following command to copy a file from a server:

scp username@remote:/file/to/get /where/to/put

Example:

scp admin@serv1.test_lab.vdura.com:/tmp/vol-list ./vol-list

Push from system to client:

1. Log in to the VDURA system.
2. Use the following command to copy a file to your client:

scp /file/to/send username@remote:/where/to/put

Example:

scp /var/tmp/vol-list auser@serv1.vdura.com:/fs/home/auser/vol-list

9.3 Log Analysis

The VDURA system actively monitors the network and keeps a historical record of events. This information is captured in system logs. If a suspected network problem is occurring, the net_mon.log, all.log, and sysstat.log logs described on the following pages are your best chance to determine root cause.

When looking at logs, be aware that, compared to other VDURA components, the net_mon.log and all.log files for Storage Nodes are located in /pan/var/log/. Furthermore, this folder will be empty until the PanCLI command journal.ctl is used. When run, this command finds all the relevant logs and places them in /pan/var/log.

9.3.1 /var/log/net_mon.log

Every ten minutes, each node in the VDURA system polls vital statistics from the network interfaces and logs them in net_mon.log. Results for ifconfig, netstat -r, and netstat -s are stored here.

ifconfig

Though an administrator should never manipulate the Ethernet interfaces on the node using ifconfig, the output of the command is useful. From this output you can see the state of the link, the effective MTU, and how failover is configured.

The example below is from a Director Node. The laggproto is set to lacp, so cc0 and cc1 are present and active. The two physical interfaces cc0 and cc1 do not have IP interfaces bound to them; they are added to the lagg0 interface to accomplish bonding. From the lagg0 laggport information, we can tell the physical interfaces which are part of the bond. From the physical interfaces, we can tell that the links are active and their speed.

Also notice the igb0 and igb1 ports. VDURA field engineers use igb0, which is part of lagg2, as a service port during installation. If you are using IPMI on the node as this example is, that connection is made on igb1.

igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether ac:1f:6b:cf:8d:18
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether a4:bf:01:65:73:c1
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
cc0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=2ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP>
ether 00:07:43:4b:91:80
media: Ethernet autoselect (25GBase-CR <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
cc1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=2ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP>
ether 00:07:43:4b:91:80
hwaddr 00:07:43:4b:91:88
media: Ethernet autoselect (25GBase-CR <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=2ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP>
ether 00:07:43:4b:91:80
inet 10.65.244.11 netmask 0xffffff00 broadcast 10.65.244.255
laggproto lacp lagghash l2,l3,l4
laggport: cc0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: cc1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lagg1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=800000<>
ether 00:07:43:4b:91:88
laggproto failover lagghash l2,l3,l4
groups: lagg
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lagg2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether ac:1f:6b:cf:8d:18
inet 169.254.12.34 netmask 0xffff0000 broadcast 169.254.255.255
laggproto failover lagghash l2,l3,l4
laggport: igb0 flags=1<MASTER>
groups: lagg
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

netstat -r

The net_mon.log also contains the output of netstat -r. This will show you all of the selected routes active on the current node. This is most useful for routing problems.

[pancli] netstat -rn
Routing tables
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 10.17.221.1 UGS 0 1918 lagg0
10.17.221.0/24 link#6 UC 0 0 lagg0
10.17.221.1 00:01:e8:8a:f0:bf UHLW 2 0 lagg0 452
10.17.221.11 00:60:dd:45:bc:4e UHLW 1 15353 lo0
10.17.221.70 00:09:03:01:b1:5b UHLW 1 799 lagg0 599
10.17.221.72 00:09:03:01:ce:58 UHLW 1 793 lagg0 600
...
10.17.221.111 00:09:03:01:c6:db UHLW 1 789 lagg0 599
10.17.221.112 00:09:03:01:b4:81 UHLW 1 778 lagg0 598
10.17.221.113 00:09:03:01:bc:3b UHLW 1 743 lagg0 599
10.17.221.114 00:09:03:01:d2:38 UHLW 1 778 lagg0 599
127.0.0.1 127.0.0.1 UH 0 3488 lo0
255.255.255.255 00:00:00:00:00:00 UHSb 0 134 lagg0
Internet6:
Destination Gateway Flags Netif Expire
::1 ::1 UHL lo0
fe80::%lo0/64 fe80::1%lo0 U lo0
fe80::1%lo0 link#5 UHL lo0
ff01:5::/32 fe80::1%lo0 UC lo0
ff02::%lo0/32 fe80::1%lo0 UC lo0

netstat -s

An historical record of tcp statistics can help determine if a failure was network related or not. The netstat -s is the delta from the last record. If you see a value of 100, it is really 100 in the last 10-minute interval. If the difference between the two runs is 0, the value is not listed.

tcp:
11944 packets sent
4213 data packets (4683796 bytes)
18 data packets (5273 bytes) retransmitted
17 data packets unnecessarily retransmitted
4690 ack-only packets (1347 delayed)
596 window update packets
2006 control packets
11307 packets received
6808 acks (for 4682649 bytes)
1607 duplicate acks
5467 packets (6040202 bytes) received in-sequence
112 completely duplicate packets (14665 bytes)
596 window update packets
2 out-of-order packets (68 bytes)
97 packets received after close
629 connection requests
944 connection accepts
1573 connections established (including accepts)
1604 connections closed (including 100 drops)
286 connections updated cached RTT on close
286 connections updated cached RTT variance on close
282 connections updated cached ssthresh on close
6808 segments updated rtt (of 5909 attempts)
18 retransmit timeouts
212 keepalive timeouts
204 keepalive probes sent
448 correct ACK header predictions
913 correct data packet header predictions
944 syncache entries added
944 completed
944 cookies sent

Look for the following values to flag possible error situations:

• completely duplicate packets: These are packets that were retransmitted by the sender. This can indicate a lost ACK or reordered packets in the write direction.
• out-of-order packets: These are packets that were reordered in the network. This can be caused by non-TCP-flow-based ECMP or LAGs. It can also be caused by small amounts of packet loss in the write direction.
• retransmit timeouts: This is the number of times we gave up on a TCP segment after not receiving an ACK. This typically means a 50ms timeout has been incurred. You should not see these in high numbers. This typically signals excessive packet loss in the read direction.

9.3.2 /var/log/all.log

A number of messages listed in the all.log file have specific bearing on network activities. Following are examples of some of these messages.

This message indicates that the layer 2 stats are incrementing beyond a preset threshold:

cpu_mon: cc0: dot3StatsFCSErrors exceed threshold (178 > 1) during previous 10 minutes

This message shows the NIC on a node losing link. This could indicate a failure of the NIC hardware on the node:

Jul 19 11:31:21 01ab8923af0003 kernel: cc0: link state changed to DOWN
Jul 19 11:31:23 01ab8923af0003 kernel: cc0: link state changed to UP

Another key message to look for is the rioe error. This is short for Report IO Error. When a PanFS client is unable to communicate to a Storage Node, it returns a rioe to the metadata manager. These errors are logged and used to isolate network failures.

pan_mgr[796]: sm: 0x02008989187f0010(SM): rioe [1269490094:732398995]: client 10.10.0.37:42634, obj I-xD02008989187f0010-xGb3582637-xU6744b3f20396696d: client_op=OP_WRITE, vcap_range=0x5a000000:0x2000000, vop_range=0x5a400000:0x400000, handle=0x26088034b920d5fa, action=0xbf400000
pan_mgr[796]: sm: 0x02008989187f0010(SM): rioe [1269490094:732398995]: comps=0x6008d07187f0007, op_range=0x3780000:0x80000, cap_range=0x3700000:0x400000, osd_op=OP_WRITE (0x7e9 (RPC: No response within the timeout period))

As shown above, each error produces a stanza of two messages. The first reports the client address and the operation that failed. The second line shows which data components (comps) failed. Each comp ID includes the data serial number of the Storage Node involved.

9.3.3 /var/log/sysstat.log

This log only applies to Director Nodes. Every hour, the mserver node records the output of sysstat storage to disk, as shown in the following partial output:

OSD IP CPU Disk Op/s Resp
Address % % msec
east-asu-1,1 10.64.8.61 1 2 28 1.76 ...
east-asu-1,2 10.64.8.228 2 2 31 1.20 ...
east-asu-1,3 10.64.8.93 1 2 29 1.42 ...
east-asu-1,4 10.64.8.88 1 1 29 123.48 ...
east-asu-2,1 10.64.8.90 1 2 30 1.47 ...
east-asu-2,2 10.64.8.226 0 0 12 0.27 ...
east-asu-2,3 10.64.8.27 1 1 30 1.22 ...
east-asu-2,4 10.64.8.212 1 1 21 1.48 ...
east-asu-3,1 10.64.8.203 1 1 30 1.40 ...
east-asu-3,2 10.64.8.6 1 2 25 1.25 ...
east-asu-3,3 10.64.8.201 1 2 28 1.21 ...
east-asu-3,4 10.64.8.4 1 2 28 1.36 ...
test Total 1 1 321 1.27 ...

The sixth column is the latency of the node. In many cases, latency can increase due to disk utilization. However, if the disk and CPU utilization are low and the latency is high, the problem is likely network-related. In the example above, we see east-asu-1,4 10.64.8.88 has high node latency, but the third and fourth column all line up with the other nodes. This indicates that the latency discrepancy is from the network stack. If the fourth column also had an outlier number, it would indicate that the latency is coming from the disk.

9.4 Diagnostic Utilities

There are a number of network diagnostics that can be run on the system from PanCLI. Refer to the PanCLI Reference Guide for more information on each command.

netperf

Both a netperf server and client are available on every node of the VDURA system. By default, the server is disabled. When enabled, it operates on port 4005 and can be accessed from any standard netperf client:

[pancli_service] netperf enable
Successfully enabled netperf server on port 4005.

A pre-compiled netperf client is included in the panfs-apps RPM that can be downloaded and installed from the support.VDURA.com portal.

Netperf is a network bandwidth test. It shows the network transfer rate between two end points. It is a memory-to-memory transfer, so there is no disk I/O to affect the results:

~$: netperf -p 4005 -H 192.168.1.10
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.100 (192.168.1.100) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6 bits/sec
524288 16384 16384 10.65 60.36

It is also possible to run the client from a VDURA node:

[pancli_service] diag_exec "pan_netperf -p 4005 -H 192.168.1.100"
Mon Jul 18 06:28:54 GMT 2015: Executing "pan_netperf -p 4005 -H 192.168.1.100"...
Mon Jul 18 06:29:04 GMT 2015: Executed "pan_netperf -p 4005 -H 192.168.1.100" successfully:
TCP STREAM TEST from (null) (0.0.0.0) port 0 AF_INET to (null) (192.168.1.100) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
524288 524288 524288 10.01 916.84

pan_net_enclosure_nettest

Pan_net_enclosure_nettest is a service mode command for performing basic network connectivity and throughput tests between nodes and between nodes and external hosts.

Network connectivity tests include small and medium pings. Optional tests include a large ping test, ssh, and netperf throughput tests if netperf is enabled on the realm. The large ping test runs by default only if the system MTU is 9000. Test results are output for each node tested and summarized by enclosure.

If pan_net_enclosure_nettest is run with any of the following options, the results are summarized by enclosure in the summary log:

--all --encl --directors --storage --dms --dbs --sms -sbs

Results for each node are shown in the detail log.

If pan_net_enclosure_nettest is run either with the –repset option or with IP addresses, the results for those nodes are shown in the summary log.

Pan_net_enclosure_nettest prints the path and names of both logs to the screen when it completes.

The small (56 bytes) and medium (1472 bytes) ping tests are always run and may not be skipped. The large (8972 bytes) ping test is run if the realm is configured for MTU 9000. The large ping test may optionally be skipped via the –nolargeping option.

The netperf test requires that netperf be enabled on the realm. To enable netperf on the realm:

[pancli] netperf enable

Netperf may optionally be disabled once pan_net_enclosure_nettest has completed with the following:

[pancli] netperf disable

Netserver must be installed on any hosts whose IP addresses are supplied on the command line if those hosts are not nodes in the realm. Netserver must be configured to listen on port 4005.

Netperf results for the local enclosure will be unusually high. The test is executed across the enclosure backplane.

Command Examples

Run all tests to all nodes in the realm:

[pancli] support servicemode yes
[pancli_service] pan_net_enclosure_nettest --all

Run all tests to the repset Directors:

[pancli] support servicemode yes
[pancli_service] pan_net_enclosure_nettest --repset

Run all tests to all Storage Nodes in the realm, in addition to the client hosts 10.70.15.105 and 10.70.15.127.

[pancli] support servicemode yes
[pancli_service] pan_net_enclosure_nettest --storage 10.70.15.105 10.70.15.127

Run all tests to all nodes in enclosures VCH-1 and VCH-4 without the netperf test:

[pancli] support servicemode yes
[pancli_service] pan_net_enclosure_nettest --encl VCH-1 --encl VCH-4 --noperf

Run all tests to all Storage Nodes in StorageSet Set 2:

[pancli] support servicemode yes
[pancli_service] pan_net_enclosure_nettest --set 'Set 2'

Run a quick connectivity test to find out which enclosures are up:

[pancli] support servicemode yes
[pancli_service] pan_net_enclosure_nettest --all --nossh
--noperf --nolargeping --quick

Test only node 10.65.23.106. 10.65.23.106 may either be a node in the realm or an external host. No ssh test will be run:

[pancli] support servicemode yes
[pancli_service] pan_net_enclosure_nettest 10.65.23.106

Below is a list of all command options. At least one option specifying a group of nodes to test is required.

Option	Description
--help	Display this usage message.
--examples	Display explanation of pan_net_enclosure_nettest use and example command options.
--all	Test all nodes in the realm.
--encl [name]	Test all nodes in the specified enclosure. Multiple –encl arguments may be used.
--repset	Test the repset member Directors.
--directors	Test all Director Nodes in the realm.
--storage	Test all Storage Nodes in the realm.
--set	Test all Storage Nodes in the specified StorageSet. Use single quote marks to enclose a StorageSet name that contains a space, as in ‘Set 1’.
--offline	Include any nodes that are not marked online in the tests. By default, non-repset member nodes that are not online are excluded. Hosts specified on the command line are assumed to be online.
--quick	Run shorter ping and netperf tests. The number of pings is reduced from five to one for each of the small, medium, and large ping tests. The netperf test is reduced from ten seconds to one second.
--nolargeping	Skip the large ping test.
--nossh	Skip the ssh test. This test is automatically skipped for external hosts.
--noperf	Skips the netperf test.
--noquery	Do not query the realm to build a list of nodes to test. A space-separated list of node IP addresses to test must be provided. If –noquery is used, all options that specify types of nodes to test are disregarded. The large ping and netperf tests are run without querying the realm to see if netperf is enabled or MTU is set to 9000. The ssh test is not run. This option is useful if the repset is down.

ping

A simple ICMP ping to test connectivity can be issued from PanCLI. By default it sends three 64-byte ICMP pings:

[pancli_service] ping compute1
PING panasas1.example.com (172.17.2.234): 56 data bytes
64 bytes from 172.17.2.234: icmp_seq=0 ttl=61 time=1.454 ms
64 bytes from 172.17.2.234: icmp_seq=1 ttl=61 time=0.818 ms
64 bytes from 172.17.2.234: icmp_seq=2 ttl=61 time=0.813 ms
--- compute1.example.com ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.813/1.028/1.454/0.301 ms

Additional ping options (for example, to determine if the MTU is correct) must be executed through diag_exec:

[pancli_service] diag_exec "ping -s 9000 -c 3 compute1"
Tue Jul 19 16:44:53 GMT 2015: Executing "ping -s 9000 -c 3 compute1"...
Tue Jul 19 16:44:55 GMT 2015: Executed "ping -s 9000 -c 3 compute1" successfully:
PING compute1.example.com (172.17.2.234): 9000 data bytes
9008 bytes from 172.17.2.234: icmp_seq=0 ttl=61 time=7.729 ms
9008 bytes from 172.17.2.234: icmp_seq=1 ttl=61 time=8.991 ms
9008 bytes from 172.17.2.234: icmp_seq=2 ttl=61 time=8.972 ms
--- compute1.example.com ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 7.729/8.564/8.991/0.590 ms

tcpdump

Tcpdump is a standard UNIX/Linux utility used to capture raw packets from the network interface. It can be used to diagnose anything from simple connectivity problems to complex network protocol interactions.

Tcpdump is available directly from PanCLI. It can be attached to any interface including the LAG interfaces. By default, it will collect 100,000 packets and exit. That limit can be modified using the -c option.

Below is a simple command to monitor all DHCP traffic on a node:

[pancli_service] tcpdump port 9876 or port 9877

In the next example, the binary output is saved to a file. When invoked in this way, the system will prompt you to upload the file after the dump is terminated:

[pancli_service] tcpdump -w /tmp/trace.bin -s 0 host 172.17.2.234
This tcpdump will stop after collecting 100000 packets.
Use "-c count " option to specify a different packet count.
tcpdump: WARNING: bge0: no IPv4 address assigned
tcpdump: listening on bge0, link-type EN10MB (Ethernet), capture size 65535 bytes

traceroute

Traceroute can also be issued directly from PanCLI:

[pancli_service] traceroute compute1
traceroute to compute1.example.com (172.17.2.234), 64 hops max, 52 byte packets
1 10.72.3.1 (10.72.3.1) 0.690 ms 0.517 ms 0.519 ms
2 * ROUTER-0.lab.example.com (10.64.0.254) 103.784 ms *
3 compute1.example.com (172.17.2.234) 0.869 ms 0.518 ms 0.513 ms

Traceroute is commonly used to determine the actual path a packet will travel in a routed network.

9.5 Configure the Maximum Transmission Unit

Maximum Transmission Unit (MTU) in bridged environments can cause some confusion. The VDURA system supports both 1500- and 9000-byte IP MTUs for Ethernet and 2048- and 4096-byte IP MTUs for InfiniBand. Like MTU settings on hosts, this is the maximum size of the IP packet that will be generated from the host. However, once the packet is sent to the Ethernet interface, it becomes larger due to Ethernet headers encapsulating the packet. If VLAN tagging will be used on the LAN, you must also account for the 4-byte p/q tags.

Let’s use a 9000-byte IP packet as an example. To pass a 9000-byte IP packet, the Ethernet infrastructure must be able to pass a larger frame. Many newer switches allow you to specify the exact number of bytes per Ethernet frame. In these cases, a common misconfiguration is to specify 9000. If this is done, 9000-byte IP packets will be lost as they appear larger when encapsulated into Ethernet frames.

If the LAN infrastructure can support the largest MTU that can be generated, it is not necessary to make sure both the clients’ and storage servers’ MTU match. Part of the TCP connection process involves negotiating the segment size. This allows a 1500-MTU client to properly communicate to a 9000-MTU server. This can be done simultaneously, with a 9000-MTU client in the mix. The key is to make sure the infrastructure in between can handle the largest frame that will pass.

In the case of routed networks, there is the possibility of two LANs with different MTUs. When a packet arrives on a router and its destination is a LAN with a smaller MTU, the IP packet is split to accommodate the destination LAN. Fragmentation of packets can significantly impact performance and should be avoided at all costs.

To make sure the MTU settings are optimal, realms configured with an MTU of 9000 should have all switches configured with an MTU of at least 9018/9022 and routers must have an IP MTU of at least 9000. For realms configured with an MTU of 1500, all switches must have an MTU of at least 1518/1522 and routers must have an IP MTU of at least 1500.

A good test to ensure that MTU is sufficient from end to end is to use the following options on the ping command:

ping -M do -s 8972 <IP address>

The -M do tells ping to not allow the packets to fragment, and the -s specifies the size of packet to send. If a simple ping succeeds, but this format fails, there is likely some component in the path that is configured with an MTU too low to pass the packet without fragmenting.

To run a jumbo ping from one of the VDURA nodes, use the following command:

diag_exec "ping -D -n -s 8972 -c 4 <IP address>"

Use the Correct MTU Values

MTU values are commonly misconfigured. Determining the maximum value may be difficult, but the user interface may provide a valid range. Ethernet adds 18 bytes or 22 bytes with an 802.1q VLAN tag, so a switch MTU of at least 9018 or 9022 is required for 9000-byte IP packets. Larger frames might be necessary for support of MPLS, so switches will usually support larger MTU sizes needed to support those larger frames.

The most important issue is that the layer-2 switch MTUs are set higher than the endpoints which are injecting traffic into the network. From the perspective of the VDURA storage system, it is always safe to set the switch MTUs to the largest values supported on the individual switch, even though that might not be exactly the same as the other switches in the same broadcast domain, as long as that value is at least 9018.

If the switch infrastructure is simultaneously being used to transport other traffic which uses larger frame sizes, then the individual switches need to support those MTUs. Note that an MTU mismatch will lead to problems which manifest as either error packets or missing packets, depending on the switch. Store-and-forward architectures will discard the bad frame (incrementing appropriate counters), whereas a cut-through architecture will propagate the bad frame (also incrementing the appropriate counters), ultimately resulting in the end station also incrementing an error packet counter.