14. S3

Note: S3 is configured with panctl or the API. Vantage does not manage S3 in this release, so this chapter has no screens.

If end-to-end encryption is enabled, S3 support is disabled (see 13.4).

This chapter shows a platform administrator how to set up, run and use the PanFS S3 service: enabling S3, gateway instances, S3 users, buckets, TLS, named access, monitoring, client tools and day-to-day operations. It describes PanFS 12.0.0. The commands and example output were run on a live realm.

14.1 Concepts

S3 runs per tenant. Each tenant has its own prefix, users, buckets, instances, certificate and network endpoints.

14.2 Logging In with panctl

panctl is installed on every director at /pan/bin/panctl. Run it as root.

The management API listens on port 8000 of one director, the one hosting the realm’s management services. On that director, panctl connects to http://localhost:8000 by default. On any other director, give that director’s address.

The password is the realm admin password.

# On the director that hosts the management API
printf '%s' "$ADMIN_PASSWORD" | panctl auth login --username admin --password-stdin

# From another director
printf '%s' "$ADMIN_PASSWORD" | panctl auth login http://<management-director-ip>:8000 --username admin --password-stdin

panctl auth whoami        # ROLES: platform-admin, IS PLATFORM ADMIN: true

The session is saved as a profile in ~/.panctl/profiles.toml. Use --profile <name> to keep several sessions.

Every S3 command in this chapter takes --tenant <short name or UUID>. The examples use a tenant whose short name is acme.

Confirmation. Commands that delete something ask for confirmation in a terminal. In a script or whenever there is no terminal, they refuse with this operation requires confirmation; pass --yes to proceed. The examples pass --yes.

API tokens for scripts. A token lets a script run panctl without a saved login:

TOKEN=$(panctl auth tokens create --name s3-automation --expires-in-days 30 --format 'jsonpath={.token}')    # shown once
echo "$TOKEN" | PANCTL_API_SERVER=http://<management-director-ip>:8000 panctl --token-stdin s3 get-config --tenant acme
panctl auth tokens list
panctl auth tokens revoke <token id> --yes

14.3 Deployment Options

14.3.1 Dedicated Tenant (Recommended)

A tenant created for S3 users (for example acme) has its own private network, the tenant subnet: - The gateways listen on the nodes’ addresses in that subnet. - Only hosts attached to the subnet can reach them.

This chapter uses this option throughout. A dedicated tenant needs an Ethernet network. Multi-tenancy is not supported on InfiniBand, so tenants cannot be created on a realm whose clients connect over InfiniBand (see the Note at the start of 9). The platform tenant (see 14.3.2) does not use a tenant subnet.

14.3.2 Platform Tenant

The built-in platform tenant can also serve S3 (see 14.17). Its gateways listen on the nodes’ own realm addresses, so any host that can reach the realm network can reach them.

14.3.3 Choices Within a Tenant

S3 is always served over HTTPS. Every instance listens on port 3000.

14.4 Prerequisites

This section gives the minimum non-S3 setup needed for S3. It does not cover these objects’ other options.

14.4.1 Tenant and Storage

panctl storagesets list                                   # note a storage set ID, e.g. 1
panctl tenants create --name acme --display-name "ACME Corporation" --email s3-admin@acme.example
panctl tenants assign-storagesets --tenant acme --storageset-id 1 --quota 200GiB

The tenant short name (--name) is fixed once created. It also appears in S3 host names and certificate names (s3-acme).

14.4.2 Tenant Subnet

Instances in a dedicated tenant need a subnet. The realm gives every node an address in it.

panctl subnets create --tenant acme --name acme-net --cidr 10.240.0.0/20
panctl subnets list --tenant acme      # ADDR COUNT = number of realm nodes that have an address

Use a range that does not overlap your other networks. It needs enough addresses for every realm node plus every client host.

14.4.3 Attaching Client Hosts to the Tenant Subnet

Every host that will use S3 for this tenant must be attached to the subnet.

1. Register the host. Pass its address on the realm network:

panctl subnets attachments create acme-net --tenant acme --underlay-ip 10.128.9.0 --hostname client-0 --format json

The response tells the host how to join:

{
  "overlay_ip": "10.240.8.1",
  "overlay_prefix": 20,
  "mtu": 1450,
  "segment_type": "vxlan",
  "segment_id": 12010160,
  "tunnel": {
    "mac": "9a:89:c2:d5:41:81",
    "udp_port": 4789,
    "vteps": ["10.128.1.1", "10.128.1.2", "10.128.2.2", "...one address per realm node..."]
  }
}

2. Configure the interface. On a Linux client, use the values from that response:

modprobe vxlan
ip link add acme0 type vxlan id 12010160 local 10.128.9.0 dev eth0 dstport 4789
ip link set acme0 address 9a:89:c2:d5:41:81 mtu 1450
ip addr add 10.240.8.1/20 dev acme0
ip link set acme0 up
for v in <every address in tunnel.vteps>; do bridge fdb append 00:00:00:00:00:00 dev acme0 dst $v; done

• dev eth0 is the interface that holds the --underlay-ip address.
• These commands do not survive a reboot. Make them persistent with your distribution’s network configuration.

3. Check it. ping 10.240.0.1 should get replies from a director.

14.4.4 Volume

Create the volume that S3 will serve. A volume is created inside the tenant you act in. To act in acme, add the global --as-tenant-admin acme option: It runs that single command inside acme on behalf of the platform administrator.

panctl volumes create --as-tenant-admin acme --name /s3data --storageset-id 1 --hard-quota 100GiB
panctl volumes list --tenant acme

14.5 Enabling S3

14.5.1 Realm Switch

panctl s3 enabled set true
panctl s3 enabled get

14.5.2 Tenant Configuration

Create the tenant’s S3 configuration with its prefix and turn it on in one step:

panctl s3 create-config --tenant acme --prefix /s3data --enabled true
panctl s3 get-config --tenant acme

ENABLED:           true
ACCESS LOG:        true
DEBUG LOG:         false
METRICS ENABLED:   true
DEFAULT PREFIX:    /s3data
DEFAULT OWNER:     uid:65534
DEFAULT GROUP:     gid:65534
DEFAULT DIR MODE:  755
DEFAULT OBJ MODE:  644
TLS:               {"ca_certificate":null,"certificate":"-----BEGIN CERTIFICATE-----...","mode":"realm-generated","private_key":"***"}
AGGREGATOR:        {"enabled":false}

Always pass --prefix together with --enabled true. create-config --enabled true without a prefix is accepted, but it leaves S3 enabled with no prefix, so instances cannot serve anything. Recover with set-config --enabled false, then --prefix, then --enabled true.

14.5.3 Settings Reference

Change settings with panctl s3 set-config --tenant acme <flags>. Only the flags you pass are changed.

The TLS, aggregator and DNS settings are covered in sections 14.9 and 14.10. They take effect without a restart.

When a change needs a restart, set-config prints a note:

note: the tenant's running S3 instances pick up access_log only after a restart ('panctl s3 instances stop-all' then 'start-all' or stop/start a single instance by iid)

Rules:

• The prefix must be an existing volume of the tenant or an existing directory inside one.
• To change the prefix, delete the tenant’s instances, set --enabled false, change --prefix, set --enabled true, then create and start the instances again (see 14.15). The prefix cannot be changed while tenant S3 is enabled. Stopping and starting the instances is not enough: They keep serving the old prefix until they are recreated.
• To disable tenant S3, stop all of its instances first.

14.6 Gateway Instances

14.6.1 Choose Nodes

panctl nodes list

NAME       DATASERIAL      SHELF ID           IP ADDRESS    STATUS  ROLE   ...
VCH-1,1    b022963e7165c9  3e7e0000404001001  10.128.1.0    online  qnode
VCH-5,1    32cdb93e7165fa  3e7e0001404001001  10.128.1.1    online  qnode
VCH-9,1    409d673e7165cc  3e7e0002404001001  10.128.1.2    online  qnode
VCH-2,1    94c5393e716630  3e7e00ff404001001  10.128.2.0    online  vosd
...

Where instances can run: - Instances run on directors (ROLE qnode). - Do not place instances on storage nodes (ROLE vosd). Such an instance is accepted but never starts. It also cannot be removed, so it blocks removing S3 from the tenant.

Limits: - A tenant can have at most one instance per node. - Different tenants can use the same node.

14.6.2 Create and Start

The iid is exactly 5 letters or digits. The same iid can be created on several nodes. Commands that name only the iid then act on every node that holds it.

panctl s3 instances create gw001 b022963e7165c9 --tenant acme     # iid, node DATASERIAL
panctl s3 instances create gw001 32cdb93e7165fa --tenant acme     # same iid on a second director
panctl s3 instances create gw004 409d673e7165cc --tenant acme     # another iid on a third director
panctl s3 instances start gw001 --tenant acme                     # starts gw001 on both nodes
panctl s3 instances start gw004 --tenant acme
panctl s3 instances list --tenant acme

INSTANCE ID  DATA SERIAL     DESIRED  OBSERVED
gw001        32cdb93e7165fa  started  running
gw001        b022963e7165c9  started  running
gw004        409d673e7165cc  started  running

A new instance is created and must be started explicitly. It usually reaches running within 30 seconds. To wait for it:

panctl s3 instances get gw001 --tenant acme --wait-for-observed=running

14.6.3 Endpoints

Endpoints appear in the JSON output:

panctl s3 instances list --tenant acme --format 'jsonpath={.instances[*].endpoint}'
# https://10.240.0.2:3000 https://10.240.0.1:3000 https://10.240.0.3:3000

Each endpoint is the node’s address on the tenant subnet, port 3000. panctl s3 instances get <iid> --tenant acme --format json shows each placement with its endpoint, observed state and health.

14.6.4 States and Health

Automatic recovery. If a gateway process exits, the node restarts it within seconds, and restart_count_60s counts the restart. After several exits in a short time, OBSERVED shows crashloop and the gateway stays down for about a minute before the node starts it again.

Availability. An instance stays on the node it was created on. For availability, run the tenant’s instances on several directors, for example the same iid on two or three directors. Then publish one DNS name per iid with an address record for each director hosting it (see 14.10.2), so clients are not tied to a single node.

14.6.5 Stop, Start, Delete

panctl s3 instances stop gw004 --tenant acme
panctl s3 instances start gw004 --tenant acme
panctl s3 instances stop gw001 --dataserial 32cdb93e7165fa --tenant acme   # one placement only
panctl s3 instances stop-all --tenant acme
panctl s3 instances start-all --tenant acme
panctl s3 instances delete gw004 --tenant acme --yes
panctl s3 instances delete gw001 --tenant acme --confirm-fanout --yes    # the iid is on more than one node
panctl s3 instances delete-all --tenant acme --yes

Deletion finishes in the background. To wait until an iid is gone:

panctl s3 instances list --tenant acme --wait-for 'it == null' \
  -o 'jsonpath={.instances[?(@.desired.instance_id=="gw004")].desired.instance_id}'

14.7 S3 Users

S3 users hold the S3 credentials. Each user has:

• a name, unique in the tenant;
• an access key, which defaults to the name;
• a uid/gid that files created through S3 are owned by. If omitted, the tenant’s --owner/--group defaults apply at creation time (65534 unless changed);
• an S3 profile:

Create and list:

panctl fs-users create --tenant acme --name alice --uid 2001 --gid 2001 --s3-profile privileged
panctl fs-users create --tenant acme --name bob
panctl fs-users list --tenant acme

ID                                    NAME   UID    GID    PROFILE     ACCESS KEY
789b3eb6-3eda-4fe6-a8c3-a72e3997b4dc  alice  2001   2001   privileged  alice
15bc5774-0c37-4f89-87c5-e9fed641189d  bob    65534  65534  user        bob

Read the secret. It is not printed at creation:

panctl fs-users s3-secret 789b3eb6-3eda-4fe6-a8c3-a72e3997b4dc --tenant acme      # ACCESS KEY / SECRET KEY

Rotate the secret. The access key stays the same. The new secret is printed once, and the old secret stops working. panctl notes that a gateway may keep accepting the old secret for a short time.

panctl fs-users rotate-s3-secret <fs_user_id> --tenant acme

Disable and re-enable, change identity, delete:

panctl fs-users update <fs_user_id> --tenant acme --s3-profile disabled
panctl fs-users update <fs_user_id> --tenant acme --s3-profile user
panctl fs-users update <fs_user_id> --tenant acme --uid 2002 --gid 2002
panctl fs-users delete <fs_user_id> --tenant acme --yes

Deleting a user does not delete its buckets: They remain, still listed with that owner. Empty and delete them first (see 14.16).

14.8 Buckets

Create. An administrator creates a bucket for an owner, given by the owner’s access key. At least one instance of the tenant must be running.

panctl s3 buckets create projects --owner alice --tenant acme
panctl s3 buckets list --tenant acme

NAME      OWNER
projects  alice

Bucket names follow the S3 rules: 3–63 characters, lowercase letters, digits, - and ..

Buckets for user-profile users. --owner accepts a user with any profile. A user-profile user cannot create buckets over S3 (AccessDenied), so create its buckets this way:

panctl s3 buckets create bobs --owner bob --tenant acme

As the owner, the user has full control of the bucket over S3: It reads and writes objects, sets the bucket policy and ACLs, and can delete the empty bucket.

Buckets created by S3 users. Users with the privileged profile can also create buckets themselves (aws s3 mb). These buckets appear in panctl s3 buckets list like any other, and the administrator can delete them.

Delete. Any bucket of the tenant can be deleted, whoever created it. The bucket must be empty. Objects are removed over S3.

panctl s3 buckets delete projects --tenant acme --yes

panctl s3 buckets commands are carried out by a running instance of the tenant. Without one they fail (see 14.18).

14.9 TLS Certificates

All S3 endpoints use HTTPS. Each tenant has one certificate, used by all of its instances (and by the aggregator, if enabled). Certificate changes take effect within seconds, without restarting instances.

14.9.1 Tenant-Provided Certificate (Recommended)

You supply a certificate issued by a CA your S3 clients already trust, and clients verify the endpoint normally.

Requirements

The certificate is checked when you submit it. If it does not meet these requirements it is refused with a message, and the current certificate stays in use. For example:

tls.certificate carries a RSA-1024 key, which ... Accepted key types: RSA (at least 2048 bits), ECDSA P-256, P-384 or P-521, or Ed25519.
tls.private_key does not match tls.certificate.
tls.private_key is not a parseable PEM private key: Password was not given but private key is encrypted

If the chain is incomplete: A certificate issued by an intermediate CA is accepted without the intermediate. Clients then fail to verify it, so always append the intermediates.

Names

The realm’s names for a tenant come from the tenant short name and the realm DNS zone. A realm-generated certificate always carries them (see 14.9.2). Include them in a tenant-provided certificate if clients use them.

Check the zone with pancli -c "dns zone" on a director, and set it with pancli -c "dns zone lab.example.com".

Without a realm DNS zone, do not rely on the realm names. Clients disagree on a wildcard over a single-label name such as *.s3-acme. The AWS CLI v2, boto3 and rclone accept gw001.s3-acme, but curl and other tools that use OpenSSL’s host name check refuse it with a hostname mismatch. Connect clients by IP address or by your own host names.

Using Your Own Host Names

Clients can use host names of your choice, for example s3.acme.example.org, whatever the realm DNS zone:

1. Publish the names in your DNS, each pointing to the endpoint addresses of the instances it should reach (see 14.6.3).
2. Put the names in the certificate’s SAN. The certificate needs no other names.
3. Point clients at https://<your name>:3000.

Example

1. Issue a certificate. This uses a lab CA; in production, use your organization’s CA. The certificate carries the realm’s names for zone lab.example.com, which section 14.10.2 publishes.

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out s3-acme.key
openssl req -new -key s3-acme.key -out s3-acme.csr -subj "/CN=s3-acme.lab.example.com"
cat > s3-acme.ext <<'EOF'
subjectAltName=DNS:s3-acme.lab.example.com,DNS:*.s3-acme.lab.example.com
extendedKeyUsage=serverAuth
keyUsage=critical,digitalSignature,keyEncipherment
EOF
openssl x509 -req -in s3-acme.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out s3-acme.crt -days 397 -extfile s3-acme.ext

2. Upload it. If an intermediate CA issued it, use cat s3-acme.crt intermediate.crt as the certificate.

jq -n --rawfile cert s3-acme.crt --rawfile key s3-acme.key \
  '{tls: {mode: "tenant-provided", certificate: $cert, private_key: $key}}' |
  panctl s3 set-config --tenant acme -f -

3. Verify from a client that trusts the CA:

aws --endpoint-url https://gw001.s3-acme.lab.example.com:3000 --ca-bundle ca.crt s3 ls

To replace the certificate, upload a new one the same way.

14.9.2 Realm-Generated Certificate (Default)

If you do not supply a certificate, the realm issues one: - it covers the names above and the tenant’s endpoint addresses; - it is valid for one year; - it is signed by the realm’s internal root CA.

Clients cannot verify a realm-generated certificate: The realm root CA is not shared outside the realm. Clients must therefore skip certificate verification (see 14.11.4). Use a tenant-provided certificate wherever clients need verified connections.

The certificate takes its names from the realm DNS zone at the time it is issued. After changing the zone, renew it.

Switch back to a realm-generated certificate or renew it before it expires (renewal is not automatic):

echo '{"tls":{"mode":"realm-generated"}}' | panctl s3 set-config --tenant acme -f -
panctl s3 tls renew --tenant acme

14.10 Aggregator and Named Access (DNS)

14.10.1 Aggregator (Experimental)

The aggregator is experimental at this stage; keep it off (the default) for regular use.

panctl s3 aggregator on --tenant acme       # experimental
panctl s3 aggregator off --tenant acme      # default

With the aggregator on: - every director serves the tenant on port 3001 of its tenant-subnet address, for example https://10.240.0.1:3001; - requests are forwarded to the tenant’s running instances, and stopped instances are skipped; - the per-instance endpoints on port 3000 keep working.

With a tenant-provided certificate, add the names or addresses clients use for the aggregator to the certificate (see 14.9.1).

14.10.2 Named Access with Your Own DNS (external)

To let clients use host names instead of IP addresses, publish the names in your own DNS:

panctl s3 dns external --tenant acme

PanFS publishes nothing in this mode. You create the records:

Getting the addresses: - Instance addresses: panctl s3 instances list --tenant acme --format json (endpoint field).

Getting the names right: - The table shows the realm’s names (see Names in 14.9.1). To use them, set the realm DNS zone to the domain you publish them in, and put them in the tenant-provided certificate. - Alternatively, publish host names of your own and put only those in a tenant-provided certificate (Using Your Own Host Names). The realm DNS zone then does not matter. - A realm-generated certificate covers only the names in the table.

Example with dnsmasq. Records for the realm above, as served by dnsmasq:

host-record=gw001.s3-acme.lab.example.com,10.240.0.1
host-record=gw001.s3-acme.lab.example.com,10.240.0.2
host-record=gw004.s3-acme.lab.example.com,10.240.0.3

Check from a client:

dig +short gw001.s3-acme.lab.example.com      # 10.240.0.1 10.240.0.2
aws --endpoint-url https://gw001.s3-acme.lab.example.com:3000 --ca-bundle ca.crt s3 ls
aws --endpoint-url https://gw004.s3-acme.lab.example.com:3000 --ca-bundle ca.crt s3 ls

Keep the records current when you add, move or delete instances.

Clients must use path-style addressing (https://gw001.s3-acme.lab.example.com:3000/<bucket>/<key>). Bucket names in the host name (virtual-hosted style) are not supported.

14.10.3 Realm-Published Names (panfs, Experimental)

panctl s3 dns panfs makes the realm publish the names itself:

• It is experimental and not recommended for production use.
• It is available only for the platform tenant. Any other tenant is refused with dns.mode=panfs is not supported for this tenant ... Use dns.mode=off or dns.mode=external.
• Clients can use the names only if they resolve through the realm’s DNS service.
• A tenant-provided certificate must carry the realm’s names (see Names in 14.9.1), including s3-<tenant>[.<zone>] with the aggregator on. Otherwise the change is refused with tls.certificate is missing SAN(s) the realm publishes with dns.mode=panfs.

panctl s3 dns off (the default) turns named access off; clients use the IP endpoints.

The current DNS mode is not shown by panctl s3 get-config.

14.11 Client Access

Clients connect to an instance endpoint (port 3000) with an S3 user’s access key and secret key.

Client settings: - Region: us-east-1. Requests signed for another region are accepted, but creating a bucket fails with InvalidLocationConstraint when the client asks for another region, as aws s3 mb does when the client’s region is not us-east-1. - Signature: Signature Version 4. Signature Version 2 is not supported. Requests signed with it are rejected: - header-signed requests with AccessDenied; - browser upload forms (see 14.12.4) with InvalidArgument; - presigned URLs with InvalidRequest. - Addressing: Path-style.

The examples use the AWS CLI v2 (installation), boto3 and rclone.

14.11.1 AWS CLI

# ~/.aws/credentials
[acme]
aws_access_key_id = alice
aws_secret_access_key = <secret key>

# ~/.aws/config
[profile acme]
region = us-east-1
endpoint_url = https://gw001.s3-acme.lab.example.com:3000
ca_bundle = /etc/pki/acme-ca.crt

export AWS_PROFILE=acme
aws s3 ls
aws s3 cp report.pdf s3://projects/docs/report.pdf
aws s3 cp big.bin s3://projects/big.bin              # large files are uploaded in parts automatically
aws s3 ls s3://projects --recursive --human-readable
aws s3 cp s3://projects/big.bin ./big.bin
aws s3 presign s3://projects/docs/report.pdf --expires-in 3600
aws s3 rm s3://projects/docs/report.pdf
aws s3 mb s3://alice-scratch                          # privileged users only
aws s3 rb s3://alice-scratch                          # bucket must be empty

Without a profile, pass the options on each call: --endpoint-url https://<address>:<port> --ca-bundle <file>.

14.11.2 Python (boto3)

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url="https://gw001.s3-acme.lab.example.com:3000",
    aws_access_key_id="alice",
    aws_secret_access_key="<secret key>",
    region_name="us-east-1",
    verify="/etc/pki/acme-ca.crt",
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
s3.put_object(Bucket="projects", Key="docs/a.txt", Body=b"hello")
url = s3.generate_presigned_url("get_object", Params={"Bucket": "projects", "Key": "docs/a.txt"}, ExpiresIn=3600)

Set signature_version="s3v4". Without it, boto3 creates presigned URLs that the gateway rejects (InvalidRequest).

14.11.3 rclone

# ~/.config/rclone/rclone.conf
[acme]
type = s3
provider = Other
access_key_id = alice
secret_access_key = <secret key>
endpoint = https://gw001.s3-acme.lab.example.com:3000
force_path_style = true
region = us-east-1

rclone --ca-cert /etc/pki/acme-ca.crt lsd acme:
rclone --ca-cert /etc/pki/acme-ca.crt copy ./data acme:projects/data

14.11.4 Connecting to a Realm-Generated Certificate

A realm-generated certificate cannot be verified (see 14.9.2), so turn verification off in the client:

aws --endpoint-url https://10.240.0.1:3000 --no-verify-ssl s3 ls

import boto3, urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
s3 = boto3.client("s3", endpoint_url="https://10.240.0.1:3000", verify=False, region_name="us-east-1")

Use a current boto3 release. With the older boto3/botocore packaged by SUSE Linux Enterprise 15 (boto3 1.26, botocore 1.29), verify=False was ignored and connections still failed certificate verification.

14.12 Supported S3 Features

14.12.1 Bucket Policies

Once a bucket has a policy, the policy alone decides access, including the owner’s. A policy that grants only public read locks the owner out of the bucket. Always include a statement for the owner:

{
  "Version": "2012-10-17",
  "Statement": [
    {"Sid": "PublicRead", "Effect": "Allow", "Principal": "*",
     "Action": ["s3:GetObject"], "Resource": ["arn:aws:s3:::pubread/*"]},
    {"Sid": "OwnerAll", "Effect": "Allow", "Principal": {"AWS": ["alice"]},
     "Action": ["s3:*"], "Resource": ["arn:aws:s3:::pubread", "arn:aws:s3:::pubread/*"]}
  ]
}

With this policy anyone can read objects anonymously, anonymous listing is refused, and the owner keeps full access. Deleting the policy makes the bucket private again.

14.12.2 Bucket ACLs

ACLs are off on a new bucket: Its object ownership is BucketOwnerEnforced, and PutBucketAcl is refused with AccessControlListNotSupported. The bucket owner turns ACLs on by changing the object ownership to BucketOwnerPreferred:

aws s3api put-bucket-ownership-controls --bucket shared --ownership-controls 'Rules=[{ObjectOwnership=BucketOwnerPreferred}]'
aws s3api put-bucket-acl --bucket shared --acl public-read      # anyone can list the bucket and read its objects
aws s3api put-bucket-acl --bucket shared --grant-read bob       # S3 user bob can list and read
aws s3api put-bucket-acl --bucket shared --acl private          # owner only again
aws s3api get-bucket-acl --bucket shared

To set an ACL when creating a bucket, pass the object ownership in the same request. Without it, the request is refused with InvalidBucketAclWithObjectOwnership, as on AWS:

aws s3api create-bucket --bucket shared2 --acl public-read --object-ownership BucketOwnerPreferred

What PutBucketAcl accepts:

A grantee that is not an S3 user of the tenant fails with InternalError. The AWS CLI retries InternalError a few times before it reports it.

A bucket ACL covers the bucket’s objects too. Objects have no ACLs of their own: GetObjectAcl and PutObjectAcl return NotImplemented, and an x-amz-acl header on an upload is ignored. A grant on the bucket therefore applies to every object in it. On AWS, a bucket ACL controls listing and object ACLs control reading.

14.12.3 Public Access

A bucket owner can make the bucket public with a public ACL (above) or with a bucket policy for "Principal": "*" (above). Every owner can, including a user-profile user that owns a bucket created for it with panctl s3 buckets create --owner.

The platform administrator has no setting that prevents this. There is no Block Public Access: PutPublicAccessBlock returns NotImplemented (below), and there is no tenant- or realm-wide equivalent. To limit public buckets, limit who owns buckets: Give the privileged profile only to users who may create buckets, and choose the --owner of buckets you create.

14.12.4 Browser Uploads (POST Object)

A browser can upload straight to a bucket with an HTML form that POSTs to https://<endpoint>/<bucket>. The form must be signed with Signature Version 4. boto3 creates the signed fields:

post = s3.generate_presigned_post(
    "projects", "uploads/photo.jpg", ExpiresIn=3600,
    Conditions=[["starts-with", "$key", "uploads/"], ["content-length-range", 1, 10485760]],
)
# post["url"]:    https://gw001.s3-acme.lab.example.com:3000/projects
# post["fields"]: key, policy, x-amz-algorithm, x-amz-credential, x-amz-date, x-amz-signature

The form sends every field of post["fields"], then the file field last. To test it without a browser:

curl --cacert /etc/pki/acme-ca.crt \
  -F key=uploads/photo.jpg -F policy=<policy> -F x-amz-algorithm=AWS4-HMAC-SHA256 -F x-amz-credential=<credential> \
  -F x-amz-date=<date> -F x-amz-signature=<signature> -F file=@photo.jpg \
  https://gw001.s3-acme.lab.example.com:3000/projects

• A successful upload returns 204. With the field success_action_status set to 201, it returns 201 and an XML body. With success_action_redirect, it redirects (303) to that URL, with the bucket, key and ETag added to the query string.
• A file larger than the content-length-range maximum is refused with EntityTooLarge. A field that breaks another condition, such as starts-with, is refused with AccessDenied.
• ${filename} in the key is not replaced with the uploaded file’s name: The object is stored under the literal key.
• A form signed with Signature Version 2 (fields AWSAccessKeyId and signature) is rejected with InvalidArgument: Bucket POST must contain a field named 'x-amz-algorithm'.

14.12.5 Object Lock

Object lock works without versioning. A bucket created with object lock is not versioned: GetBucketVersioning returns VersioningNotConfigured. On AWS, object lock always turns versioning on. Here each key has a single copy, and a lock protects that copy:

Applications that need versioned object lock cannot use these buckets, for example backup software that checks for versioning or writes new versions of locked keys.

aws s3api create-bucket --bucket records --object-lock-enabled-for-bucket
aws s3api put-object --bucket records --key r1.txt --body r1.txt \
  --object-lock-mode GOVERNANCE --object-lock-retain-until-date 2026-12-31T00:00:00Z
aws s3api get-object-retention --bucket records --key r1.txt
aws s3api put-object-legal-hold --bucket records --key r2.txt --legal-hold Status=ON
aws s3api put-object-lock-configuration --bucket records \
  --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=GOVERNANCE,Days=30}}'
aws s3api delete-object --bucket records --key r1.txt --bypass-governance-retention

Protections are enforced as soon as they are set:

• Bucket default retention protects every object in the bucket, including objects uploaded before it was set. Its period counts from when the default was set, not from each object’s upload. get-object-retention does not report it for an object without retention of its own (NoSuchObjectLockConfiguration). Remove the rule with put-object-lock-configuration --object-lock-configuration ObjectLockEnabled=Enabled.
• Bypassing GOVERNANCE retention needs the --bypass-governance-retention option (header x-amz-bypass-governance-retention: true) and a bucket policy that grants the caller s3:BypassGovernanceRetention. The bucket owner has no implicit right to bypass: With no such policy, the delete is refused. Once the policy grants it, the caller can also overwrite GOVERNANCE-protected objects without the option. The policy decides all access to the bucket, so include the owner’s other permissions in it (above).

Object lock is not a tamper-proof (WORM) guarantee:

• put-object-retention accepts an earlier retain-until date, for COMPLIANCE as well as GOVERNANCE, and needs no bypass permission. Once the new date passes, the object can be deleted. A COMPLIANCE default retention can likewise be removed from the bucket.
• It applies only to S3 requests. Over the filesystem (see 14.13), locked files can be changed and deleted with ordinary file permissions.

14.12.6 Not Implemented

The following bucket features are not available. Their API calls return 501 NotImplemented, so a client that tries to configure them gets an error and nothing is applied.

14.12.7 Server-Side Encryption

Data is stored unencrypted. The x-amz-server-side-encryption* headers on uploads (SSE-S3, SSE-KMS and SSE-C, for example aws s3 cp --sse AES256) are accepted and ignored: The upload succeeds, the file under the prefix holds the plain data, and an SSE-C object can be read back without its key. Do not rely on them.

14.13 S3 and File Access to the Same Data

S3 data is ordinary PanFS data under the tenant’s prefix:

Ownership. Files and directories created through S3 get the first of these that applies:

1. object metadata x-amz-meta-uid, x-amz-meta-gid, x-amz-meta-mode on the upload (uid/gid 0 is not accepted);
2. the S3 user’s uid/gid (set when the user is created; see 14.7);

and mode --obj-mode (files) and --dir-mode (directories). A bucket created with panctl s3 buckets create is owned by its owner’s uid/gid.

Reading the data as files. A host attached to the tenant subnet (see 14.4.3) can mount the tenant volume. The director list comes from panctl subnets mount-info acme-net --tenant acme:

mount -t panfs -o "rmlist=(10.240.0.1;10.240.0.2;10.240.0.3)" panfs://10.240.0.1/s3data /mnt/acme
ls -l /mnt/acme/projects/docs

-rw-r--r-- 1 2001 2001  11 Oct 12 11:40 small.txt
-rw------- 1 3001 3001   1 Oct 12 11:40 owned.txt      # uploaded with x-amz-meta-uid/gid/mode

Files written over the filesystem: - Files and subdirectories written inside an existing bucket appear as objects. Objects that did not come through S3 have no ETag. - A new top-level directory under the prefix does not become a bucket.

14.14 Monitoring and Logs

Metrics. With --metrics-enabled true (default), the gateways publish request counts, errors and throughput. VDURA does not ship Grafana. To view them, import the VersityGW S3 Metrics sample dashboard into a Grafana instance that you run yourself (see the PanFS Metrics Reference Guide). - In your Grafana, open the dashboard (path /d/panfs-s3/versitygw-s3-metrics). - Select the tenant by its UUID (panctl tenants list), then the instance. - Panels: Request rate (success and failed), error ratio, throughput, objects created and removed, failures by action and HTTP status.

Access log. With --access-log true (default), every request is logged on the node that served it, in /var/log/versitygw/access_<pid>.log, in S3 server access log format. With the experimental aggregator on, its periodic health checks also appear there, as 403 AccessDenied lines for /.

The access log setting takes effect after the instances are restarted.

14.15 Day-to-Day Operations

When realm S3 is turned off while tenants still have S3 enabled, panctl warns:

WARNING:                  1 tenant(s) still have S3 enabled at the tenant level; realm-wide disable does not turn them off.
TENANTS WITH S3 ENABLED:  acme

14.16 Removing S3 from a Tenant

1. Remove all objects and buckets.
   • Users empty their buckets over S3. They can also delete their own buckets over S3.
   • The administrator deletes the remaining empty buckets with panctl s3 buckets delete <name> --tenant acme --yes.
2. Delete the instances, then the configuration:

panctl s3 instances delete-all --tenant acme --yes
panctl s3 instances list --tenant acme --wait-for 'it == null' -o 'jsonpath={.instances[*].desired.instance_id}'
panctl s3 set-config --tenant acme --enabled false
panctl s3 delete-config --tenant acme --yes

3. Delete the S3 users with panctl fs-users delete <fs_user_id> --tenant acme --yes.

14.17 S3 on the Platform Tenant

The built-in platform tenant (short name platform) can serve S3 with the same commands, using --tenant platform.

Differences from a dedicated tenant:

• No subnet or client attachment. Endpoints are the nodes’ own realm addresses (e.g. https://10.128.1.0:3000), reachable from any host on the realm network.
• Volume. The volume is created without --as-tenant-admin, at the realm root. The platform tenant needs a storage set assignment first:

panctl tenants assign-storagesets --tenant platform --storageset-id 1 --quota 100GiB
panctl volumes create --name /s3platform --storageset-id 1 --hard-quota 50GiB
panctl s3 create-config --tenant platform --prefix /s3platform --enabled true
panctl fs-users create --tenant platform --name ops --s3-profile privileged
panctl s3 instances create pgw01 b022963e7165c9 --tenant platform
panctl s3 instances start pgw01 --tenant platform
panctl s3 buckets create ops-data --owner ops --tenant platform

• DNS. Named access mode panfs (experimental) is available (see 14.10.3).

14.18 Troubleshooting