AWS S3 Command Builder

Amazon S3 CLI in depth: buckets, keys, storage classes, and CLI patterns

Amazon S3 (Simple Storage Service) was launched in March 2006 as the very first public AWS service, predating EC2 by months. It introduced the world to large-scale object storage — a third storage paradigm distinct from block (EBS, raw disks) and file (NFS, EFS). Instead of mounting a filesystem, you talk to S3 via a flat HTTP API: PUT, GET, DELETE, LIST. Two decades later S3 stores trillions of objects, serves tens of millions of requests per second per region, and underpins everything from Netflix's master tapes to your terraform state file.

The AWS CLI is the official command-line wrapper around the S3 API. Version 2 (released 2020) is the recommended distribution — installed as a self-contained binary, with auto-completion, SSO support, and a faster JSON parser. Version 1 (Python wheel) still works but is in maintenance mode.

Core concepts: bucket, key, region, storage class

• Bucket — top-level namespace, globally unique across all AWS accounts. Names are 3-63 characters, lowercase letters, digits, hyphens, and dots; must start and end with a letter or digit. DNS-compatible (used as bucket.s3.region.amazonaws.com).
• Key — the path of an object inside a bucket. Max 1024 bytes UTF-8. S3 has no real directories; logs/2026/05/27.log is a single key, but the console fakes folders by splitting on /.
• Region — geographical home of the bucket. us-east-1 (N. Virginia, default), sa-east-1 (São Paulo), eu-west-1 (Ireland), etc. Data never leaves the region unless you replicate it.
• Storage class — tier that trades retrieval speed for cost: STANDARD (default), STANDARD_IA (infrequent access, ~45% cheaper, retrieval fee), ONEZONE_IA (single AZ), GLACIER_IR (instant retrieval), GLACIER (minutes-hours), DEEP_ARCHIVE (~$1/TB/month, 12 h retrieval), INTELLIGENT_TIERING (auto-moves based on access).

High-level vs low-level: aws s3 vs aws s3api

The CLI exposes two S3 namespaces:

• aws s3 — high-level filesystem-style commands: cp, mv, rm, ls, sync, mb, rb. Use this for day-to-day uploads, downloads, and syncs.
• aws s3api — thin wrapper around the raw HTTP API. Every operation maps 1:1 to a method (put-object, get-object-acl, list-objects-v2, create-multipart-upload). Use this for fine-grained control: setting tags, lifecycle, replication, object lock, presigned URLs with custom expiry.

Common commands

# List objects
aws s3 ls s3://my-bucket/path/

# Upload a single file
aws s3 cp file.txt s3://my-bucket/uploads/

# Download a directory tree
aws s3 cp s3://my-bucket/logs/ ./logs/ --recursive

# Mirror a local folder, deleting remote files that vanished locally
aws s3 sync ./dist s3://my-bucket --delete --exclude '.DS_Store'

# Delete an object
aws s3 rm s3://my-bucket/old/file.log

# Create / remove a bucket
aws s3 mb s3://new-bucket --region sa-east-1
aws s3 rb s3://old-bucket --force

Useful flags

• --profile name — use a non-default credentials profile from ~/.aws/credentials.
• --region us-east-2 — override the default region for one call.
• --storage-class GLACIER — upload directly to a cold tier (saves a lifecycle transition).
• --sse AES256 or --sse aws:kms --sse-kms-key-id alias/my-key — server-side encryption.
• --acl public-read — legacy ACL; prefer bucket policies for new buckets.
• --metadata Key=Value — custom user metadata (stored as x-amz-meta-* headers).
• --dryrun — preview what sync/cp would do without touching S3.
• --exact-timestamps / --size-only — change how sync decides if a file is up-to-date.
• --exclude '*.tmp' --include 'logs/*' — combine patterns; order matters (later wins).

Performance and cost

The CLI automatically uses multipart upload for files larger than ~8 MB (configurable via aws configure set default.s3.multipart_threshold 16MB) and runs up to 10 parallel upload threads. For very large transfers, enable S3 Transfer Acceleration (uploads land at the nearest CloudFront edge) and bump max_concurrent_requests. For browser uploads, generate presigned URLs with aws s3 presign s3://bucket/key --expires-in 3600 — the user uploads directly to S3 without ever touching your server.

Cost has three drivers: storage ($0.023/GB/month STANDARD, $0.00099 DEEP_ARCHIVE), requests ($0.005/1000 PUT, $0.0004/1000 GET), and data transfer out (free to AWS in same region, $0.09/GB to the internet). Lifecycle rules can transition objects to cheaper tiers automatically — set them up on day one.

FAQ

cp vs sync — which one should I use? Use cp --recursive for one-shot copies. Use sync when re-running: it compares timestamps and sizes and only uploads changed files, making it much faster for incremental backups.

Can the same CLI talk to non-AWS S3 services? Yes — Wasabi, Backblaze B2, MinIO, and Cloudflare R2 all expose an S3-compatible API. Add --endpoint-url https://s3.us-west-002.backblazeb2.com and the CLI will route requests there.

How do I accelerate a multi-gigabyte upload? Increase the multipart chunk and concurrency: aws configure set default.s3.multipart_chunksize 64MB and default.s3.max_concurrent_requests 30. If the bottleneck is latency rather than bandwidth, enable Transfer Acceleration on the bucket.

How do I make an object public? Modern best practice is a bucket policy granting s3:GetObject to * on a specific prefix, combined with disabling S3 Block Public Access. The legacy --acl public-read still works but is blocked by default on new buckets since 2023.

Why does my aws s3 ls return nothing even though the bucket exists? Either your IAM identity lacks s3:ListBucket, or the bucket is in a different region than your default — try --region or check with aws s3api get-bucket-location --bucket name.

Is data encrypted at rest? Since January 2023, every new object is encrypted with SSE-S3 by default. For stricter control, switch the bucket to SSE-KMS with a customer-managed key — gives you per-key access logs and rotation.