Storage API: Access Key, Bucket Policy, and Presigned URL

You're getting a 403 Forbidden error with a SignatureDoesNotMatch body, but you've checked the key and secret ten times and they're correct. Or worse: aws s3 ls against your own endpoint works fine, but the same request from inside Python code or from another server doesn't respond. The problem is almost always one of three things: how the request is signed, the bucket policy, or the URL you've given the client. This article opens up exactly those three.

What S3 protocol compatibility means and where it breaks

When we say a service is "S3-compatible," it means its endpoint understands HTTP commands signed with AWS Signature Version 4. This is the official definition, not a marketing claim. In practice, it means you can use boto3, aws-cli, rclone, or s3cmd and just change the endpoint.

But compatibility isn't binary. Where it actually breaks:

• Virtual-hosted style vs. path-style. The URL https://bucket.example.com/key is not the same as https://example.com/bucket/key. If a DNS wildcard isn't configured for the subdomain, the first form returns a TLS error or NoSuchBucket.
• Multipart Upload. Some implementations don't support UploadPartCopy and only have UploadPart.
• Versioning and Object Lock. These are optional and not available everywhere.
• Signature headers. If a reverse proxy tampers with the Host or x-amz-date header, the signature breaks and that same SignatureDoesNotMatch appears.

Practical tip: first of all, test with aws s3api get-bucket-location --bucket mybucket --endpoint-url https://s3.example.com. If this works but your code doesn't, the problem isn't the protocol, it's your SDK.

Access key: why you can't see the secret and how to store it

An access key has two parts: an Access Key ID that's like a username, and a Secret Access Key that's shown only once. If you lose it, there's no way to recover it; you have to create a new key. This behavior is intentional, not a product flaw.

Three common mistakes in key management:

1. Putting the secret in an environment variable inside the Dockerfile. It can be read with docker history.
2. Using one key for all services. If one service leaks, everything leaks.
3. Not rotating keys. A key that's been working for two years is a hidden risk.

The right pattern: create a separate user for each service with least privilege, keep the secret in a secret manager or a file with chmod 600 permissions, and put key rotation on the calendar. If you're on cloud infrastructure, instance-based IAM roles are better than static keys, because the key never sits on disk.

A subtle point: many people think that because they haven't hardcoded the key in code and put it in .env, they're secure. But if .env has been committed to git, it's the same disaster. Check with git log --all --full-history -- .env.

Bucket policy: the difference between policy and ACL, and which to choose

There are two access control mechanisms: Bucket Policy (a JSON document attached to the bucket itself) and ACL (object-level access list). My recommendation is clear: turn off ACL and use only Bucket Policy. ACL is practically deprecated in AWS and managing it at scale is a nightmare. Go for ACL in only one case: when you have a legacy client that doesn't understand policy and only sends ACL.

A minimal policy for public read of a specific folder:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PublicReadAssets",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::mybucket/assets/*"
    }
  ]
}

Notice two things in this document. First, Principal: "*" means anyone on the internet. Second, assets/* means only that folder, not the whole bucket. If you change the star to mybucket/*, all your private files become public too. This is exactly what we see in data leaks.

And one warning: s3:ListBucket is different from s3:GetObject. If you only grant GetObject, no one can see the file list, but if they know the filename, they can download it. For sensitive files, this isn't enough.

Presigned URL: how to grant temporary access without making the bucket public

A Presigned URL is a URL signed with your secret key that's valid for a limited time. This way the bucket stays private but the user can download a specific file.

With AWS CLI:

aws s3 presign s3://mybucket/report.pdf \
  --expires-in 3600 \
  --endpoint-url https://s3.example.com

The number 3600 is seconds, meaning one hour. The maximum value in AWS Signature V4 is seven days (604800 seconds). If you want more, you need another mechanism like CloudFront Signed URL.

Here's the mistake people make: they store the presigned URL in the database, and after it expires, the user gets a Request has expired error and the team thinks the service is broken. A presigned URL is a short-lived token, not a permanent link. Generate it on the fly and send it.

Second point: the signature is sensitive to request headers and parameters. If you copy the URL and add a parameter, the signature breaks. Also, if the clock of the server generating the URL differs from the storage service's clock by more than 15 minutes, the signature is rejected. Check the clock with timedatectl status or chronyc tracking.

Choosing between public URL, presigned, and proxy

For public static files, a public bucket with a CDN is the best choice, because it eliminates server bandwidth costs and lowers TTFB. For everything else, presigned URLs. Choose a proxy only when you need precise logging or watermarking.

If traffic is picking up and you want to put a caching layer in front of storage without changing code, a content delivery network is the logical choice. And if you want to set up this whole flow on your own infrastructure, object storage is the starting point.

A few numbers and settings you'll actually run into

• The maximum size of an object in a single PUT request is usually 5 gigabytes. For larger, Multipart is required.
• Each Multipart part (except the last) is at least 5 megabytes. If you send smaller, you get an EntityTooSmall error.
• The maximum number of parts is 10000. With 5-megabyte parts, the practical ceiling is about 50 gigabytes; for larger files, increase the part size.
• The allowed clock difference for signatures is 15 minutes.
• By default, aws s3 cp automatically uses Multipart for files over 8 megabytes.

A subtle trap: if you leave a Multipart upload half-finished, the incomplete parts stay in the bucket and incur storage costs without appearing in the object list. Check with aws s3api list-multipart-uploads --bucket mybucket and set a lifecycle rule to clean them up automatically.

If you're working on Kubernetes and want to mount storage as a volume, managed Kubernetes makes this easier. For monitoring error rates and request latency, see the cloud monitoring and alerts guide; an alert on the 403 rate will reveal a policy problem much sooner than a user complaint.

Frequently asked questions

Why do I get a SignatureDoesNotMatch error despite having the correct key?

In ninety percent of cases, your system clock differs from the storage service's. V4 signatures are time-sensitive, and a difference of more than 15 minutes causes rejection. Check the clock with timedatectl status and enable NTP.

If the clock is correct, the next possibility is that a reverse proxy is modifying the Host or x-amz-* header. The signature is calculated over these headers.

What's the difference between a presigned URL and a public link?

A public link works for anyone, always. A presigned URL is only valid for the specific duration you set, and after that returns Request has expired. For sensitive files, always use presigned.

The maximum validity period in Signature V4 is seven days. For longer than that, you need to use a CDN signing mechanism.

Should I keep ACL enabled on the bucket?

No, unless you have a legacy client that only sends ACL. Bucket Policy is more readable, versionable, and auditable. ACL is applied individually at each object level and becomes nearly impossible to manage at scale.

Why are incomplete files taking up space in my bucket?

Because a Multipart Upload was left half-finished. These parts don't appear in the object list but are stored. Find them with list-multipart-uploads and set a lifecycle rule to delete them automatically after a few days.

Next step: create a separate key for each service, turn off ACL, and set an alert on the 403 error rate in your monitoring. These three things alone will eliminate most of the problems I've written about here.