Cloud & Infrastructure

Object Storage: Cost, Access, and File Migration

If your server disk is full of uploaded files, this guide clarifies the real decision: cost, public or signed access, and the safe migration path.

Cloud & Infrastructure

Your server disk is full and df -h shows a number close to 100% on the / partition. The largest chunk belongs to the folder where users upload photos and videos. If you've been keeping files alongside the application code until now, you've reached the point where you need to decide: enlarge the disk or move the files to object storage. This article is about that decision, not about defining the technology.

Why enlarging the disk isn't a permanent answer

Enlarging the disk is the fastest route and works for a few months. The problem is that every time you enlarge it, the backup grows too. A server with 500 GB of uploaded files generates 500 GB of backup traffic every night, and if the backup lives on the same infrastructure, both cost and recovery time go up. In practice, I've seen a team that adds disk once a month end up, after a year, with a server whose full recovery takes several hours.

Object storage changes this pattern. The file is separated from the compute machine, is accessible via a URL or key, and versioning and lifecycle are managed by the service itself. But it isn't free, and if you don't choose correctly, its cost can exceed that of the disk.

The cost model: what you don't see on the first invoice

Object storage cost has three components: stored volume, number of requests, and egress traffic. The first component is predictable. The second and third are not, and that's where the biggest mistakes happen.

A real example: a site with 200 GB of product photos, if it serves images directly from the bucket to the user, generates several terabytes of egress per month. If the same images are served behind a CDN, a large portion of this traffic is cached and the egress cost drops noticeably. That's why in a correct architecture, the bucket is the source and the content delivery network is the delivery path to the user.

The second component, the number of requests, becomes expensive when your code fetches a full listing on every visit. If the application calls ListObjectsV2 on every page load to check whether a file exists, the request count rises with site traffic, not with file volume. You see this in the access log, not in the volume dashboard.

A practical comparison of three options

OptionDominant costWhere it worksWhere it fails
Local server diskDisk and backup volumeSmall files, frequent access, single-server applicationEndless growth, heavy backup, hard horizontal scaling
Object storageVolume + requests + egressLarge files, archive, media, backupSmall files with frequent random access
Attached block diskAllocated volumeDatabases, files requiring high IOPSSharing between multiple servers, cheap archive

My choice for user-uploaded files is object storage. For databases and temporary processing files, block disk. If your application reads several million small files under 10 KB with random access per day, object storage is a bad choice and the same local disk works better.

Public versus signed access

The second question, which almost always follows the cost question: should I make the bucket public or not? The short answer is that the default should be private, and only what truly needs to be public should be made public.

In a private bucket, access is done via a signed URL. With the AWS CLI it's generated like this:

aws s3 presign s3://my-bucket/uploads/report.pdf --expires-in 3600

The number 3600 seconds means one hour of validity. You can give this URL to the user without exposing the access key. If you work with an SDK, the equivalent is getSignedUrl with the same expiry parameter.

A common error you see in the log is:

AccessDenied: Access Denied
status code: 403

and it usually means you've written the bucket policy wrong, not that the file doesn't exist. A minimal policy for public read of just one folder:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": "*",
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::my-bucket/public/*"
  }]
}

Note that Resource is restricted to public/*. If you write my-bucket/*, everything becomes public, including the database backup you put there last week.

This is where they go wrong

The real mistake I've seen several times: for convenience, the team makes the bucket public so that image links work in the admin panel. A few months later, an automated scanner reads the bucket listing and internal files, including database dumps, get indexed. The sign isn't in the access permissions either; it's in the egress traffic bill that suddenly multiplies. If you see this pattern in the dashboard, check the bucket policy first, then the CDN.

Migration without downtime: a practical path

Don't do the migration overnight. The correct order is to first move the old, rarely accessed files, then write new files directly to the bucket, and finally change the read path.

  1. Take an initial copy with aws s3 sync /var/www/uploads s3://my-bucket/uploads --exclude "*.tmp". This command only transfers the differences on subsequent runs.
  2. Change the application so it writes new files to the bucket and stores their path in the database. Old files are still read from disk.
  3. Write a read layer that tries the bucket first and, if not found, reads from disk. Keep this layer for a few weeks.
  4. After you're confident, make the disk read-only and eventually delete it.

Don't skip the third step. That layer is what, if the sync was incomplete, returns the file from disk to the user instead of a 404 error. For database backup you need a different pattern too: put the mysqldump output in a separate bucket with the date in the filename, and define a lifecycle for that bucket so old versions are deleted automatically.

Where this decision connects to a larger infrastructure

Once the files are separated from the server, the server is no longer stateful and you can rebuild or scale the machine without worrying about losing uploads. This is exactly the condition that makes horizontal autoscaling practical; without it, every new instance has an incomplete copy of the files.

If you work on Kubernetes, the correct pattern is using PersistentVolume for the database and a bucket for media. Combining these two with managed Kubernetes makes rebuild and recovery easier, because the bucket itself lives outside the cluster.

For backup and recovery, do a real test: download a 5 GB file from the bucket and measure the time. If recovery takes several hours, the problem is in the architecture, not the service. Record this number as a metric in your monitoring and alerting so you know when it slows down.

If you're still choosing a platform, object storage is a logical starting point and the rest of the machines can stay on cloud infrastructure (IaaS). For projects where file volume is still small, the same cloud server with sufficient disk is a simpler choice and postpones migration to later months.

Frequently asked questions

Can I serve site files directly from object storage?

Yes, but for public and high-traffic files it's better to put a CDN in front of the bucket. Serving directly from the bucket works, but egress traffic cost and latency for users far from the bucket's region increase.

For private files, always give a signed URL with a short expiry and don't store it in the database, because the link is dead after expiry.

How do I know whether object storage is more expensive than disk for my site?

Put three numbers side by side: total file volume, monthly egress traffic, and request count. If egress traffic is high, the CDN makes the difference. If the request count is high but file volume is low, disk is probably the cheaper choice.

Collect one month of access logs and then decide. Deciding based on guesswork usually ends up costing more with the bucket.

Is object storage suitable for database backup?

Yes, and one of its best uses is exactly this, provided you define a lifecycle. Without it, old versions accumulate and the volume cost rises for no reason.

Build the filename with date and time and set the retention policy to, say, 30 days. Also actually test recovery once a month.

How long does file migration take and will the site go down?

Migration time depends on file volume and bandwidth, and with aws s3 sync you can do it in several stages. Downtime isn't necessary if you enable the dual read layer before changing the write path.

The safe order is: initial copy, new writes to the bucket, dual read, and finally disk deletion. Test each stage separately.

The next step is simple: tonight, create a private bucket, apply the minimal policy, and migrate only the uploads folder from the past month with sync. If that one month runs without problems, migrate the rest.

ServerNet Support

ServerNet engineering & editorial team — specialists in infrastructure, networking and web hosting.

Cloud Infrastructure (IaaS)
Share:

Comments 0

No comments yet — be the first!

Leave a comment

Related service

Cloud Infrastructure (IaaS)

Servers, private networks, firewalls and storage — all API-driven and billed hourly. Infrastructure as code that scales with you.