Setting up object storage

Why Is Object Storage Essential for Your Application?

When your application grows, storing files on the server's disk no longer works. Limited space, difficult backups, and slow concurrent access are three major problems you'll encounter quickly. Object storage solves these issues with a different architecture: each file is stored as an "object" with a unique identifier, and instead of a physical path, you access it via a URL and an access key.

In this article, you'll learn how to set up object storage from scratch: creating a bucket, generating access keys, security configurations, and finally connecting from a real application. You can run all the examples directly.

Step 1: Creating a Bucket — The First Step in Object Storage

A bucket is a container where objects are placed. The bucket name must be unique across your entire storage space because it becomes part of the file's final address. For example, if the bucket name is my-app-media, a file's address will look like this:

https://bucket.my-provider.com/my-app-media/images/profile.jpg

To create a bucket in the object storage management panel, you typically follow these steps:

1. Go to the "Buckets" section and click "Create New Bucket."
2. Enter the bucket name using lowercase letters, numbers, and hyphens (no spaces or special characters).
3. Select the geographic region — the closest one to your primary users.
4. Set access type to "Private" and configure it later with a Policy.

Setting the Access Policy (Bucket Policy)

After creating the bucket, you need to specify who has access to what. The simplest case is allowing public read access for a specific folder:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-app-media/public/*"
    }
  ]
}

This policy only allows read access to files inside the public/ folder. The rest of the files remain private. If you're working with S3-compatible services (like MinIO or Ceph), this JSON structure is directly usable.

Step 2: Generating Access Keys and Configuring Security

To connect your application to object storage, you need two values: Access Key and Secret Key. These act like a username and password and must be handled carefully.

In the management panel, you can usually create a new key pair from the "Access Keys" section. After generation, the Secret Key is only shown once — so save it in a secure place immediately.

Best Security Practices for Keys

• Do not place keys in application code or public files like .env that get committed to Git.
• Create a separate key for each environment (development, testing, production).
• Limit each key's access — for example, only to a specific bucket and only read operations.
• If a key is leaked, immediately deactivate it and create a new one.

Step 3: Connecting from an Application — Practical Example with Python

Now that the bucket and keys are ready, it's time to connect. For this example, we'll use the boto3 library, which is the standard for communicating with S3-compatible services. First, install the library:

pip install boto3

Then create a configuration file:

import boto3
from botocore.client import Config

s3 = boto3.client(
    's3',
    endpoint_url='https://your-storage-endpoint.com',
    aws_access_key_id='YOUR_ACCESS_KEY',
    aws_secret_access_key='YOUR_SECRET_KEY',
    config=Config(signature_version='s3v4'),
    region_name='us-east-1'
)

Now you can upload a file:

s3.upload_file(
    'local-image.jpg',
    'my-app-media',
    'public/profile.jpg',
    ExtraArgs={'ContentType': 'image/jpeg'}
)

And to download:

s3.download_file(
    'my-app-media',
    'public/profile.jpg',
    'downloaded-image.jpg'
)

Generating Temporary URLs for Private Files

If a file is private and you want to grant a user access for a limited time, use a presigned URL:

url = s3.generate_presigned_url(
    'get_object',
    Params={'Bucket': 'my-app-media', 'Key': 'private/report.pdf'},
    ExpiresIn=3600  # one hour
)
print(url)

This URL includes a digital signature and expires after one hour. It's perfect for displaying profile images or downloading invoices.

Step 4: Configuring CORS for Browser Access

If your application has a JavaScript frontend and wants to connect directly from the browser to object storage, you need to configure CORS. Without this setting, the browser blocks the request and you'll get a No 'Access-Control-Allow-Origin' header error.

In the management panel, find the CORS section and add these settings:

[
  {
    "AllowedOrigins": ["https://my-app.com"],
    "AllowedMethods": ["GET", "PUT"],
    "AllowedHeaders": ["*"],
    "ExposeHeaders": ["ETag"],
    "MaxAgeSeconds": 3000
  }
]

In this example, only the domain my-app.com is allowed access. For development, you can also add http://localhost:3000.

Step 5: Error Handling and Troubleshooting

When working with object storage, you'll definitely encounter a few common errors. Know them to resolve them quickly:

403 Forbidden Error

This usually means the access key is incorrect or the policy doesn't allow it. First, check the keys, then review the bucket policy. If you're using a temporary URL, make sure it hasn't expired.

404 Not Found Error

Either the file doesn't exist or the path (Key) is wrong. Note that the Key includes folder names. For example, if you uploaded the file to public/, the Key should be public/filename.jpg, not just filename.jpg.

SignatureDoesNotMatch Error

This error means your system's time is not synchronized with the server. Synchronize the time with the following command:

sudo ntpdate pool.ntp.org

Or on more modern systems:

sudo timedatectl set-ntp true

Summary and Next Steps

Setting up object storage isn't difficult, but you shouldn't take security and configuration details lightly. Summary of steps:

1. Create a private bucket and configure the policy precisely.
2. Generate separate access keys for each environment and keep them secure.
3. Use a standard library (like boto3) for connection.
4. For private files, generate temporary URLs instead of public access.
5. Enable CORS only for allowed domains.

If you're looking for a reliable cloud infrastructure for your projects, ServerNet offers various services in cloud and storage that can meet your needs. But regardless of the service provider you choose, the principles discussed in this article are the same across all S3-compatible services, and you can transfer your skills to any platform.

Now it's your turn: create a bucket, upload your first file, and enjoy how simple it is. If you encounter an error, review the troubleshooting section again — your answer is probably there.