All writing
6 min readNaren

Create a pre-signed URL for a Civo object store

Upload a file to a Civo object store with s3cmd, then hand out a link that works for exactly one hour and then stops.

InfrastructureToolingSecurity

Introduction

The civo object store is a managed service that is an S3-compatible system for storing unstructured data objects. All data objects created or uploaded are private by default. To share any specific data, you may have to make it public, but it creates a challenge as the object becomes accessible to anyone who has the URL. As time goes on, keeping track of who has that URL becomes its own problem. In most instances, you don't want to do that.

To rescue ourselves from this, you can make use of pre-signed URLs. Pre-signed URLs offer a flexible way to share objects with anyone by providing them with temporary access that has an expiry time that is decided by you.

In this guide, you will create a new civo object store in your account, then upload files to it, and create a pre-signed URL with an expiration time for that uploaded file in your civo object store. Later you will test the created pre-signed URL and then delete your object store.

Prerequisites

To successfully follow along, You will need an active Civo account, so if you have not signed up yet, create one now.

You will also need the listed command line tools installed in your machine to follow along successfully. They are:

  • Civo CLI: is a tool to manage and administer resources in your civo account. If you already have civo preinstalled, run the CLI command civo update to ensure you have the latest version.
  • s3cmd: for interacting with the object stores or another S3 client of your choice. This guide will assume you are using s3cmd.

This guide also assumes you have familiarity with the civo object store. If not, this getting started guide on using the Civo object store covers it.

Quick Setup

In this section, you will quickly create a new credential for the object store, creation of a new object store, and then upload a file to that object store which will become the premise for creating pre-signed URLs. You will be using civo CLI and s3cmd to perform all the actions. This guide assumes you have already set up civo CLI with your API key. If not, you can refer to this guide.

Creating a new object store credential

Civo object stores are region-specific. In this guide, you will be using the NYC1 region. Let's check the current region you are in via civo CLI.

$  civo region ls
+------+-------------+----------------+---------+
| Code | Name        | Country        | Current |
+------+-------------+----------------+---------+
| NYC1 | New York 1  | United States  | <=====  |
| FRA1 | Frankfurt 1 | Germany        |         |
| LON1 | London 1    | United Kingdom |         |
| PHX1 | Phoenix 1   | United States  |         |
+------+-------------+----------------+---------+

Note: The Civo Object store is not available in PHX1 region.

Note: If the result shows you're in a different region, you can switch to NYC1 region with this command: civo region current NYC1 .

Let's create a new credential:

$ civo objectstore credential create SIGN_CRED
Creating Object Store Credential SIGN_CRED in NYC1

Now let's list to see the newly created credentials:

$  civo objectstore credential ls
+-----------+----------------------+--------+
| Name      | Access Key           | Status |
+-----------+----------------------+--------+
| SIGN_CRED | EXAMPLEACCESSKEY1234 | ready  |
+-----------+----------------------+--------+

Creating a new object store

You will create a new object store in the NYC1 region with civo CLI:

$ civo objectstore create super-store --size 500 -n SIGN_CRED --wait
Created Object Store super-store in NYC1 in 0 min 1 sec
Created default admin credentials, access key is EXAMPLEACCESSKEY1234, this will be deleted if the Object Store is deleted. To access the secret key run: civo objectstore credential secret --access-key=EXAMPLEACCESSKEY1234

Let's look at our newly created object store with the civo CLI:

$ civo objectstore ls
+--------+-------------+------+---------------------------+--------+
| ID     | Name        | Size | Object Store Endpoint     | Status |
+--------+-------------+------+---------------------------+--------+
| 3a84f6 | super-store |  500 | objectstore.nyc1.civo.com | ready  |
+--------+-------------+------+---------------------------+--------+

You should see similar output once the object store is created and up. Now let's add some files to it.

Uploading files to object stores

You will be uploading files to the object store using the command line tool s3cmd mentioned in the prerequisite section, along with the details and credentials of the object store.

Setting up credentials for s3cmd

To interact with the Civo object store you created, you'll need s3cmd to be made aware of the credentials and location. You can either use environment variables like this tutorial, or input credential values directly into the s3cmd configuration which you can create by running s3cmd --configure and following the prompts.

You can use the Civo CLI to read the credential details out of your account.

Next, confirm the CLI can see your object store:

$ civo objectstore show super-store

                   ID : 3a84f6f9-e014-4fe4-a803-a259b1f03702
                 Name : super-store
                 Size : 500
Object Store Endpoint : objectstore.nyc1.civo.com
               Region : NYC1
           Access Key : EXAMPLEACCESSKEY1234
               Status : ready
To access the secret key run: civo objectstore credential secret --access-key=EXAMPLEACCESSKEY1234

The Access Key of the store's credential in the command above can be used with the following CLI command to get the environment variable commands needed for s3cmd to work:

$ civo objectstore credential export  --access-key=EXAMPLEACCESSKEY1234
# Tip: You can redirect output with (>> ~/.zshrc) to add these to Zsh's startup automatically
export AWS_ACCESS_KEY_ID=EXAMPLEACCESSKEY1234
export AWS_SECRET_ACCESS_KEY=EXAMPLEsecretkeyEXAMPLEsecretkeyEXAMPLE1
export AWS_DEFAULT_REGION=NYC1
export AWS_HOST=https://objectstore.nyc1.civo.com

For the purposes of this tutorial, copy the lines beginning with export from your command's output and paste them to your terminal to create the environment variables for this session.

As S3 was developed by Amazon Web Services (AWS), tools commonly use the AWS variable names for other S3 hosts that do not run on AWS. This ensures interoperability between object store providers, where the Access Key ID/Secret Key credential mechanic is used throughout.

Using s3cmd to upload a file to an object store

Create a file to upload to the object store you have just gained access to. As an example, you can create a short text file:

$ echo "Hello folks this is epic. Our object store supports signed URL. You're viewing this file because of that." >> sign_demo.txt

The s3cmd syntax to upload a file to a store is s3cmd put FILE [FILE...] s3://STORENAME[/FOLDERNAME]. Assuming you named the store super-store, the command would be

$ s3cmd --host=${AWS_HOST} --host-bucket=s3://super-store put sign_demo.txt s3://super-store/signed/

upload: 'sign_demo.txt' -> 's3://super-store/signed/sign_demo.txt'  [1 of 1]
 110 of 110   100% in    0s   407.48 B/s  done

In this command, the structure is as follows:

  • The --host argument specifies the object store host, that is the Civo region where the store was created.

  • The --host-bucket argument names the object store in the host region as an S3 URI, in this case "super-store".

  • The put command tells s3cmd to upload file to be specified.

  • sign_demo.txt is the file to be uploaded.

  • s3://super-store/signed/ is the path in the object store to upload the file to.

Now, when you look on the web interface, you can see that a folder called signed/ has been created and it contains the file you uploaded.

Creating Signed URL

You have uploaded the file to the object store and it is private. Signed URLs are a great way to share private files with limited time access set by you.

Let's create a signed URL for that file using s3cmd:

$ s3cmd --host=${AWS_HOST} --host-bucket=s3://super-store  signurl s3://super-store/signed/sign_demo.txt $(echo "`date +%s` + 3600" | bc)

http://objectstore.nyc1.civo.com/super-store/signed/sign_demo.txt?AWSAccessKeyId=EXAMPLEACCESSKEY1234&Expires=1678881433&Signature=QUuj6WIx7aX3Q5QuW3%2BmNW2S6u4%3D

In this command, the structure is as follows:

  • The --host argument specifies the object store host, that is the Civo region where the store was created.

  • The --host-bucket argument names the object store in the host region as an S3 URI, in this case "super-store".

  • The signurl command tells s3cmd to create a signed URL with an expiry specified.

  • s3://super-store/signed/sign_demo.txt is the path in the object store with the file that needs to be shared.

  • $(echo "`date +%s` + 3600" | bc) this will create a UNIX timestamp with ease. You can supply the expiration time duration in seconds. In this example that is one hour, so 60 * 60 = 3600 seconds.

This output is the signed URL and can be shared. The expiry time set for this example is one hour. You can increase or decrease it as per your requirements.

Let's download the file with curl:

 $ curl -L -O http://objectstore.nyc1.civo.com/super-store/signed/sign_demo.txt\?AWSAccessKeyId\=EXAMPLEACCESSKEY1234\&Expires\=1678881433\&Signature\=QUuj6WIx7aX3Q5QuW3%2BmNW2S6u4%3D
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   164  100   164    0     0    263      0 --:--:-- --:--:-- --:--:--   266
100   107  100   107    0     0     71      0  0:00:01  0:00:01 --:--:--    71

You can see the file downloaded in your current directory where you execute this command.

The URL will not provide access to the file after it reached the set expiration time. In this case 1 hour. You can test this by reducing the time to just 1 minute and test it before and after expiration with ease.

Cleaning up

Now that you have learnt about how to create a signed URL, you can clean up the created resources.

Deleting the object store and credential with civo cli:

$ civo objectstore  delete super-store -y
The objectStore (super-store) has been deleted

$ civo objectstore credential delete SIGN_CRED -y
The objectStoreCredential (SIGN_CRED) has been deleted

Once the store is deleted, all the files in it are pruned. It is an irreversible process. Be sure you have backed up any necessary files. Note that you cannot delete credentials that are assigned to active object stores - the store itself must be deleted before the credential can be removed.

Conclusion

Creating a signed URL for an object store is a crucial aspect of securing access to resources stored in the cloud. The Civo object store has this feature enabled by default. By using this signed URL, you can grant temporary access to authorized users or applications to perform certain actions on your object store. This approach offers a highly secure and flexible way to manage access to your cloud storage resources, while also reducing the need for managing access credentials or managing complex access control policies.