How can I have a basic idea of S3 using Golang? This is the question that led to the birth of this article. If you are like me and want to know how S3 works using Golang in an easy and understandable way, this article is for you.

Prerequisites

You should already have an AWS Account.
You should have an IAM user with full s3 bucket permission.

What is S3?

Amazon S3 (Simple Storage Service) is a scalable and highly durable cloud storage service provided by AWS. It allows users to store and retrieve data, objects, and files over the internet, making it ideal for backup, data archiving, content distribution, and as a foundation for various cloud-based applications. S3 provides a simple and cost-effective way to store, organize, and secure data while offering high availability and low-latency access.

Creating an S3 bucket on AWS

Open AWS Console
Type S3 in the search bar

3. Click on create a bucket on the left side of the panel

4. Fill in the data, remember the region in this case it is ap-southeast-1. It will be used when we set the environment before coding.

5. Uncheck the block of all public access (just for development)

6. Create the bucket

After bucket creation, you will see a list of buckets where you can see the bucket you just created.

AWS Credentials

In order to send requests to the AWS S3 bucket we need to have keys that will be used in the next steps. We can get them from IAM>User>your_username and select local code then click next. Add a description and create the keys, then click on the download button to get the CSV credentials file.

Now you need to add them to the environment, in my case I added them to the debug file of VSCode. My launch.json looks like this.

{
 // Use IntelliSense to learn about possible attributes.
 // Hover to view descriptions of existing attributes.
 // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
 "version": "0.2.0",
 "configurations": [
 {
 "name": "Launch Package",
 "type": "go",
 "request": "launch",
 "mode": "auto",
 "program": "${fileDirname}",
 "env":{
 "AWS_ACCESS_KEY_ID":"access key from file you just downloaded",
 "AWS_SECRET_ACCESS_KEY": "secret key from file you just downloaded",
 "AWS_REGION": "region you have set for your s3 while creating it"
 }
 }
 ]
}

Code

It's time to start coding. Let's create a new Golang project and start writing code.

Upload Files

Follow the following steps

Create a session.
Create an uploader instance.
Open the file that you need to upload and save its instance in a variable.
Call the upload function of the uploader by providing the bucket name, file name, and file instance.

var uploadFileName = "fileup.txt" // this file will be uploaded
var bucketName = "bucketname" // name of your bucket


// upload file
func uploadFile() error {
 awsSession := session.Must(session.NewSession())

 uploader := s3manager.NewUploader(awsSession)

// just getting the path of current folder
 filePath, err := runCommand("pwd")
 if err != nil {
 return err
 }

// concatinating the complete file path and filename
 uploadFile, err := os.Open(filePath + "/" + uploadFileName)
 if err != nil {
 return err
 }

// upload the file
 _, err = uploader.Upload(&s3manager.UploadInput{
 Bucket: aws.String(bucketName),
 Key: aws.String(uploadFileName),
 Body: uploadFile,
 })
 if err != nil {
 return err
 }

 return nil
}

Download Files

Follow the following steps

Create a session.
Create a downloader instance.
Open the file that you need to download and save its instance in a variable.
Call the download function of the downloader by providing the bucket name and file name.

var dowloadFileName = "file1.txt" // this file will be downloaded
var bucketName = "bucketname" // name of your bucket

func downloadFile() error {
// create session
 awsSession := session.Must(session.NewSession())

// create downloader instance
 downloader := s3manager.NewDownloader(awsSession)

// create file that will where the data will be stored
 downloadFile, err := os.Create(dowloadFileName)
 if err != nil {
 return err
 }

// download the file
 _, err = downloader.Download(downloadFile, &s3.GetObjectInput{
 Bucket: &bucketName,
 Key: &dowloadFileName,
 })
 if err != nil {
 return err
 }

 return nil
}

List buckets

Follow the following steps

Create session.
Create a new S3 Client.
Get the list of the buckets.
Print Names of buckets.

func listBuckets() error {
// create session
 awsSession := session.Must(session.NewSession())

// create s3 client
 s3client := s3.New(awsSession)

// list the s3 buckets
 result, err := s3client.ListBuckets(nil)
 if err != nil {
 return err
 }

// listing buckets
 for _, val := range result.Buckets {
 fmt.Println(*val.Name)
 }

 return nil
}

List Objects

Follow the following steps

Create session.
Create a new S3 Client.
Get the list of the objects/ items stored in S3.
Print Names of objects/items.

func listObjects() {
// create session
 awsSession := session.Must(session.NewSession())

// create s3 client
 s3client := s3.New(awsSession)

// list the objects/ items in the bucket
 result, err := s3client.ListObjectsV2(&s3.ListObjectsV2Input{
 Bucket: &bucketName,
 })
 if err != nil {
 return err
 }

// print the 
 for _, val := range result.Contents {
 fmt.Println(*val.Key)
 }

 return nil
}

Conclusion

In summary, this article has guided you through the essentials of working with Amazon S3 in Go. We began with prerequisites, explored the significance of S3, and covered steps for creating an S3 bucket on AWS. Managing AWS credentials was highlighted, followed by practical code examples for everyday S3 tasks like uploading, downloading, and listing objects. This article provides a solid foundation for integrating Amazon S3 into your Go projects, enabling efficient data storage and retrieval in a cloud environment. With this knowledge, you’re well-equipped to leverage the power of S3 and Go in your applications.

AWS S3 and Golang

Table of Contents:

Prerequisites

What is S3?

Creating an S3 bucket on AWS

AWS Credentials

Code

Download Files

List buckets

List Objects

Conclusion