- read

AWS S3 and Golang

Rao Talha 43

Photo by Lucas van Oort on Unsplash

How can I have a basic idea of S3 using Golang? This is the question that led to the birth of this article. If you are like me and want to know how S3 works using Golang in an easy and understandable way, this article is for you.

Table of Contents:

1. Introduction
- AWS S3 and Golang
- Purpose of the Article

2. Prerequisites
- AWS Account
- IAM User Setup

3. Understanding Amazon S3
- What is S3?
- S3’s Role in Cloud Computing

4. Creating an S3 Bucket on AWS
- Accessing AWS Console
- Steps to Create an S3 Bucket

5. AWS Credentials
- Obtaining AWS Access and Secret Keys
- Setting Up Environment Variables

6. Coding with Golang
- Uploading a File to S3
- Downloading a File from S3
- Listing S3 Buckets
- Listing Objects/Items in an S3 Bucket

7. Conclusion
- Summary of Key Takeaways

Prerequisites

  1. You should already have an AWS Account.
  2. You should have an IAM user with full s3 bucket permission.

What is S3?

Amazon S3 (Simple Storage Service) is a scalable and highly durable cloud storage service provided by AWS. It allows users to store and retrieve data, objects, and files over the internet, making it ideal for backup, data archiving, content distribution, and as a foundation for various cloud-based applications. S3 provides a simple and cost-effective way to store, organize, and secure data while offering high availability and low-latency access.

Creating an S3 bucket on AWS

  1. Open AWS Console
  2. Type S3 in the search bar
search s3 on aws console

3. Click on create a bucket on the left side of the panel

create bucket

4. Fill in the data, remember the region in this case it is ap-southeast-1. It will be used when we set the environment before coding.

fill the data of bucket

5. Uncheck the block of all public access (just for development)

uncheck block public access

6. Create the bucket

After bucket creation, you will see a list of buckets where you can see the bucket you just created.

AWS Credentials

In order to send requests to the AWS S3 bucket we need to have keys that will be used in the next steps. We can get them from IAM>User>your_username and select local code then click next. Add a description and create the keys, then click on the download button to get the CSV credentials file.

Now you need to add them to the environment, in my case I added them to the debug file of VSCode. My launch.json looks like this.

{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Launch Package",
"type": "go",
"request": "launch",
"mode": "auto",
"program": "${fileDirname}",
"env":{
"AWS_ACCESS_KEY_ID":"access key from file you just downloaded",
"AWS_SECRET_ACCESS_KEY": "secret key from file you just downloaded",
"AWS_REGION": "region you have set for your s3 while creating it"
}
}
]
}

Code

It's time to start coding. Let's create a new Golang project and start writing code.

Upload Files

Follow the following steps

  1. Create a session.
  2. Create an uploader instance.
  3. Open the file that you need to upload and save its instance in a variable.
  4. Call the upload function of the uploader by providing the bucket name, file name, and file instance.
var uploadFileName = "fileup.txt" // this file will be uploaded
var bucketName = "bucketname" // name of your bucket


// upload file
func uploadFile() error {
awsSession := session.Must(session.NewSession())

uploader := s3manager.NewUploader(awsSession)

// just getting the path of current folder
filePath, err := runCommand("pwd")
if err != nil {
return err
}

// concatinating the complete file path and filename
uploadFile, err := os.Open(filePath + "/" + uploadFileName)
if err != nil {
return err
}

// upload the file
_, err = uploader.Upload(&s3manager.UploadInput{
Bucket: aws.String(bucketName),
Key: aws.String(uploadFileName),
Body: uploadFile,
})
if err != nil {
return err
}

return nil
}

Download Files

Follow the following steps

  1. Create a session.
  2. Create a downloader instance.
  3. Open the file that you need to download and save its instance in a variable.
  4. Call the download function of the downloader by providing the bucket name and file name.
var dowloadFileName = "file1.txt" // this file will be downloaded
var bucketName = "bucketname" // name of your bucket

func downloadFile() error {
// create session
awsSession := session.Must(session.NewSession())

// create downloader instance
downloader := s3manager.NewDownloader(awsSession)

// create file that will where the data will be stored
downloadFile, err := os.Create(dowloadFileName)
if err != nil {
return err
}

// download the file
_, err = downloader.Download(downloadFile, &s3.GetObjectInput{
Bucket: &bucketName,
Key: &dowloadFileName,
})
if err != nil {
return err
}

return nil
}

List buckets

Follow the following steps

  1. Create session.
  2. Create a new S3 Client.
  3. Get the list of the buckets.
  4. Print Names of buckets.
func listBuckets() error {
// create session
awsSession := session.Must(session.NewSession())

// create s3 client
s3client := s3.New(awsSession)

// list the s3 buckets
result, err := s3client.ListBuckets(nil)
if err != nil {
return err
}

// listing buckets
for _, val := range result.Buckets {
fmt.Println(*val.Name)
}

return nil
}

List Objects

Follow the following steps

  1. Create session.
  2. Create a new S3 Client.
  3. Get the list of the objects/ items stored in S3.
  4. Print Names of objects/items.
func listObjects() {
// create session
awsSession := session.Must(session.NewSession())

// create s3 client
s3client := s3.New(awsSession)

// list the objects/ items in the bucket
result, err := s3client.ListObjectsV2(&s3.ListObjectsV2Input{
Bucket: &bucketName,
})
if err != nil {
return err
}

// print the
for _, val := range result.Contents {
fmt.Println(*val.Key)
}

return nil
}

Conclusion

In summary, this article has guided you through the essentials of working with Amazon S3 in Go. We began with prerequisites, explored the significance of S3, and covered steps for creating an S3 bucket on AWS. Managing AWS credentials was highlighted, followed by practical code examples for everyday S3 tasks like uploading, downloading, and listing objects. This article provides a solid foundation for integrating Amazon S3 into your Go projects, enabling efficient data storage and retrieval in a cloud environment. With this knowledge, you’re well-equipped to leverage the power of S3 and Go in your applications.