Amazon s3
Amazon Simple Storage Service (Amazon S3) is a scalable cloud storage service provided by Amazon Web Services (AWS). It's designed to store and retrieve any amount of data from anywhere on the web. Amazon S3 is widely used for a variety of purposes, including data backup, hosting static assets for websites, and as a data storage backend for various applications
Cloud Object Storage
There are many ways to store data on the cloud, but the easiest are the object stores. All three major cloud providers have them:
This serves as an excellent entry point for files, as you can often store them without extensive advance planning or capacity considerations. Moreover, these object stores boast exceptional resilience, offering multiple layers of data durability and high availability.
Amazon S3 Terminology:
- Identity and Access Management (IAM) – Controls for making and controlling who and what can interact with your AWS resources.
- Access Keys – These are your access credentials to use AWS. These are not your typical username/password — they are generated using access identity management.
- Bucket – A grouping of similar files that must have a unique name. These can be made publicly accessible and are often used to host static objects.
- Folder – Much like an operating system folder, these exist within a bucket to enable organization.
Creating an Access Key
For NiFi to have permission to write to S3, we must set it up with an access key pair. There are many ways to do this, but the best practice is to create a new IAM user. To get to the IAM user screen, navigate to the IAM homepage
- Select "Add user" and check "Programmatic access"
- Enter a new name such as "NiFi_demo"
- Click "Next: Permissions"
- Click "Create Group" and you will be presented with a list of permissions you can add to this new user
- Enter a group name such as "Nifi_Demo_Group"
- Next to filter policies, search for S3 and check "AmazonS3FullAccess" > Click "Create Group"
- At the bottom right, select "Next: Tags" > Click through to "Next: Review"
- Click "Create user" to finish making an IAM User
The access key ID and secret access key are very important to setting up your data transfer. You can download them as a .CSV file or save them somewhere safe.
IMPORTANT: Be sure to record your secret access key as this is the only time it can be viewed.
Creating an S3 Bucket
Now that we have AWS credentials, we require a location to store them. In essence, we need to establish a new S3 bucket if you don't already have one. Navigate to the AWS S3 Console.
- Click “+ Create Bucket”
- Enter a unique bucket name and the region you are creating it in
- Click through until the bucket is created (default options are fine to use)
- Click on your new bucket and you should be able to see its contents — which will be empty