Provision AWS Infrastructure using Terraform & Jenkins – GitOps

0 / 104
Terraform Gitops pipeline setup

In this article we will learn about Infrastructure As Code (IAC).



We will provision AWS infrastructure using Terraform & Jenkins. We are using Jenkins for creating a pipeline and deploying terraform code.



In this example we are creating s3 bucket and ec2 instance using terraform. You can similarly add any other resource.



Infrastructure as Code – It is the process of configuring resources automatically in a user interface as opposed to manually managing infrastructure in a file, or files.



Gitops – GitOps is a framework for operations that adds infrastructure automation to DevOps best practises, including version control, collaboration, compliance, and CI/CD, which are often used for application development.



Terraform – Terraform is HashiCorp’s infrastructure-as-code service. It is a tool for safely modifying and managing infrastructure in a repeatable manner.



Jenkins- An open-source DevOps solution for continuous integration/continuous delivery and deployment (CI/CD) automation that uses the Java programming language. Implementation of pipelines, often known as CI/CD workflows, uses it.



Terraform state files are created locally by default. This is not ideal when multiple people are working on a project. We will use Amazon S3 to store remote state files with DynamoDB state locking and consistency checking.



Your state files will be stored in the S3 bucket. The DynamoDB table allows you to lock the state file to prevent multiple people from writing to it at the same time.



Steps:



1.      Create AWS account and get the details of AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY



 

2.      Install AWS CLI on Linux system

 

3.      Create s3 bucket to store terraform state

 

4.      Create DynamoDB table to lock your Terraform state file

 

5.      Create GitHub repo to store terraform config files and Jenkinsfile

 

6.      Install Jenkins

 

7.      Create Jenkins credentials for github and aws (AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY)

 

8.      Create and run Jenkins’s pipeline

 

Procedure:



1.      Create AWS account and get the details of AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY

Generate new Access ID and Secret or access existing one using –



http://console.aws.amazon.com



No alt text provided for this image

Copy or Download of AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY



No alt text provided for this image

2.      Install AWS CLI on Linux system



Follow below steps on your Linux system to download and install aws cli –



$ python –version

$ curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"

$ unzip awscli-bundle.zip

$ sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws

$ which aws

$ aws configure

3.      Create s3 bucket to store terraform state and enable versioning

Create aws s3 bucket –

$ aws s3 ls

$ aws s3api create-bucket --bucket "tfs-s3-bkt1" --region us-east-1

$ aws s3 ls

$ aws s3api put-bucket-versioning --bucket "tfs-s3-bkt1" --versioning-configuration Status=Enabled

Output ->

[prayag@panel tf-aws]$ aws s3 ls
[prayag@panel tf-aws]$ aws s3api create-bucket --bucket "tfs-s3-bkt1" --region us-east-1
 {
    "Location": "/tfs-s3-bkt1"
 }

[prayag@panel tf-aws]$ aws s3 ls
2022-07-13 14:35:46 tfs-s3-bkt1
[prayag@panel tf-aws]$ aws s3api put-bucket-versioning --bucket "tfs-s3-bkt1" --versioning-configuration Status=Enabled

You can also create s3 bucket using AWS GUI Console

No alt text provided for this image

4.      Create DynamoDB table to lock your Terraform state file

$ aws dynamodb create-table --table-name tf-backend-lock --attribute-definitions AttributeName=LockID,AttributeType=S --key-schema AttributeName=LockID,KeyType=HASH --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

Output ->

[prayag@panel tf-aws]$ aws dynamodb create-table --table-name tf-backend-lock --attribute-definitions AttributeName=LockID,AttributeType=S --key-schema AttributeName=LockID,KeyType=HASH --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=

  {
   "TableDescription": {
       "AttributeDefinitions": [
           {
               "AttributeName": "LockID",
               "AttributeType": "S"
           }
       ],
       "TableName": "tf-backend-lock",
       "KeySchema": [
           {
               "AttributeName": "LockID",
               "KeyType": "HASH"
           }
       ],
       "TableStatus": "CREATING",
       "CreationDateTime": "2022-07-13T14:39:33.285000+05:30",
       "ProvisionedThroughput": {
           "NumberOfDecreasesToday": 0,
           "ReadCapacityUnits": 5,
           "WriteCapacityUnits": 5
       },
       "TableSizeBytes": 0,
       "ItemCount": 0,
       "TableArn": "arn:aws:dynamodb:us-east-1:768149447218:table/tf-backend-lock",
       "TableId": "eab0955e-1870-45c3-96b7-82810f987b3b"
   }
}
[prayag@panel tf-aws]$5

You can also create DynamoDB table using AWS GUI Console

No alt text provided for this image

5.      Create GitHub repo to store terraform config files and Jenkinsfile

You can clone my repo –

$ git clone https://github.com/prayag-sangode/tf-aws.git
No alt text provided for this image

We are using s3 bucket and dynamodb table which we created in earlier steps in provider.tf

$ cat provider.tf

terraform {

 backend "s3" {

   bucket = "tfs-s3-bkt1"

   key   = "terraform.tstate"

   region = "us-east-1"

    dynamodb_table = "tf-backend-lock"

 }

}

provider "aws" {

 region = "${var.region}"

}

6.      Install Jenkins

On Jenkins host install git & terraform –

$ sudo yum install -y yum-utils

$ sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo

$ sudo yum -y install terraform

$ sudo yum -y install git

Access Jenkins using http://IP:8080

In Jenkins install terraform and aws plugin

No alt text provided for this image
No alt text provided for this image

7.      Create Jenkins credentials for github and aws (AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY)

Create GitHub Credentials.

Create AWS credentials in Jenkins –

In secret add your key – AWS_ACCESS_KEY_ID. Select kind as Secret text

No alt text provided for this image

In secret enter AWS_SECRET_ACCESS_KEY. Select kind as Secret text

No alt text provided for this image

So, we have below 3 credentials –

No alt text provided for this image

In Jenkinsfile we are using AWS credentials variables as below –

 pipeline {

   environment {

       AWS_ACCESS_KEY_ID    = credentials('AWS_ACCESS_KEY_ID')

       AWS_SECRET_ACCESS_KEY = credentials('AWS_SECRET_ACCESS_KEY')

   }

   agent any

   tools {

      terraform 'terraform'

   }

8.      Create and run Jenkins’s pipeline

 Create Pipeline, add github credentials for repo and add github repo

No alt text provided for this image

Run pipeline and check if build is successful –

No alt text provided for this image

Check AWS console if EC2 instance and S3 bucket is created –

No alt text provided for this image
No alt text provided for this image

Check AWS console if terraform state file exists in s3 bucket (used in terraform backend ) –

No alt text provided for this image

Check DynamoDB lock table item –

No alt text provided for this image

I hope you found this to be useful in some way. I’ll be back with some more interesting new cloud and Devops articles soon.

Comments

comments


***Linux, Cloud & Devops Architect & Technical Content Writer*** I am a Linux Enthusiast and Supporter/Promoter of Open Source Technology with over 12+ years of experience in Linux, Cloud and Devops. I am A Technical Content writer for various sites like : Hostbread & Golibrary

Related Posts