Instance Code

{
  "MaxCount": 1,
  "MinCount": 1,
  "ImageId": "ami-0089b8e98cd95257d",
  "InstanceType": "t3.micro",
  "EbsOptimized": true,
  "NetworkInterfaces": [
    {
      "DeviceIndex": 0,
      "AssociatePublicIpAddress": true,
      "SubnetId": "subnet-086a045ade7eae99f",
      "Groups": [
        "sg-0c2e3fdd288a74d3a"
      ]
    }
  ],
  "TagSpecifications": [
    {
      "ResourceType": "instance",
      "Tags": [
        {
          "Key": "Name",
          "Value": "frontend"
        }
      ]
    },
    {
      "ResourceType": "spot-instances-request",
      "Tags": [
        {
          "Key": "Name",
          "Value": "frontend"
        }
      ]
    }
  ],
  "InstanceMarketOptions": {
    "MarketType": "spot",
    "SpotOptions": {
      "InstanceInterruptionBehavior": "stop",
      "SpotInstanceType": "persistent"
    }
  },
  "PrivateDnsNameOptions": {
    "HostnameType": "ip-name",
    "EnableResourceNameDnsARecord": true,
    "EnableResourceNameDnsAAAARecord": false
  }
}

MongoDB Installation and Configuration

In RoboShop, the database management system MongoDB has been chosen to store and manage data. Here's how to install and configure it:

Step 1: Set up MongoDB repository

Before installing MongoDB, we need to set up its repository. Create a new file called mongo.repo in the directory /etc/yum.repos.d/ using the following command:

sudo vim /etc/yum.repos.d/mongo.repo

Then, add the following lines to the file:

[mongodb-org-4.2]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.2/x86_64/
gpgcheck=0
enabled=1

Step 2: Install MongoDB

Once the repository is set up, we can install MongoDB using the following command:

sudo yum install mongodb-org -y

Step 3: Start and enable MongoDB service

Start and enable the MongoDB service using the following commands:

sudo systemctl enable mongod
sudo systemctl start mongod

Step 4: Update MongoDB configuration

By default, MongoDB only listens to connections from the local machine (localhost). In order to allow connections from other servers, we need to update the MongoDB configuration file.

Open the configuration file /etc/mongod.conf in your text editor:

sudo vim /etc/mongod.conf

Find the line that starts with bindIp and change the value from 127.0.0.1 to 0.0.0.0.

# network interfaces
net:
  port: 27017
  bindIp: 0.0.0.0  # Listen to local interface only, comment to listen on all interfaces.

Step 5: Restart MongoDB service

After updating the MongoDB configuration file, restart the MongoDB service to apply the changes:

sudo systemctl restart mongod

That's it! MongoDB is now installed and configured to accept connections from other servers.

User Management

Types of Users

1. Root User: Superuser with unrestricted system access. Home directory: /root.

2. Regular Users: Non-privileged users with home directories in /home/username.

Creating Users

bashsudo useradd john
sudo passwd john

User Configuration Files

• /etc/passwd: User account information

• /etc/shadow: Encrypted passwords and account expiration data

• /etc/group: Group information

Group Management

Groups facilitate managing permissions for multiple users simultaneously.

Creating Groups and Adding Users

bashsudo groupadd developers
sudo usermod -aG developers john

Key Management Commands

• useradd: Add new user

• passwd: Set/change user password

• usermod: Modify user account

• groupadd: Add new group

• groups: Display user's group memberships

• deluser: Delete user

• delgroup: Delete group

Permissions and Ownership

Linux uses a permission model to control file and directory access.

File Ownership

• Owner: User who owns the file

• Group: Group that owns the file

Change ownership:

bashsudo chown user:group filename

File Permissions

Permissions are represented as:

text-rwxr-xr--

• First character: File type

• Next three: Owner permissions

• Following three: Group permissions

• Last three: Others' permissions

Permissions include:

• r: Read

• w: Write

• x: Execute

Changing File Permissions

Using chmod command:

Symbolic mode:

bashchmod u+rwx filename
chmod g-w filename
chmod o=rx filename

Numeric mode:

bashchmod 755 filename
chmod 644 filename

Practical Scenarios

1. Creating a New User and Assigning to a Group

    bashsudo useradd john
    sudo passwd john
    sudo groupadd developers
    sudo usermod -aG developers john
    groups john
   

2. Changing File Ownership and Permissions

    bashtouch example.txt
    sudo chown john:developers example.txt
    chmod 754 example.txt
    ls -l example.txt
   

Comprehensive Example

bashsudo useradd alice
sudo passwd alice
sudo groupadd engineers
sudo usermod -aG engineers alice
groups alice
touch project.txt
sudo chown alice:engineers project.txt
chmod 664 project.txt
ls -l project.txt

This comprehensive guide provides a solid foundation for managing users, groups, and file permissions in Linux environments, essential for maintaining system security and access control.

SOURCE="/home/ubuntu/Test01"
DESTINATION="/home/ubuntu/Test02/"
DATE=$(date +%Y-%m-%d_%H-%M-%S)

# Create backup directory and copy files

mkdir -p $DESTINATION/$DATE
cp -r $SOURCE $DESTINATION/$DATE
echo "Backup completed on $DATE"

Explanation

• SOURCE: The directory to be backed up. • DESTINATION: The directory where the backup will be stored. • DATE: Captures the current date and time to create a unique backup folder.

• mkdir -p $DESTINATION/$DATE: Creates the backup directory if it does not exist.

• cp -r $SOURCE $DESTINATION/$DATE: Copies the contents of the source directory to the backup directory.

• echo "Backup completed on $DATE": Outputs a message indicating the completion of the backup.

Scheduling the backup with Cron

To schedule regular execution of the backup script, utilize the crontab editor by running the following command:

crontab -e

Once in the editor, add the following line to configure the backup schedule:

text* * * * * /path/to/backup_script.sh

This configuration will execute the backup script every minute5. Modify the cron schedule parameters to align with your desired backup frequency.

Disk Usage Monitoring Script

Script Overview

This Bash script monitors disk usage across partitions and issues warnings when usage exceeds a predefined threshold.

bash#!/bin/bash

THRESHOLD=80

df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' | while read output;
do
    usage=$(echo $output | awk '{ print $1}' | cut -d'%' -f1)
    partition=$(echo $output | awk '{ print $2 }')
    if [ $usage -ge $THRESHOLD ]; then
        echo "Warning: Disk usage on $partition is at ${usage}%"
    fi
done

Functionality Breakdown

1. Threshold Setting: The script initializes with a disk usage threshold of 80%.

2. Disk Usage Data Collection: Utilizes df -H to retrieve disk usage information in a human-readable format.

3. Data Filtering: Employs grep to exclude non-essential filesystem entries.

4. Data Extraction: Uses awk to isolate usage percentages and partition names.

5. Iterative Processing: Processes each filtered entry using a while loop.

6. Usage Calculation: Extracts the numerical usage percentage from each entry.

7. Partition Identification: Isolates the partition name for each entry.

8. Threshold Comparison: Compares the usage against the predefined threshold.

9. Alert Generation: Outputs a warning message for partitions exceeding the threshold.

Service Health Check

This script checks if a specified service is running and starts it if not.

bash#!/bin/bash

SERVICE="nginx"

if systemctl is-active --quiet $SERVICE; then
    echo "$SERVICE is running"
else
    echo "$SERVICE is not running"
    systemctl start $SERVICE
fi

Explanation:

• SERVICE: Specifies the name of the service to check (nginx in this example).

• systemctl is-active --quiet $SERVICE: Checks if the service is running.

• If the service is running, it prints a confirmation message.

• If it is not running, it prints a message and attempts to start the service.

Network Connectivity Check

This script checks network connectivity to a specified host.

bash#!/bin/bash

HOST="google.com"
OUTPUT_FILE="/home/ubuntu/output.txt"

if ping -c 1 $HOST &> /dev/null
then
    echo "$HOST is reachable" >> $OUTPUT_FILE
else
    echo "$HOST is not reachable" >> $OUTPUT_FILE
fi

Explanation:

• HOST: Specifies the hostname to check.

• OUTPUT_FILE: Defines where to write the output.

• ping -c 1 $HOST &> /dev/null: Pings the host once, suppressing output.

• Depending on the ping result, it writes a reachability status to the output file.

Database Backup

This script creates a backup of a specified MySQL database.

bash#!/bin/bash

DB_NAME="mydatabase"
BACKUP_DIR="/path/to/backup"
DATE=$(date +%Y-%m-%d_%H-%M-%S)

mysqldump -u root -p $DB_NAME > $BACKUP_DIR/$DB_NAME-$DATE.sql

echo "Database backup completed: $BACKUP_DIR/$DB_NAME-$DATE.sql"

Explanation:

• DB_NAME: Specifies the database to back up.

• BACKUP_DIR: Defines where to store the backup.

• DATE: Captures the current date and time for a unique filename.

• mysqldump command creates a SQL dump of the database.

• The echo statement confirms the backup completion and location.

System Uptime Check

This simple script displays the system's uptime.

bash#!/bin/bash

uptime -p

Explanation:

• uptime -p: Prints the system uptime in a human-readable format.

Listening Ports Monitor

This script lists all listening ports and their associated services.

bash#!/bin/bash

netstat -tuln | grep LISTEN

Explanation:

• netstat -tuln: Lists all TCP and UDP listening ports.

• grep LISTEN: Filters the output to show only listening ports.

Automatic Package Updates

This script updates and cleans up system packages.

bash#!/bin/bash

apt-get update && apt-get upgrade -y && apt-get autoremove -y && apt-get clean
echo "System packages updated and cleaned up"

Explanation:

• apt-get update: Updates the package list.

• apt-get upgrade -y: Upgrades all installed packages.

• apt-get autoremove -y: Removes unnecessary packages.

• apt-get clean: Cleans up the package cache.

• The echo statement confirms the completion of updates and cleanup.

HTTP Response Time Monitor

This script checks HTTP response times for specified URLs.

bash#!/bin/bash

URLS=("https://www.devopsshack.com/" "https://www.linkedin.com/")

for URL in "${URLS[@]}"; do
    RESPONSE_TIME=$(curl -o /dev/null -s -w '%{time_total}\n' $URL)
    echo "Response time for $URL: $RESPONSE_TIME seconds"
done

Explanation:

• URLS: An array of URLs to check.

• The for loop iterates over each URL.

• curl command fetches each URL and measures the total response time.

• The script prints the response time for each URL.

System Process and Memory Usage Monitor

This script displays the top processes by memory usage.

bash#!/bin/bash

ps aux --sort=-%mem | head -n 10

Explanation:

• ps aux: Lists all running processes.

• --sort=-%mem: Sorts processes by memory usage in descending order.

• head -n 10: Displays only the top 10 processes.

These scripts provide valuable tools for various DevOps tasks, from system monitoring to backup and maintenance operations.

Instance Setup Code

{
  "MaxCount": 1,
  "MinCount": 1,
  "ImageId": "ami-0089b8e98cd95257d",
  "InstanceType": "t3.micro",
  "EbsOptimized": true,
  "NetworkInterfaces": [
    {
      "DeviceIndex": 0,
      "AssociatePublicIpAddress": true,
      "SubnetId": "subnet-086a045ade7eae99f",
      "Groups": [
        "sg-0c2e3fdd288a74d3a"
      ]
    }
  ],
  "TagSpecifications": [
    {
      "ResourceType": "instance",
      "Tags": [
        {
          "Key": "Name",
          "Value": "frontend"
        }
      ]
    },
    {
      "ResourceType": "spot-instances-request",
      "Tags": [
        {
          "Key": "Name",
          "Value": "frontend"
        }
      ]
    }
  ],
  "InstanceMarketOptions": {
    "MarketType": "spot",
    "SpotOptions": {
      "InstanceInterruptionBehavior": "stop",
      "SpotInstanceType": "persistent"
    }
  },
  "PrivateDnsNameOptions": {
    "HostnameType": "ip-name",
    "EnableResourceNameDnsARecord": true,
    "EnableResourceNameDnsAAAARecord": false
  }
}

Configuring Server

The frontend service in RoboShop is responsible for serving the web content over Nginx. This service includes the web frame for the web application and serves static content. To accomplish this, a web server is needed. In this case, the developer has chosen Nginx Web Server, which we will install by running the following command:

yum install nginx -y

After installation, we need to start and enable the Nginx service:

systemctl enable nginx 
systemctl start nginx

Once the service is up and running, we can access it through a browser to ensure that default content is being served. To remove the default content, run:

rm -rf /usr/share/nginx/html/*

Next, we need to download the frontend content by running:

curl -o /tmp/frontend.zip https://roboshop-artifacts.s3.amazonaws.com/frontend.zip

The content can then be extracted using the following command:

cd /usr/share/nginx/html 
unzip /tmp/frontend.zip

After the frontend content has been extracted, we need to create an Nginx Reverse Proxy Configuration file. Open the file using the following command:

vim /etc/nginx/default.d/roboshop.conf

Add the following content to the file:

proxy_http_version 1.1;
location /images/ {
  expires 5s;
  root   /usr/share/nginx/html;
  try_files $uri /images/placeholder.jpg;
}
location /api/catalogue/ { proxy_pass http://localhost:8080/; }
location /api/user/ { proxy_pass http://localhost:8080/; }
location /api/cart/ { proxy_pass http://localhost:8080/; }
location /api/shipping/ { proxy_pass http://localhost:8080/; }
location /api/payment/ { proxy_pass http://localhost:8080/; }

location /health {
  stub_status on;
  access_log off;
}

Note that the localhost should be replaced with the actual IP address of the component server to avoid failures on the Nginx Server.

Finally, restart the Nginx service to load the changes to the configuration file:

systemctl restart nginx

Frontend of the Roboshop is set up now.

• Install AWS CLI for Linux on the computer.

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

• Install zip/ Unzip to unzip the aws cli package

sudo apt-get install zip (This should also install unzip option too)

• Unzip the file, currently should be in the same directory.

Unzip awsclive2.zip

• Navigate to the file and run the command

sudo ./aws/install

• Create an Admin user through IAM (Identity Access Management).

• To get the Access Key ID

• Secret Key

(Note: Google is your best friend if you don’t know this search for it.)

• Need to know the specific region (for this I will be using us-east-1)

• Create keypair through AWS CLI

aws ec2 create-key-pair --key-name awsclikey --query 'keyMaterial' --output text > awsclikey.pem

• Change the permission of the key to it has read-only access

chmod 400 awsclikey.pem

• A filter name and value pair that is used to return a more specific list of results from a described operation.

aws ec2 describe-key-pairs --key-names awsclikey

• We will need a security group to manage the traffic that will be using the EC2 instances.

aws ec2 create-security-group --group-name awscligroup --description "group created with awscli" --vpc-id vpc-0e4167f02a110a4ed

Note: Get the Default VPC Id from the console

• Define the inbound rules now

aws ec2 authorize-security-group-ingress --group-id sg-0b2ba2fe2deb772b6 --protocol tcp --port 0-65535 --cidr 0.0.0.0/0

• Ingress means input bounds rule

• Group id is what we created just now.

• Protocol tcp which we will be using

• Port has to define the range 0-65535

• Define the IP address 0.0.0.0/0

• To perform the next step we will need the Image Id (a specific image we will be using for our instance. For this demo we will be using Ubuntu Linux LTS 20).

aws ec2 run-instances --image-id ami-0b93ce03dcbcb10f6 --count 2 --instance-type t2.micro --key-name awsclikey --security-group-ids sg-0b2ba2fe2deb772b6 --subnet-id subnet-013a2aa48b12f80b2

• The EC2 instance will be ready shortly.

Note: The user can create an IAM account with the same email as the one used to log in to the AWS Console Account. The user must switch from Root to IAM at the login window.

• Once Logged in Navigate to the Search bar and Type IAM or Identity Access Management

• The user will be presented with the following dashboard, where users, groups, policies and etc can be managed.

How to create Users in IAM

Users are known as entities, the purpose of creating users is to give the team or the environment access to the services in AWS. We can manage the users with the help of IAM.

• Click Users under Access management

• Click on Add Users

• Add user name, under Set user details.

(U1 is for demonstration purposes)

• The user will have two ways to provide the access to the U1,

• Access Key - Programmatic Access (User will connect using CLI, SDK, and other development tools to connect)

• Password - AWS Management Console Access (users will log in using AWS web console)

• Let's use Password - AWS Management console access.

• We can Add a user to the Group here or create a group for the user if it doesn't already exist.

• We can copy permissions from the existing user

• We can attach existing policies here as well

• We will just create a user at this time.

• Click on Next: Tags Button

• Add tags is optional and it is used to organize, track and control access for the user. It could be the user's email, description, or job title.

• Click on the Next: Review button, to continue

• The user should Review all the inputted information and Click Create User.

• A user account will be created, You can either download the CSV and share it with the users successfully or Email the CSV file to the user.

Creating Groups in IAM

A Group is known as an identity that has all the IAM

users in a single environment. The groups are used to specify the permissions for multiple users. The groups make permission to manage and apply to multiple users.

• To Create Groups Navigate to the left panel, under Access Management.

• Click on User groups.

• Give the group a specific name (ec2-access)

• You can also add users to the group, but it doesn’t have to be now.

• Under Attach permission policies, there are many permissions to choose from, we need to create a group for users to give only read-only permissions.

• Select AmazonGlacierReadOnlyAccess

• Click Create Group

• The group ecs-access is created under the group with the defined permissions.

• Users can create folders in the S3 bucket and upload the files and folder in the S3 bucket

• In order to share the document from S3, Click on a specific file to open it.

• Navigate to the Objects Action Tab

• Click on Share with a pre-signed URL

• You can define if you want to share for minutes or hours.

• After creating a pre-signed URL

• Copy the URL with the intending party and do remind them the file/folder is available for a certain amount of time.

In the AWS console search for S3.

• Click Create Bucket

• Under General Configuration give your bucket a unique name.

• Leave the Object Ownership as the default

• Under Block Public Access settings for this bucket, leave it as the default

• Leave Bucket Versioning as default too.

• Skip the Tags and leave the Default encryption as default too.

• Don’t need to make any changes in Advance, Click on Create a bucket

• You should have your first AWS S3 bucket.

SSH is also known as Secure Shell Protocol, which allows sys admins or users a secure way to access their systems over the unsecured network. You need ad public key and a private key to connect to the security systems.

The public is a key that will be used to decrypt the private key and usually gets saved in the secure credentialing manager on the user's system.

The private key is an encrypted key that the users will have it saved somewhere safe on their computers.

The process demonstrated here is done on Linux, if you follow along you won’t fail.

• Navigate to cd ~/.ssh

• ssh-keygen -o -t key(RSA) -C "your GitHub email address"

• If it asks you for a location just hit enter or return because by default ssh will look for the key here. /home/devbox/.ssh

• The location of the ssh key cat id_key(rsa).pub

• Copy and paste it into the git hub setting > ssh key.

Git is a popular and widely used version control system that allows developers to keep track of changes made to their code over time, collaborate with others, and work on multiple versions of their code.

What is a GitHub?

GitHub is a web-based platform that provides hosting for Git repositories and offers additional features such as issue tracking, pull requests, and code reviews. It allows developers to share and collaborate on their code with others.

What is Version control?

Version control is a system that allows developers to keep track of changes made to their code over time, and enables them to revert to previous versions or collaborate with others on the same codebase.

Types of Version Controls

There are two main types of version control systems: centralized and distributed.

• Centralized version control systems store code in a central server, while distributed version control systems create multiple copies of the repository on different computers.

• Distributed version control offers several advantages over centralized version control, such as the ability to work offline and make commits locally, easier collaboration and branching, and a faster workflow.

Everything Combined

Git is a version control system that allows developers to keep track of changes made to their code, GitHub is a platform that provides hosting for Git repositories and additional features, version control is a system that allows developers to manage and track changes made to their code over time, there are two main types of version control systems (centralized and distributed), and distributed version control offers several advantages over centralized version control.

10 Basic git Commands

1. git init: Initializes a new Git repository in your current working directory.

2. git add: Adds changes made to your code to the staging area, which prepares them to be committed.

3. git commit: Commits changes to the repository and creates a new snapshot of the code.

4. git status: Shows the current status of the repository, including which files have been modified or added, and which files are ready to be committed.

5. git log: Shows a history of all the commits made to the repository, along with their messages and other details.

6. git branch: Lists all the branches in the repository, and shows which branch you are currently on.

7. git checkout: Switches to a different branch or a specific commit.

8. git pull: Updates the local repository with changes made in the remote repository.

9. git push: Sends changes made locally to the remote repository.

10. git clone: Creates a copy of a remote repository on your local machine.

Git has many more powerful features and commands that can help you manage your code and collaborate with others.

Task 1: New Repository in Github and Cloning it locally.

A GitHub account already exists. I created the repository Day-8-task on git. I will clone the repo and make changes to the README.md file and commit the changes to GitHub.

• To clone the repository git clone followed by the repository link

git clone https://github.com/Arafique458/Day-8-Task.git

• To configure the credentials, I will be using a Secret Access token to connect to the repository.

git config --global user.name "arafique458"

git config --global user.email "arafique458@gmail.com

• To access the cloned directory

cd Day-8-task

• To make changes to the README.md file

vim README.md

• After making changes check the status of the file modified.

• To stage, the changes git add . (this will stage all the files in the directory or we can specify the file want to stage)

  

  • To commit the change we made to the README.md file git commit -m "what were changes"

• To push the changes to the main repository in the GitHub git push origin main (I was prompted to enter credentials, I used the email address of the git hub and Access token)

Task 2: File in the repository changed and commit was made in the repository using Git

GitHub repository before making a commit

GitHub repository before making a commit

Task 3: Pushed Changed / Commit Screenshot of GitHub repository