When you're starting out in DevOps, it's easy to get overwhelmed by tools like Kubernetes, Prometheus, or Terraform. But real DevOps thinking starts much simpler:

👉 Can your system detect problems?

👉 Can it alert you?

👉 Can it fix itself?

In this project, we built a System Monitoring + Auto-Healing Script on Google Cloud (GCP) that:

Monitors CPU, Memory, and Disk usage. Detects if Nginx goes down. Automatically restarts it, sends alerts to Slack, logs everything for visibility

This is a hands-on beginner project that introduces you to observability and self-healing systems, which are core DevOps concepts.

Project Architecture

We used a simple setup:

GCP Virtual Machine (Ubuntu) Bash script for monitoring Cron job for automation, Slack webhook for alerts, Log file for tracking events

Step 1: Create a GCP VM

We created a virtual machine using Google Cloud:

OS: Ubuntu 22.04

Machine type: e2-micro (free tier friendly)

Enabled HTTP traffic

Then connected via SSH:

gcloud compute ssh your-instance-name

Step 2: Install Required Tools

We installed Nginx (for simulation) and sysstat:

sudo apt update sudo apt install -y nginx sysstat

Start and enable Nginx:

sudo systemctl start nginx sudo systemctl enable nginx

Step 3: Create the Monitoring Script

We created a Bash script:

nano monitor.sh

The script performs:

i. CPU Check

CPU_USAGE=\((top -bn1 | grep "Cpu(s)" | awk '{print 100 - \)8}' | cut -d. -f1)

ii. Memory Check

MEM_USAGE=\((free | awk '/Mem/ {printf("%.0f"), \)3/$2 * 100.0}')

iii. Disk Check

DISK_USAGE=\((df / | awk 'NR==2 {print \)5}' | sed 's/%//')

iv. Nginx Health Check

if ! systemctl is-active --quiet nginx; then

sudo systemctl restart nginx

fi

v. Slack Alerts

We used a Slack webhook:

curl -X POST -H 'Content-type: application/json' --data '{"text":"Alert message"}' YOUR_WEBHOOK_URL

Step 4: Make Script Executable

chmod +x monitor.sh

Step 5: Automate with Cron

We scheduled the script to run every minute:

crontab -e

Add:

• * * * * * /home/your-username/monitor.sh

• Step 6: Simulate Failures Nginx Failure

• sudo systemctl stop nginx

Result:

Script detects failure Restarts Nginx Sends Slack alert 🔥 High CPU Usage yes > /dev/null &

This project teaches real DevOps fundamentals:

Observability Metrics (CPU, memory, disk) Logs Alerts

Automation Cron jobs Continuous monitoring

Self-Healing Systems Detect failure Recover automatically

Production Thinking Don’t wait for users to report issues Systems should detect and react

🎥 Full Walkthrough Video
https://drive.google.com/drive/folders/1OAXNUZsIjBp7igSE\_Xz608shn-hV2pQo?usp=sharing

Final Thoughts

This project may look simple, but it introduces the mindset that defines DevOps:

Build systems that monitor themselves, alert you, and recover automatically

If you're new to DevOps, this is one of the best practical starting points.

In this guide, I’ll walk you through:

• Setting up a VM on GCP

• Hardening it like a real server

• Breaking it (intentionally)

• And recovering it like an SRE

Project Overview

The goal was simple:

Build a secure server → break it → recover it.

But what actually happened?

• I locked myself out of SSH completely

• Even GCP browser SSH stopped working

• I had to mount the disk on another VM to recover it

This is real-world failure simulation.

Step 1: VM Setup on GCP

I created a VM instance using Ubuntu.

Immediately after setup, I ran a custom bash script to prepare the system:

#!/bin/bash
sudo apt update && sudo apt upgrade -y

This ensured the system was up-to-date from the start.

Step 2: Install Nginx

sudo apt install nginx -y
sudo systemctl status nginx

At this point, the server was live and accessible via its public IP.

Step 3: Server Hardening

This is where things got serious.

Create a new user

sudo adduser devops
sudo usermod -aG sudo devops

Setup SSH Key Authentication (Ed25519)

On my local machine:

ssh-keygen -t ed25519 -C "devops-access"

Then I added the public key to:

/home/devops/.ssh/authorized_keys

And fixed permissions:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Disable Password Login

Edited:

sudo nano /etc/ssh/sshd_config

Set:

PasswordAuthentication no
PermitRootLogin no
PubkeyAuthentication yes

Firewall Setup (UFW)

sudo ufw allow OpenSSH
sudo ufw allow 'Nginx Full'
sudo ufw enable

Install Fail2Ban

sudo apt install fail2ban -y
sudo systemctl enable fail2ban

Step 4: Breaking the System (The Fun Part)

This is where things went sideways—in a good way.

❌ SSH Lockout

I added this to my SSH config:

AllowUsers somefakeuser

Then restarted SSH:

sudo systemctl restart ssh

Result:

• SSH access denied

• GCP browser SSH failed

• Total lockout

❌ Kill Nginx

sudo pkill nginx

Now:

• Website down

• Server inaccessible

What Went Wrong?

The line:

AllowUsers somefakeuser

Whitelists SSH users.

Since somefakeuser didn’t exist: 👉 No one could log in

🛠️ Step 5: Recovery (The Real Learning)

At this point, normal recovery methods failed.

Even GCP SSH depends on a working SSH config.

So I had to go deeper.

Disk Recovery Strategy

1. Stopped the original VM

2. Detached its boot disk

3. Attached it to a rescue VM

4. Mounted it manually

sudo mount /dev/sdb1 /mnt/recovery

Fix SSH Config Offline

sudo nano /mnt/recovery/etc/ssh/sshd_config

Removed:

AllowUsers somefakeuser

Replaced with safe config:

PasswordAuthentication yes
PermitRootLogin no
PubkeyAuthentication yes

Restore VM

• Detached disk from rescue VM

• Reattached it as boot disk

• Restarted original VM

Result

• SSH access restored ✅

• Nginx restarted ✅

• Server fully recovered ✅

Key Lessons

1. SSH Misconfiguration = Total Outage

One wrong line can lock you out completely.

2. Always Test Before Restarting SSH

sudo sshd -t

3. Keep a Backup Access Path

Cloud console ≠ guaranteed access.

4. Disk Recovery is a Lifesaver

Mounting a disk on another VM is a real-world recovery technique.

5. Think Like an SRE

This wasn’t just setup—it was:

• Failure simulation

• Incident response

• System recovery

Final Thoughts

This project changed how I think about servers.

It’s easy to follow tutorials. It’s different when:

• You break things

• You get locked out

• And you still recover

That’s where real learning happens.