Skip to content

Commit

Permalink
feature : clarify messages and introduce CHECKPOINTs
Browse files Browse the repository at this point in the history
  • Loading branch information
patternhelloworld committed Nov 8, 2024
1 parent 569ee1c commit 5186b7b
Show file tree
Hide file tree
Showing 5 changed files with 135 additions and 55 deletions.
77 changes: 39 additions & 38 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,11 @@

> One Simple Zero-Downtime Blue-Green Deployment with your Dockerfiles
Deploying web projects should be [simple, with high availability and security](https://github.com/Andrew-Kang-G/docker-blue-green-runner?tab=readme-ov-file#Quick-Guide-on-Usage).

- Use ``the latest Release version`` OR at least ``tagged versions`` for your production, NOT the latest commit of the 'main' branch.
- In production, place your project in a separate folder, not in the samples folder, as they are just examples.

## Table of Contents
- [Process Summary](#process-summary)
- [Features](#features)
- [Requirements](#requirements)
- [Quick Start with Samples](#quick-start-with-samples)
Expand All @@ -34,7 +33,6 @@ Deploying web projects should be [simple, with high availability and security](h
- [Consul](#consul)
- [USE_NGINX_RESTRICTION on .env](#use_nginx_restriction-on-env)
- [Advanced](#advanced)
- [Process Summary](#process-summary)
- [Gitlab Container Registry](#gitlab-container-registry)
- [Upload Image (CI/CD Server -> Git)](#upload-image-cicd-server---git)
- [Download Image (Git -> Production Server)](#download-image-git---production-server)
Expand All @@ -47,6 +45,44 @@ Deploying web projects should be [simple, with high availability and security](h

---

## Process Summary

- Term Reference
- ``All`` means below is "App", "Nginx", "Consul&Registrator".
- ``(Re)Load`` means ``docker run.... OR docker-compose up``.
- ``State`` is ``Blue`` or ``Green``
- More is on [Terms](#terms)
- Load Consul & Registrator, then the App, and finally Nginx to prevent upstream errors.


```mermaid
graph TD;
A[Initialize and Set Variables] --> B[Backup All Images]
B --> C[Check the .env File Integrity]
C --> D[Build All Images]
D --> E[Create Consul Network]
E --> F{Reload Consul if Required}
F -- Yes --> G[Reload Consul]
F -- No --> H[Load Your App]
G --> H[Load Your App]
H --> I[Check App Integrity]
I --> J{Reload Nginx if Required}
J -- Yes --> K[Check Nginx Template Integrity by Running a Test Container]
J -- No --> L[Check All Containers' Health]
K --> L[Check All Containers' Health]
L --> M{Set New State Using Consul Template}
M -- Fails --> O[Run Nginx Contingency Plan]
M -- Success --> N[External Integrity Check]
O --> N[External Integrity Check]
N -- Fails --> P[Rollback App if Needed]
N -- Success --> Q["Remove the Opposite State (Blue or Green) from the Running Containers"]
P --> Q["Remove the Opposite State from the Running Containers"]
Q --> R[Clean Up Dangling Images]
R --> S[Deployment Complete]
```
![img5.png](/documents/images/img5.png)

## Features

- **No Unpredictable Errors in Reverse Proxy and Deployment**
Expand Down Expand Up @@ -492,43 +528,8 @@ bash check-source-integrity.sh
- **For the properties of 'environment, volumes', use .env instead of setting them on the yml.**
- Set ```USE_MY_OWN_APP_YML=true``` on .env
- ```bash run.sh```
## Process Summary
- Term Reference
- ``All`` means below is "App", "Nginx", "Consul&Registrator".
- ``(Re)Load`` means ``docker run.... OR docker-compose up``.
- ``State`` is ``Blue`` or ``Green``
- More is on [Terms](#terms)
- Load Consul & Registrator, then the App, and finally Nginx to prevent upstream errors.
```mermaid
graph TD;
A[Initialize and Set Variables] --> B[Backup All Images]
B --> C[Check the .env File Integrity]
C --> D[Build All Images]
D --> E[Create Consul Network]
E --> F{Reload Consul if Required}
F -- Yes --> G[Reload Consul]
F -- No --> H[Load Your App]
G --> H[Load Your App]
H --> I[Check App Integrity]
I --> J{Reload Nginx if Required}
J -- Yes --> K[Check Nginx Template Integrity by Running a Test Container]
J -- No --> L[Check All Containers' Health]
K --> L[Check All Containers' Health]
L --> M{Set New State Using Consul Template}
M -- Fails --> O[Run Nginx Contingency Plan]
M -- Success --> N[External Integrity Check]
O --> N[External Integrity Check]
N -- Fails --> P[Rollback App if Needed]
N -- Success --> Q["Remove the Opposite State (Blue or Green) from the Running Containers"]
P --> Q["Remove the Opposite State from the Running Containers"]
Q --> R[Clean Up Dangling Images]
R --> S[Deployment Complete]
```
## Gitlab Container Registry
### Upload Image (CI/CD Server -> Git)
Expand Down
Binary file added documents/images/img5.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
53 changes: 38 additions & 15 deletions run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@
set -eu

source use-common.sh

display_checkpoint_message "Checking versions for supporting libraries...(1%)"

check_bash_version
check_gnu_grep_installed
check_gnu_sed_installed
Expand All @@ -18,7 +21,7 @@ sudo bash prevent-crlf.sh
git config apply.whitespace nowarn
git config core.filemode false

sleep 3
sleep 1

source ./use-app.sh
source ./use-nginx.sh
Expand Down Expand Up @@ -181,16 +184,19 @@ backup_to_new_images(){

_main() {

# [A] Get mandatory variables
display_checkpoint_message "Initializing mandatory variables... (2%)"

cache_global_vars
# The 'cache_all_states' in 'cache_global_vars' function decides which state should be deployed. If this is called later at a point in this script, states could differ.
local initially_cached_old_state=${state}
check_env_integrity

echo "[NOTICE] Finally, !! Deploy the App as !! ${new_state} !!, we will now deploy '${project_name}' in a way of 'Blue-Green'"
display_checkpoint_message "Deployment target between Blue and Green has been decided... (3%)"
display_planned_transition "$initially_cached_old_state" "$new_state"
sleep 2

# [A-1] Set mandatory files
## App
display_checkpoint_message "Setting up the app configuration 'yml' for orchestration type: ${orchestration_type}... (5%)"
initiate_docker_compose_file
apply_env_service_name_onto_app_yaml
apply_docker_compose_environment_onto_app_yaml
Expand All @@ -200,8 +206,13 @@ _main() {
if [[ ${skip_building_app_image} != 'true' ]]; then
backup_app_to_previous_images
fi


## Nginx
if [[ ${nginx_restart} == 'true' ]]; then

display_checkpoint_message "Since 'nginx_restart' is set to 'true', configuring the Nginx 'yml' for orchestration type: ${orchestration_type}... (7%)"

initiate_nginx_docker_compose_file
apply_env_service_name_onto_nginx_yaml
apply_ports_onto_nginx_yaml
Expand All @@ -216,7 +227,9 @@ _main() {
fi


# [A-2] Set 'Shared Volume Group'
display_checkpoint_message "Performing additional steps before building images... (10%)"

# Set 'Shared Volume Group'
# Detect the platform (Linux or Mac)
if [[ "$(uname)" == "Darwin" ]]; then
echo "[NOTICE] Running on Mac. Skipping 'add_host_users_to_host_group' as dscl is used for user and group management."
Expand All @@ -227,27 +240,32 @@ _main() {
fi
fi

# [A-3] Etc.
# Etc.
if [[ ${app_env} == 'local' ]]; then
give_host_group_id_full_permissions
fi
if [[ ${docker_layer_corruption_recovery} == 'true' ]]; then
terminate_whole_system
fi

# [B] Build Docker images for the App, Nginx, Consul


if [[ ${skip_building_app_image} != 'true' ]]; then
load_app_docker_image
display_checkpoint_message "Building Docker image for the app... ('skip_building_app_image' is set to false) (12%)"
load_app_docker_image
fi


if [[ ${consul_restart} == 'true' ]]; then
load_consul_docker_image
display_checkpoint_message "Building Docker image for Consul... ('consul_restart' is set to true) (14%)"
load_consul_docker_image
fi

if [[ ${nginx_restart} == 'true' ]]; then
load_nginx_docker_image
display_checkpoint_message "Building Docker image for Nginx... ('nginx_restart' is set to true) (16%)"
load_nginx_docker_image
fi


if [[ ${only_building_app_image} == 'true' ]]; then
echo "[NOTICE] Successfully built the App image : ${new_state}" && exit 0
fi
Expand All @@ -259,17 +277,21 @@ _main() {
(echo "[ERROR] Just checked all states shortly after the Docker Images had been done built. The state the App was supposed to be deployed as has been changed. (Original : ${cached_new_state}, New : ${new_state}). For the safety, we exit..." && exit 1)
fi

# [C] docker-compose up the App, Nginx, Consul & * Internal Integrity Check for the App
# docker-compose up the App, Nginx, Consul & * Internal Integrity Check for the App
display_checkpoint_message "Starting docker-compose for App, Nginx, and Consul, followed by an internal integrity check for the app... (40%)"
load_all_containers

# [D] Set Consul

display_checkpoint_message "Reached the transition point... (65%)"
display_immediate_transition ${state} ${new_state}
./nginx-blue-green-activate.sh ${new_state} ${state} ${new_upstream} ${consul_key_value_store}

# [E] External Integrity Check, if fails, 'emergency-nginx-down-and-up.sh' will be run.
display_checkpoint_message "Performing external integrity check. If it fails, 'emergency-nginx-down-and-up.sh' will be executed... (87%)"
re=$(check_availability_out_of_container | tail -n 1);
if [[ ${re} != 'true' ]]; then

echo "[WARNING] ! ${new_state}'s availability issue found. Now we are going to run 'emergency-nginx-down-and-up.sh' immediately."
display_checkpoint_message "[WARNING] ! ${new_state}'s availability issue found. Now we are going to run 'emergency-nginx-down-and-up.sh' immediately."
bash emergency-nginx-down-and-up.sh

re=$(check_availability_out_of_container | tail -n 1);
Expand All @@ -281,6 +303,7 @@ _main() {


# [F] Finalizing the process : from this point on, regarded as "success".
display_checkpoint_message "Finalizing the process. From this point, the deployment will be regarded as successful. (99%)"
if [[ ${skip_building_app_image} != 'true' ]]; then
backup_to_new_images
fi
Expand All @@ -304,7 +327,7 @@ _main() {
echo "[NOTICE] Delete <none>:<none> images."
docker rmi $(docker images -f "dangling=true" -q) || echo "[NOTICE] Any images in use will not be deleted."

echo "[NOTICE] APP_URL : ${app_url}"
display_checkpoint_message "[NOTICE] APP_URL : ${app_url}"
}

_main
11 changes: 9 additions & 2 deletions use-app.sh
Original file line number Diff line number Diff line change
Expand Up @@ -272,7 +272,7 @@ check_availability_inside_container(){

for (( retry_count = 1; retry_count <= ${total_cnt}; retry_count++ ))
do
echo "[NOTICE] ${retry_count} round health check (curl -s -k ${app_https_protocol}://$(concat_safe_port localhost)/${app_health_check_path})... (timeout : ${3} sec)" >&2
echo -e "\033[1;35m[NOTICE] ${retry_count} round health check (curl -s -k ${app_https_protocol}://$(concat_safe_port localhost)/${app_health_check_path})... (timeout: ${3} sec)\033[0m" >&2
response=$(docker exec ${container_name} sh -c "curl -s -k ${app_https_protocol}://$(concat_safe_port localhost)/${app_health_check_path} --connect-timeout ${3}")

down_count=$(echo ${response} | grep -Ei ${bad_app_health_check_pattern} | wc -l)
Expand All @@ -281,7 +281,14 @@ check_availability_inside_container(){
if [[ ${down_count} -ge 1 || ${up_count} -lt 1 ]]
then

echo "[WARNING] Unable to determine the response of the health check or the status is not UP, or Check the REDIRECT_HTTPS_TO_HTTP param in .env (*Response : ${response}), (${container_name}, *Log (print max 25 lines) : $(docker logs --tail 25 ${container_name})" >&2
echo -e "\033[1;35m[WARNING] Unable to determine the health check response, or the status is not UP. Please check the following:\033[0m
\033[1;35m1) Verify the REDIRECT_HTTPS_TO_HTTP parameter in .env\033[0m
\033[1;35m2) Confirm your app's database connection\033[0m
\033[1;35m3) Review your health check settings in .env.\033[0m
\033[1;35m*Response:\033[0m ${response}
\033[1;35m*Container:\033[0m ${container_name}
\033[1;35m*Logs (last 25 lines):\033[0m $(docker logs --tail 25 ${container_name})" >&2


else
echo "[NOTICE] Internal health check of the application succeeded. (*Response: ${response})" >&2
Expand Down
49 changes: 49 additions & 0 deletions use-common.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,55 @@ set -eu
source ./validator.sh
source ./use-states.sh


display_checkpoint_message() {
local message=$1
printf "\033[1;34m[CHECKPOINT] %s\033[0m\n" "$message" # Display message in bold blue
}

# Function to display a transition message between states in a Blue-Green deployment
display_planned_transition() {
local current_state=$1
local target_state=$2

# Clear the screen and set text to bold blue
echo -e "\033[1;34m" # Bold blue text

echo "─────────────────────────────"
echo " Current State (${current_state})"
echo "─────────────────────────────"
echo " |"
echo " >> Transition planned <<"
echo " v"
echo "─────────────────────────────"
echo " Target State (${target_state})"
echo "─────────────────────────────"
echo -e "\033[0m" # Reset text style
}

display_immediate_transition() {
local current_state=$1
local target_state=$2

# Display the state transition diagram with a bold blue message
echo -e "\033[1;34m" # Bold blue text

echo "─────────────────────────────"
echo " Current State (${current_state})"
echo "─────────────────────────────"
echo " |"
echo " >> Immediate Transition <<"
echo " v"
echo "─────────────────────────────"
echo " Target State (${target_state})"
echo "─────────────────────────────"
echo -e "\033[0m" # Reset text style
echo -e "\033[1;32m" # Bold green text for emphasis
echo ">>> Transition to ${target_state} is now being executed <<<"
echo -e "\033[0m" # Reset text style
}


to_lower() {
echo "$1" | tr '[:upper:]' '[:lower:]'
}
Expand Down

0 comments on commit 5186b7b

Please sign in to comment.