End-to-End Project: HiveBox

Almost everyone loves honey, and we at DevOps Hive love it too and appreciate the role that bees play for the planet! Because bees are essential to people and planet.

For that reason, in this roadmap our main hands-on project will be for the bees! We will utilize the technology and open source software to build an API to track the environmental sensor data from openSenseMap, a platform for open sensor data in which everyone can participate.

tip

If you are looking for the full roadmap including this project, back to the getting started page.

Project Badge

Push it forward and add the project badge in your repo to support the community! ⭐

Code:

[![Dynamic DevOps Roadmap](https://devopshive.net/badges/dynamic-devops-roadmap.svg)](https://github.com/DevOpsHiveHQ/dynamic-devops-roadmap)

Preview:

---

Approach

This project follows the same Dynamic MVP-style mindset used in the roadmap. Which aims to cover the whole Software Development Life Cycle (SDLC). That mean each phase of this project will cover all aspects of the DevOps areas like planning, coding, containers, testing, continuous integration, continuous delivery, infrastructure, etc.

This project works the best in Pairing mode where you have another person helps you whenever you stuck (like a mentor or so). But in case you don't have that, always back to the Related Module where it has more details could help you in the project.

note

Each phase of this project is tackled gradually part of the roadmap modules. But it's also standalone, and you can work on it if you have the required knowledge for each phase.

danger

Remember, the goal is not only to implement the project but also to learn industry best practices in the DevOps field! Just applying this project without understanding what the problems to solve will not help you to be a DevOps Engineer!

Goal

This project aims to build a real system from code to infrastructure and everything in between in but iterative phases

⭐ Click on the diagram to see it animated ⭐

The goal of this project is to build a scalable RESTful API around openSenseMap but customized to help beekeeper with their chores. The API output should be in JSON. You will start with a basic implementation, then extend the whole system to handles thousands of requests per second. But always remember, every decision has a cost.

You can get senseBox IDs by checking the openSenseMap website. Use 3 senseBox IDs close to each other (you can use this one 5eba5fbad46fb8001b799786 as starting point). Just copy the IDs, you will need them in the next steps.

---

Phase 1

Roadmap Module: Welcome to the DevOps World

• Understand your role in this project and how you work with other teams.
• Brush up on your knowledge about Software Project management (Hint: What is agile project management? And Introduction to Software Product Management).
• Decide which Agile methodology you will use. (Scrum, Kanban, Scrumban, etc.).
• Document as you go. Always assume that someone else will read your project at any phase.
• Avoid Scope Creep! Make it work, then make it right, then make it fast!
• Each change should be done in order and using pull requests (don’t push directly to the main branch!).
• Don't forget! You are managers of one!

---

Phase 2

Roadmap Module: Basics - DevOps Core

2.1 Tools

• Git
• VS Code
• Docker

2.2 Code

• Create GitHub repository for the project.
• Implement the code requirements.

Requirements:

• Create a function that print current app version. It should print the version then exit the application.
• Use Semantic Versioning for the app version starting with v0.0.1.

2.3 Containers

• Create Dockerfile for the project.
• Build the Docker image and run it locally.

2.4 Testing

• Locally, run the app container and ensure that it returns the correct value.

---

Phase 3

Roadmap Module: Start - Laying the Base

3.1 Tools

• Hadolint and VS Code hadolint extension
• Pylint and VS Code Pylint extension

3.2 Code

• Use Conventional Commits for Git commits.
• Familiarize yourself with openSenseMap API.
• Implement the code requirements (Hint: Flask Quickstart or FastAPI) .
• Write unit tests for all endpoints.

Requirements:

Version:

• Endpoint: /version
• Parameters: No parameters.
• Requirements:
  • Returns the version of the deployed app.

Temperature:

• Endpoint: /temperature
• Parameters: No parameters.
• Requirements:
  • Return current average temperature based on all senseBox data.
  • Ensure that the data is no older 1 hour.

3.3 Containers

• Apply Best Practices for containers (Hint: Docker Best Practices).

3.4 Continuous Integration

• Create a GitHub Actions workflow for CI.
• Add step to lint code and Dockerfile.
• Add step to build the Docker image.
• Add step to unit tests.
• Setup OpenSSF Scorecard GitHub Action and fix any issues reported by it.

3.5 Testing

• In the CI pipeline, call the /version endpoint and ensure that it returns the correct value.

---

Phase 4

Roadmap Module: Expand - Constructing a Shell

4.1 Tools

• Kind
• Kubectl

4.2 Code

• Implement the code requirements.
• Write integration test (Hint: 3 ways to test your API with Python).

Requirements:

General:

• The senseBox should be configurable via env vars.

Metrics:

• Endpoint: /metrics
• Parameters: No parameters.
• Requirements:
  • Returns default Prometheus metrics about the app.

Temperature:

• Endpoint: /temperature
• Parameters: No parameters.
• Requirements:
  • Add "status" field based on the temperature average value.
    • Less than 10: Too Cold
    • Between 11-36: Good
    • More than 37: Too Hot

4.3 Containers

• Create KIND config to run with Ingress-Nginx.
• Create Kubernetes core manifests to deploy the application.

4.4 Continuous Integration

• Run code integration tests.
• Run SonarQube for code quality, security and static analysis (Hint: Use SonarQube Quality Gate check action, also consider Semgrep).
• Run Terrascan for Kubernetes manifest misconfigurations and vulnerabilities (Hint: Terrascan GitHub Action).
• Apply Best Practices for CI (Hint: Best practices when adopting continuous integration).

4.5 Continuous Delivery

• Create a GitHub Actions workflow for CD.
• Add step to release by pushing a versioned Docker image to a container registry (Hint: Use GitHub Container registry).

---

Phase 5

Roadmap Module: Transform - Finishing the Structure

5.1 Tools

• Kind
• Kubectl

5.2 Code

• Implement the code requirements.

Requirements:

General:

• Add a caching layer using Redis based project Valkey.
• Add a storage layer using S3-Compatible project MinIO and store the data periodically every 5 minutes.

Store:

• Endpoint: /store
• Parameters: No parameters.
• Requirements:
  • By default, the application will store the data every 5 minutes, but by calling this endpoint, it should store the data directly on MinIO.

Metrics:

• Endpoint: /metrics
• Parameters: No parameters.
• Requirements:
  • Extend the Prometheus metrics by adding custom metrics based on the code logic.

Readyz:

• Endpoint: /readyz
• Parameters: No parameters.
• Requirements:
  • Returns HTTP 200 unless:
    • 50% + 1 of the configured senseBoxes are not accessible.
    • AND caching content is older than 5 min.

5.3 Containers

• Create a Helm chart for the application (Hint: Awesome Helm List).
• Create Kustomize manifests for the infrastructure resources like KeyVal (Hint: Awesome Kustomize List).
• Review Kubernetes Security Best Practices (Hint: Kubernetes Security Best Practices with tips for the CKS exam).
• Configure the Kubernetes app manifest to use /readyz as a readiness probe.

5.4 Infrastructure as Code

• Deploy Grafana agent to collect logs and metrics (Hint: Create Grafana Cloud free account to Use Loki and Grafana).
• Create a Kubernetes cluster using Terraform IaC (Hint: Use free tier from any Cloud provider).

5.5 Continuous Integration

• Create End-to-End test (Hint: Venom, a declarative test suite).
• Run KIND cluster, deploy the app and infrastructure, then run the End-to-End test.

5.6 Continuous Delivery

• Apply Best Practices for CD.

---

Phase 6

Roadmap Module: Optimize - Keep Improving

This phase a free-style user-defined enhancements which means the enhancements could be related to any part of the project.

Here are some highly recommended suggestions:

• Deploy the application in Declarative GitOps style using Argo CD.
• Prepare for production by setup DNS (ExternalDNS) and Certificates (Cert-Manager).
• Automate dependency updates with Dependabot.
• Move all external services (Like Grafana.com and Terraform Cloud) to Kubernetes cluster using open-source solutions.

Here are some extra suggestions:

• Setup Kyverno for Policy as Code.
• Build Multi-environment Kubernetes clusters (Dev, Stage, and Prod) with Terraform and Kustomize.
• Use TestKube for better testing execution.
• Develop a Kubernetes Operator to handle the app operations (Hint: Introduction to Kubernetes Operators).

---

Phase 7

Roadmap Module: Capstone Project

In this phase, you will design and impelment your own project similar to HiveBox project but with an idea and a tech stack of your choice. Check the Capstone Project page for more details.