Ever since I read "The architecture behind a one man SaaS" and "Tools and services I use to run my SaaS" I thought it would be cool to write about what powers breadNET and how I have things setup!
Let's start.
We will first take a look at what breadNET is, as the about breadNET page is pretty bad and is pending a re-write.
breadNET (Yes, it's typed like that) started as my home lab project and business venture where I would host FOSS software like Kanboard, Bookstack, Jellyfin, passbolt etc. Basically the stuff I use day to day. Sadly this never took and another company came in and took this opportunity, gap in the market if you will, and did a pretty good job of it. Kudos!
You can still by all means contact me to have me host these things for you for the cost of the server! Contact me via email or linkedin or what ever and we can work something out!
From there I decided just to change the site to a blog as for my job (just like everyone who works in IT) I spend a lot of time on google, and wanted to give back to the communities who rely on resources for help with things.
Enough chit-chat, let's dive in!
We will break this down in to a few categories:
- Hosted solutions
- Config management
- Servers and Software
- Backups
- Logging/ Monitoring
- Applications
- Cost
- Workflows
Hosted Solutions
There are things that are just not best to host your self when you're looking for the best uptime avalible.
Coudflare : DNS and DDOS protection
Digitalocean : VPS hosting for mail server and web server (Highly recommend)
OVH : Internal app hosting
Codefresh : Ci/Cd pipelines
Terraform cloud : Remote state for Terraform
Wasabi : S3 compliant backups for cheap (but very reliable)
Namecheap : Really good pricing for domains and my GO TO for anything domain related (Except DNS)
Config Management
This is the bane of my existence. In my ideal world anything I do I should be able to delete it and have it up and running again on Monday. (Let's be honest, this is more around me messing something up lol)
"Even if you lose all one day, you can build all over again if you retain your calm!" - Thuan Pham, former CTO of Uber.
Terraform : This is what I use for creating cloud deployments, all the way from a load balancer to a database as a service, terraform can do it
Ansible : This is what I use for provisioning my servers and getting them up to operating standards. Also use it on a cron job to keep all my servers up to date.
Gitlab : This is where all my code lives. I don;t know why I use this over github, but I like it :)
Servers and software
This little section is about the servers and software that power this place
Ubuntu : The choice OS for any server I deploy. This is what I grew up with, and this is what I know very well.
nginx : Once again, this is what I grew up with and know well. This powers EVERY web server I have ever deployed. Unless it's apache then that wast me!
mariadb : This is my go to database engine for any database that I require. If an application allows me to use mariadb, you bet I will use it!
rclone : This is an important piece of software I use for synching data between many different services, s3, drive, gcs etc...
intelliJ : This is the most beautiful IDE I have ever used, strongly recommend
direnv : Allows setting environment variables per directory, great for terraform and projects that need env variables
Backups
This is the most important part of any business or lab. Without backups, nothing is really important.
I follow the 3-2-1 rule, and I suggest you do!
The off site backups are designed to be used if something was to burn down or the backup generators not actually doing what they're meant to or routers just going "nah bro" - Okay, I'm done shitting on the cloud proving why you should have many backups!
backuppc : I know, the site looks bad and the UI is old, but boy does this software haul ass. Highly reccomend
S3 : See Wasabi from Above
Raspberry pi and a harddrive and a solid connection: This serves as the UK backup location
Monitoring/ Logging
This is one of those things that are often overlooked, but when shit hits the fan things go wrong, being able to look at a graph and point to a spike and go "yeah that's fucked broken" really helps, especially if you're able to then dial down in to each service and see what's happening.
Zabbix : Providing agent metrics, mtr, snmp and everything I can jam in to it, in one place as well as alerting
Datadog : Monitoring for cloud environments, little pricey but free tier is DECENT
libreNMS : as I move all resources to the cloud, this will be decommissioned, but really good for network monitoring where Zabbix just wont cut it.
Elastic stack : Coming soon! (I think?)
Applications
This is the stuff I use day to day, and will happily host for you if you pay me to do it.
- Ghost
- This is what runs my beautiful site
- Bookstack
- KB and how to articles
- Gitlab
- Source code and config managemnt lives here
- Jira
- Project managment software and a good attempt to organize my life
- Jellyfin
- Media server for all my legally sourced movies
- Grocy
- Manages my food
- firefly-iii
- Manages and makes me feel bad for spending money
- Passbolt
- Password manager
- Matomo
- Provides website analytics
- AWX
- Ansible tower for server stuff and updates
Cost
I've never done an exact break down but a rough estimate would look like
| Item | Count | Cost | Occurrence | Total Monthly | Total Yearly |
|---|---|---|---|---|---|
| Digital Ocean Droplet | 2 | $5 | Monthly | $10 | $120 |
| OVH Instance | 3 | £2.99 | Monthly | £10.76 | £129 |
| Wasabi Storage | idk | ($6 to 12) let's say $9 | Monthly | ~$9 | $108 |
So all in all, it costs me around about £295 at the time of writing this
(April the 10th at 3:33am like an idiot, my (new) girlfriend will be here in like 8 hours and this is what I decide to do... let's see how long she can survive seeing someone who works in IT and takes their hobbies very very seriously)
Now the reason I don't know about Wasabi is due to it being how much I use and delete per month. They don't charge upload and download so I can do that as much as I want, more so for storage and if you delete 1TB tomorrow, you pay for that TB for 3 months. Eh, sucks but i'm yet to find a better offering that is so simple.
Workflows
This is a strange one to write about as I am constantly learning new technology and moving things around, but let's look at an example that we're currently working on!
Moving my sheeeet to the cloud!
(shameless self plug below)
Bradley Stannard
Bradley Stannard
I have decided that I want to be able to simply deploy DNS records with minimal pain and agg, and to do this it would be best to do it though Terraform and then if I hadn't over complicated it enough, decided to automate the process of actually deploying it!
For this I have used Terraform, Gitlab and Codefresh
Below is an example of the codefresh.yml
version: '1.0'
stages:
- checkout
- prepare
- deploy
steps:
main_clone:
title: Cloning main repository...
stage: checkout
type: git-clone
repo: '<bang your repo url here>'
revision: master
git: gitlab
SetupAuth:
image: alpine:3.9
title: Configuring Auth
stage: prepare
commands:
- export TF_VAR_cloudflare_email=$CLOUDFLARE_EMAIL
- export TF_VAR_cloudflare_api_key=$CLOUDFLARE_API_KEY
DeployWithTerraform:
image: hashicorp/terraform:light
title: Deploying Terraform plan
stage: deploy
commands:
- terraform init -backend-config="token="$token""
- terraform apply -auto-approve
Second workflow would be creating infrastructure for a deployment
This way I ensure that terraform is uniform, and where the module already exists, I don't have to fart around with some strange issues.
Future plans are to fully opensource all code I write and move any secrets to environment vars so modules can be used anywhere!
My end goal here is to have everything under git control and cicd so I just describe something as code and then boom, it appears 3 minutes later.
You can hire me via Upwork or emailing me for weekend projects!