One of the core functionalities of this tool is to get a list of systems from an API. It then performs a search on the list based on given system name(s) and presents relevant information about them.

While at it, I thought that it might be worth exploring the possibility of making the search concurrent. Not that performance matters for this particular tool, but because I saw an opportunity to get a bit more out of my comfort zone and experiment with something that is still quite confusing to me. It could potentially improve the tool even though it is uncertain whether the performance gain would be significant enough to justify the implementation. At the end of the day, the worst that could happen is to get the code thrown away and still learn something.

The first test

The search is implemented as a simple linear search function. Therefore, I decided to take the easiest and most obvious approach, making the least amount of changes to the code in order to call the search function concurrently and compare it to the initial code.

For the first attempt, the goal was to test two functions that perform the same task: one non-concurrent and one concurrent. I used a sample list with 65 systems loaded from a JSON file.

As a self-proclaimed lazy noob, instead of completely changing the original function, I created a new one based on the original and made the desired changes.

// Initial Code
func searchName(name string, systems []System) *System {
	for _, s := range systems {
		if s.Name == name {
			return &s
		}
	}

	return nil
}

func SystemsByName(fileName string, names []string) (systems []System, err error) {
	allSystems, err := AllSystems(fileName)
	if err != nil {
		return nil, err
	}

	for _, name := range names {
		if system := searchName(name, allSystems); system != nil {
			systems = append(systems, *system)
		} else {
			return systems, fmt.Errorf("%w: %s", ErrorNameNotFound, name)
		}
	}

	return systems, nil
}

// The new funcion
func GoSystemsByName(fileName string, names []string) (systems []System, err error) {
	allSystems, err := AllSystems(fileName)
	if err != nil {
		return nil, err
	}

	systemsChan := make(chan System, len(names))
	errorChan := make(chan error, len(names))
	var wg sync.WaitGroup

	for _, name := range names {
		wg.Add(1)
		go func(systems []System, name string) {
			defer wg.Done()

			if system := searchName(name, allSystems); system != nil {
				systemsChan <- *system
			} else {
				errorChan <- fmt.Errorf("%w: %s", ErrorNameNotFound, name)
			}
		}(allSystems, name)
	}

	go func() {
		wg.Wait()
		close(systemsChan)
		close(errorChan)
	}()

	for system := range systemsChan {
		systems = append(systems, system)
	}

	if len(errorChan) > 0 {
		return systems, <-errorChan
	}

	return systems, nil
}

For the benchmark, I randomly searched for some names that I knew existed in the list, as well as one name that didn’t exist.

// Benchmarking
func BenchmarkSystemsByName(b *testing.B) {
	for i := 0; i < b.N; i++ {
		_, _ = SystemsByName("randomNameX", "randomNameY", "randomNameW", "randomNameZ", "foo")
	}
}

func BenchmarkGoSystemsByName(b *testing.B) {
	for i := 0; i < b.N; i++ {
		_, _ = GoSystemsByName("randomNameA", "randomNameY", "randomNameB", "randomNameG", "foo")
	}
}

The results:

$go test -bench=.
goos: linux
goarch: amd64
pkg: toolbox/pkg/govcm
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
BenchmarkSystemsByName-8            2815            395575 ns/op
BenchmarkGoSystemsByName-8          2424            491929 ns/op
PASS
ok      toolbox/pkg/govcm       2.409s

Non-Concurrent: 2815 operations, with an average of 395575 ns per operation.

Concurrent function: 2424 operations, with an average of 491929 ns per operation.

Well… This was unexpected! The non-concurrent version of the search performed better than the concurrent one. But… this doesn’t seem right. The concurrent search should be faster, right? What am I doing wrong?!

As someone who is driven by curiosity, I made the decision to delve further and explore the next most apparent thing.

Moar Data!

Maybe its just my code that sucks, but for sake of simplicity, let’s just assume not, after all, when making the function I added some prints to it and actually saw it working as intended and it passed all the unit tests in the original code! When did this ever go wrong riiiiight? let’s move on …

The next most obvious thing is to test it with more data and try to look for a change in the benchmark values. From my understanding, concurrency also adds some overhead, so it may not provide any benefits for smaller amounts of data.

After reading a bit more about benchmarking in Go and finding benchstat, I decided to change the way I was doing the benchmark a bit, ensuring that everything is tested in the same way, making it easier to analyze the results. I refactored the tests to only test the SystemsByName() function, and the concurrent and non-concurrent functions have the same name, with one of them always commented out depending on what I want to test.

For the test cases, I decided to benchmark against datasets with 100, 1000, 10000, 100000, and 1000000 entries. I will search for the first element in the list, an element in the middle of the list, the last element of the list, and a non-existing element.

package main

import "testing"

const (
	file100 = "data/systems_100.json"
	file1000 = "data/systems_1000.json"
	file10000 = "data/systems_10000.json"
	file100000 = "data/systems_100000.json"
	file1000000 = "data/systems_1000000.json"
)

// Benchmarking
func BenchmarkNameSearch(b *testing.B){
	b.Run("100 Systems", func (b *testing.B) {
		names := []string{"system0", "system50", "system99", "foo"}
		for i := 0; i < b.N; i++ {
			_, _ = SystemsByName(file100, names)
		}
	})

	b.Run("1000 Systems", func (b *testing.B) {
		names := []string{"system0", "system500", "system999", "foo"}
		for i := 0; i < b.N; i++ {
			_, _ = SystemsByName(file1000, names)
		}
	})

	b.Run("10000 Systems", func (b *testing.B) {
		names := []string{"system0", "system5000", "system9999", "foo"}
		for i := 0; i < b.N; i++ {
			_, _ = SystemsByName(file10000, names)
		}
	})

	b.Run("100000 Systems", func (b *testing.B) {
		names := []string{"system0", "system50000", "system99999", "foo"}
		for i := 0; i < b.N; i++ {
			_, _ = SystemsByName(file100000, names)
		}
	})

	b.Run("1000000 Systems", func (b *testing.B) {
		names := []string{"system0", "system50000", "system999999", "foo"}
		for i := 0; i < b.N; i++ {
			_, _ = SystemsByName(file1000000, names)
		}
	})
}

Now that I have determined how I want to benchmark, I just need the actual data to run the benchmarks against. So, I have created a quick and simple Python script to generate multiple JSON files with different numbers of entries. This data also resembles the data returned by the original API.

#!/bin/python

import json
import uuid
import random
import sys

total = int(sys.argv[1])
file=(f'systems_{total}.json')
regions = ["eu-north-1", "eu-west-1", "eu-west-2", "us-west-1"]
states = ["STATE", "PLAN_FAILED", "RESUME_FAILED", "PAUSED", "STORAGE_CREATION_FAILED", "FOO_FAILED"]
auxStates = ["PLANING", "RESUMING", "PAUSING", "TO_BE_PLANED", "TO_BE_PAUSED"]
types = ["team", "lab"]
versions = ["1.2.3-gec414e3c24", "2.3.2-bde194c123a", "3.4.0-foobarbaz"]
items = []

for i in range(total):
    item = {
        "id" : str(uuid.uuid4()),
        "name" : (f'system{i}'),
        "region" : random.choice(regions),
        "state" : random.choice(states),
        "auxState" : random.choice(auxStates),
        "apiEndpoint" : (f'https://system{i}.mydomain.com'),
        "systemType" : random.choice(types),
        "systemVersion" : random.choice(versions)
    }
    items.append(item)

data = {"items": items}

with open(file, "w") as f:
    json.dump(data, f, indent=2)

Now that I have the data, it’s time to run the tests. This time, I will also run them with different settings: default, 20 seconds, 30 seconds, 20 times, and 30 times. It is important to note that the test environment is not perfect, but neither is the world, and I am not looking for super detailed results. All I want to find are some hints about whether it’s worth using concurrency on linear searches and, if so, at what point.

Benchstat

I will not provide all the results here, but the results were consistent across all the different benchmark settings.

$benchstat test_default.txt test_concurrent_default.txt
goos: linux
goarch: amd64
pkg: concurrency
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
                             │ test_default.txt │     test_concurrent_default.txt      │
                             │      sec/op      │    sec/op     vs base                │
NameSearch/100_Systems-8           248.9µ ± ∞ ¹   262.5µ ± ∞ ¹       ~ (p=1.000 n=1) ²
NameSearch/1000_Systems-8          2.445m ± ∞ ¹   2.604m ± ∞ ¹       ~ (p=1.000 n=1) ²
NameSearch/10000_Systems-8         26.01m ± ∞ ¹   26.33m ± ∞ ¹       ~ (p=1.000 n=1) ²
NameSearch/100000_Systems-8        259.4m ± ∞ ¹   259.0m ± ∞ ¹       ~ (p=1.000 n=1) ²
NameSearch/1000000_Systems-8        2.553 ± ∞ ¹    2.534 ± ∞ ¹       ~ (p=1.000 n=1) ²
geomean                            25.36m         25.97m        +2.42%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05

Even though the benchmark results indicate that the non-concurrent search is generally more efficient than the concurrent search, it is possible to observe that an inversion starts to occur after reaching 100,000 items. At this point, the concurrent search becomes slightly more efficient, although the difference is not significant.

However, just because concurrency is widely available and relatively easy to implement, it does not mean that we should always use it. We should be mindful of when and how we should use it because it may actually harm performance.

Wait, Karen, wait, before you start throwing rocks… when I say performance, I don’t just mean code performance. Your team’s performance may also suffer. When opting for a concurrent solution, your team now has to deal with the added complexity in the code when things go wrong, without any real benefit. One could argue that when dealing with data sets larger than 10k items, it may be better to study and learn about other algorithms instead.

Perhaps that’s a story for another time!

If you’ve made it this far in this article, congratulations! I hope it was at least useful. To me, these results still don’t make a whole lot of sense, but I guess it’s time to study a bit more.

Are my results far off from reality? Am I completely wrong? What am I missing? Feel free to reach out to me and let me know what you think!

The code and test results can be found here

Before starting, I think it’s important to go back a little in time to explain why I’m taking on this challenge.

Throughout all my years in the IT field, I never seriously dedicated myself to learning a programming language. I learned a bit of Pascal, C++, and Visual Basic in school, which gave me a basic understanding of programming, but never sparked my interest in learning more, and to be honest, I don’t even remember anything! Maybe because school projects were boring and never motivated me to learn more, or maybe because I was more interested in hardware than software development. A few years later, when I started focusing more on Linux, I learned Bash, which has been quite useful for automating various tasks, including creating several tools for my team in our daily activities in production environments.

However, I’ve been having more and more ideas for different projects (some more foolish than others), and the need to learn a programming language arose! I started learning Python, and my interest grew. I did some small and fun projects, but decided to slow down, change course, and learn Go! Why? Simply because! All the tools I work with daily are written in Go, and after some time thinking, I consider Go to be the most suitable language for the projects I have in mind. Maybe I’m wrong, but… as my grandfather used to say, “Knowledge doesn’t take up space!”

After several days studying the basics of Go, I decided to start solving the problems of Advent of Code and started with the year 2015, which brings us to today’s topic and discovery!

The problem

You can learn more about the challenge here. But, in a very brief summary, the problem consists of calculating the floor of an elevator based on a series of parentheses that indicate if the elevator goes up or down. For example, if the elevator starts on floor 0 and the input is ((( or (()(()(, the elevator will go up to floor 3. If the input is ()), the elevator will go down to floor -1.

Ooh! This is easy, I thought in my innocence!

First, a function called elevator that receives the instructions and returns the floor where the elevator ends up.

func elevator(instructions string) int{
	var floor int
	for _, instruction := range instructions {
		if instruction == '(' {
			floor++
		} else if instruction == ')' {
			floor--
		}
	}

	return  floor
}

And then I just have to ask “Santa Claus” to input the instructions, print the result and it’s done!

func main() {
	var instructions string
	fmt.Print("What floor you want to go?: " )
	fmt.Scan(&instructions)

	lastFloor := elevator(instructions)
	fmt.Println("final Floor: ", lastFloor)
}

I introduce the instructions and, whoa! I’m a machine! Everything works perfectly! (())) returns 3 and ()) returns -1.

I take the challenge input and… Wrong!!! For more experienced programmers, this might be quite obvious, but for little innocent me, it was enough to spend the afternoon banging my head against the wall.

First, I focused on the fact that the result was wrong and decided to add an extra print, hoping to understand what was going on.

func main() {
	var instructions string
	fmt.Print("What floor you want to go?: " )
	fmt.Scan(&instructions)
  fmt.Println("instructions: ", instructions)

	lastFloor := elevator(instructions)
	fmt.Println("final Floor: ", lastFloor)
}

Hey! The output of the instructions is not the same as the input! What’s happening here? And if I read from a text file? Will I have the same problem? I changed the code to read from a text file:

func main() {
	instructions, err := os.ReadFile("input.txt")
	if err != nil {
		panic(err)
	}

	floor := elevator(string(instructions))
	fmt.Println("Final floor:", floor)

This time, the result was different. I entered the result on the website and… Correct! But why? What’s going on here? I got the correct answer for the exercise, but I still couldn’t understand what was happening. So, I decided to compare the result of the input made by the user and the input read from the text file.

My first idea was to check if the strings had the same length and compare the result of both.

func main() {
	var instructionsFromInput string
	fmt.Print("What floor you want to go?: " )
	fmt.Scan(&instructionsFromInput)
	fmt.Println("instructions: ", instructionsFromInput)
	fmt.Println("Length from input: ", len(instructionsFromInput))

	instructionsFromFile, err := os.ReadFile("../01/input.txt")
	if err != nil {
		panic(err)
	}
	fmt.Println("Lenght from file: ", len(instructionsFromFile))
	lastFloorFromInput := elevator(instructionsFromInput)
	lastFloorFromFile := elevator(string(instructionsFromFile))
	fmt.Println("Final Floor From input: ", lastFloorFromInput)
	fmt.Println("Final Floor from file: ", lastFloorFromFile)
}

EUREKA! I’m a noob! Or am I not?

$go run main.go
What floor you want to go?: (((())))()((((((((())()(()))(()((((()(()(((()((()((()(()()()()()))(((()(()((((((((((())(()()((())()(((())))()(()(()((()(()))(()()()()((()((()(((()()(((((((()()())()((((()()(((((()(())()(())((())()()))()(((((((())(()())(()(((())(()))((())))(()((()())))()())((((())))(()(((((()(())(((()()((()((()((((((((((())(()())))))()))())()()((((()()()()()()((((((())())(((()())()((()()(((()()()))(((((()))(((()(()()()(()(()(((())()))(()(((()((())()(()())())))((()()()(()()(((()))(((()((((()(((((()()(()())((()())())(()((((((()(()()))((((()))))())((())()()((()(()))))((((((((()))(()()(((())())(())()((()()()()((()((()((()()(((())))(()((())()((((((((()((()(()()(((())())())))(())())))()((((()))))))())))()()))()())((()())()((()()()))(()()(((()(())((((())())((((((((()()()()())))()()()((((()()))))))()((((()(((()))(()()())))((()()(((()))()()())())(((())((()()(())()()()(((())))))()())((()))()))((())()()())()())()()(()))())))())()))(())((()(())))(()(())(()))))(()(())())(()(())(()(()))))((()())()))()((((()()))))())))()()())((())()((()()()))()(((()(()))))(())()()))(((()())))))))))(((())))()))())()))))()()(((())))))))()(()()(()))((()))))((())))((()((())))())))()()(()))())()(()((()())(()(()()())())(()()))()))))(()())()()))()()()()))(()(()(()))))))()(()))()))()()(()((())(()(())))()(((())(())())))))()(()(()))))()))(()()()(())()(()(())))()))))()()(((((())))))())()())())())()())()))))()))))))))())()()()()()()())))()))((())()))())))()((())()))))()))())))))))())()()()))()()(()((((()(((((((()(())((()())((()()))()))))(())))()()()(())((())()())))(())))(())))(((()()))()(())(((()(()))((())))())()))((((()))())()))))))))()(())())))(()))()(()()))())()()(())())))())()()(()())))()((()())(()(())(())))))))))))))(()))))()))))))()()())(()(((((()(()())))())()))(()))()))(()()))()())(()))())()(())((()()))))))())))())()(((())))(()(()))()()))()(()))))))((()())(()))))))()())))()()))))))))((((((((()()()(()))))))()())))())))()()((())()))((())(())))())())))()()()((()((()(())))())()(())))))))))()())))()()()()()()))()))((())())(()(()))))))(()()))()))(())))()))))))))))))(()))))))))()))))()))()())()))()()))))))()))))((()))))(()))())()(())))(()())((((()())))()))))(()))()(()()(())))))())))))()))))))())))())))))())))())())))())(()))))(())()(())))())()))((()()))))))())))((())))))))())))(())))))()()())))))())))))()))))))()))()()()(()(((()())())())(()))())))))((()(())(()))))))))(())))()()()())())(()))))()()()))()))())())())()(())))()(((()((((())))))))()))))))))))))))))))))((())()())(()))))()()))))))(()()(())())))())))((())))((())))))))))))))()))))()(()))))))())))))()))(()()())(()())))))))))()))))))(())))))()()))()())(((())))()))(()))))))))(())())))())))())())())()()))((())()(())()())()))()())(())(()))))()())))(()(((()))))))()(()())()()()))()))))))))()()()(())()())()(((((()))()())())(()))))()()()(())))())))()((()())))(()))())()(()())())(()))()()))((()()))((()()()()())))(())()))(()(())))((()()))))))))())))))))())()()))))))))))))))))(())()(())(())()())())()))()(()))))())())))))()())()(()))()()(())))(())())))))(()))))))))))))))())())(())(())))(((()))()))))())((())(()))())))))))())))))())))()))()))))))))))))())()))))()))))((()))(())))()(())))(())()))()))())))())))))))()(()())())))()()())))(())))))(()))))))))))))(()))()))()))())))(((()()()(())((()())))()())(((()))(())()))((()()()())))())(())(()))))()(((((())))(()))())())))))))((((()()()))())())()(()(()())))))))))()())())))(())))()())(((()(())())()()))())())))))))((()())((()()(()))(()(())))()))()))(()))(()))()()(()(((())((((()))()(()))((())()(()(()())()(()))()())))))(()))()))())()())))())))(())))((())(()())))))()))(())(()))()())()(()()((()(()))))))()(())(()())(())()))(((())()))(()()(()()()))))(()(())))()))))())))))())(()()()()()()(((())))(()()))()((())(((((()()())))(()))(()))()()))(((())())()(((()()()()))))(()))(())())))()())(()()())())))))))()))))((())))()())(()))(()(()))())))))())(())))))()()())())()))()()(())))(()))(())((((((())(()))(()))())()))(()()(())))()))(()()))()))()(())))(())))((()(()))(())()()())())))(((()()())(())()))))))()(((()(((((()()(((())(())))())()((()))))((()())()(())(((())))(((()((()(()(()))(()()))())(()))(())(())))()))))))((((()))()((((()(()))()))()()))))()(()(()))()(()((()(((()(()()(((()))))()(((()(()(()(((()(()())())()()(()(()())())(()((((())(()))()))(((((()()())(())()((()()())))()()(((()()))()((((((((()(())))())((()))))(())))(()))))((()((((()()(())(((((()))(((((((((((((()())))((((()(((()((())())()))((()))()(()()((()()()()(()()(()(()(((())()(()((((((()((()()((())()((((()((()()(()()())((()()()((()((())()(()(((()((())((((())(()))((()(()))(()())()((((((((()(((((((((((()))(()(((()(()()()((((())((())()())()))(())((())(()))(((()((()(())))(()))))((()()))))((((()(()(()())(()(())((((((((()((((()((()(((((()))())()(()))(()()((()(())(((((()(())()(((((()()))))))()(((())()(()()((((())()((())((()(((())(((()))((()()((((()(())))))((()((((()((()((()(((())((()))(((((((()(((()((((((((())()))((((())(((((()((((((((()(((()((()(((()()(((()((((((()()(()((((((((()()(()(()(())((((()())()))))(((()))((((())((((()())((()(())()((()((((((()((((((()(())))()())(((())())())()(())()(()())((()()((((())((((((())(()(((((()((((())()((((()(()(())(()())(((())()((())((((()))()((((((())(()(((()(((()((((((()(((()))(()()())())((()((()())()((((())(((()(()(((((((((())(())))()((()()()()(())((()))(((((((()(((((((((()(()))))(()((((((((()((((()((()()((((((()()(((((((()(()(())()(())((()()()((()(((((()())()(((((()())()()((()(()())(()()()(((()()(((((()((((((()()((()(()()()((((((((((((()((((((((()()(((()())))()(((()()(())())((((()((((()((((()()()(())(())((()(()(((((((((((((((()(())(())))))()()))((()(((()(())((()(((()(()()((((()()(((()(((()(((((()()((()(()(((()))((((((()((((((((()((()((())(((((()(((())(())())((()()))((((())()()((()(((()(((((()()(((()))(((()(()(((((((((((((()))((((((((()(((()))))())((((((((((((())((())((()())(((())((())(()((((((((((()(((())((()()(()((())(((((((((((()))((((((((((((()(()())((()((()((()(()(((()((((((((()()(()((()(()(((()))((()))(((((((((((((()(())((((((())(((()(())(()(()(()((()()))((((()((((()((((())))())((((()((((()))((((((()((((((()((()(((())))((())(()))(()((()((((()((()(((()()))((((()()()(((((((())(((())(()))())((((()())(((()(((((((((((()(()(()((()(((((((((((((((()()((((()((((((((()(((()()((()((((()))(((()(())((((((()((((())()((((()((()))(())()(()(((()((())())((((((()(()(())())(((())(()(()())(((((()((()((())()())(())))(((()(())))))))(((()(((()))()((()(((()()((()())()()))())))(((()))(()(((()(((((((((()(()(((((()()(((()())()()))))()(((()))(((()(()(()(()(()))()(())()))(()(((())))(()))))))))))(())((()((())((()(())()(())((()()((((()()((()()))((())(((()((()(())(())))()(()(((((()((()))())()(((((()()(((()(()((((((())(()))(())()))((()(()()))(())())()))(((())))(()((()(((())(())())))((()()((((((((((((((()((()(()()(()(((()))())()()((()()()(())(()))(()())(((())((())()(())()()(()()(())))((()(((()))))(((()()(()()))())((()((())()))((((()()()())((())))(((()(())(((((()(((((()((()(()((((()()(((()()()(((()())(((()()((((())(()))(((()))(())())((()))(((()((()))(((()()((())((()(((((()((((()()())((()))()((((()((()(()()()(

Length from input:  4095
Lenght from file:  7001
Final Floor From input:  -633
Final Floor from file:  232

Hmm… interesting. When I pass the elevator instructions through the input in the terminal, the result is different because some instructions are missing! Here everything started to make more sense and, after a little research, I found out that the terminal buffer has a limit of 4096 bytes, and everything beyond that is cut off.

Wow, what a big noob! Usually, when I know that the input is large, I always use a file, especially since I usually get the data from somewhere, save it to a file, and then read and process it. But in this case, since Santa Claus enters an elevator and places instructions for it, I thought it would be nicer to take the input from the website and pass it directly through the terminal, but no!

Part 2

Regarding part 2, it was fairly easy and all the code for this exercise is available on my GitHub repository.

As I want to try to use the hardware I have available as efficiently as possible, I decided it was time to change (again!!). Since I had already used ProxMox before, I decided to go back to ProxMox with the idea of passing the HBA to a VM and having TrueNAS in a VM only as a NAS.

ZFS on ProxMox

A little research and I found out that ProxMox supports ZFS, which made me jump for joy. I didn’t think twice and got to work!

One of the pools with 2 250GB SSDs in zraid0 has been experiencing some errors and since I don’t want to risk losing anything there (even though it’s not important), I decided to make some upgrades.

Hardware Upgrade

1 x NVMe Samsung 990 EVO 1TB for my main Windows machine, where my local storage needs are a bit higher due to gaming and video editing.

And the NVMe that was in the main machine goes to the server along with a 1TB SSD that I had lying around for another project I never started.

And my current setup is:

Desktop (Main machine), dual boot, 1x TB NVMe Windows + 500GB NVMe Linux

Main server: Proxmox, zraid2 22TB + zraid0 500GB SSD + zraid0(single disk) 1TB + zraid0(single disk) 500GB NVMe.

Transition to ProxMox

After planning the transition, it was time to export the pools in TrueNAS to ensure that I could import everything properly after switching to ProxMox, prepare a USB with ProxMox ISO, and install the new system.

After installing ProxMox, the pools were immediately recognized and imported, but since I wanted to change the pool names, I exported them again, imported them with new names, and created new pools.

I moved the datasets to their new destinations.

zfs create pool/datasetname
rsync -arv oldpool/datasetname newpool/datasetname

Install Docker and Docker Compose, which is what I use on my home server.

Change the Docker Compose with the new paths for the mounts.

docker-compose up -d

And as expected… some errors… OpenTelemetry Collector, port 8006 is already in use…

I have the containers configured to send logs with Fluentd format to the OpenTelemetry Collector container on port 8006 and by default, ProxMox has the interface on port 8006. It doesn’t seem that simple to change this, so let’s go for the easier option… change the OpenTelemetry Collector configuration:

fluentforward:
    endpoint: 0.0.0.0:8007

And change the container configuration.

sed -i s/8006/8007/g
docker-compose up -d

Again and… Boom! everything working again!

The little snowflake!

Downloading Linux ISOs is being done with a Raspberry Pi using Deluge in Docker and connected to a VPN 100% of the time.

ProxMox doesn’t have any of these NAS, NFS/Samba, etc. capabilities. Well… it does, but not in the same way as a NAS operating system.

I thought of an LXC container with Debian, install Deluge, install Wireguard, wg-quick0, Deluge services, and so on!

But I didn’t like it… too much to do manually, I’m a very lazy guy, remember?

A little more research and I found out that ZFS supports NFS and Samba sharing natively, without major complications… EASY!

zfs set sharenfs="rw=@192.168.1.20/24,no_root_squash,no_subtree_check" tempstorage/Downloads

Update the fstab on the Raspberry Pi with the new share location, mount -a, done!

ZFS Trimming & Scrubbing

This is easy… unlike TrueNas which had to be configured, ProxMox is already set up by default!

root@proxmox:~# cat /etc/cron.d/zfsutils-linux
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# TRIM the first Sunday of every month.
24 0 1-7 * * root if [ $(date +\\\\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/trim ]; then /usr/lib/zfs-linux/trim; fi

# Scrub the second Sunday of every month.
24 0 8-14 * * root if [ $(date +\\\\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ]; then /usr/lib/zfs-linux/scrub; fi

Backups

Although TrueNas makes this task easier, it’s not exactly more complicated on ProxMox either. ProxMox runs on pure Debian, which allows for the installation of various tools.

For automatic ZFS snapshots, apt install zfs-auto-snapshot. By default, it already configures the cronjobs in /etc/cron.* to keep several snapshots, intra-hourly (every 15 minutes), hourly, weekly, and monthly. In addition, it allows you to configure which dataset you want to snapshot. Just use the command zfs set com.sun:auto-snapshot=fale|true snapshot name.

zfs get com.sun:auto-snapshot
NAME                                                           PROPERTY               VALUE                  SOURCE
HomeCloud                                                      com.sun:auto-snapshot  false                  local
HomeCloud/Docker                                               com.sun:auto-snapshot  true                   local
HomeCloud/Docker@zfs-auto-snap_frequent-2023-04-30-2315        com.sun:auto-snapshot  true                   inherited from HomeCloud/Docker
HomeCloud/Docker@zfs-auto-snap_hourly-2023-04-30-2317          com.sun:auto-snapshot  true                   inherited from HomeCloud/Docker
HomeCloud/Docker@zfs-auto-snap_frequent-2023-04-30-2330        com.sun:auto-snapshot  true                   inherited from HomeCloud/Docker
HomeCloud/Docker@zfs-auto-snap_frequent-2023-04-30-2345        com.sun:auto-snapshot  true                   inherited from HomeCloud/Docker
HomeCloud/Docker@zfs-auto-snap_frequent-2023-05-01-0000        com.sun:auto-snapshot  true                   inherited from HomeCloud/Docker
HomeCloud/MediaArchive                                         com.sun:auto-snapshot  true                   local
HomeCloud/MediaArchive@zfs-auto-snap_frequent-2023-04-30-2315  com.sun:auto-snapshot  true                   inherited from HomeCloud/MediaArchive
HomeCloud/MediaArchive@zfs-auto-snap_hourly-2023-04-30-2317    com.sun:auto-snapshot  true                   inherited from HomeCloud/MediaArchive
HomeCloud/MediaArchive@zfs-auto-snap_frequent-2023-04-30-2330  com.sun:auto-snapshot  true                   inherited from HomeCloud/MediaArchive
HomeCloud/MediaArchive@zfs-auto-snap_frequent-2023-04-30-2345  com.sun:auto-snapshot  true                   inherited from HomeCloud/MediaArchive
HomeCloud/MediaArchive@zfs-auto-snap_frequent-2023-05-01-0000  com.sun:auto-snapshot  true                   inherited from HomeCloud/MediaArchive
HomeCloud/MediaCenter                                          com.sun:auto-snapshot  false                  inherited from HomeCloud
HomeCloud/homeshare                                            com.sun:auto-snapshot  false                  inherited from HomeCloud
nvmestorage                                                    com.sun:auto-snapshot  false                  local
ssdstorage                                                     com.sun:auto-snapshot  false                  local
ssdstorage/minecraft                                           com.sun:auto-snapshot  true                   local
ssdstorage/minecraft@zfs-auto-snap_frequent-2023-04-30-2315    com.sun:auto-snapshot  true                   inherited from ssdstorage/minecraft
ssdstorage/minecraft@zfs-auto-snap_hourly-2023-04-30-2317      com.sun:auto-snapshot  true                   inherited from ssdstorage/minecraft
ssdstorage/minecraft@zfs-auto-snap_frequent-2023-04-30-2330    com.sun:auto-snapshot  true                   inherited from ssdstorage/minecraft
ssdstorage/minecraft@zfs-auto-snap_frequent-2023-04-30-2345    com.sun:auto-snapshot  true                   inherited from ssdstorage/minecraft
ssdstorage/minecraft@zfs-auto-snap_frequent-2023-05-01-0000    com.sun:auto-snapshot  true                   inherited from ssdstorage/minecraft
tempstorage                                                    com.sun:auto-snapshot  false                  local
tempstorage/Downloads                                          com.sun:auto-snapshot  false                  inherited from tempstorage
tempstorage/Misc                                               com.sun:auto-snapshot  false                  inherited from tempstorage
tempstorage/Resolve                                            com.sun:auto-snapshot  false                  inherited from tempstorage
tempstorage/minecraft                                          com.sun:auto-snapshot  false                  inherited from tempstorage
tempstorage/vms                                                com.sun:auto-snapshot  false                  inherited from tempstorage

Since snapshots are not backups, a cronjob with rsync to the backup storage does the trick!

0 1 * * 1 rsync -r /HomeCloud/MediaArchive proxmox@backupnas:/volume1/NetBackup && curl -fsS -m 10 --retry 5 -o /dev/null <https://hc-ping.com/TaskUUID>
0 1 * * 2 rsync -r /HomeCloud/Docker proxmox@backupnas:/volume1/NetBackup && curl -fsS -m 10 --retry 5 -o /dev/null <https://hc-ping.com/TaskUUID>

And to be honest, I find this method much more efficient!

I never look at the TrueNas dashboard and never know exactly if everything worked or not, I just believe that the notification email will arrive if something goes wrong.

In this case, if rsync runs successfully, it sends a ping to healthchecks.io, if it fails, it doesn’t send and I receive notifications on Discord and email. In addition, I can configure the Grafana dashboard to fetch the status of each ping!

Final Result

The main pool zraid2 22TB: everything that’s important!

The secondary pool Stripe 550GB SSD: temporary shares, test VMs, and those Linux ISOs that come through torrents!

The secondary pool 1TB SSD Single Disk: VMs / Containers that require faster disks (game servers, for example).

The 500GB NVMe Pool: Windows VM as a secondary Windows machine.

What am I going to do with all this? Well… I don’t know yet, I’ll find out soon, but it will certainly fill up again when I start making skate and snowboard videos again.

TL;DR

Although ProxMox is not designed to serve as a NAS, it is quite flexible and capable of playing that role.

The virtualization capabilities benefit me more than NAS features, considering that all my file sharing goes through NextCloud running in Docker. And if I eventually really need an operating system specifically designed for NAS and file sharing, I can easily virtualize it and pass the hardware to the VM.

The migration was quite simple and took me no more than a couple of hours to get everything operational. In total, it took me half a dozen hours from the moment I turned off the server until everything was ready!

Finally, I wanted to leave an updated diagram with this change, but I don’t know what I did with the original. So, see here. Everything is the same, but instead of TrueNAS, I’m using ProxMox.

I will use this website as an example, but it applies to virtually everything I maintain, from my personal websites, projects, PiVPN, or other applications.

Development: it works on my PC!

Some of the things, like the website, blog, and CV, are developed by me using a variety of tools. I don’t have a very specific criterion for choosing, but usually the choice is made based on what I want to learn. My focus is on learning more about cloud, DevOps and GitOps, testing new applications, pipelines, a little bit of everything! This is where the learning journey begins and usually results in more serious implementations within the company where I work.

The process is very similar for almost everything. In the case of this blog, I currently use HUGO to generate the website. I won’t go into details about what HUGO is and does, but in a very simple way, each article is a markdown document that is then used to generate a static website.

For other applications, it depends. But usually, I always develop with the awareness that nothing runs on my computer and it will have to run somewhere. Usually, everything ends up inside Docker images.

Version Control: Git is truth, Git is life, GitOps, GitOps, GitOps!

Since there is no DevOps or GitOps without version control, for that, I use Git. To host the code, I use GitLab for private repositories and GitHub for OpenSource projects. Yes… I can have everything on the same, but that way I’m depriving myself of knowledge, and this journey is all about knowledge!

By using both, I not only know both platforms relatively well, but I also get to know their CI/CD tools (GitlabCI, GithubActions, and even TravisCi in the case of PiVPN).

Back to the topic! After testing the application, website,

Version Control: Git is truth, Git is life, GitOps, GitOps, GitOps!

Since there is no DevOps or GitOps without version control, for that, I use Git. To host the code, I use GitLab for private repositories and GitHub for OpenSource projects. Yes… I can have everything on the same, but that way I’m depriving myself of knowledge, and this journey is all about knowledge!

By using both, I not only know both platforms relatively well, but I also get to know their CI/CD tools (GitlabCI, GithubActions, and even TravisCi in the case of PiVPN).

Back to the topic! After testing the application, website, script - whatever you want to call it - I commit the code. Each commit follows the Angular commit style (Hold on a bit, you’ll soon understand why!). I push the code into the “test” branch and then, later, merge into the “master” branch.

And that’s where the manual work ends. I open a beer and let the machines do the work!

Pipelines and Docker: DevOps, a lifestyle!

As soon as the code “lands” on one of the defined branches, the CI/CD tools begin their perfect dance in perfect synchrony! GitHub Actions or GitlaCI are responsible for running these pipelines, which run (more!) tests if necessary.

They build the container image and finally run Semantic-release to automatically calculate and version the code. For this, this tool analyzes each commit and calculates which will be the next version based on them and according to the rules of semantic versioning. Remember the Angular Commit style? Well, that’s why it’s important! Finally, it tags the containers and pushes them to the registry (GitLab, DockerHub, GitHub)!

Kubernetes, Flux CD, Watchtower

Everything running in the cloud is currently operating on a Kubernetes cluster in Google’s cloud (GCP).

I use Terraform to create infrastructure and FluxCD to manage everything running in the cluster.

FluxCD constantly monitors the Git repository where the configurations of the applications running in the cluster are stored and reverts any changes that are not included in the configurations that exist in the repository. This means that if I apply kubectl apply NewApplication.yaml directly from my PC, the changes will be reverted minutes later in true GitOps style!

However, if I commit the configuration and push it to the repository without applying it, FluxCD will detect the new application and apply it automatically! Yay!!!

In addition, FluxCD monitors the container registries and whenever a new version of the application is released, it updates the configurations, commits, pushes to the repository, and applies which in turn causes Kubernetes to update the application containers!

commit 844ec31b7d74838f50b72836d4d1c8c2fc2dd84Continuation:

Author: fluxcdbot <fluxcdbot@4s3ti.net>
Date:   Thu Mar 16 20:27:51 2023 +0000

    pivpn/docs:1.4.0

diff --git a/pivpn/03-docs.yml b/pivpn/03-docs.yml
index 78f5c97..495269e 100644
--- a/pivpn/03-docs.yml
+++ b/pivpn/03-docs.yml
@@ -22,7 +22,7 @@ spec:
         node_pool: "cfplayground-global"
       containers:
       - name: pivpn
-        image: pivpn/docs:1.3.0 # {"$imagepolicy": "flux-system:pivpn-docs"}
+        image: pivpn/docs:1.4.0 # {"$imagepolicy": "flux-system:pivpn-docs"}
         imagePullPolicy: Always
         ports:
         - containerPort: 80

That’s beautiful!

The same applies to everything hosted on my servers but not developed by me, from PrivateBin, Whiteboard, monitoring tools to helm charts!!

What runs in the home lab is a simpler process. I use the :latest tag for everything, and Watchtower takes care of the rest!

Summing up!

The content may not fall here very often, and to tell the truth, the quality of it matters little to me. What really matters to me is the quality of what happens behind this “simple website,” and I’ll even say more! Blessed artificial intelligence that helped me write, structure, correct, and translate this article!

But what I want to emphasize is that a “simple website” sometimes is all it takes to distinguish a professional from a good professional! A “simple website” is all it takes to gain knowledge and understand a wide range of important tools in the professional life, from a little bit of code, whatever language it may be, infrastructure, cloud, CI/CD tools!

In today’s world, where there is more and more work to be done and the demand for new features is increasing, there is less and less time for maintenance and an increasing need to automate routine tasks! Nobody wants to have to do the same things like robots over and over again, that’s what machines are for!

Now, I’m just going to drink a beer while the machines take care of putting this article online! And spend a little more time thinking and questioning myself, “how am I going to improve all of this and automate even more?”

It’s been a few years since my last post about my little laboratory.

5 years later, after a change of country, several movings, and a job change, the home laboratory went through several iterations until its current format.

Unfortunately, I don’t have pictures of all stages, but there was a Small NAS,

The number of raspberry pi has increased, had 2 Dell enterprise servers, one for storage and another for virtualization,

I’ve gone through several operating systems, including TrueNAS with FreeBSD, ProxMox, Open Media Vault, and Unraid.

And it currently looks like this:

The server:

• Operating System: TrueNAS Scale
• Enclosure: Inter-Tech 4F28 Mining Rack 4U 19"
• PSU: Corsair CV Series CV550 550W 80 PLUS Bronze
• Motherboard: X470 Gaming Pro Carbon (Yup, gamin!! the motherboard moved from my main machine to server)
• CPU: AMD Ryzen 7 2700X (Like the motherboard, it moved from my main rig)
• Memory: Kingston FURY Beast 16GB 3,200MHz DDR4 SDRAM DIMM 288-pin
• SATA Controller: 1 x Startech 8P6G-PCIE-SATA-CARD
• Disks: 7x4TB HDD (some WD others Toshiba) + 3x250GB SSD
  • configured in a raidz2 pool, which leaves me with a total of 18.5TB usable space and 2 simultaneous drives fault tolerance.
  • SSD disks are configured in stripe (raid0).

On the shelves:

• 1 x Synology DS412+, receive daily backups from main server,
  • has 4 Disks (2 x 3TB + 2 x 4TB) configured in an SHR pool (Synology Hybrid RAID) with 9TB usable and 1 disk fault tolerance
• 4 x Raspberry pi 3 running everything that you can’t turn off when the main server is down, or that for some reason, I can’t run there (more about in another post)
• 1 x Teldus stick to connect “smart” equipment that works with radiofrequency.
• 1 x Unifi Switch USW-Flex-Mini where the raspberry pi and teldus stick are connected
• 1 x Unifi AP Lite Access Point

Rack and accessories:

• Startech 12U Open Frame
• 2 x Digitus Rackhylla Shelves 19" 15kg
• 4 x Digitus 1U Snap-in Panels
• 2 x Deltaco Panels 19-14 with brushes
• 1 x Pair of Startech 1U Adjustable Rails
• 1 x Extension (rPDU) 19" Powersocket 9Xoutlets 2M Black

Why only one server?

As you can tell, I didn’t get here by magic! The laboratory is a learning and day-to-day aid tool, so I try to do as much with as little complexity as possible.

Currently, I spend less time managing hardware and more and more time working with the cloud, designing resilient and high-availability systems, optimizing systems and processes, automating tasks and developing tools to meet specific needs. This being said, if before I wanted to learn more about virtualization and how to automate the installation of operating systems to isntall and run applications. Today those tasks are ofloaded to the cloud, applications run in containers, and everything is automated as much as possible in order to spend less time maintaining and more time learning (or sometimes gaming).

The only critical stuff in the lab is the data stored in the server which is safeguarded by regular backups, both in and off-site. The rest can afford to be down for as long as necessary because all uptime critical applications run in the cloud (mainly GCP).

Plans for the future?

For now I don’t plan big changes, the next investments will undoubtedly go through a UPS and a bigger switch with PoE.

What am I running in the lab?

I’ll make a more elaborate post about it, but for now I’ll leave it here:

Shortly after rolling out support for multiple domains was asked to redirect a few more secondary domains.

According to my last post, it should be easy, right?

Not so fast … I went ahead and added the new domains to the list

  alternative_domains = [
    {
        domain = "alternative.com"
        zone   = "alternative.com"
    },
    {   domain = "www.alternative.com"
        zone   = "alternative.com"
    },
    {   domain = "alternative.se"
        zone   = "alternative.se"
    },
    {   domain = "www.alternative.se"
        zone   = "alternative.se"
    },
    {   domain = "alternative.site"
        zone   = "alternative.site"
    },
    {   domain = "www.alternative.site"
        zone   = "alternative.site"
    }
  ]

Terraform apply … and … drum roll …

Error: Error modifying LB Listener Rule: ValidationError: A rule can only have '5' condition values
	status code: 400, request id: 19c18fa3-ed0d-4ca3-8b3e-d10b5221ac77

here is the problem:

A Load balancer can have up to 100 Listener rules, and each listener rule can have up to 5 conditions.

Adding all alternative domains to the same listener rule exceeded that limit as soon as a few more were added to the list.

resource "aws_alb_listener_rule" "redirect_alternative_domains" {
  count = length(var.alternative_domains) > 0 ? 1 : 0
  listener_arn = aws_alb_listener.listener_443.arn
  priority     = 97
  action {
    type             = "redirect"
    target_group_arn = aws_alb_target_group.dashboard2_target_group.id
    redirect {
      host        = local.env_domain
      status_code = "HTTP_302"
    }
  }
  condition {
    host_header {
      values = [for ad in var.alternative_domains : ad.domain]
    }
  }
}

After a bit on the whiteboard, I could see 2 possible solutions.

The easy way out:

Just create one Listener rule per each forwarding condition. This means that www.domain.com would be one listener rule, and domain.com would be another rule until it reaches the 100 Listener rules limit.

The Optimal solution:

Create one rule per unique hosted zone, each rule containing the conditions for that specific zone.

This means each listener rule would contain a condition for both domain.com and www.domain.com

Rolled up my sleeves, created a new variable

distinct_ad_zones   = distinct([for d in var.alternative_domains : d.zone]) 

This one would serve as a condition to determine how many Listener rules I want to create

count = length(var.alternative_domains) > 0 ? length(local.distinct_ad_zones) : 0

And changed the header values to lookup at the map mapping

condition {
  host_header {
    values = [lookup(local.domain_to_zone_map, local.distinct_ad_zones[count.index])]
}

This is where things got hairy!

Being honnest, at this point, I cannot even remember if this was the first condition I tried or not. The point is, it was not working as expected!

My hopes were the lookup also worked the other way around but were only getting one of the values and not both of them, so I created another mapping:

zone_to_domain_map = zipmap(local.all_zones, local.all_domains)

leaving with two maps, one mapping domain to hosted zone, and another from hosted zone to the domain

{
domain.com      = domain.com,
www.domain.com  = domain.com,
domain.xyz      = domain.xyz,
www.domain.xyz  = domain.xyz,
domain.io       = domain.io,
www.domain.io   = domain.io
}

and

{
domain.com = domain.com,
domain.com = www.domain.com,
domain.xyz = domain.xyz,
domain.xyz = www.domain.xyz,
domain.io = domain.io,
domain.io = www.domain.io
}

I was also expecting the lookup function to return all the values for any given key with the same name, (local.distinct_ad_zones[count.index]) for example, for zone domain.com I would get both domains, the domain.com and the www.domain.com but that was not the case! I could only get either www.domain.com or domain.com in each listener rule or all of them in all listener rules. Never each pair per listener rule.

After some time back and forth, I noticed depending on the map I was looking up to, I was getting either one or the other, and a hackish solution just jumped up.

resource "aws_alb_listener_rule" "redirect_alternative_domains" {
  count = length(var.alternative_domains) > 0 ? length(local.distinct_ad_zones) : 0

  listener_arn = aws_alb_listener.listener_443.arn

  action {
    type             = "redirect"
    target_group_arn = aws_alb_target_group.dashboard2_target_group.id

    redirect {
      host        = local.env_domain
      status_code = "HTTP_302"
    }
  }

  condition {
    host_header {
      values = [
        lookup(local.domain_to_zone_map, local.distinct_ad_zones[count.index]),
        lookup(local.zone_to_domain_map, local.distinct_ad_zones[count.index])
      ]
    }
  }
}

Can you notice what I did there? yup … I simply lookup at both maps and I get the values I need where I need them!

I am sure this is not the best solution and someday, it will come back to haunt me! Let’s hope I don’t need to add a 3rd domain in the same zone! 🙈

At work, most (if not all) infrastructure is deployed with Terraform, clicky-clicky actions are not at all encouraged and I got asked about the possibility of generating a new TLS certificate to forward users from a given domain to the main one.

Well, it will be an easy ride! the certificate is already created and validated with Terraform. A few tweaks in the module I will quickly be done with it.

Created a new variable so that we can provide the values that we need on the environment-specific code:

variable "secondary_domains" {
  type        = list(string)
  description = "Secondary domains to create the certificate for"
  default     = []
}

Updated an existing one that generates the environment-specific alternative names to include it

env_alternative_names = var.primary_env == "true" ? flatten(["*.${var.domain}", var.secondary_domains]) : ["*.${var.environment}.${var.domain}"]

Also squeezed in a fix to suppress a deprecation warning, merged to master, and got a new release, v1.3.0.

• Applied to dev … Cool, working fine
• Applied to Staging … all roses and rainbows!
• Applied to pro … Certificate validation Failed! Crap!! I Completely missed this one, should have seen this one coming!

The good thing about having versioned modules and good life-cycle policies on the resources?

Rolled back to v1.2.0, all was back as it was before, with no disruptions, and got back to the drawing board!

And … Yes! This was in Production!!! cool isn’t it?

So, Now I have a problem! How do I get this to work?

The problem:

Now I have Multiple domains to include in a certificate and to apply the DNS Validation records on multiple AWS route53 Hosted zones and since I am at it it should be dynamic enough to add as many as we need in the future.

This is quite a common practice, so I am sure someone else has figured it out!

It is time to dress in the Kimono and practice all the lessons from the ancient masters in the art of google-fu!

It didn’t take long to find a module that does exactly what I wanted ringanta/terraform-aws-acm-multiple-hosted-zone

However, this does not entirely fit our design, I cannot just grab this module and use it, but the good thing about open source? It is open!

That being said, I still spent a good amount of time looking into other approaches and ultimately ended up deep diving into the one above!

I always aim for simplicity. But after some playing back and forth with variables ended up with the most complex solution I have implemented on Terraform yet (I believe) clean and elegant enough to be understood!

The Solution

Some things might not make sense as I go through them, but as in any good movie, everything unfolds and fits nicely together, so please bear with me for a sec!

1 - Generate the environment-specific alternative name (we were doing this before).

env_alternative_name = var.primary_env == "true" ? "*.${var.domain}" : "*.${var.environment}.${var.domain}"

2 - Update the alternative_domains variable and allow module users to provide a list with all the domain => zone mappings

variable "alternative_domains" {
  type        = list(map(string))
  description = "Secondary domains to create the certificate for"
  default     = []
}

3 - Create a list with all the alternative names for the certificate.

For this, I made the local.env_alternative_name part of a list with a single item then, a loop through the domains inside the alternative_domains variable creates another list with the alternative domains provided by the module input and finally concatenate all of them together into a single list!

all_alternative_names = concat(
    [local.env_alternative_name],
    [for an in var.alternative_domains : an.domain]
  )

3 - create a list of all the domains in the correct order. This one follows the same logic as before, but this time the first element needs to be the primary domain of the environment, then the environment-specific alternative name, and then all the provided alternative domains.

all_domains = concat(
    [local.env_domain],
    [local.env_alternative_name],
    [for d in var.alternative_domains : d.domain]
  )

4 - create a list of all the hosted zones in the correct order, following the same logic as before, first the primary domain hosted zone then the environment-specific hosted zone, and then the hosted zones for each alternative domain provided externally into the alternative_domains variable.

all_zones = concat(
    [var.zone_name],
    [var.zone_name],
    [for z in var.alternative_domains : z.zone]
  )

5 - create a list of all unique zones using the distinct() function.

distinct_zones = distinct(local.all_zones)

6 - Create a new variable that maps each distinct DNS Hosted zone with their respective IDs. These IDs are unique per hosted zone and will be needed later to create validation DNS Records at a later stage.

zone_name_to_id_map = zipmap(local.distinct_zones, data.aws_route53_zone.acm_zones[*].zone_id)

Each zone ID is then grabbed with a data source

data "aws_route53_zone" "acm_zones" {
  count = length(local.distinct_zones)
  name         = local.distinct_zones[count.index]
  private_zone = false
}

7 - create a variable that maps each Domain to their respective DNS Hosted zones.

domain_to_zone_map = zipmap(local.all_domains, local.all_zones)

What kind of sorcery is going on here? remember all those lists we have created before?

Let’s pick up one of the lists from before and understand how it works.

So the list all_domains Will include the following example values:

["domain.com", "*.domain.com", "alternative.com", "www.alternative.com"]

the list all_zones will contain the following example values:

["domain.com", "domain.com", "alternative.com", "alternative.com"]

Now, from the terraform documentation,

zipmap, constructs a map from a list of keys and a corresponding list of values.

Remember what I said before about needing them in order? this is the reason why!

when using the function domain_to_zone_map = zipmap(local.all_domains, local.all_zones) we are creating a map called domain_to_zone_map that will contain the following example values:

{
"domain.com" = "domain.com",
"*.domain.com" = "domain.com",
"alternative.com" = "alternative.com",
"www.alternative.com" = "alternative.com"
}

At this point I am done with manipulating variables and sorting all the pieces of the puzzle, here is how it looks all together:

variables.tf

locals {

  ... Other Local variables ...

  env_alternative_name = var.primary_env == "true" ? "*.${var.domain}" : "*.${var.environment}.${var.domain}"

  all_alternative_names = concat(
    [local.env_alternative_name],
    [for an in var.alternative_domains : an.domain]
  )

  all_domains = concat(
    [local.env_domain],
    [local.env_alternative_name],
    [for d in var.alternative_domains : d.domain]
  )

  all_zones = concat(
    [var.zone_name],
    [var.zone_name],
    [for z in var.alternative_domains : z.zone]
  )

  distinct_zones      = distinct(local.all_zones)
  zone_name_to_id_map = zipmap(local.distinct_zones, data.aws_route53_zone.acm_zones[*].zone_id)
  domain_to_zone_map  = zipmap(local.all_domains, local.all_zones)

}

Making it work!

Let’s put it all into action and take it one bit at a time again.

1 - Create the TLS certificate, providing the environment domain name and all the alternative domain names

# certificate creation
resource "aws_acm_certificate" "cert" {
  domain_name               = local.env_domain
  subject_alternative_names = local.all_alternative_names
  validation_method         = "DNS"

  lifecycle {
    create_before_destroy = true
  }
}

2 - Create the necessary DNS records to validate the certificate generated above.

• Loop through each domain validation option and assign it to loop internal variables.
• Create the records with that data using the mappings generated before to know exactly which hosted zone ID to use.

# certificate validation
resource "aws_route53_record" "cert_validation" {
  depends_on = [aws_acm_certificate.cert]

  for_each = {
    for dvo in aws_acm_certificate.cert.domain_validation_options : dvo.domain_name => {
      name   = dvo.resource_record_name
      record = dvo.resource_record_value
      type   = dvo.resource_record_type
      domain = dvo.domain_name
    }
  }

  allow_overwrite = true
  name            = each.value.name
  type            = each.value.type
  records         = [each.value.record]
  zone_id         = lookup(local.zone_name_to_id_map, lookup(local.domain_to_zone_map, each.value.domain))
  ttl             = 60
}

3 - And finally we loop through all those records and validate the certificate

resource "aws_acm_certificate_validation" "cert" {
  depends_on              = [aws_acm_certificate.cert, aws_route53_record.cert_validation]
  certificate_arn         = aws_acm_certificate.cert.arn
  validation_record_fqdns = [for record in aws_route53_record.cert_validation : record.fqdn]
}

Here is all of it pieced together (including the Data part that was mentioned above)

# certificate creation
resource "aws_acm_certificate" "cert" {
  domain_name               = local.env_domain
  subject_alternative_names = local.all_alternative_names
  validation_method         = "DNS"

  lifecycle {
    create_before_destroy = true
  }
}

# certificate validation
resource "aws_route53_record" "cert_validation" {
  depends_on = [aws_acm_certificate.cert]

  for_each = {
    for dvo in aws_acm_certificate.cert.domain_validation_options : dvo.domain_name => {
      name   = dvo.resource_record_name
      record = dvo.resource_record_value
      type   = dvo.resource_record_type
      domain = dvo.domain_name
    }
  }

  allow_overwrite = true
  name            = each.value.name
  type            = each.value.type
  records         = [each.value.record]
  zone_id         = lookup(local.zone_name_to_id_map, lookup(local.domain_to_zone_map, each.value.domain))
  ttl             = 60
}

resource "aws_acm_certificate_validation" "cert" {
  depends_on              = [aws_acm_certificate.cert, aws_route53_record.cert_validation]
  certificate_arn         = aws_acm_certificate.cert.arn
  validation_record_fqdns = [for record in aws_route53_record.cert_validation : record.fqdn]
}

4 - Last but not least, I added a new Load Balancer rule to redirect all the provided var.alternative_domains to the correct main one.

resource "aws_alb_listener_rule" "redirect_alternative_domains" {
  count = length(var.alternative_domains) > 0 ? 1 : 0
  listener_arn = aws_alb_listener.listener_443.arn
  priority     = 97
  action {
    type             = "redirect"
    target_group_arn = aws_alb_target_group.dashboard2_target_group.id
    redirect {
      host        = local.env_domain
      status_code = "HTTP_302"
    }
  }
  condition {
    host_header {
      values = [for ad in var.alternative_domains : ad.domain]
    }
  }
}

Extra points if you know what’s wrong with the code above! Don’t know? read more about it here

After all of This, committed everything and ran a couple of plans on other environments to make sure there were no unexpected surprises and it would not be changing any environment not requiring this change. Then proceeded to run a plan against Prod to check if the changes were the ones I was expecting.

Pushed everything to my feature branch, double-checked the pipeline for anything that I might have missed, and shared it with my team. After a quick review and discussion we got it merged to master, the pipelines made their magic, and v1.3.1 was released!

Applying this to production was smooth sailing! no errors and now we can have any alternative domains redirecting to the main one by providing a list with the mapping of all the alternative domains and their hosted zone name.

alternative_domains = [
    {
        domain = "alternative.com"
        zone   = "alternative.com"
    },
    {   domain = "www.alternative.com"
        zone   = "alternative.com"
    },
    {   domain = "alternative2.com"
        zone   = "alternative2.com"
    }
  ]

I have seen a couple of different approaches to this one around the internet, but none of them made much sense to me. There are better ones? Probably there are! But despite its complexity, this solution seemed quite elegant and easy enough to understand without risking the introduction of any breaking changes in the module!

Do you have any other ideas? How would you have solved this? feel free to reach out and let me know.

“How can I run R in the cloud? R code takes so much time, and I cannot use my laptop for anything else.”

I’ve learned there are plenty of reasons why R code can take a long time to run. Regardless of the skill level and project complexity, the fact is, you have certainly been in a situation where the computing power isn’t enough, and you can only hope nothing happens to your laptop and takes the results of weeks with it.

I know nothing about R, I don’t even work with it, but I have friends that do! And they reached out for help.

Unlike many subscription services not accessible to the average joe, cloud providers are quite an affordable way to access any hardware 24/7. You start up the server, run the code, and when done, just “trash” and pay only for the time you have used it.

Sounds easy, right?

Unfortunately, this requires learning an overwhelming amount of new skills, navigating through a maze of “computer” dark magic terminologies, and getting bored of doing the same things every time you need to start a new RStudio server instance.

The goal is to get you up and running without getting overwhelmed by the amount of information, having to deal with VPCs, Subnets, EC2 instances, AMIs, or even installing the whole thing yourself! ezRStudio simplifies the entire process by joining all the moving parts and creating all the needed resources, providing you with an RStudio server running on an AWS EC2 instance in seconds. The only requirements are an AWS Account, configured AWS CLI tool, and terraform or skip the last two and use tools like terraform cloud or env0 making the whole thing even easier (More about this later)!

As most things I build, ezRStudio is fully open source, and you can find it on https://github.com/4s3ti/ezrstudio

I am currently supporting AWS only, but I am more than happy to make it work with other cloud providers, given the project gets enough visibility usage.

Please let me know what you think about it! Feel free to open a Discussion, request features, or even pull requests!

TLDR; AzureDevops is not the most intuitive platform to work with, but it is not that bad. Some days we only complicate things, and if I had paid attention to the errors from the start, I would have saved myself many hours of running in circles

Project Settings

Open the Project => Project settings => Repositories => Security tab => Select “… Build Service” => Set the following settings to allow: “Contribute”, “Create Branch”, “Create Tag”, “manage notes” and “Read”

Configuração da “pipeline”

pool:
  vmImage: ubuntu-latest

### Rest of the pipeline ###
### Release stage is skipped on Pull Requests, manual triggers and Scheduled events
- stage: release
    condition: and(succeeded(), ne(variables['Build.Reason'], 'Schedule'), ne(variables['Build.Reason'], 'PullRequest'), ne(variables['Build.Reason'], 'Manual'))
    jobs:
    - job: release
      steps:
      - checkout: self
        persistCredentials: true
        clean: true
      - script: |
          npx semantic-release          
       
        displayName: semantic-release

Configurações da semantic-release

{
  "name": "aws-vpc",
  "plugins": [],
  "release": {
    "branches": [ "master", "release", "test", "dev" ]
  },

  "branches": [
    {name: 'master'},
    {name: 'release', channel: 'release', prerelease: true},
    {name: 'test', channel: 'test', prerelease: true},
    {name: 'dev', channel: 'dev', prerelease: true}
  ]
}

With the new version of the website and the time I spent learning how to use HUGO, the idea of converting the cv template came up therefore allowing the usage of HUGO for both sites and making the whole process easier, not only for me, but for everyone who uses it.

Now it’s all very simple, whenever an update is made to the template I just have to update the Git submodule and submit the update to the private repository. At that point the CI/CD tools take care of the rest (more on this topic sometime …).

You can access the project at: https://github.com/4s3ti/terminalcv

Here is the video of the whole process, you can also watch it in real time at https://twitch.tv/4s3ti

This being said on 4s3ti.net v2.0 I merged the site and the blog under the same roof and is now generated with HUGO instead of static HTML (which was used on the site) and Pelican (which was used for the blog ).

The choice of HUGO to generate the content was not a big reason, he just wanted to try something new and HUGO being one of the most popular “frameworks” with plenty of themes available made the choice simple.

The theme used to build the site is completely Open Source and available at https://github.com/4s3ti/hugo-theme-hello-4s3ti

VPN at home/office

Let’s try to make it simple, you will want to set up a VPN server at your home or office if your objective is to access your home or office network and devices when you are elsewhere.

This setup is also useful when you are out of home or office and connecting to the internet through public networks such as a pub, bar, cafe, public transportation, this way all traffic is encrypted by your VPN tunnel and protected from whoever might be sniffing the network traffic.

Because all the traffic is only encrypted between your computer and your VPN server, this setup DOES NOT protect your traffic from your ISP.

How can i setup a VPN at home?

If this is what you need then please take a look at PiVPN it makes the whole process of setting up a VPN at home way more simple and all you need is a raspberry pi.

VPN Server placed outside of home/office

Trying to keep it simple again, this is where you want to have your server placed if you want privacy above everything and you don’t require access to any local devices when you are outside of your home/office.

You will want this one especially if you want to hide your traffic from your ISP.

This means that wherever you are when you connect to the VPN Server you will have your traffic fully encrypted and hidden from your ISP, or any other network where you might be connecting from.

How can i use this kind of VPN server?

To use this kind of VPN you can search the internet for paid VPN services, or if you want to have your own, just fire up a VPS server and build your own, you can also use PiVPN for this.

Hope this post helps you to understand VPNs a little bit better and which one to use regarding server placement.

After 2 years (yes last time I have posted something was more than 2 years ago!!), a new job, a country move and lots of ups and downs, I am now happy to have to resume all the projects set on hold and coming back with a couple of new ones as well!

Now I have a better picture of the path forward, therefore, the blog got renamed from colacoweb to 4s3ti which will be my nickname from now on.

As games have been always a big part of my life, I decided to start a project on twitch.tv/4s3ti and combine both of my passions! IT and Gaming, so hit that follow button, and watch out for some live streams with gaming and tech talks! feel free to come over!

Some content might be arriving on youtube as well, so stay tuned on My youtube channel as well!

I have now made a few updates to nice-blog theme and I am again maintaining PiVPN (spoiler alert, next post will be exactly about VPN!)

That’s all for now, stay tuned for more news!

All usage instructions are availabe on Nice-blog readme file.

Like the project? Would like to contribute? Check out Nice-Blog Github repo at:

nice-blog

Usage instructions on GitHub repo.

nice-blog

The answer is simple, for learning purposes!

There are some tools like OpenStack / puppet / Chef / Ansible / Nagios / FreeNAS among many other tools involved in running a data center, that are quite useful on today’s IT infrastructures which, unfortunately, you won’t learn until you get the chance to use them at work or until you really need those tools.

Having this said, no, I don’t work in a place where I have the chance to use such tools, no, I really don’t need them, and a huge NO, I don’t have the money to invest in servers and networking equipment, but YES, I have the will to learn and I want to learn how to use and set-up those tools and, at least have a basic understanding of their usage and functionalities, so … I decided to make small, low budget, (kind of) Data Center at home powered with raspberry pis’ and perhaps use it to support the three VPS I already own.

Now comes the interesting part,

What am I really using?

Between some slavaged parts that I already had at home and some I bought for this project, what is already in place and Working:

3 x Raspberry Pi 3 with Fedora 25 ARM (One was a christmas gift)
3 x Raspberry Pi 3 cases.
1 x TP-Link Archer C7 with OpenWRT (will make a post about it latter, meanwhile if you want to install OepnWRT in an Archer C7 or another supported router, look at the OpenWRT wiki which is very well documented)
1 x tp-link 5-Port Gigabit Easy Smart Switch (TL-SG105E)
3 x 5v 2.4amps mobile phone chargers bought from the local chineese store.
2 x 32GB MircoSD Card which I already had, and are the cheapest ones I could find
1 x 16GB MicroSD Card (once again the cheapest one I could find)
1 x USB ffan that was gifted to me sometime ago

What’s still missing?

A Network storage

Until I can afford to buy a decent NAS, I will use a Docking Station that I’ve been using for several years with a couple of salvaged hard drives plugged via USB to one of the pis'

A KVM capable computer to use as OpenStack server

well … this one will surely be the last one, I have yet to find the best way to do it, maybe I’ll salvage some parts from my laptops graveyard, or hope I can afford to buy better hardware for it.

There follow some pics how is it looking so far:

To overcome this issue I felt the need to develop a script that takes publicly referenced IP addresses from blocklist.de and blocks them in the firewall before they even try to attack the server, acting in a more preventive way instead of reacting to a given set of events.

IPBL is still a project in development which I intend to improve in order to automate the ip blocking at the maximum, further objectives goes by creating an install script which creates a systemd timer or service ( yet to find the best way to do it ) that runs the script every given amount of time, e.g every 8 hours, in order to keep the blacklist updated by removing unlisted ips and adding new ips and a config file that allows costumization of the script behavior.

If you like the project and want to contribute either with coding or suggestions feel free to open an issue or a pull request in the script Github repo which is available at:

https://github.com/cfcolaco/IPBL

This article will tell you how to reset the root password in any Fedora OS family, well… never tried it in Red Hat but I expect it to work the same way as for Fedora and CentOS.

So if you are standing in your terminal window, just CTRL+ALT+DEL … Reboot now … Shutdown -r … Reboot the system using which method you like more.

When you reach the grub menu, press E and search for the line starting with “linux” that maps the paths for vmlinuz root and at the end of that line type

rw init=/bin/bash

(obs: don’t forget to add \ (backslash) if you want to start on a new line or for any reason you need to add a new line. )

Reboot Pressing Ctrl-X

And there you go, after booting you will be in “rescue mode” or for the more conservative ones, init 1 ; runlevel 1.

Now you just need to:

passwd

Type the new password and confirm it and then:

touch / .autorelabel

This step is needed otherwise your new password will not work.

/sbin/reboot -f

You can now use your new root password after system boot up.

First things first!

I don’t like to have to write “sudo” at the beginning of each command I give to a Linux machine. Especially when it comes to servers where we have to be always using files and directories that require root privileges. SO just…

su root

But … Isn’t it dangerous? well.. it is as dangerous as sudo if you don’t know what you’re doing.
And if there’s someone reading this, I expect it to be

1. ME, HELLO FUTURE ME READING THIS!!! Yes, I will use my own blog from time to tome to remember some things I know I’ve already done but I don’t remember exactly how they’re done.

2. Someone with at least some Linux knowledge.
   Having this said, let’s get back to the track, and if you don’t like to be logged in as root, then just use sudo at the beginning of each line.

   apt-get install openvpn easy-rsa

Copy Easy-Rsa to OpenVpn directory

mkdir /etc/openvpn/easy-rsa/
cp -r /usr/share/easy-rsa/* /etc/openvpn/easy-rsa/

Optional Step

The next step not mandatory, however it can make the whole process faster and easier, sparing you from having to answer all the certificate questions everytime you generate a new certificate

Edit vars file in /etc/openvpn/easy-rsa and change

export KEY_COUNTRY="COUNTRY"
export KEY_PROVINCE="PROVINCE"
export KEY_CITY="CITY"
export KEY_ORG="KEYORIGIN"
export KEY_EMAIL="EMAIL@SOMETHING.COM"
export KEY_OU="OFFICE"
export KEY_NAME="KEYNAME"

Generate Server and Client certificates

Yes, we want privacy, for privacy we need to encrypt our communications and have a safe way to authenticate in the server, otherwise, everyone will be able to use it, we don’t what that to happen.

cd /etc/openvpn/easy-rsa/
source vars
./clean-all

Create CA certificate

./build-ca

answer the questions according to your needs

Server Private Key

./build-key-server servername

Answer the questions and answer yes to the two yes or no questions.

Build Diffie Hellman

What the hell is Diffie Hellman? Well… I will not explain it here, just Google It.

./build-dh

Create openvpn ta.key for ssl connection

openvpn --genkey secret keys/ta.key

Copy Certificates to “etc/openvpn”

cd keys
cp myservername.crt myservername.key ca.crt dh2048.pem ta.key /etc/openvpn

Create client Certificates

Check if you are into /etc/openvpn/easy-rsa if not, cd into /etc/openvpn/easy-rsa

source vars
./build-key client1

answer the questions and answer yes like previous certs

Copy ca.crt client.crt client.key and ta.key to client machine

mkdir clientcrts
cp ca.crt ta.key clientcrts && mv client.crt client.key clientcrts/

Move clientcrts to a location where you can reach through scp

Example

let’s say that we are with root user and that your server does not allow you to do SSH login with root user. move clientcrts to a place where you can easily reach via ssh with the ssh user.

mv clientcrts /home/sshuser/ && chown -R sshuser:sshuser /home/sshuser/clientcrts

Copy the files from the server to a local machine.

scp -r sshuser@serveraddress:/path/to/cliencrts /Local/machine/Path

OpenVPN server setup

Copy sample config file

sudo cp /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz /etc/openvpn/
cd /etc/openvpn
gzip -d server.conf.gz

Edit server.conf

port 1194

Or you can change it to whatever you like.

Point to the right cert paths

ca ca.crt
cert myservername.crt
key myservername.key
dh dh2048.pem

Uncoment

#tls-auth ta.key 0

obs: the 0 means the key direction is 0, it means in the client u have to set up key direction 1.

Drop openvpn user pervileges and change

;user nobody
;group nogroup

to

user nobody
group nogroup

Force clients traffic to be forwarded trough our vpn server

push "redirect-gateway def1 bypass-dhcp"
push "remote-gateway SERVER IP ADDRESS"

Give DNS to clients

push "dhcp-option DNS 8.8.8.8"
push "dhcp-option DNS 8.8.4.4"

NOTE

these are google DNS servers if you like, you can change it to whatever DNS you want. OpenVPN should be OK now, but there are more configs that need to be done. in order to VPN work correctly.

Enable Ip forwarding

edit /etc/sysctl.d/sysctl.conf and uncoment

#net.ipv4.ip_forward=1

and then do

sysctl -p /etc/sysctl.conf

Firewall

If you are using ubuntu server, probably you will have ufw, which must be configured to forward all packets from tun0 to eth0, otherwise our clients wont be able to reach the internet.

edit /etc/ufw/before.rules after editing the file it should look like this:

#
# rules.before
#
# Rules that should be run before the ufw command line added rules. Custom
# rules should be added to one of these chains:
#   ufw-before-input
#   ufw-before-output
#   ufw-before-forward
#

# START OPENVPN RULES
# NAT table rules
\*nat
:POSTROUTING ACCEPT [0:0]
# Allow traffic from OpenVPN client to eth0
-A POSTROUTING -s 10.8.0.0/8 -o eth0 -j MASQUERADE
COMMIT
# END OPENVPN RULES

# Don't delete these required lines, otherwise there will be errors
\*filter
...

NOTE: this lines need to be like this example. the rules must be in the begining of the before.rules file. (arround line 10)

Change Default Forward policy to ACCEPT

edit /etc/default/ufw and change the following line to be like this.

DEFAULT_FORWARD_POLICY="ACCEPT"

Open UFW ports to allow trafic and users to connect

ufw allow 1194/udp (or another port if you are using a custom port)

Just in case, Add the following rules also.

ufw allow OpenSSH ( not mandatory but, just in case )
ufw allow 22 ( or your ssh port if have ssh in a different port)
ufw disable
ufw enable

Now you should be good to go and browse the web without with relative security, privacy, and anonymity.

systemctl start openvpn@server

if you want to enable the VPN at startup

systemctl enable openvpn@server

Hope you found this post usefull. Have fun!

Template git repo: https://github.com/cfcolaco/nice-blog

It’s Forked from Daniel Hauck terminal CV, had to tweak it a bit, to better suit my needs. Feel free to use it.