Rick van Lieshout ~ Mastermindzh

What is Software Architecture (Really)?

Tue, 14 Oct 2025 00:00:00 GMT

A new category on rickvanlieshout.com: Software Architecture

This new category is about the real side of software architecture. Not buzzwords, not picture-perfect diagrams, but the messy, human part of designing systems that actually work.

I’ve spent many years working with different kinds of architectures, from tightly coupled monoliths to sprawling microservice landscapes. Along the way, I’ve learned that architecture isn’t about doing things the “right” way, it’s about understanding why you’re doing them in the first place, and how they fit into your team and product’s reality.

Note that this applies both to enterprise systems as well as open-source initiatives.

What software architecture actually is

Software architecture is the set of decisions that define how a system behaves, evolves, and is understood over time. It’s not the framework, not the diagram, and not the shiny technology you picked. It’s the reasoning behind those things, and how they hold up when the system changes.

If software design is about the details, like how classes interact, how APIs are structured, and how code is organized, architecture is about direction. It’s the compass that helps the team navigate change without losing its bearings.

Good architecture provides just enough structure for a team to move fast, without locking them into a corner. It’s the scaffolding around the code, the shared mental model that keeps complexity in check.

But architecture doesn’t exist in isolation. The same design that works beautifully in one place can fail completely in another, which brings us to the one factor most people overlook: context.

Why context always matters

There’s no single best architecture, only architectures that make sense in a specific context. A perfect design on paper might fail completely when you drop it into the wrong environment. The right architecture depends on your product stage, your constraints, and, most importantly, your team.

At Frontliners, we’ve seen this up close. We took on the task of replacing a monolith that had been in production for more than 25 years. It was deeply intertwined with the business, used everywhere, and full of edge cases that only existed because of decades of real-world use. To make matters worse, these edge cases are often only known to some users, others just “follow what they’ve been taught”.

We didn’t have a large or deeply experienced team at the time, but we did have a strong sense of purpose. We knew we wanted to modernize, to move toward something distributed and scalable, but we also knew we couldn’t do it all at once. That meant making tough decisions, again and again, weighing what we could achieve now against what would still make sense later. This is especially true from a technical perspective as during this tumultuous time we’ve switched both CPO and CEO multiple times. They all offered new and shiny things, and tech was left trying to realize them within increasingly shorter timelines.

We focused on the people we had, their strengths, their limits, their growth potential, and built an architecture that could grow with them. It wasn’t perfect, but it gave us some momentum, and that momentum led us to something sustainable.

Still, even with the best intentions, it’s easy to lose perspective once you’ve found a path that seems to work. The next challenge we faced wasn’t technical at all, it was cultural.

When good principles turn into dogma

During our early microservices phase, we fell into a trap many teams do. We started creating “rules” that sounded right, but didn’t always fit reality. Things like “you can only do X” or “you can NEVER depend on another service”. And whilst I, as the architect or CTO, never called these “rules” (rather: guidelines), when they were communicated between layers of old <> new developers they often turned into “rules”. Rules like that can be comforting because they feel like control, but they’re often just fear in disguise. Take something like route calculation. When you have hundreds of parameters and thousands of routes, you can’t pre-compute every possible scenario. Sometimes you need to depend on another service, and that’s fine. Architecture should adapt to problems, not deny their existence.

The problem wasn’t the technology, it mostly never is, it was the mindset. Without a clear product goal, people cling to certainty. Those “rules” gave us a sense of safety, but they also made change harder. We eventually had to tear those walls down and rebuild our way of thinking, together.

That rebuilding forced us to look inward and ask hard questions about who we were as a team. Because architecture isn’t just about systems, it’s about people.

Architecture and team maturity: balancing trade-offs

Every architecture exists in the shadow of the team that builds it. A highly mature, cross-functional team can handle complexity. A newer or smaller team cannot, no matter how good the intentions are.

If your architecture outpaces your team’s ability to understand or maintain it, it’s not a good architecture for you. Progress sometimes means taking two steps forward and one step back. You might accept a short-term compromise, a “bad” thing, to enable the next leap forward.

And that’s okay. Because architecture isn’t a competition, it’s a conversation. The best systems evolve through collaboration, not commandments. Let the team make mistakes, learn, and recover. If you enforce every decision from above, you’ll gain consistency, but lose creativity and ownership.

Every engineer should be part of that conversation. They don’t have to think about architecture every day, but they should care about it. Curiosity and challenge keep architecture alive.

Yet even with a healthy mindset, time changes everything. The longer a team works in the same system, the easier it becomes to stop questioning it.

Choice blindness and the value of fresh eyes

As teams settle into a certain way of working, patterns start to feel “normal.” Pain points fade into the background, awkward workarounds become invisible (even desired!). Over time, everyone forgets that things could be different. I call this choice blindness, when familiarity blinds you to your own design decisions.

That’s why new hires are so valuable. They don’t carry the same assumptions. They look at your system and ask the uncomfortable questions: “Why does this work like that?” “Is that rule still needed?” “Has the context changed since we made that decision?”

Sometimes those questions sting. But they’re essential, because context does change. Teams evolve, products evolve, constraints evolve, and what once was a good choice might be holding you back now.

Helping teams see those patterns and guide that evolution is what good architects do best.

The role of the architect

An architect’s job therefore isn’t to dictate (though they are often given that power), it’s to clarify. They connect the big picture to the team’s day-to-day, make trade-offs explicit, and keep people aligned. The most valuable skill an architect has isn’t technical, it’s adaptability.

A good architecture isn’t static. It bends with the product, the people, and the business. And that adaptability only exists when the team is part of the conversation. Architecture that’s understood by everyone lasts longer, because it belongs to everyone.

And that shared ownership matters, because architecture isn’t something you finish, it’s something you continually shape.

Architecture is ongoing, not final

Architecture is never done. It’s a living process that shifts with each decision, sprint, and release. That constant evolution can look chaotic from the outside. To stakeholders or customers, it might even seem like we’re changing direction all the time.

But that’s exactly what makes a system resilient, it changes with its context. You can’t plan your way to perfection, you can only evolve your way there.

The key is communication. When we explain why we’re making architectural changes, and how they serve the product long-term, that “technical mess” starts to look like healthy adaptation. A system that never changes is a dead one.

That’s something we learned first-hand. At Frontliners, we saw how both people and systems evolve, and how one without the other simply doesn’t work.

Wrapping up

At Frontliners, we started with a thirty-year-old monolith and a small, still-growing team. We worked through complexity, limitations, and doubt. We made decisions that weren’t perfect, but they kept us moving.

Now, we have the right people and the right balance. We’re building something sustainable and great. What helped us wasn’t just the technical architecture, but its ability to change with its context.

That, to me, is what software architecture really is.

In the next article, we’ll look at how these ideas translate into structure, comparing monoliths, distributed monoliths, and microservices, and when each one actually makes sense in the real world.

A Hidden Life of Pain, Sorrow, Misery and Rampant Emotions

Sun, 04 May 2025 00:00:00 GMT

⚠️ Content Warning
This blog contains themes of death, grief, mental health, and emotional trauma.
It is raw, honest, and may be deeply triggering for some.
Please read with care and stop if it becomes too heavy. You're not alone.

A Hidden Life of Pain, Sorrow, Misery and Rampant Emotions

Some kinds of missing are temporary. A friend you haven’t seen in a while. A conversation left unfinished. A plan postponed.

But then there’s another kind.

The kind that settles into your bones. The kind that haunts you during the quiet hours. The kind that never finds a conclusion.

That kind is not absence. It’s gone. Irrevocable, final, cruel. A door slammed shut that you still find yourself knocking on, long after you’ve forgotten why you started.

It never opens. And still, you try the handle every now and then.

In the background of every moment

Losing a loved one doesn’t stay in one room of your house though, it isn’t one door that is locked. It leaks under doors and through cracks. You find it in the most unexpected places.

You’re sipping coffee with someone, and they mention a trip they took. Suddenly, you’re remembering the trip you never took. The one you’d planned. The one that cancer took from you.

They talk about a shared playlist. You remember the song you danced to in the rain, soaking wet but laughing anyway. The same song you now skip every time it comes on because it hurts too much

Someone laughs about chipped mugs. You think of the one she cracked and called “vintage,” and how it’s still at the back of your cabinet. Untouched. Sacred.

It’s like walking through a museum where only you know what’s behind the glass.

They talk, they laugh, the world goes on,
Yet I sit with shadows, from dusk to dawn.
A smile here hides a silent scream,
A haunted heart, a broken dream.

These are what I call mind pops. Short for Involuntary Autobiographical Memory Chains. I wrote a journal-like book once, and later a smaller one. I wanted to share them. I tried. But I was never brave enough to.

Mind pops come uninvited. And they come hard. Here are some examples from my book:

The eyeliner she never quite got even.
The way she tucked me in, thinking I was asleep.
The fake wedding we planned just to see what colors we’d choose.
Your love for Twilight. And the moment we realized, quietly, you’d never get to read the next book—if there ever was one.
The mole you’ve had removed and I pretended not to notice. I did miss it so.
The mixture of guilt, remorse and happiness in your eyes as conscious life ebbed away.

They’re not just memories. They’re grenades. Quiet ones. You never know when they’ll detonate. Nor whether they will ruin your entire day.

A flicker in time, unbidden and wild,
A whisper of laughter, a memory smiled.
Then tears arrive with no request,
The heart’s old wounds laid bare, unblessed.

Unfair doesn’t even begin to cover it

We grow up believing that the world has some kind of order to it. That bad things happen, sure—but not this bad. Not this unfair. You lose someone who made life feel infinite. You watch them disappear into the haze of hospital lights and soft-spoken specialists.

Cancer doesn’t care if you’re young. Or kind.
Or if you had a hundred plans left. It just takes.
And keeps taking.

And you’re left with jewelry in a drawer. Her necklace still smells like her.
Sometimes I take it out. I hold it. Smell it. Let it transport me. I don’t even know if I should still be doing that. But it’s a thread. One of the last ones.

This necklace, cold, rests in my hand,
The last soft echo of a silent land.
I breathe her in, though she’s not near,
A ghost in scent, in touch, in tear.

The masks we wear

For a long time, no one knew. Most don’t, still.

People see the version of you they expect: Smiling. Joking. Coping. Existing.
But behind the eyes, you’re unraveling. Constantly. Because grief doesn’t pause.
Not really.

Sharing this is hard. Not because I don’t want to, but because it makes you so vulnerable. Because as you’re writing or speaking, memories pop up uninvited. They derail your words. They choke your sentences. They just pop in… bastards…

And even when you manage to share, others often don’t understand. They compare. They say things like “I lost someone too,” but it’s different. This kind of grief? This kind of loss? It’s heavier. It cuts deeper. And hearing it treated like it’s the same as all the rest—that makes you angry.

Here’s something I once wrote, when I thought no one would ever read it:

I’m surrounded by many figures, but still alone…
No one sees me, nor the pain behind my mask.
They see the smiling, happy guy I show them.
You can’t blame them though, how can I let them see the darkness in my heart?
It would scare them, for it has been torn apart.

I didn’t want to scare people. Or burden them. So I said nothing.
For years.

Strange rituals of survival

Grief makes you do strange things too..
One of my telltale behaviors is that I start eating, lots… but there are weirder ones.
More unexpected ones…

I’ve spoken out loud to pictures. Imagined new conversations. Tried to conjure her in dreams. I’ve spent hours crafting memories that never happened, just to feel like I got one more day. I name items, characters, and other things in games after her. If I eat alone, at a restaurant, I order what she would’ve. (who do I kid, I sometimes do it with company too…)

And I wrote a song. It’s unfinished though—like she was.

🎵 (Refrain)
A strand of blonde hair, as delicate as sun,
A mystery found, in silence it’s spun.
But memory, like a fleeting breeze,
Fades away as a haze, in the time we seize.

🎶(Verse)
Days pass like rustling leaves,
Stories fade, like quiet thieves.
How swiftly memory slips away,
Like an old melody, lost in the fray.

A mysterious gate, in the labyrinth of the mind,
Locked and hidden, what secrets behind?

As I’m writing this blog however, I have a sudden urge to add sound to the lyrics, it’ll still be unfinished, but a little farther along. I used both a piano and a viola to give it a more interesting sound than just the piano.

Your browser does not support the audio element.
score | midi | flac

The final moments and the words that stayed

There are memories too hard to sit with for long. But they come back, as clear as yesterday.

The quiet nights filled with endless conversation. Plans made, even though we both knew we’d never see them through. Fantasies of places we’d never travel, children we’d never have, shows we’d never finish.

She once told me something that still echoes today:

“You have to grieve me, for a while… and then you move on.
You find someone else to love. Someone who makes you feel something again.
You deserve that.”

I shook my head. “I can’t. I won’t. Not because I don’t want to feel again. But because abstinence, to me, feels like remembering you as you are. Eternal.”
We both cried after that. For different reasons, I think.

My words weren’t true though

But life went on. And I did try. I did have relations after her. And every single one felt… off. Wrong. Like stepping into a life that wasn’t meant to fit me anymore.

They weren’t bad people. In fact, I think some of them really tried to care. But the weight of comparison was impossible to escape. The shadow she left was too wide.

Sometimes I’m mad at myself for trying. Other times, I’m mad at myself for failing to let them in. And always,…always,.. I feel like I betrayed her just a little.

I know grief isn’t rational. It doesn’t follow clean lines.
And maybe that’s what hurts most of all—knowing I tried, and still came back to the same place.
Alone, but full of someone who isn’t here.

One day… maybe.

Why I’m finally speaking

It’s taken me over a decade to find the strength to say any of this. And maybe the only reason I can now… is her.

But also—my mother.
She’s terminally ill. And while it’s “natural” for a parent to go before their child, it’s still excruciating.
We don’t know how long she’s got left, but I know we’ll make the best of it. For both of us.

I do find myself ashamed to admit that it hasn’t hit me quite the same way. But it’s not lesser. It’s just different. A different wound on the same, torn, body.
And this time, I’m watching it happen while knowing exactly what’s coming. And I’m scared.
For everyone.

I think often of my brothers. For them, this may be their first real loss. Their first brush with the permanence of death.
And it breaks me in new ways to imagine them feeling what I felt, without ever seeing it coming. I’ll try to be there for them, as you do.

But it’s at least part of why I’m writing now. That’s why I’m finally saying all this.
Because pain grows in the dark.

In a moment of foolish bravery, I once planned to publish my journaled thoughts. That book. That attempt to show what this kind of loss looks like.
I didn’t. I couldn’t.

But maybe I can share this blog.

On those who want(ed) to leave

I need to say something with a very serious tone now, and I do hope you read it gently and thoroughly:

I’ve met people over the years who’ve carried this same grief. And some of them have found themselves teetering.

On the edge.

Wondering if there’s peace on the other side of absence.
Let me say this clearly: I have never thought of taking that road for myself. If anything, I would ask for eternal life. Even with the grief.
But if you are someone who has stood on that edge:

Don’t. Please.

The pain is real. The grief is heavy. But life—even broken life—has light in it still. And sometimes, all you need is one person willing to sit with you in the dark.

Let me be that person for a moment. Just long enough to remind you: the door might be closed, but the room isn’t empty.

If you are on the edge. Seek help. ^{(dutch | english)} Whatever you do, don’t pass on the pain.

What help looks like

Over the past two years, I’ve slowly—achingly—learned to live with it. And honestly, I’m happier now than I’ve ever been since.

I’ve had friends. Real ones. Some with the training to guide me through the murk. Others who simply sat there while I unraveled, trying their best.

They didn’t try to fix me. They didn’t rush me. They just showed up. They had always been there, I just didn’t allow them to be there for me before.

I’ve learned that grief isn’t a wall to climb or a puzzle to solve. It’s a landscape.

You don’t conquer it.
You walk it. One aching step at a time.
And if you’re lucky… you don’t walk it alone.

To those who held me together

There’s a quiet kind of heroism in being the one who stays. Not the one who fixes. Not the one with the right words. Just… the one who stays.

To those of you who sat with me in silence, who didn’t flinch at the weight of my words, who let me cry without needing to understand why. thank you.

To those who asked how I was and meant it. Who asked again when I lied the first time. Who sent messages I never responded to, but always read. Thank you.

To those who reminded me that it was okay to laugh. To love again. To mess up. To heal slow. Thank you.

And to the ones who didn’t even know they helped— you did.

You all held the thread when I couldn’t anymore. And maybe that’s the reason I’m still happy… Still holding her memory, but not drowning in it.

Before I finally sign off

This entire blog was something I never thought I could write. And what comes next,this next part, is something I swore I never would share.

But here we are.

What follows is the foreword and some pages from the book I wrote years ago. A raw collection of thoughts, pain, patterns, and memory. I’ve always wanted to share it, and I’ve always been too scared. Too protective. Too convinced no one would understand.

My grief has changed, not disappeared, never that. But shifted. And time has a way of loosening things you once held too tight. Of showing you that even pain deserves to breathe.

So this is me, still afraid—but doing it anyway.
This is me, defiant in the face of silence.
This is me, opening the door just a little wider.

What comes next is the truth as I once wrote it, in a different voice, at a different time (2021-2022)—but still very much me.

A Peek into the Pain and Misery

The foreword to a book I once wrote titled “A Peek into the Pain and Misery”

We, humans, perceive the vast majority of other humans we encounter as bystanders, observers and otherwise side characters in our own lives. Yet, each and every one of them has a life of comparable complexity to our own. Realizing that for the first time is an emotion called “sonder”.

This book will give you a small glimpse into my life. By the end, you might find that I’m more complex than you’ve thought till now. That there’s another side to the happy, carefree guy I show the world for the majority of my time.

Truth is, that life hasn’t always been easy. In fact, I’d say that after the turning point my life has never felt easy anymore. In fact, for at least 10 years now I’ve been living with prodigious amounts of a single emotion: bereavement.

And though I’ve tried not to let the bereavement take over my life, or in fact make it known to others. it’s always been there. I’ve effectively come to think of it as a zit. Constantly there, constantly annoying and once it’s gone away it’s only a matter of time before it shows up again somewhere else on your body.

Ten years is truly a long time. In it, I’ve tried countless things to forget my sorrow. I’ve written songs and poetry. I’ve indulged in life’s greatest things. I’ve thrown myself into the pursuit of knowledge and I’ve even tried to seclude myself, thinking that I’d become a burden to the world. However, In trying to cope with it I’ve also made many mistakes. I’ve experienced more loss, extirpated meaningful connections and, to my deepest regret, hurt others. And even now, 10 full years later I’m still burdened by the same emotions, though now compounded by the feelings and mistakes of the past 10 years.

Furthermore, During these 10 years, on nearly every day I’ve been haunted by something I’ve come to know as Involuntary Autobiographical Memory Chains, or as I now like to call them: mind pops. Mind pops are sudden memory flashes, seemingly triggered at random, about things I’ve experienced prior. These mind pops never seem to go away and can alter my mood for the rest of the day with the emotions they invoke.

I’ve actually been able to capture the entire story of the past 10 years in a book, of sorts, titled “a hidden life of pain, sorrow, misery and rampant emotions”. And though I’ve always wanted to share the entire story, I’ve never been strong enough to do so. Not 10 years ago, not now, and maybe not ever.

But… The urge to share… something… is still there. So on the 1st of January 2021, I set out on a challenge. I wanted to, at least, share a part of the story. Not one from a distant past though, one of constant remembrance. One of pain and pleasure. One I’ve lived for every day of the entire year. One I’ll live for every day for many years to come.

Every page henceforth will contain one “mind pop” that I’ve had on the date listed. So turn over this page and join me on my struggle from January 1st through December 31st of 2021.

A postscript

If you’ve read this far—thank you. That alone means more than you know.

- Rick, mastermindzh, lycan, xxxroosjexxx, and other alias’ you might know me from.

Infrastructure as Code - Simplifying My Homelab Setup with Portainer GitOps

Sat, 03 May 2025 00:00:00 GMT

After writing my last blog post about using Infrastructure as Code for uptime monitoring with UptimeKuma, I felt inspired to dive deeper into my broader homelab story.

From Complexity to Clarity

When I started building my homelab, it was all about experimenting and having fun. Kubernetes, Ansible, HA clusters spread across multiple nodes… I dove into everything headfirst. The concept of Infrastructure as Code (IaC), something I fell deeply in love with thanks to Kubernetes, quickly became the backbone of my ideal setup.

Yet, as much as I enjoyed the complexity and elegance Kubernetes brought to my experiments, I soon realized something pretty obvious: when it comes to crucial services like Jellyfin for movies or Home Assistant for managing my smart home, the last thing I needed was an over-engineered, highly available Kubernetes cluster requiring constant babysitting.

The Before Times

It reminded me of simpler times, the “before times”, before Docker and Kubernetes even existed. Back then, my homelab consisted of multiple Ubuntu servers, each handling different parts of my setup, along with a single, reliable main server named “Erebus,” using Greyhole for redundancy. Even when Docker came along (and later Kubernetes), I adopted them immediately for non-critical, experimental projects, but crucial services always remained comfortably on one stable server. After all, Kubernetes-level scalability was never really necessary for my home media or automation and I could still use all the benefits of Docker.

Hades, the Storage Behemoth

Today, this philosophy still holds true. My critical services run smoothly on a single Unraid server, my trusted main storage server named “Hades.” On Hades, it’s a pleasantly chaotic mix of Unraid’s Docker plugin, community apps, and various Docker Compose files. Each container has decentralized configuration variables, with updates effortlessly handled by Unraid’s auto-update plugin.

Here’s the ironic twist: despite having Kubernetes nodes running non-critical, playful stuff, all my serious storage still ends up on Hades. Honestly, nothing else compares. Hades dwarfs every other node in my network storage-wise, think around 200TB of free space at any given time. Even Kubernetes relies heavily on Hades for its back-end storage.

Simplifying with Portainer GitOps

Still, this fragmented setup felt manual and messy. I missed the simplicity and clarity of Infrastructure as Code. Thankfully, Portainer’s Ops feature came to my rescue, allowing me to blend the massive storage and ease of use from Unraid with structured IaC management. To fully leverage this, I registered for 3 free nodes of Portainer Business Edition, unlocking their powerful GitOps feature.

Portainer’s GitOps polls my private Git repository for updates to Docker Compose files at regular intervals, automatically deploying any changes. It seamlessly integrates with Docker Hub, pulling the latest container images and ensuring my stacks remain updated without manual intervention. This automation keeps my services fresh and secure with minimal effort.

The UptimeKuma example from my last blog is one such stack I recently migrated to this new, cleaner workflow.

The Best of Both Worlds

Now, I have the best of both worlds—a robust and neatly structured setup embracing Infrastructure as Code without sacrificing ease of use or storage headaches. In other words, I could keep the things I liked best about my Kubernetes (the predictability and IaC parts) and got rid of its worst (multiple nodes, manual updates).

On to the next adventure!

Infrastructure as code - Service uptime monitoring

Sat, 18 Jan 2025 00:00:00 GMT

If there’s one thing I’ve always embraced, it’s the philosophy of working smarter, not harder. Infrastructure as Code (IaC) has been one of the cornerstones of my career. a perfect blend of laziness and the pursuit of predictability. From my earliest days experimenting with Docker back in 2014 to leading a platform team at my current company, IaC has proven invaluable for simplifying complexity, ensuring reproducibility, and enabling automation.

Below you’ll find my journey and why I think every engineer should embrace it, eventually walking you through one of the latest automations I set up at home: Automatically creating uptime monitors in UptimeKuma based on Docker labels.

The Early Days: Discovering Docker

Back in 2014, I stumbled upon Docker, and it was a game-changer. At the time, I was frustrated by the manual and error-prone process of setting up environments for my code (especially on other people’s machines…). Docker offered a way to reliably recreate these environments with simple, declarative configuration files. Suddenly, I could spin up a development environment in minutes and be confident it would work exactly the same on another machine.

In 2017, I taught several university classes on Docker, emphasizing the importance of reproducibility. My students learned how to containerize applications and ensure their work could be shared and run anywhere, regardless of underlying infrastructure. Reproducibility wasn’t just a technical advantage; it became a mindset I encouraged in every project.

And yes.. for those that know me well, I did go overboard in some of the details and even ended up explaining the entire Copy-On-Write (COW) nature of the Docker filesystem in those classes… But ah well, people learned a lot :P.

Home Servers: The Personal Lab

My passion for IaC extended to my home servers. By 2017, nearly everything I ran at home was Docker-based. I created dozens of bash scripts and system services to orchestrate my personal infrastructure. Whether it was media servers, backup systems, or development environments, everything was automated and version-controlled. Even my own computers mostly became IaC based as I figured out that part of the Linux community was saving their setups (and install instructions, usually) in so called “dotfiles”. To this day you can still find my setup (and its changes when I switch machine) in my Dotfiles on Github.

All in all, my home setup became a sandbox for testing new ideas and tools, many of which eventually found their way into my professional work as well.

Scaling IaC in the Workplace

At my current company, I introduced Docker about eight years ago. It was a gradual process, but within a year, we had our first Kubernetes cluster running. This transition wasn’t just about adopting new tools; it was about embedding the principles of IaC into the organization’s culture.

Over time, I spearheaded the creation of a dedicated platform team. With an architect/Product Owner and four DevOps engineers, this team took IaC to the next level. They implemented robust CI/CD pipelines, infrastructure monitoring, and scalable deployment patterns. Some of these practices mirrored what I had done at home, while others were tailored to the unique needs of the business. The result was a resilient and predictable infrastructure that supports rapid development and deployment.

Setting Up Uptime Monitoring with UptimeKuma

One of the most satisfying aspects of IaC is the ability to automate even the smallest tasks. Take uptime monitoring, for example. At home and at work, ensuring that services are available is critical. Recently, I’ve been using my old UptimeKuma instance, a self-hosted monitoring tool that’s as powerful as it is user-friendly, a lot more after introducing it to some friends (who started homelabbing) and at work.

Setting up UptimeKuma is straightforward, you can simply use our old friend Docker:

docker run -d --name uptime-kuma -v ./data/uptimekuma:/app/data -p 3001:3001 louislam/uptime-kuma

And access the Dashboard by navigating to http://localhost:3001 to configure your monitors. But that isn’t automatic enough for me, I like to put my things in compose files for home usage.

Automating uptime monitors with AutoKuma

AutoKuma allows us to set labels on our Docker containers that will then automatically generate monitors in UptimeKuma.

One of my containers (UptimeKuma actually) has the following labels attached:

labels:
  kuma.monitoring.group.name: "Monitoring"
  kuma.uptime_kuma.http.parent_name: "monitoring"
  kuma.uptime_kuma.http.name: "Kuma status monitoring"
  kuma.uptime_kuma.http.url: "http://${HOST_IP}:3001"

This actually does 2 things:

creates a group with the key/id monitoring and the name Monitoring
Adds a monitor with the key/id uptime_kuma to UptimeKuma with the type http, name Kuma status monitoring, and url http://${HOST_IP}:3001

Adding these labels, whilst AutoKuma is running and configured to pick up labels starting with kuma is enough for monitors to show up (after restarting the containers). All in all, my docker-compose.yml file for both UptimeKuma and AutoKuma now looks like this:

services:
  autokuma:
    image: ghcr.io/bigboot/autokuma:master
    restart: unless-stopped
    environment:
      AUTOKUMA__KUMA__URL: http://${HOST_IP}:3001
      AUTOKUMA__KUMA__USERNAME: ${KUMA_USERNAME}
      AUTOKUMA__KUMA__PASSWORD: ${KUMA_PASSWORD}
      AUTOKUMA__TAG_NAME: AutoKuma
      AUTOKUMA__TAG_COLOR: "#42C0FB"
      AUTOKUMA__DEFAULT_SETTINGS: |-
        docker.docker_container: {{container_name}}
        http.max_redirects: 10
        *.max_retries: 3
      AUTOKUMA__DOCKER__LABEL_PREFIX: kuma
      AUTOKUMA__MIGRATE: true
    volumes:
      - ${APP_DATA}/autokuma:/data
      - ${DOCKER_SOCKET}:/var/run/docker.sock
    depends_on:
      - kuma

  kuma:
    image: louislam/uptime-kuma:1
    volumes:
      - ${APP_DATA}/uptimekuma:/app/data
      - ${DOCKER_SOCKET}:/var/run/docker.sock
    ports:
      - 3001:3001
    restart: unless-stopped
    labels:
      kuma.monitoring.group.name: "Monitoring"
      kuma.uptime_kuma.http.parent_name: "monitoring"
      kuma.uptime_kuma.http.name: "Kuma status monitoring"
      kuma.uptime_kuma.http.url: "http://${HOST_IP}:3001"

Problems… the compose file doesn’t work :O

If you’d try to run the compose file in the previous chapter, even after replacing all the variables, things likely still won’t work. This is because AutoKuma relies on the credentials for UptimeKuma which we setup during the initial launch of UptimeKuma. Unfortunately, that account is the only account we can currently set up in UptimeKuma since it doesn’t have user management. And yes, that means we have to use these same credentials to get AutoKuma to work, which also prohibits us from enabling MFA.

Anyway, after setting up the kuma account, simply adjust both the KUMA_USERNAME and KUMA_PASSWORD variables and restart the containers. After doing so we should see the monitor appear in UptimeKuma:

neat!

The Future of IaC

Infrastructure as Code isn’t just a technical approach; it’s a philosophy that prioritizes automation, reproducibility, and simplicity. Whether you’re managing a home lab or a global platform, IaC provides the tools and practices to build resilient systems with minimal effort.

For me, IaC has been a journey of continuous learning and experimentation. From Docker to Kubernetes, from bash scripts to dedicated platform teams, the principles remain the same: automate everything, document everything, and embrace the predictability that code brings to infrastructure.

If you haven’t already, give UptimeKuma and AutoKuma a try. Their combination is an excellent example of how IaC can simplify even the most mundane tasks, leaving you more time to focus on what really matters… or just to be a little lazier.

LegoLeague - Battling it out with little robots made from Lego

Mon, 18 Sep 2023 00:00:00 GMT

The 2023 LegoLeague battle board

A little before the start of the summer period my colleague Jan-Kees came up to me and proposed the idea of an internal LegoLeague. After a little bit of back-and-forth about what LegoLeague exactly was I agreed it would be a great idea to bring some colleagues together and have a little fun during work hours. Pretty much right away excitement got the better of us and we started drafting a plan… when? where? how? prizes? teams!?? A lot. But First…

What is LegoLeague?

LegoLeague is an event where teams come together to build, program and test a Lego Spike robot to perform autonomous tasks. The tasks range from collecting “power cores”, pushing or pulling levers, to dropping Lego blocks in a hopper to get some bonus points.

All of this is accompanied by a theme, in our case energy production, which adds a layer of fun to solving these challenges. The video below shows a teaser of 2022-2023’s “SuperPowered” event, the very thing we based our LegoLeague on:

For each task a team completes successfully they will score points. Teams get 2 minutes and 30 seconds (2:30) to score as many points as they can. The video below outlines some of the challenges the teams have available to them with a little explanation of how they function and even some info on the scores. This video is especially useful for teams that don’t want to peruse the entire rule book but still wanted to figure out an optimal strategy.

To build the robot each team gets a Lego Spike Prime set and nothing more! In the original event, teams had access to a little more Lego in various kits but for our little event at INFORIT, we were only able to source the basic kits in time. Having said that, we wouldn’t have been able to get our hands on any of the kits at all if it wasn’t for the “PReT” organization in Weesp. Thanks guys!

A Lego spike set laid out on display

As for the programming… Teams are free to use either ”scratch” or “Python” to program their robots. Scratch being much easier for those who don’t spend all day behind their computer working on software development whilst Python allows for more flexibility in what can be achieved. Spoiler alert: all teams chose scratch, it’s much more accessible for non-developers!

The competition at INFORIT

So yeah, we now know what a LegoLeague is and we have the approval to host the competition at INFORIT, now what? Well, Jan-Kees got to work building the arena and setting it up and I set out to gather team members. Given that it was the summer break I had to hustle to get teams together but eventually we managed to create some beautiful teams.

Fortunately, INFORIT values employee happiness, autonomy and continuous improvement. As such I was able to offer the 2 Friday afternoons free of day-to-day work, to work on the robot whilst providing catering, music, and the like to the teams whilst they are participating!

In the end, we were successful in creating 4 teams of at least 3 people who were all mixed with developers and non-developers to keep the balance fair. Meanwhile, Jan-Kees also finished building the arena which you can see in the header of this blog or the pictures in the picture section.

Pictures

Of course, we’ve taken loads of pictures during the event but those are mostly for internal use. Below I’ve included some pictures of the event that are all cleared for social media distribution with a little description of what’s happening in each picture to, hopefully, convey some of the vibes of the event itself.

one of the competing robots

A contestant calculating exactly where to go. Such a pathfinder!

code -> test -> repeat. Soooo important

This team might have overcomplicated things a tad. Looks impressive though.

Winners, losers, etc

In the end, this is still a competition. People competed hard to come out on top and 1 team did so successfully. Congratulations guys!

All of our winners! (excluding myself 😢)

Nah, we are all winners in the end

Having 4 teams gave us the unique opportunity to reward them all with something nice. The 3 teams that got podium places all got their medals and their rewards and the team in fourth place got their consolation prize.

Luckily though, we’re all winners, even the Jury members and the organizers. This event has brought us all closer together and improved the connection between our colleagues. I, personally, had a blast and would love to do similar things in the future.

That’s it again, thanks for reading and see you next time!

Flashing the LSI-9211 to IT mode using an EFI shell

Wed, 12 Apr 2023 00:00:00 GMT

I’ve been building a new storage-oriented server for a while now and have yet again decided to go with (3 +1 backup) LSI 9211 raid controllers.

The reason I keep going for these specific raid controllers is quite simple. The card can theoretically support (8x500MB) 4GB/s in throughput divided over 8 drives (2x SFF-8087) which is near the maximum for consumer hard drives.

Using 3 of these will allow me to use 24 drives in a single chassis without an expander whilst getting pretty good performance. The fourth card that I got, and flashed, is simply used as a backup or debug card in case of any errors.

Downloading the files

Before we can start flashing we will need to download some files. The table below lists all the files with a brief description of what we’ll use them for.

File	Description
EFI flashing utility	Flashing utility
EFI ROM image	Card ROM file
SAS2008 Firmware	IR and IT firmware for the 9211

Prepping the flash drive

Make sure that you have a flash drive that is 32GB or smaller and that it is formatted as FAT32. Next, we’ll add the following files to the root of the flash drive:

Installer_P20_for_UEFI/sas2flash_efi_ebc_rel/sas2flash.efi
UEFI_BSD_P20/uefi_bsd_rel/Signed/x64sas2.efi
9210-8i_Package_P20_IR_IT_FW_BIOS_for_MSDOS_Windows/sasbios_rel/mptsas2.rom
9210-8i_Package_P20_IR_IT_FW_BIOS_for_MSDOS_Windows/Firmware/HBA_9210_8i_IT/2108it.bin

Flashing with EFI

Because we are going to flash using EFI we don’t really need anything besides the built-in EFI shell from our motherboard. Simply insert the flash drive and go into the BIOS or use the “select boot device” functionality to select the EFI shell:

selecting the EFI shell boot option

After the shell has booted successfully you can use the map command to see all available devices. In the image below you’ll see that my flash drive is listed as fs1:

a list of devices note that `fs1` is my flash drive

To switch into the drive simply enter the drive name on the command line followed by a semicolon:

fs1:

Next, we’ll determine that our adapters are available and we will simultaneously take note of their identifier, we can do so by issuing the following command:

sas2flash.efi -listall

a list of adapters, I flash them one at a time so I only have card '0' available to me

Next, we can empty the current firmware with the following command:

sas2flash.efi -o -e 6

bye, bye firmware 😢

Now we’ll simply flash the new (IT-mode) firmware on it with the following command:

sas2flash.efi -o -f 2108it.bin -b mptsas2.rom

1 card down, 3 more to go!

Wait patiently until the operations have completed. Note that it might seem like the steps repeat, just wait until it is completely done and exits back to the shell by itself.

Multiple cards and conclusion

You can rerun the commands for different cards (by adding -c x where x is the card number from listall) or by simply physically switching the current one out for a different one.

That’s all there is too it, if you need any help feel free to hit me up!

Uncovering a new hobby that might just take over my entire house

Tue, 14 Feb 2023 00:00:00 GMT

The problem

Even though my house is fully furnished, one could say it’s quite light on decorations. You see, I’m not really one for many of the popular decoration pieces like candles, picture frames, animal skins, or decorative statues.

The only thing I really like is stone and woodwork. This means that I have enough decorations to fill the downstairs, kind off. You see, some of the decorations that I also like are plants (mainly because they go in large stone pots) but I’m not great at keeping them alive.

(Un)fortunately, there is more to my house than just the downstairs. There are 2 more floors that are, once again, furnished but not really decorated. The only room that is pretty well decorated is my office, which has various little nerdy things on little black shelves and 12 beautiful Displates.

One of the main problem areas is my bedroom. I don’t have any entertainment in my bedroom (by choice.. it’s a place to sleep, not watch telly!) but I do have some FJallbo shelving units. Of course, those shelves are mostly empty… they are only adorned by some books, clothes and a few stuffed animals.

The solution, or so I thought

Pretty little orchids!

I don’t know when exactly it happened but a few months ago I stumbled across “Lego plants” on Youtube. Plants that look quite good and don’t die. The first plant I got was the little bonsai and the second one were the orchids. After that, I quickly picked up the cacti and bouquet.

And as you can probably figure out by now… a new hobby was born… Lego. Lego allows me to fill up the empty space with whatever I want whilst keeping things interesting and giving me something to do to boot.

Christmas time’s a coming

After building a few smaller items I felt ready to tackle a “project”, one that would allow me to play with Legos and at the same time get a few friends and family members off my back about not having Christmas decorations. That’s right. I set out to build a Christmas village from lego.

I emptied a shelf, picked up some white base plates, procured a string of small Christmas lights, and bought some Christmassy-looking Lego sets:

The result of a few days of work and the Christmas village has been completed:

Happy little Christmas village

A new room to build Lego

Now that the Lego virus has taken hold of me I am quickly growing tired of needing to move stuff off my dining table just to build some Legos. Likewise, I dislike having to move half-finished Lego builds just to use my dining table. It was time to upgrade my spare bedroom with a desk and some other things to build a comfortable space.

I bought the following items:

Which, with some effort, got turned into the following room:

First build of the room, no display or audio yet

A little trolley to hold the new Lego sets

I added a monitor so I could watch some series or listen to music.

Sound from just the monitor won't do so I added a nice JBL speaker

the finished room in use, building the Harry Potter special edition owl

The space I’ve created with this Ikea build works very well for building Legos! The mounted desk light (which I got from Amazon) has also proven to be quite handy. I can move the light however I like, be it to shed some more light on the little booklet with instructions or on the build itself to illuminate some of the darker pieces at night. (brown/black) Turning on both lights (there’s one hidden behind the monitor/speaker) ensures that there’s enough light in the room that I don’t have to turn on the main light which makes the room feel nice and cozy.

More of my creations

I’d like to round off this blog post by showing some of the other builds I’ve since completed:

Saturn V

Shout out to Hedwig! (and Ravenclaw)

The desk felt a bit empty so I added some flowers to spice things up

Two of the succulents in decorative pots with some shells

Adopting conventional commits and standard versions in a git and npm codebase

Mon, 07 Nov 2022 00:00:00 GMT

At my place of work, INFORIT, we are talking about adopting conventional commits as a company standard. However… debating commits is a tricky endeavor, everyone seems to have their own opinion about it and pretty much no one is following even the basic git rules (e.g title + body). It is precisely because of all of this that an earlier proposal about standardizing commit messages brought us nothing.

Now though, a solid 2 years later, our company has grown tremendously! Not just in actual size, but also in knowledge. The developers that were resistant to standardization, hell… even git, have since come around to many of the ideas that were implemented. Especially after seeing their, often new, colleagues thrive with it.

After seeing one of the new hires use conventional commits and their team being fine with it I decided to take the plunge and submit a proposal for it myself. Though the proposal only covers “conventional commits” and generating changelog entries from it I will be explaining the entire process of adopting conventional commits and standard versions in this blog post.

Getting started

Before we can do anything we need a git repository. I’ll be using a hosted git repository (on Github) so I will clone it:

git clone git@github.com:Mastermindzh/conventional-commits-showcase.git

But, if you want to follow along without a hosted repository you can also manually create it yourself:

mkdir conventional-commits-showcase # create a folder with the name "conventional-commits-showcase"
cd conventional-commits-showcase
git init

What exactly are “conventional commits”

”Conventional commits” is a specification that allows developers to write commits that are easily parsable by both developers and systems alike. Or in their own words:

The Conventional Commits specification is a lightweight convention on top of commit messages. It provides an easy set of rules for creating an explicit commit history; which makes it easier to write automated tools on top of. This convention dovetails with SemVer, by describing the features, fixes, and breaking changes made in commit messages.

According to the specifications the commit message should be structured as follows:

<type>[optional scope]: <description>

[optional body]

[optional footer(s)]

The specification also allows us to specify a “footer” with breaking changes and ticket references. A full example, with a breaking change and a fictional ticket number of 95043 would look like this:

feat(api): A test title for a change to the API

I made a few changes to the API. Now I have to write a body.
This is a looooooong body.
Multiple lines :O

BREAKING CHANGE: Well, I have to write something here... so :)
fixes #95043

Let’s get the dependencies installed

First, let’s make sure our repository contains an npm environment. If not, run:

npm init

Now that we have a package.json file we can start installing some dependencies:

npm install --save-dev husky @commitlint/cz-commitlint commitizen commitlint standard-version

That will give us all the dependencies we need during this blog:

Dependency	function
Husky	Allows us to easily manage and execute git hooks
commitlint	Checks whether our commit messages meet the conventional commits specification
@commitlint/cz-commitlint	adapter for commitlint to use commitizen
commitizen	Provides a prompt during commit to fill in necessary information
standard-version	Semver based release management and changelog generator based on conventional commits

Setting up the commitlint and commitizen configuration

First, let’s add a script to the package.json that will do our commit for us:

{
  "scripts": {
    "commit": "git-cz"
  }
}

Now, when we call npm run commit it will actually run git-cz which will start our commitizen prompt. You can try it if you want, it won’t actually do anything but throw an error right now. We can also configure commitizen in the package.json as well, to do so we add commitizen to the package.json’s config object:

{
  "config": {
    "commitizen": {
      "path": "@commitlint/cz-commitlint"
    }
  }
}

If you ran npm run commit before and got an error, you can now do it again and face another, but different, error!

Now we’ll set up the actual commit linting. Start by creating a file commitlint.config.js with the following content:

module.exports = {
  rules: {
    "body-leading-blank": [1, "always"],
    "body-max-line-length": [2, "always", 100],
    "footer-leading-blank": [1, "always"],
    "footer-max-line-length": [2, "always", 100],
    "header-max-length": [2, "always", 100],
    "subject-case": [2, "never", ["start-case", "pascal-case", "upper-case"]],
    "subject-empty": [2, "never"],
    "subject-full-stop": [2, "never", "."],
    "type-case": [2, "always", "lower-case"],
    "type-empty": [2, "never"],
    "type-enum": [2, "always", ["first type", "second type"]],
  },
  prompt: {
    questions: {
      type: {
        description: "This is the first prompt!",
      },
    },
  },
};

As you can see we have configured a whole bunch of things, but there are 2 things that matter for now:

a single “question” in our prompt.
a few types (first and second type)

Staging a few files (git add <file>) and running npm run commit will show us our prompt and a choice between the two types:

output of 'npm run commit'

Now, our current config isn’t very useful, so let’s add some more types and a few additional questions (the defaults):

module.exports = {
  rules: {
    "body-leading-blank": [1, "always"],
    "body-max-line-length": [2, "always", 100],
    "footer-leading-blank": [1, "always"],
    "footer-max-line-length": [2, "always", 100],
    "header-max-length": [2, "always", 100],
    "subject-case": [2, "never", ["start-case", "pascal-case", "upper-case"]],
    "subject-empty": [2, "never"],
    "subject-full-stop": [2, "never", "."],
    "type-case": [2, "always", "lower-case"],
    "type-empty": [2, "never"],
    "type-enum": [
      2,
      "always",
      [
        "build",
        "chore",
        "ci",
        "docs",
        "feat",
        "fix",
        "perf",
        "refactor",
        "revert",
        "style",
        "test",
        "content",
        "blog",
      ],
    ],
  },
  prompt: {
    questions: {
      type: {
        description: "Select the type of change that you're committing",
        enum: {
          content: {
            description: "A new piece of content",
            title: "Content",
            emoji: "✨",
          },
          feat: {
            description: "A new feature",
            title: "Features",
            emoji: "✨",
          },
          fix: {
            description: "A bug fix",
            title: "Bug Fixes",
            emoji: "🐛",
          },
          docs: {
            description: "Documentation only changes",
            title: "Documentation",
            emoji: "📚",
          },
          style: {
            description:
              "Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)",
            title: "Styles",
            emoji: "💎",
          },
          refactor: {
            description: "A code change that neither fixes a bug nor adds a feature",
            title: "Code Refactoring",
            emoji: "📦",
          },
          perf: {
            description: "A code change that improves performance",
            title: "Performance Improvements",
            emoji: "🚀",
          },
          test: {
            description: "Adding missing tests or correcting existing tests",
            title: "Tests",
            emoji: "🚨",
          },
          build: {
            description:
              "Changes that affect the build system or external dependencies (example scopes: gulp, broccoli, npm)",
            title: "Builds",
            emoji: "🛠",
          },
          ci: {
            description:
              "Changes to our CI configuration files and scripts (example scopes: Docker, Kubernetes, npm, git)",
            title: "Continuous Integrations",
            emoji: "⚙️",
          },
          chore: {
            description: "Other changes that don't modify src or test files",
            title: "Chores",
            emoji: "♻️",
          },
          revert: {
            description: "Reverts a previous commit",
            title: "Reverts",
            emoji: "🗑",
          },
        },
      },
      scope: {
        description: "What is the scope of this change (e.g. component or file name)",
      },
      subject: {
        description: "Write a short, imperative tense description of the change",
      },
      body: {
        description: "Provide a longer description of the change",
      },
      isBreaking: {
        description: "Are there any breaking changes?",
      },
      breakingBody: {
        description:
          "A BREAKING CHANGE commit requires a body. Please enter a longer description of the commit itself",
      },
      breaking: {
        description: "Describe the breaking changes",
      },
      isIssueAffected: {
        description: "Does this change affect any open issues?",
      },
      issuesBody: {
        description:
          "If issues are closed, the commit requires a body. Please enter a longer description of the commit itself",
      },
      issues: {
        description: 'Add issue references (e.g. "fix #123", "re #123".)',
      },
    },
  },
};

As you can see we have a fair few questions, each with its own description. The one that will be new to most people is “content”, which is a custom type that I added for this very blog. Reenacting our example commit from before will look like this:

Looking good so far!

That’s pretty nice already! Creating our example commit is nice and easy now. Next, let’s add a git hook to stop us from messing up.

Setting up git hooks

Without the git hook, we can simply make a commit that doesn’t adhere to conventional commits:

mkdir test-files && touch test-files/test1
git commit -m "this is not conventional :O"

Yikes! That is not what we want! Let’s quickly revert it:

git reset --soft HEAD~1

Now, let’s add a Husky git hook to combat future commits that are unconventional:

npx husky-init && npm install

This will automatically create the .husky/ directory. Let’s create a file in it called commit-msg and add commitlint:

#!/usr/bin/env sh
. "$(dirname -- "$0")/_/husky.sh"

npx --no-install commitlint --edit "$1"

Let’s try our unconventional commit again:

touch test-files/test2
git add test-files/test2
git commit -m "this is not conventional :O"

nice! It throws an error :)

Great, our commits seem to be working. Let’s commit all the things we have now before we move on, crucially I will be using the regular git commit and providing the conventional commit syntax myself:

git commit -m "ci(git): Added conventional commits and some dependencies"

Releases

Now that we’ve successfully set up conventional commits we want to use those to release our first version and generate a changelog to boot. First, let’s add some npm scripts again:

{
  "scripts": {
    "release": "standard-version",
    "release:minor": "standard-version --release-as minor",
    "release:patch": "standard-version --release-as patch",
    "release:major": "standard-version --release-as major"
  }
}

Now, let’s add a configuration file for standard-version so that it will generate changelog entries. Create a file named .versionrc.json with the following content:

{
  "types": [
    {
      "type": "feat",
      "section": "Features"
    },
    {
      "type": "fix",
      "section": "Bug Fixes"
    },
    {
      "type": "content",
      "section": "New content",
      "hidden": true
    },
    {
      "type": "chore",
      "hidden": true
    },
    {
      "type": "docs",
      "hidden": true
    },
    {
      "type": "style",
      "hidden": true
    },
    {
      "type": "refactor",
      "hidden": true
    },
    {
      "type": "perf",
      "hidden": true
    },
    {
      "type": "test",
      "hidden": true
    }
  ],
  "commitUrlFormat": "https://github.com/mastermindzh/conventional-commits-showcase/commits/{{hash}}",
  "compareUrlFormat": "https://github.com/mastermindzh/conventional-commits-showcase/compare/{{previousTag}}...{{currentTag}}"
}

The first thing we declare is an array of types, for every type we can define a “section” and whether it is hidden or not. All hidden types will not show up in the changelog, the others will be grouped by type and will have a header with the text specified in the “section” field. Let’s try it:

git add .versionrc.json package.json
git commit -m "ci(git): Set up standard-version"

# and let's add a feature
touch test-files/test3
git add test-files/test3
git commit -m "feat: Feature test3"

note: Where I’ve been using git commit -m you can also use npm run commit to get the interactive git prompt. Now, let’s release a major version:

npm run release:major

our first release!

The output is kind enough to tell us how to push our changes to the remote so let’s just do it:

git push --follow-tags origin master

Now let’s have a look at our git log:

we have our own commits + a new release commit

Now, let’s also look at our changelog:

very neat!

That looks great! standard-version automatically added a header for our version, sub-headers for our types and even linked our commits to the remote as configured in .versionrc.json.

Let’s play a little

Now that we have a successful setup we can try a few commits and releases. Let’s start with 2 bugfixes and a new feature:

# our first bugfix
touch test-files/test4
git add test-files/test4
git commit -m "fix: fixed test 4 by doing fancy stuff"

# our second bugfix
touch test-files/test5
git add test-files/test5
git commit -m "fix: test5 was difficult, but we got it"

# and finally a new feature
touch test-files/test6
git add test-files/test6
git commit -m "feat: Added feature number 6! So nice"

Now we’ll release this as a minor, because we added a new feature. Execute

npm run release:minor

our updated changelog

breaking changes

All is well, but let’s introduce a breaking change:

touch test-files/test7
git add test-files/test7
npm run commit

Output:

> conventional-commits-showcase@1.1.0 commit
> git-cz

cz-cli@4.2.5, @commitlint/cz-commitlint@17.2.0

? Select the type of change that you're committing: feat
? What is the scope of this change (e.g. component or file name) (press enter
to skip): (max 96 chars)
 (0)
? Write a short, imperative tense description of the change: (max 96 chars)
 (38) feature 7, with a few breaking changes
? Provide a longer description of the change (press enter to skip):
 We broke some stuff, woopsie
? Are there any breaking changes?: Yes
? Describe the breaking changes:
 api endpoint 'test7' now returns json instead of xml
? Does this change affect any open issues?: No
[master e7f7b0e] feat: feature 7, with a few breaking changes
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 test-files/test7

Which gives us the following changelog:

everyone will see this message, no users will complain, no way!

Conclusion

Phew! That was quite a long post. Hopefully, you’ve managed to set up conventional commits and/or standard-releases to your liking and if not there’s always the comment section. The end-result can be found on Github, the Changelog is especially pretty.

My blog hasn't gotten an update in 5 years, here's why.

Sun, 30 Oct 2022 00:00:00 GMT

I’ve been rebuilding the old ”rickvanlieshout.com” for over 5 years now… But the website isn’t that big. It could be done way faster and it has been done way faster. The thing is, I rewrote it several times, each time with a different design and updated packages.

Changing the theme (round #1)

Shortly after finishing the original website its design already felt stale. Sure, it had some nice colors and I really liked the mobile view (it read beautifully on a smartphone), but it still felt off. Nothing on the website gave me a wow effect, most of it was “ok” and some of it was not to my liking. I never quite put my finger on the why, but it just didn’t feel right.

I did put a lot of time into optimizing and designing the bits I found important, the biggest time sinks were multi-language support and print mode. In the end, I even ended up making a compromise on the multi-language support because I didn’t want to blog in Dutch. One of the things I’m glad I did, however, was working on styling print mode. It wasn’t because I expect many people to print my website or articles, it’s because I expect some of them to use “read-focus” extensions or apps like pocket as I do so often myself. I also suspect the occasional pdf download, most of which I probably did myself on my resume page though.

One of the biggest annoyances that I tried solving with a new theme was dark mode. Hell, I swung completely in the opposite direction and bought the all-dark Hikma theme. I rebuilt the entire site in that theme, but never published. The next chapter will explain why.

Moving to Gatsby

Right around the time I started thinking about uploading the new theme I also found GatsbyJS. Gatsby was in its infancy back then, barely out of beta and a little rough around the edges. Considering I like to live on the edge, software-wise anyway, I decided to go for it. rickvanlieshout.com-hikma-gatsby was born and I worked on it for a few short weeks.

In the end, that also burned out because, again, the theme felt stale.

Upgrading to Gatsby 3 and back to the old theme

I’d had enough! I needed to finish my website and I needed to do it quickly. I picked up the new Gatsby V3, my old theme, and a renewed mental vigor and started coding away. The first prototypes came into existence very quickly and looked fine.

Alas, I burnt out on the project almost as quickly as I had started. One of the reasons was the theme… but there was a more serious reason this time as well.

Server-side rendering, the old theme, and a stubborn software architect

During the rewrite to Gatsby V3 I encountered various SSR issues. The most notable of them all was not being able to save and apply the color scheme without annoying flashes on page loads. I chalked it up to a CSS issue and left it for what it was at first, but I simply couldn’t go live like this.

We just discovered my other biggest flaw. Perfectionism. Because I am responsible for code and product quality at my job and have also heavily emphasized that part of my contributions in any of the open-source projects I’ve worked on, I felt that I couldn’t release something that wasn’t perfect. And as most of you will know, you can chase perfection forever. There is always a flaw, always something to improve.

In my mind, I could fix the quality issues, but I’d still be left with the theme. I also realized I’d end up at the same point I started… Same design, and the same missing features, just a bunch of time later. This wasn’t working either.

Finding a new design and a new starting point

One day I found myself procrastinating on a whole bunch of things by looking at fancy CSS themes. I hadn’t really thought about it, but I’d ended up looking at new portfolios and blog designs. I’d found a few good candidates but didn’t really make a choice yet.

Was I embarking on the journey once again? Or would I give up before I started? Eventually, I decided to organize my thoughts and figure out why things failed before, it roughly came down to:

I can’t go about and make it perfect on the first try
- No translations
- Print mode for all the important pages
- Dark mode can come later
I didn’t like the design, I wanted something black and white which could be reversed for dark mode
I didn’t like the previous efforts of using Gatsby for just the posts and not the static pages
I needed a starting point, no matter how “good” it was, I needed a place to start writing again.
I needed a new website fast because the old one was breaking down (PHP updates caused the API to break)

Eventually, I stumbled upon ”lumen”. A beautiful blog project with amazing typography and a relatively modern Gatsby setup. It wasn’t perfect, it missed a few bells and whistles, but it did tick off a lot of boxes. I decided to fork the project and try it out, adding bits and bobs of content to see how my writing style would look with the new theme.

I liked it.

Not all was perfect however

As with all things in life, this project would need some work. I found a few things that were lacking which I wanted to fix before releasing a new website. I was determined not to fall into the same trap as before however and structured the work better than I ever had before.

Things I changed

Below you’ll find a list of some of the biggest things I’ve changed.

I added a Dark mode with a little react-toggle to switch between light/dark.
Blog posts now show a header that includes the “back to articles” button and my name.
Added code block theming for both light and dark mode
I added support for tables with a bit of styling around them (slight indent and row styling)
I added the ability to quote others in a beautiful way
Medium like image zooming (click any of the images above)

The posts query bug

Here and there I also encountered some bugs. Most of them got fixed in my copy but one of them was big enough that I felt a need to contribute to the project. The “posts query” bug.

Before the fix, there was a bug in the post-query file that simply looked through all markdown files to find pages with a slug. This, however, is troublesome if you have more than the 3 example pages because the default page sizing will fail. You see, the code that fetches the blogs does so by filtering out real “posts”, but the pagination queries do not. This meant that, given enough pages, the pagination would overflow and tell the user that there were more posts available than there were. If a user were to click on “next” they would be presented with an empty page.

In the end, it was a tiny bug, but I submitted a PR to fix it in the upstream repository.

making it my own

Though the original codebase gave me a great starting point I have decided not to follow the upstream contributions but instead forge my own future. Doing so allows me to keep my codebase clean and allows me to deviate from the choices made by Alexander.

Alexander is slowly introducing more features than I need. One of the newer feature commits (a dark mode) even included his own state management library “diesel” for which I see absolutely no fit in the project.

As such, I have decided to simply skip pulling from upstream and maintain the website myself.

The future is bright

Now that I have this new blog it should be easier for me to write some articles. Let’s hope I will find time to write more than I have in the past 5 years!

My dog, Eddy, died today. He will be missed.

Fri, 30 Sep 2022 00:00:00 GMT

This is Eddy, January 30, 2011

Unfortunately, I’ve had to put my dog Eddy down today. Eddy was my best friend. Not just my best furry friend, but my best friend period. He never failed to bring me joy, he was always there when I needed him and he helped me overcome some of life’s biggest obstacles just by existing.

Putting him down was very hard for me. Eddy had been in a rough shape for at least a week but I wasn’t ready to give up on him. As a matter of fact, I didn’t want to give up until after he was put down. To say I cried like a newborn would be an understatement. I’ve felt pain like this in my life before, and I swore I would never let it happen again but Eddy managed to nestle his way too deep in my heart.

I love you Eddy. Always have. Always will. Rest in peace my friend.

Eddy

2009-10-18 - 2022-09-30

Why Eddy had to be put down

Such a cute boy with his ribbon, July 2016

Of course this isn’t the first time Eddy has had some issues with his health. During the course of a life many things can happen and some of those will inevitably be bad. I’ve included a picture of the time Eddy had to carry a ribbon through his torso to prevent the wound from closing. Not the prettiest sight but it was worth it because he pulled through. Apart from that Eddy also had to take medicine twice a day for various other illnesses, and he was very good at hiding his pills too… Later Eddy lost a lot of his teeth and also a big part of his vision.

The issues with Eddy were much more serious this time though. Eddy has had stomach ulcers for a long time now and whilst they are somewhat treatable with medication they will almost always develop to become a bigger issue. In his last few days Eddy began spewing blood, having very dark excrements and generally being a tad cranky. Not traits usually associated with the happy little dog I knew. Just before Eddy was put down the veterinarian pointed out a hard lump in his stomach, thought to be a cancerous growth. I felt it and remembered feeling something very similar before. I knew then that we were making the right call.

mixed feelings

I am not often an emotional wreck, but today I was. Though, after some thought I realized something that I’d realized before:

For something to cause me so much pain it had to have brought me joy. Memories are fading in and out, my vision goes blurry. I’m coming short of breath… The only thing keeping me sane is knowing I have memories of us, the pictures captured a mere glimpse of it. But my heart captured everything.

- from the poem thoughts by Rick van Lieshout

Very professional doggy. Early 2022

And though those feelings are true. I also can’t help but shake the sorrow I feel. I realize, all too well, that I’ve lost the last thing on this planet that I chose to truly love. Family ties are great and all, but I chose to spend my life with Eddy. I chose to take care of him and to love him until I couldn’t anymore.

A little over 2 years ago I moved out of my parent’s house and I unfortunately couldn’t take Eddy with me. Not only because my job doesn’t really allow me to take the best emotional care (9-5..) of a dog but also because Eddy is old. Eddy simply didn’t want to move. He knew his surroundings, even though he didn’t see very well, and didn’t really want to change them up. Though I went to see him every week (and my parents too…) I still didn’t like being away from him for the rest of the week.

I have debated getting a pet of my own. But as I said before I would feel sorry for the animal because I wouldn’t be home much. I also thought of getting a cat, but I just like dogs more. Now that Eddy has left me completely I will just have to see where life takes me. I sure could use a buddy at times so maybe I will end up adopting a pet anyway.

Farewell

In the end, though, I don’t think anything can truly replace Eddy. I got Eddy an urn so I can have him in my life for the rest of mine. Sleep well, friend.

Eddy peacefully "sleeping" post mortem

The urn

It took way too long for the urn to be filled with pictures but in the end, I’ve gotten it done and I hung Eddy’s urn in my bedroom.

A lovely memory of Eddy

Server management tips: Keep 1 sata port available

Fri, 01 Dec 2017 00:00:00 GMT

Servers… Both a joy and a pain. This week my server has just been a pain though… What should’ve been a relaxing night with some easy peasy hard drive replacements quickly turned into hours of downtime and lots of yelling and screaming :(

During my holiday stay (at Tropical Islands in Germany, pictures can be found here), I decided I’d replace the last 3 500 GB drives in my server array (don’t ask me why I was thinking about my server on holiday). So, as soon as I got home, I decided to look around and order some of those beautiful WD Red drives. I considered upgrading my parity drive as well, but I’d have to buy 3 TB models and that seemed like more of a hassle than I was willing to take on at the moment. (I should have.)

So, the drives arrived, I ran a quick benchmark followed by a quick drive test and all seemed fine with the first drive. At this point, I decide to open up the server and replace the first 500 GB hard drive. After a lot of gymnastics (trying to read drive labels while the drives are in the drive bays) I finally figured out which drive I should replace and I went to work on that drive. I was careful not to touch any of the other cables to avoid accidentally unplugging them and I thought I had done a good job. I remember thinking: “Drive replaced, cables still connected, not much hassle so far. Let’s turn it on”. And so I did.

Prepare for trouble

My good mood was rudely interrupted by a loud beep followed by three, rapid, beeps. At first, I thought it was the regular old “fan not spinning” error (I use several ways to limit fan speed) but on further inspection, I realised that the fan was spinning just fine. Things were about to turn bad…

The drive I had inserted was fine, the OS picked it up and it was performing well. One of the other drives had failed though… So I shut it back down, reconnected the old drive (the one I replaced) and replaced the failing drive with a brand new one. A parity sync is necessary at this point.. Sigh..

After replacing the “failed” drive and letting the machine run for a while I got an email from my server, emails from my server are usually bad, stating that a drive had failed. At first, I was going to dismiss it as I had just replaced one and figured the email was simply delayed. Upon closer inspection, I found out that another hard drive had failed! So I logged back into the server and inspected the problem. The new drive had failed… even though it previously passed my tests with flying colours!

Something was up… and after countless hours of fighting with my server I finally figured it out. The SATA port on the controller was faulty. It didn’t error, it didn’t smoke, it looked fine. It even held on to the cable just fine (cause of those tiny little lips on SATA cables) so it was kinda hard to find out that the SATA port was faulty.

a random yellow SATA cable

So, with a way to reproducibly “fail” known good drives it was just a matter of plugging the SATA cable into another port and go on with my day… or so I thought…

Little did I know that my server had used up all its SATA ports! I didn’t have a single one left! In the end, I had to go out, pay way too much for a raid card that I didn’t need and work on the server till dawn in order to fix my mistakes.

Please don’t make the same mistake… make sure you have a replacement SATA port (or raid card for that matter).

Live and Learn I guess, Live. And. Learn.

Scala Day 4 - A challenge!

Wed, 05 Apr 2017 00:00:00 GMT

The book doesn’t offer a day 4, my school, however, does. Or rather, it wants us to “create” a day 4. The main objective of day 4 is to “create a smart goal which fits the language and challenges you”. For this purpose I came up with the following goal:

For the “4th” day in the “Seven Languages in Seven Weeks” book, I want to create a sitemap builder which prioritises URLs based on their number of occurrences.

I picked this specific goal because Scala has awesome functionality for collections, as such it should make creating a site map a piece of cake! All the code I created for this fourth day can be found on Github or at the end of this blog post.

My journey with Scala (and the book…)

Scala, for the most part, is pretty ok. The development tools aren’t up to par with those of say Java, Python or even .net but with a bit of hassle, we can make do. The Scala collections are wicked though! I’ve done some really crazy things using the collections already and I’m sure there is more to come. (if anyone ever wants me to write a bit of Scala that is)

The concurrency chapter was a big failure, the book is old (2010) and covers the older API. This meant that I had to cast aside the book and create the examples with the new API myself. A drag, to say the least.

As far as functional goes, my opinion hasn’t changed much. I still prefer OO, maybe because I grew up doing everything the OO way or maybe because functional programming is just not for me. Whatever the case I still prefer OO. But… I’ve gained a new appreciation for the functional languages so I guess you could call this experiment a success.

Anyways, that was it for me guys, hope you enjoyed and see you later!

import java.io.{File, PrintWriter}

import scala.collection.immutable.ListMap
import scala.io.Source

object PageLinkLoader {
  //I stole this regex from somewhere for a previous project, don't remember where

  // regex commented out because pretify can't handle it
  //val hrefRegex = """<a\s+(?:[^>]*?\s+)?href=(["'])(.*?)\1""".r

  // merge two maps
  def mergeMap[A, B](ms: List[Map[A, B]])(f: (B, B) => B): Map[A, B] =
    (Map[A, B]() /: (for (m <- ms; kv <- m) yield kv)) { (a, kv) =>
      a + (if (a.contains(kv._1)) kv._1 -> f(a(kv._1), kv._2) else kv)
    }

  // run through all links on a page and add them to a list
  def getLinks(url: String, urls : Map[String, Int], visited : List[String] = List[String]()) : Map[String, Int] = {

    val content = Source.fromURL(url)(io.Codec("ISO-8859-1")).mkString
    val links = hrefRegex.findAllIn(content).matchData.toList.map(_.group(2).toString)

    // filter out / , # , http://url and anything else not starting with http:// or www.
    val filteredList = links.filter(
        l => l.startsWith("/") ||
        l.startsWith("#") ||
        l.startsWith(url) ||
      l.contains(url.replace(url.split("/").last, ""))
    )

    // create new mutable list and fill with visited
    var newVisited = scala.collection.mutable.ListBuffer.empty[String]
    newVisited ++= visited.filter(p => !p.equals(url))

    //add to newVisited
    filteredList.foreach(s => {
      if(!newVisited.contains(s)){
        newVisited += s
      }
    })

    // convert list to map, count occurrences
    val foldedLinks = filteredList.foldLeft(Map[String, Int]())((map, link : String) =>
      map + (link -> links.count(_.equals(link)))
    )

    newVisited = newVisited.filter(s => !s.equals(url))

    if(newVisited.isEmpty){
      // return merged map
      mergeMap(List(urls, foldedLinks))((v1, v2) => v1 + v2)
    }else{
      getLinks(newVisited(0),mergeMap(List(urls, foldedLinks))((v1, v2) => v1 + v2), newVisited.toList)
    }
  }

}

object MyApp extends App{

  val urls = List(
    "http://servers.rickvanlieshout.com/scalatest"
  )

  def getSiteMap() = {

    var siteMap : Map[String, Int] = Map[String, Int]()

    for (url <- urls) {
      siteMap = PageLinkLoader.getLinks(url, siteMap)
      siteMap.foreach (x => println (x._1 + "-->" + x._2))
      createSiteMap(url, siteMap)
    }
  }

  // creates a sitemap, uses Java which is wicked.
  def createSiteMap(url: String, urls : Map[String, Int]) = {
    // start off with a priority of 1.0 (ranges from 1.0 to 0)
    var currentPriority = 1.0

    //sort descending
    val descUrls = ListMap(urls.toSeq.sortWith(_._2 > _._2):_*)

    // write to a file called "sitemap_for_domain.txt"
    val domainRegex = """^(?:https?:\/\/)?(?:[^@\/\n]+@)?(?:www\.)?([^:\/\n]+)""".r
    val pw = new PrintWriter(new File("sitemap_for_"+domainRegex.findAllIn(url).matchData.toList.map(_.group(1)).mkString+".txt" ))
    //write xml header
    pw.write("\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<urlset\n      xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\"\n      xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\n      xsi:schemaLocation=\"http://www.sitemaps.org/schemas/sitemap/0.9\n            http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd\">")

    // for each url write an url block with priority, then decrease priority
    descUrls.foreach(s => {
        pw.write(
          "\n<url>\n  " +
            "<loc>" + s._1 + "</loc>\n" +
            "<priority>" + currentPriority + "</priority>" +
            "</url>"
        )
        // decrease priority if possible
        if(currentPriority >= 0.1){
          currentPriority -= 0.1
        }
      }
    )

    // write the closing statement for a sitemap
    pw.write("\n</urlset>")
    pw.close()
  }

  getSiteMap()

}

Scala Day 3 - Concurrency is key!

Tue, 04 Apr 2017 00:00:00 GMT

Day 3 promises to go into “hard things” like XML and concurrency. So far I’ve been liking Scala and if it can really simplify concurrency then it might earn a permanent slot in my toolkit.

XML

We can express XML just as easily as we do a string, the example below shows how to express some XML and how to access its inner text. Note, however, that you need to add the “scala-xml” jar file to your build solution as it no longer comes included with Scala.

scala> val movies =
     | <movies>
     | <movie genre="scifi">Star Trek</movie>
     | <movie genre="fairytale">Star Wars</movie>
     | </movies>
movies: scala.xml.Elem =
<movies>
<movie genre="scifi">Star Trek</movie>
<movie genre="fairytale">Star Wars</movie>
</movies>

scala> movies.text
res0: String =
"
Star Trek
Star Wars
"

We can use the ” \ ” operator to search for specific entries, which we can then access just as with a list:

scala> val movieNodes = movies \ "movie"
movieNodes: scala.xml.NodeSeq = NodeSeq(<movie genre="scifi">Star Trek</movie>, <movie genre="fairytale">Star Wars</movie>)

scala> movieNodes(0)
res1: scala.xml.Node = <movie genre="scifi">Star Trek</movie>

To check the “genre” attribute of the first item in movieNodes we run:

scala> movieNodes(1)
res3: scala.xml.Node = <movie genre="fairytale">Star Wars</movie>

scala> movieNodes(1) \ "@genre"
res4: scala.xml.NodeSeq = fairytale

As we can see it returned the correct genre, fairytale, for the movie “Star Wars”. So far Scala delivers on its promise to make “hard things” like XML easy. Let’s see what else we can do.

Pattern matching, guards and Regex

Scala will use pattern matching often, such as when you parse XML or pass messages between threads. The simplest form of pattern matching can be achieved with:

def doChore(chore: String): String = chore match {
  case "clean dishes" => "scrub, dry"
  case "cook dinner" => "chop, sizzle"
  case _ => "whine, complain"
}
println(doChore("clean dishes" ))
println(doChore("mow lawn" ))

We define two chores and an alternative. We then call the method with a valid and invalid parameter. This will match the valid one and return “scrub, dry”, the second won’t match and will return “whine, complain”.

Scala also supports guards in the case statement. Using this to calculate factorials, for instance, we can filter out weird numbers (like 0 and negatives) while matching all the other positive numbers to x, and thus return it’s factorial.

def factorial(n: Int): Int = n match {
  case 0 => 1
  case y if n < 0 => 0
  case x if x > 0 => factorial(n - 1) * n
}
  println(factorial(-2))
  println(factorial(0))
  println(factorial(5))

We can also use regular expressions to do our matching:

scala> val reg = """^(F|f)\w*""".r
reg: scala.util.matching.Regex = ^(F|f)\w*

scala> println(reg.findFirstIn("Fantastic"))
Some(Fantastic)

scala> println(reg.findFirstIn("not Fantastic"))
None

We can even combine XML and the matching features like so:

(movies \ "_" ).foreach { movie =>
  movie match {
    case <movie>{movieName}</movie> =>
      println(movieName)
    case <short>{shortName}</short> =>
      println(shortName + " (short)" )
  }
}

val movies = <movies>
   <movie>The Incredibles</movie>
   <movie>WALL E</movie>
   <short>Jack Jack Attack</short>
   <short>Geri's Game</short>
</movies>

This will go through every item in the tree (_) and match it to either movie or short. If it’s a movie it will just print its text, otherwise, it’ll print its text + (short).

In order to get this to work however I had to download the scala.xml library, as it no longer comes included with Scala since Scala 2.11.

Concurrency

Finally! We get to do something with concurrency, now Scala can really show me what it’s worth!

The book starts off by telling me about Actors, which have pools of threads and queues, and message passing. When you send a message (using the ! operator) you place an object on its queue. The actor then reads the message and takes action. Usually, the actor uses a pattern matcher to detect what it has to do before it starts doing something.

The book provides a sample application but that plain doesn’t work. Upon investigating the issue I discovered that Scala’s built-in concurrency feature is deprecated in favour of Akka. This saddens me beyond belief. I have already worked with Akka and had expected Scala to offer me something else. This also means that the rest of the book is not going to be useful anymore.

Stubborn as I am, even with a closing deadline, I decided I wasn’t going to let a stupid book stop me and decided to update the code to work with Akka. All of which can be found at Github.

This disaster can be summed up by a fantastic quote though (and I love quotes):

All that is necessary is to accept the impossible, do without the indispensable, and bear the intolerable.

- Norris, Kathleen, As quoted in: The Litchfield Co-operator, Issues 116-163 (1945)

Self-study

The theoretical question for the day is: For the sizer program, what would happen if you did not create a new actor for each link you wanted to follow? What would happen to the performance of the application?

And the answer is: it would be just as slow as the sequential one.

The practical assignment is: Take the sizer application and add a message to count the number of links on the page.

I did this by creating another Loader and calling that in my UrlActor. The code can, once again, be found on Github or in the code block below.

object PageLinkLoader {

  def getLinks(url: String) : Int = {
    val content = Source.fromURL(url)(io.Codec("ISO-8859-1")).mkString
    val links = hrefRegex.findAllIn(content).matchData.toList.map(_.group(2))

    links.size
  }

  //I stole this regex from somewhere for a previous project, don't remember where
  val hrefRegex = """<a\s+(?:[^>]*?\s+)?href=(["'])(.*?)\1""".r

}

So that’s it, right? Wrong! My school assignment requires me to do one more assignment. What assignment is that? Well you’ll just have to wait on the next blog to find out! Laters!

Scala Day 2 - Let's get functional

Mon, 03 Apr 2017 00:00:00 GMT

I feel like I covered a lot in the first chapter but that I haven’t yet found the real power of functional programming. Day 2 promises to change that by diving right into functional programming.

Val vs Var

I’ve already looked at this on day 1, looking back at the chapter I realize I wasn’t yet meant to find out about the difference between “val vs var”. In short, val is immutable while var is mutable.

The remainder of the chapter emphasizes the importance of using val, especially while designing an application with concurrency in mind.

This basic design philosophy is the key element that differentiates functional programming from object-oriented programming: mutable state limits concurrency.

- Tate, Bruce, Seven Languages in seven weeks (p 155)

Collections

In the first blog I already pointed out some of the features I had discovered about collections. I used some of these awesome features in my day 1 self-study.

The second chapter goes into more detail about collections. First up are lists.

Lists

Scala’s lists, of type List, are ordered collections of elements with random access. One of the things the book immediately goes into is the ability to store different types of objects in the list. Take a look at the interpreters response when I try to create a list of two strings and an integer:

scala> List("one", "two", 3)
res6: List[Any] = List(one, two, 3)

This returns a list with data type “Any”, which is the catchall data type for Scala. To access an item we use the ”()” operator. Think of it as the list being a method on which you “call” a get function. Doing so will return an “Any” object, which to me is weird because it’s type could be inferred as well.

scala> List("one", "two", 3)(2)
res7: Any = 3

Trying to access a number outside the list will throw either a “NoSuchElement” exception (if the index specified is too high) or a regular old “IndexOutOfBoundsException”. (if the index is below 0)

Sets

A set is like a list, but it doesn’t have any explicit order. We can specify a set with the Set keyword like so:

scala> val animals = Set("lions", "tigers", "bears")
animals: scala.collection.immutable.Set[java.lang.String] = Set(lions, tigers, bears)

Adding or subtracting from it is as easy as using the + or - operator:

scala> animals + "armadillos"
res0: scala.collection.immutable.Set[String] = Set(lions, tigers, bears, armadillos)

scala> res0 - "tigers"
res2: scala.collection.immutable.Set[String] = Set(lions, bears, armadillos).

Combining sets can be done with the ”++” operator (or list.union) and the ”—” operator (or list.diff) will return a list of differences between the two lists:

scala> animals ++ Set("armadillos", "raccoons")
res3: scala.collection.immutable.Set[String] = Set(bears, tigers, lions, armadillos, raccoons)

scala> animals -- Set("lions", "bears")
res4: scala.collection.immutable.Set[String] = Set(tigers)

Set intersection can be done with “List.intersect” like so:

animals.intersect(Set("armadillos","raccoons","lions","tigers"))
res8: scala.collection.immutable.Set[String] = Set(lions, tigers)

As I’ve mentioned before, Sets, unlike lists, are independent of order. This rule will mean that equality for sets is different. Therefor evaluating two sets with the same elements in a different order will return true:

scala> Set(1, 2, 3) == Set(3, 2, 1)
res9: Boolean = true

Maps

A map is a key-value pair, similar to hashmaps in Java. Specifying a map of ordinal numbers can be done with:

val ordinals = Map(0 -> "zero", 1 -> "one", 2 -> "two")
ordinals: scala.collection.immutable.Map[Int,String] = Map(0 -> zero, 1 -> one, 2 -> two)

We can also specify the type for a HashMap like so:

import scala.collection.mutable.HashMap
val map = new HashMap[Int, String]

We can then add items like so:

map += 4 -> "four"
map += 8 -> "eight"

List methods

Scala supports anonymous functions, I’ve used this to solve some problems in the Tic-Tac-Toe game from the first blog.

Apart from the “foreach” method, there are some other useful list methods:

Function	What it does
List.head	Returns the first element
List.tail	Returns all but the first element
List.last	Returns the last element
list.init	Returns all but the last element (recursion)
List.reverse	Reverses the list
List.drop(2)	Removes the first two objects.

Count, map filter and Others

Scala has many other functions that manipulate lists in various ways. I won’t go over these but the code examples below should give you an idea of the possibilities.

scala> val words = List("peg", "al", "bud", "kelly")
words: List[String] = List(peg, al, bud, kelly)

scala> words.count(word => word.size > 2)
res0: Int = 3

scala> words.filter(word => word.size > 2)
res1: List[String] = List(peg, bud, kelly)

scala> words.map(word => word.size)
res2: List[Int] = List(3, 2, 3, 5)

scala> words.forall(word => word.size > 1)
res3: Boolean = true

scala> words.exists(word => word.size > 5)
res4: Boolean = false

We can also parameterise our anonymous functions so that we can sort a list based on the first letter of each word. We do that like this:

words.sortWith((s, t) => s.charAt(0).toLower < t.charAt(0).toLower)

We can also sort by word size:

words.sort((s, t) => s.size undefined

foldLeft

The foldLeft method takes an initial value and a code block. It will take the initial value and for each item in the list run the code block passing back in the calculated value. Let me show you:

val list = List(1, 2, 3)
val sum = (0 /: list) {(sum, i) => sum + i}

Initially, /: takes the initial value, 0 , and the first element of list , 1 ,and passes them into the code block. sum is 0 , i is 1 , and the result of 0 + 1 is 1 .
Next, /: takes 1 , the result returned from the code block, and folds it back into the calculation as sum . So, sum is 1 ; i is the next element of list , or 2 ; and the result of the code block is 3 .
Finally, /: takes 3, the result returned from the code block, and folds it back into the calculation as sum . So, sum is 3 ; i is the next element of list , or 3 ; and sum + i is 6 .

We can achieve the same thing by using the foldLeft method like so:

list.foldLeft(0)((sum, value) => sum + value)

Self-Study

I’m starting to like Scala more and more, the practical challenge for this week was really fun albeit a bit simple. The theoretical questions for day 2 are:

A discussion on how to use Scala files
What makes a closure different from a code block

Once again these questions are trivial if you’ve actually paid attention while reading the book. Anyway, here are the answers:

The results of the second day’s practical assignments can found on Github or below:

/**
   Self-Study day 2, practical assignments:

   • Use foldLeft to compute the total size of a list of strings.

   • Write a Censor trait with a method that will replace the curse words
     Shoot and Darn with Pucky and Beans alternatives. Use a map to
     store the curse words and their alternatives.

   • Load the curse words and alternatives from a file.

*/

trait Censor {

  val curseWords = Map("shoot" -> "pucky", "darn" -> "beans")

  /**
    * Awesome function which will build a curseWords map from a file.
    * @param input string input
    * @param path curseWords file
    * @return clean string
    */
  def censorTextUsingFile(input: String, path: String) : String = {
    // neat little statement which will read a file from a path, get its lines
    // for each line it will split on "=" and put the values in the map
    val myMap = io.Source.fromResource(path).getLines.foldLeft(Map[String, String]())((map, line) =>
      map + (line.split("=")(0) -> line.split("=")(1))
    )

    censorText(input, myMap)
  }

  /**
    * censors input text using the builtin curseWords map or @param map
    * @param input string to be censored
    * @param map optional, map of curse words
    * @return censored string
    */
  def censorText(input: String, map: Map[String, String] = curseWords) : String ={
    val words = input.split(" ")

    words.map(word => map.getOrElse(word.toLowerCase, word)).mkString(" ")

  }
}

object MyApp extends App with Censor{

  // print combined size of a list of strings
  val list = List("one", "two" , "three")
  println(list.foldLeft(0)((sum, value) => sum + value.length))

  val strToCensor = "Shoot !, this darn thing won't work!"

  // censor using builtin map
  println(this.censorText(strToCensor))

  // censor using curseWords.txt file from resources folder
  println(this.censorTextUsingFile(strToCensor, "curseWords.txt"))

}

Scala Day 1 - The Basics

Sun, 02 Apr 2017 00:00:00 GMT

So day 1 has arrived, I’ve opened the book and I’ve started looking at the assignments in front of me. The book starts off by telling us how to install Scala so that is what I’ll start with today.

Installing Scala

Being on Linux the first thing I did was use my package manager of choice (Pacman) to check whether there was a Scala package in my repositories. Sure enough, it found a package (20MB in size) and proceeded to download and install said package.

While Pacman started collecting all the little dots, I took a quick look at the Scala download page. On there I am greeted with a friendly message telling me the binary for my system is ready to download. Right below the download button is a link on how to install Scala. Curiosity got to me and I decided to open the link. The installation steps briefly explain that there are 2 binaries in the compressed folder which are of importance: “scala” and “scalac”. To anyone coming from Java, these should be fairly familiar seeing as Java uses “java” and “javac” for their interpreter and compiler respectively. (note: the installation steps assume you know how to add PATH variables).

The next message on this page tells me about a popular build tool called sbt so I went ahead and installed that as well.

The final step on this page recommends me to install ”The Scala IDE” or use the IntelliJ plugin. Seeing as I adore the JetBrains product line I’ve opted to choose the latter.

While browsing the downloads page I noticed Pacman had beat his level. (a.k.a collected all the dots. a.k.a the install has finished) Which means it’s high time for me to continue on with the assignments.

Scala types

The next step in the book guides me through using the interpreter to execute basic bits of code. These range from a simple hello world to entering some arithmetics. In the code block below you can see both my input and the interpreter’s output.

scala> println("hello, surreal world")
hello, surreal world

scala> 1 + 1
res1: Int = 2

scala> 5 + 4 * 3
res2: Int = 17

scala> 5. + (4.*(3))
res3: Int = 17

Two things struck me as odd about these results. First up the word “Int” is written with a capital I. This is weird to me because in Java “int” is a base type, not a class (it uses the Integer wrapper class). Next up is the “ResX” line, does that just mean result or is there something more to find out about it? After a quick Google search I found out the “res” is an immutable variable which we can use like so:

res0: Int = 10

scala> res0 + 10
res1: Int = 20

Type inference, arrow notation, “var vs val” and GitHub

At this point the code is getting slightly more complicated and typing it all directly into the interpreter would be a fool’s effort. Therefore I’ve switched to using IntelliJ with the scala plugin and .scala files. All the files can be found on GitHub.

After fiddling a bit with the examples I’ve found a few things I already like in Scala:

Type inference. To me, not having to specify the type of a variable is strangely satisfying.
Anonymous function syntax support (a.k.a arrow notation).
First class ranges
1 line methods without braces
companion objects (extending.scala on Github)
No semicolons! (although I sometimes use them by accident)

Inheritance is weird

Inheritance is one of the many joys from the Object-Oriented programming paradigm and I’m glad to read Scala supports inheritance too. The way Scala does it, however, is rather weird in my opinion. Take a look at the following code:

class Person(val name: String) {
  def talk(message: String) = println(name + " says " + message)
  def id(): String = name
}
class Employee(override val name: String, val number: String) extends Person(name) {

  override def talk(message: String) {
    println(name + " with number " + number + " says: " + message)
  }

  override def id():String = number.toString

}

As you can see we have to specify the parameter list of the class we want to extend. Also, note that we don’t have to specify the type of said parameters. Another peculiarity is the “override” keyword. Apparently, it is mandatory to prevent the creation of new methods due to mis-spelling … lame.

Traits

Traits, in short, resemble a Java interface combined with an implementation.

We will start by defining a trait “Bad” for the Person class we’ve defined earlier:

trait Bad{
  def curse() = println("Object oriented languages are cool too!");
}

Next, we’ll override the person class with a new class and add the “Bad” trait:

class BadPerson(override val name:String) extends Person(name) with Bad

Now if we create a new object of BadPerson we can call the “curse” method.

val badPerson = new BadPerson("Mr. bad")
badPerson.curse

We can also add multiple traits to a class by repeating “with TRAIT” like so:

class IndecisivePerson(override val name: String) extends Person(name) with Nice with Bad

Self-Study

At the end of every day, the book ends with a few assignments, some are theoretical and some are practical.

Day 1’s theoretical questions are:

Find the Scala API
Find a comparison of Java and Scala
Find a discussion of “val” vs “var”

My answers to these are:

Finally, we get to move on to the practical assignment for the week. For the assignment, I have to create a “Tic-tac-toe” game for 2 players.

Writing this game has been really interesting. I tried using as few variables as possible and I tried to use Scala-specific features. One of these features I have already fallen in love with: collections. Scala collections have methods similar to LINQ in c# and they are extremely useful. Examples of these collection methods can be found in the “boardHasWinner” method in the code below.

The complete code for this game, including the bonus problem, can also be found at Github.

object TicTacToe {

  var currentPlayer = 'X';
  val playerOne = 'X'
  val playerTwo = 'O'


  val board = Array(
    Array('_', '_', '_'),
    Array('_', '_', '_'),
    Array('_', '_', '_')
  )

  /**
    * Flips current player to the other player.
    */
  def flipPlayers = {
    if(currentPlayer.equals(playerOne)){
      currentPlayer = playerTwo
    }else{
      currentPlayer = playerOne
    }
  }

  /**
    * read user input and play a move
    * @param error set to true if user made a mistake and has to re-enter his position
    */
  def readLine(error : Boolean = false) : Unit = {
    if(error){
      System.err.println("Please enter a value ON the grid which has no mark yet.")
    }

    println("Please enter the x position you want to place your mark at")
    val x = scala.io.StdIn.readLine().toInt

    println("Please enter the y position you want to place your mark at")
    val y = scala.io.StdIn.readLine().toInt

    if((x > 0 && x < 4) && (y > 0 && y < 4) ){
      play(x,y)
    }else{
      readLine(true)
    }
  }

  /**
    * checks whether pos(x,y) is a valid position
    * if it's valid it will put currentPlayer on that position
    * if it isn't it will ask the user for new input using the readLine method
    * @param x coordinate
    * @param y coordinate
    */
  def play(x : Int, y : Int) = {
    if(board(x - 1)(y - 1).equals('_')){
      board(x - 1)(y - 1) = currentPlayer
      flipPlayers
    }else{
     readLine(true)
    }
  }

  /**
    * Checks whether the board is full
    * @return true if board is full, else false
    */
  def boardIsFull : Boolean = {
    board.foreach { row =>
        // if any row contains an underscore we're obv not full yet so we can return false;
        if(row contains '_'){
          return false;
        }

    }
    return true;
  }

  /**
    * main game loop
    */
  def run = {
    while(!boardIsFull && !boardHasWinner){
      printBoard
      readLine()
      if(boardHasWinner){
        flipPlayers // reverse last player flip
        println("Congrats " + currentPlayer + " you have won this game of Tic-Tac-Toe")
      }
    }
    if(!boardHasWinner){
      println("Sorry, it's a draw!")
    }
    // print the board once more so the players can see the final score
    printBoard
    // exit execution
    System.exit(0)
  }

  /**
    * Checks whether the board has a winner
    * @return true if board has a winner else false.
    */
  def boardHasWinner : Boolean = {

    //check horizontals
    board.foreach {row =>
      if(row(0) != '_' && row.forall(c => c.equals(row(0)))){
        return true
      }
    }

    // check diagonals
    val topLeftBottomRight = Array(board(0)(0), board(1)(1), board(2)(2))
    val bottomLeftTopRight = Array(board(0)(2), board(1)(1), board(2)(0))

    if(topLeftBottomRight(0) != '_' && topLeftBottomRight.forall(c => c.equals(topLeftBottomRight(0)))){
      return true
    }
    if(bottomLeftTopRight(0) != '_' && bottomLeftTopRight.forall(c => c.equals(bottomLeftTopRight(0)))){
      return true
    }
    // if no winner has been found, return false.
    return false
  }

  /**
    * Prints a visual representation of the board and who's turn it is
    */
  def printBoard = {
    board.foreach { row => println("" + row(0) + " | "
      + row(1) + " | "
      + row(2) )}

    println(currentPlayer + " is playing")
  }

}

// let's play!
TicTacToe.run

Scala Day 0 - What is Scala?

Thu, 30 Mar 2017 00:00:00 GMT

One of the final assignments for a school course called “APP” (Algorithms Programming language & Paradigms) was/is to learn a new programming language using the book ”Seven languages in Seven weeks“.

The idea of this book is that you can get comfortable with a programming language in just a week. As you can probably guess from the title the book covers 7 languages in total and I will be covering at least 1 (Scala).

What is Scala

Scala is a hybrid language that tries to bridge the gap between object-oriented languages like Java and functional languages like Haskell.

Why did I pick Scala for the assignment?

I have never really liked functional languages (except Javascript) much. It’s a hard paradigm for me to wrap my head around in most languages and it gets tiring quickly. I thought having an assignment (for which I will be graded) to focus on might help me stay on the right path.

So… if you wanted to go functional why not go Haskell you ask? Well, purely functional programming is something which just doesn’t “speak” to me, combining it with an OO approach might just be the thing that makes the gears click. At the very least it should provide a gradual learning curve to apply functional programming ideas to my code.

Besides my personal reasons, there are some objective reasons why Scala is “better” than Haskell:

Scala is much more popular than Haskell, as such chances of encountering it “in the wild” are severely increased.
Scala painlessly integrates with pre-existing java libraries.
Scala uses the Java Virtual Machine (JVM), this has numerous benefits.
Like Java, Scala is statically typed and the syntax somewhat resembles that of Java too.

Well that’s it for today folks, check back in for the next blog within a few days!

My extended leave, status updates and other stuff

Fri, 02 Sep 2016 00:00:00 GMT

So….. last blog post was in December of last year. Does that mean the blog is dead? Well I’m happy to report that is NOT what it means! In this blog post I’ll address why I’ve been on an extended leave and I have some exciting news to boot!

Good news always comes first!

Before I start on my rant about being away for so long I’d better tell you all some exciting news. And boy do I have quite a bit of it!

First of all, if you hadn’t noticed yet, the personal website has gotten a facelift! I have added a new “Software” page, this page will contain software I made without a contract and open source projects I’ve worked on. Furthermore, I have updated the CV with some new job information and a couple of snazzy logos. The Portfolio page got trimmed down a bit (moved some stuff to Software) and the contact page has gotten a nice new look with an actual form instead of just social buttons. Lastly, I added an “Other sites” drop-down menu. This menu will house links to various other websites of which I am the owner / co-owner. As some of those keen-eyed people among you will have noticed I also upgraded my “Disclaimer” page and added a new “RSS feed” to the footer of the site.

All in all, quite a few changes which will allow me to further separate Mi-soft.nl from my personal website. (update to Mi-soft.nl expected somewhere later this year hehe)

More good news!?

Oh yes! I have recently started working on Three, yes Three, new projects. I will probably blog about all of these projects but in case I don’t here’s a little overview:

First up is a new Python (Flask) REST API which makes getting Linux OS information a breeze! I started this project because I wanted to recreate my old hardware info system (written in Java) and generally improve it overall. I started out working on this by myself but soon joined up with my old friend Janco (more on him later). We haven’t done all too much about it but in its current state it can already serve most, if not all, CPU and Memory information. Though the best feature of this new API is arguably its name. I named it after the person I have cherished for over six years now, Sammy.

I also finally bit the bullet regarding “Linfinity”. For a while now I’ve been wanting to create a “software install script generator” for Linux which I can use to install Linux software easily on a multitude of systems. Linfinity will be a web tool which will be able to generate software install scripts for various different distributions (aiming for at least: Arch, Ubuntu and Raspbian).

Lastly, I have some “maker” news. A friend and I have decided to try our hand at creating a Raspberry Pi workplace for the masses. We found an absolutely perfect display to use in combination with the Raspberry pi and have devised a way to mount it and the Rasberry Pi in a single chassis. Cables will be mostly routed within the unit and it all can be powered by a single standard 2.1mm adapter! More news about this soon!

Moving away from Digitalocean

As of today I no longer have any Digitalocean droplets. Not because they offer a bad service but because they are too expensive for what I was using them for. The only real purpose my droplet still had was handling go8.nl and it’s traffic. I have since moved go8.nl over to Sohosted purely because it is a lot cheaper :).

The rekindled friendship?

As some of you might know Janco and I completely split up about a year and a half ago after a rather lengthy fight between us. In this year and a half apart we both calmed down a bit and after a good talk we’ve decided to pick things up again. So far everything is going well and we are working on keeping it that way. Even though programming together will certainly add to both of our stress levels , and probably lead to a couple of smaller fights, we have decided to team up to create Sammy, my new REST API for extracting Linux OS information.

One of my other good friends (Jelle) also happened to come by some extra free time these past months and we’ve been using that time to explore the magical world of Tamriel (ESO online) and kick some butt (Awesomenauts) together.

So all these things combined have been taking up most of my free time and motivation. Which leaves none for writing blogs unfortunately. I will try to improve upon this though.

Well that’s it for me folks, Have a great day and may happiness surround you :)

TekTree part 4 - programming

Thu, 24 Dec 2015 23:00:00 GMT

In the last installment of the series I show you how to program the arduino to work with the TekTree we built. I would advise each and every one of you to download my sketch and try it but after that start experimenting with your own designs!

Testing

Before we start writing complicated code we first need to design a very simple test sketch for the leds. The script I used can be found below or at Github under the following directory: Testing -> LedTest

The code is quite simple. It sets all led’s as output and runs through them. back and forth

 /*
   Rick van Lieshout
   24-12-2015
   LICENSE: MIT
 */

 const int DELAYTIME = 100;
 const int NUMBEROFPINS = 11;
 const int pinArray[NUMBEROFPINS] = {0,1,2,3,4,5,6,7,8,9,10};

 void setup() {
   //initiate pins
   for (int thisPin = 0; thisPin < NUMBEROFPINS; thisPin++) {
     pinMode(pinArray[thisPin], OUTPUT);
   }
 }

 void loop() {
   // loop from the lowest pin to the highest:
   for (int thisPin = 0; thisPin < NUMBEROFPINS; thisPin++) {
    // turn the led on:
    digitalWrite(pinArray[thisPin], HIGH);
    // wait for a bit
    delay(DELAYTIME);
    // turn the led off:
    digitalWrite(pinArray[thisPin], LOW);
   }
   // loop from the highest pin to the lowest:
   for (int thisPin = NUMBEROFPINS - 1; thisPin >= 0; thisPin--) {
    // turn the led on:
    digitalWrite(pinArray[thisPin], HIGH);
    // wait for a bit
    delay(DELAYTIME);
    // turn the led off:
    digitalWrite(pinArray[thisPin], LOW);
   }
 }

Assuming all went well you should now see the led’s come on 1 by 1 and create a little wave effect (first pattern down!)

Now that we know it all works let’s clean up the code and add some pattern handling.

The finished code / result

The complete code can once again be found on github or directly below the video. That’s it for now. Merry christmas and a happy new year folks!

The code that makes it work

/*
  Rick van Lieshout
  24-12-2015
  LICENSE: MIT
*/

/** Constant value's, these have to be changed manually  **/
const int MAXSEQUENCES= 20;     // number of steps in the largest sequence. Could make it 999 but that would affect performance
const int NUMBEROFPINS = 11;    // number of leds
const int NUMBEROFPATTERNS = 5; // number of patterns specified
const int DELAYPOT = 0;         // number of pin that pot is connected to
const int BUTTONPIN = 12;       // number of pin that button is connected to
const int pinArray[NUMBEROFPINS] = {0,1,2,3,4,5,6,7,8,9,10}; // pins where leds are connected

/** Regular variables, program can alter these **/
int previousbuttonstate = 0;  // used to detect whether button is pressed
int buttonstate = 0;          // used to detect whether button is pressed
int pattern = 0;              // pattern to start with

/** massive array which holds the patterns **/
const byte LEDsequence [NUMBEROFPATTERNS][MAXSEQUENCES][NUMBEROFPINS] = {
  {
  {0,0,0,0,0,0,0,0,0,0,0},
  },
  {
  {1,1,1,1,1,1,1,1,1,1,1},
  },
  {
  {1,0,0,0,0,0,0,0,0,0,1},
  {0,1,0,0,0,0,0,0,0,0,1},
  {0,0,1,0,0,0,0,0,0,0,1},
  {0,0,0,1,0,0,0,0,0,0,1},
  {0,0,0,0,1,0,0,0,0,0,1},
  {0,0,0,0,0,1,0,0,0,0,1},
  {0,0,0,0,0,0,1,0,0,0,1},
  {0,0,0,0,0,0,0,1,0,0,1},
  {0,0,0,0,0,0,0,0,1,0,1},
  {0,0,0,0,0,0,0,0,0,1,1}
  },
  {
  {1,0,0,0,0,0,0,0,0,0,1},
  {0,1,0,0,0,0,0,0,0,0,1},
  {0,0,1,0,0,0,0,0,0,0,1},
  {0,0,0,1,0,0,0,0,0,0,1},
  {0,0,0,0,1,0,0,0,0,0,1},
  {0,0,0,0,0,1,0,0,0,0,1},
  {0,0,0,0,0,0,1,0,0,0,1},
  {0,0,0,0,0,0,0,1,0,0,1},
  {0,0,0,0,0,0,0,0,1,0,1},
  {0,0,0,0,0,0,0,0,0,1,1},
  {0,0,0,0,0,0,0,0,1,0,1},
  {0,0,0,0,0,0,0,1,0,0,1},
  {0,0,0,0,0,0,1,0,0,0,1},
  {0,0,0,0,0,1,0,0,0,0,1},
  {0,0,0,0,1,0,0,0,0,0,1},
  {0,0,0,1,0,0,0,0,0,0,1},
  {0,0,1,0,0,0,0,0,0,0,1},
  {0,1,0,0,0,0,0,0,0,0,1}
  },
  {
  {1,0,0,0,0,0,0,0,0,1,1},
  {0,1,0,0,0,0,0,0,1,0,1},
  {0,0,1,0,0,0,0,1,0,0,1},
  {0,0,0,1,0,0,1,0,0,0,1},
  {0,0,0,0,1,1,0,0,0,0,1},
  {0,0,0,1,0,0,1,0,0,0,1},
  {0,0,1,0,0,0,0,1,0,0,1},
  {0,1,0,0,0,0,0,0,1,0,1},
  {1,0,0,0,0,0,0,0,0,1,1}
  }
};

/** Setup only gets run once **/
void setup(){
  //set all pins as input/output
  initializePins();
}

/** Loop is constantly running **/
void loop() {
  // read button state
  buttonstate = digitalRead(BUTTONPIN);
  // if button state changed
  if(buttonstate != previousbuttonstate){
    previousbuttonstate = buttonstate;
     // check value and add to pattern
     if(buttonstate == HIGH){
      if(pattern +1 == NUMBEROFPATTERNS){
        pattern = 0;
      }else{
        pattern++;
      }
     }
  }

  // Loop through all the arrays and toggle pins on or off.
  for (int sequence=0;sequence<getSequences(pattern);sequence++) {
    for (int pin=0;pin<NUMBEROFPINS;pin++) {
      digitalWrite(pinArray[pin], LEDsequence[pattern][sequence][pin]);
    }
    // delay between sequences, should use millis if you want instant response on the button
    delay(getDelay(DELAYPOT));
  }
}

/** Count the number of sequences in a specific pattern **/
int getSequences(int pattern){
  byte amount = 0;
  for(byte i = 0; i<MAXSEQUENCES; i++){
    int total = 0;
    for(int pin=0; pin<NUMBEROFPINS; pin++){
      total+=LEDsequence[pattern][i][pin];
    }
    if(total>0){
      amount++;
    }
  }
  return amount;
}

/** get value of pot **/
int getDelay(int potnr){
  return analogRead(potnr);
}

/** Set all pins as either output or input **/
void initializePins(){
  pinMode(BUTTONPIN, INPUT);
  for (int pin=0;pin<NUMBEROFPINS;pin++) {
    pinMode(pinArray[pin], OUTPUT);     // we make all the declarations at once
  }
}

TekTree part 3 - Wiring

Thu, 24 Dec 2015 00:00:00 GMT

In the 3rd part of my TekTree project I’ll show you guys how to wire everything. This time I even used separate colours for the power, ground and signal wires. So no more complaining! ^_^ All in all I think the result looks absolutely wicked. Even without the Arduino case which has yet to come in the mail. (and probably won’t fit)

Wiring part 1 - schematics

Before I started to jam wires in every nook and cranny I could find I had to come up with a plan. Most of that plan has been put into a “Fritzing” sketch which can be found below and on Github. The only thing I couldn’t figure out how to model on fritzing is the actual Christmas tree and the power cord for the Arduino.

The Arduino will get power from a regular adaptor. The only modification I made to the adapter was putting a button on one of the wires. The only reason I did that is because I’m too lazy to unplug the TekTree when I want to turn it off :)

Why so many wires?

I want the project to be “manageable” for everyone. To do this I kept the wiring as simple as possible. Every LED has 2 prongs, 1 goes to ground 1 goes to the Arduino. Simple. Same goes for the potentiometers, 3 legs, 1 power 1 ground 1 signal. The only leds I did bundle together (in terms of wiring) is the star. I did this to create a single star instead of multiple led’s.

Well you could’ve made it way prettier… and

Well yes, as all of us know I’m perfectly capable of designing a refined build. I didn’t want to go with a pcb this time however because that would make the barrier to entry way higher on this project. And that is something I definitely don’t want.

I also didn’t “group” all my ground wires together. (led -> led -> led -> board) This would’ve made the project a lot neater but it wouldn’t be anywhere near as service friendly. (which was kinda the idea…)

Well then, I feel like I prepared you guys enough to show the (complicated looking, but very simple) wiring. Check it out below!

Wiring part 2 - Actual wiring

Below you’ll find the pictures I took while I was wiring the TekTree.

TekTree part 2 - Lasercutting

Sat, 19 Dec 2015 23:00:00 GMT

Last friday I actually got to cut the parts for my TekTree! So excited! This is by far the biggest “piece” I have ever laser cut but luckily everything went well. This in part is thanks to my brilliant friend Ron who helped me create the sketches for the laser cutter. Thanks again Ron!

Top tip! You don’t have to laser cut the designs. Just print the designs with a regular printer and stick it on some wood. Then just use something like a bandsaw to cut the pieces!

Laser cutting video

I’ve gotten so much positive feedback on my last laser cutting video that I decided to make another one. This one is a little different though. It’s a compilation of a few different cuts.

TekTree assembly

Below you’ll find a series of pictures I took while assembling the TekTree.

the baseplate

bottom of the assembly

bottom with box closed

the tree inserted into the base

the arduino and breadboard will just be hanging in the back

TekTree part 1 - TekTree - a christmassy surprise project

Sat, 19 Dec 2015 00:00:00 GMT

I’m very excited to announce a new maker project to you guys ! It will be simple, and thus reproducible, yet fun and quite possibly even useful. (oh, and a tad crazy)

So? What is it?

Well… basically I’m going to assemble and program a little christmas tree thing.

I got the idea after my parents put up their christmas tree. My mom opted for a more “modern” (so I’ve been told) kind of tree. This “modern” tree is basically a home made christmastree-y looking thing bolted together with a few screws. (picture below) At first I didn’t like the idea of a more modern, less authentic, christmas tree but once it was actually up it didn’t look half bad. There were some things missing though. It just didn’t make sense to me to create something new yet be limited by old stuff. For example: my parents put a string of lights around the wooden boards, why not integrate them? Or why not add a dimmer? Or make it follow some flashy patterns? As you might expect by now I am basically going to recreate, with a bit more finesse, my parent’s christmas tree.

a picture of my parents' new christmas tree

Open source?

Of course! All code will be open source! All other stuff (drawings etc.) will also be opened up to the public. Everything can be found on Github once again.

Step 1 - Design

I’ve thought long and hard about what kind of “controller” I wanted to use. And of course I immediately wanted to use a Raspberry pi. But I asked myself, why a pi? Why not use something simpler (like an arduino) to control the lights and switches. After some serious brainstorming I decided to go with an arduino in the end. I figured the Raspberry pi zero would be too cumbersome to work with for a simple christmas tree. Besides, I have a lot more uses for a pi zero than I have for an arduino, and I got more arduino’s laying around than pi zero’s at the moment.

Digital drawing

After deciding to use an arduino I had another big decision to make. Should I follow my parents’ example? Or go a different route? In the end I decided my way was better (kids…) and so I decided to laser cut my wooden designs. One of the other big changes I made is to the base. Whereas my parents opted for something rugged I’m going for a slightly more “finished” look. This means that my base is nice and round with a few fun shapes on it and I even speculated about making the circle sit flush with the ground.(which would mean creating bendable wood! A challenge in and of itself.)

After that came my struggle with the drawing software. I tried creating a drawing in Coreldraw Pro… that was… a painful experience to say the least. Then I tried drawing with a program called InkScape, while better it still didn’t yield any results. Finally I decided to give up and ask for help from a friend.

Luckily my friend Ron Nabuurs is brilliant when it comes to designing a drawing which can be lasercut. So after explaining my idea to Ron he decided to help me out. A picture of what we ended up with can be found below. The source file we used can be found on git.

Designs for the laser cutter

So what now ?

Well… now I have to actually laser cut the pieces and assemble the tree. If we haven’t made any mistakes (or I conveniently forget to mention them to you guys) I should be able to put out another blog this weekend showing you guys both the laser cutting and a test build of the TekTree.

Making things

Sat, 17 Oct 2015 00:00:00 GMT

Even though this week has been extremely busy I wanted to bring you guys a little blog because the things I’ve been doing are truly wicked.

Special assignment

Let’s start with the boring, non maker, stuff. Around 1 pm I received a frantic phone call and a couple of messages from different people about “something”. When I finally got around to dealing with them I discovered something truly awesome.

As some of you know, next semester I will be working on a (school issued) project. This usually involves a group of 4-5 students working on a single big task for about 8 weeks. This time however it will be slightly different, me and 4 other students have been asked to work on a different project than all the other students. Turns out we have been recommended by our teachers and this company, of which we do not know the name yet, has decided to give it a go.

We do not yet know what this special assignment will entail but one thing we know for sure. WE ARE EXCITED ABOUT IT !

Maker thingies

Even though I love software development, deep down I am also still a maker. And a maker likes to “make” things (go figure…), so when I got asked to do a project involving a Raspberry Pi I couldn’t refuse. Even though I had no idea what exactly the project was going to entail I enlisted anyways.

Soon after I had my first meeting with the team and the supervisor, and before long the goal of the project became clear. We were responsible for designing a workshop involving the Raspberry Pi for kids between the age of 12 and 18 years(middle school basically). One of the things we have to do is compare the Raspberry Pi to several other educational boards. This is especially important because schools don’t tend to get anywhere near sufficient resources to even teach programming / IT related stuff, let alone spend their money on fun stuff like a Raspberry pi.

So we came up with “pong”. We are going to create simple controllers (based on potentiometers) and some kind of Raspberry Pi hat to which we can connect them. Besides pong we also speculated about creating our own Raspberry Pi image. I quickly pointed out however that maintaining an image shouldn’t be high on our priority lists and that we might have to think about a different way to solve this problem. What we eventually decided upon was an “Install” script which can be run by either the teacher or some kind of maintainer which will install all dependencies and some example sketches.

The code

Throughout the entire project we will be using a github repo to sync changes. One of the benefits of a github repo, apart from the actual git stuff, is that we automatically share all our code / designs. This means that anyone who is interested can either reproduce or follow along with us.

Enough talking, show us something

Well alright then, we’ve got a couple of things “ready” to show others. The first of which are the design schematics of our controllers and our “hubs”.

fritzing wiring diagram of the controller

We quickly decided upon changing the hub design to a more elegant “hat” (hat being a board you can attach to the Raspberry Pi, not the fedora kind). A render of one such hats can be found below.

render of the raspberry pi hat

We also started working on the controllers. The final design will be compromised of a few “boxes” between which telephone wires run to connect them together. My fantastic soldering / glueing job on the inner workings of a controller and the laser cutted wooden box can be found below.

Back of the controller

bottom of the controller

potentiometer used inside the controller

The front of the controller with the logo

the inside of an assembled controller

The left side of the controller

The right side of the controller

The top of the controller

The laser cutting process

For those interested in the laser cutting process I have also prepared a video:

The end

Well that’s all for today folks, I’ll keep you updated!

I got a new thing !!!! ~ Raspberry pi 7" official touchscreen

Fri, 25 Sep 2015 00:00:00 GMT

Hey guys, as some of you know I was really, and I mean really, excited when they announced the official raspberry pi screen. Now why is that might you ask? Everything the raspberry pi people have released so far has been really solid. The boards, the sd cards, the wifi-dongle, all of them work absolutely fantastic! So as soon as I saw that the official screen was available for pre-order in the UK I ordered myself one and started waiting.

What’s in the box?

First up is the box itself:

picture of the official Raspberry Pi 7" touchscreen display

The box resembles the original packaging from a standalone raspberry pi which is a nice little touch. The picture on the box does not however resemble the pi screen at all which I personally kinda like but it could possibly be seen as false advertisement.

As soon as I opened the box I found a little booklet which contained some health and safety regulations. I have not taken a picture of the booklet because… well it’s a booklet…

Lifting the first bit of red foam reveals yet another piece of red foam in which several electronic bits are embedded. I did take a picture this time:

neatly packed pcb and accessories

From top left to bottom right we’ve got:

4 silver stand-offs
The logic board for both touch and display
Flat ribbon cable to connect the logic board to the pi
4 cables to (optionally) connect the pi to the logic board. (more on this later)

After carefully extracting all the components I yanked out the red foam and found what I was really looking for. The official pi screen nicely protected in bubble wrap (the only way a display should be packaged).

finally! A screen.

After emptying the box I laid out all the pieces and checked whether my set was complete. (it was) In the picture below you’ll see all the contents of the original raspberry pi 7” touchscreen laying next to a raspberry pi B v2. Looking good so far !

all components inside the box

Assembly was easy and straightforward. The only thing I struggled with was the stand-offs. The stand-offs connect the board directly to the metal back of the screen, something which would fry an ordinary pc motherboard. Instead of monkeying around (like usual) I decided to seek guidance and googled a bit. Being one of the early adopters comes at a price though. There was hardly any information on the screen, let alone assembly instructions. I’m sure that by the time I upload this post there is plenty of information about this particular subject but in case you are interested. Yes. You do screw the pi directly into the back of the screen with the included standoff screws.

My case

I bought my pi at Pimoroni and I ordered the case/stand there as well. The exact case I ordered is here. The case is a laser cut layer based case. This means the individual parts of the case are actually flimsy but all bolted together it forms a nice solid block. After applying the case to my screen it looked like this:

fully assembled and ready to hold the screen!

As you can see there is plenty of room for the logic board and the flat cable is dangling freely. One fun thing to note are those plastic screws (which over tighten and break easily, hence why they included a spare screw).

Attaching the pi

The pi itself is connected to the logic board by screwing it onto the stand-offs. Quick tip: connect both ribbon cables before you connect the pi! After attaching the pi I ended up with the following result:

don't mind the wires!

The red and black cables provide power (and ground) to the pi from the logic board. This means that you can power the entire thing (pi + screen) with a single micro USB cable. Neat for those that don’t have a spare micro USB cable. You could also connect the pi with an ordinary micro USB cable to the logic board since it has a USB port. Connecting the pi separately might be required to use some high power GPIO peripherals.

Booting it up

The moment had arrived, I plugged the pi into a battery pack (creating a makeshift tablet-like device) and waited. First boot wasn’t all that promising… Touch didn’t work and I had an annoying little square in the top right. This was however not my first rodeo with a pi and I quickly worked out that the little square indicates insufficient input power. I did a quick update and after rebooting touch worked wonders! The only thing my makeshift tablet test was missing was some sort of input method for the keyboard. For this I used a program called “matchbox”, which was available in the default repository. After playing around with touch for a bit I switched to a solid 2 amp power supply and the little, coloured square disappeared, as suspected.

That’s nice and all… But we want a video

Of course I prepared a little video, wouldn’t be a showcase if I hadn’t now would it. So here goes:

In closing

So, conclusion time. I really like the pi screen, I have been using it for a while now (hence the delayed release of this blog.) and it hasn’t failed me yet. The screen is solid and bright enough for my use case and the touch works wonders. The price point is a little high (88 euro’s) but I suspect the price to go down quite a bit when it becomes available in stores in the Netherlands. That’s it for today’s blog and I’ll see you all later !

What I've been doing

Fri, 11 Sep 2015 00:00:00 GMT

My goals

Last blog we talked about a couple of goals I set for myself. As usual, not all of these have been completed. But instead of focusing on the ones I didn’t complete I’ll focus on the ones I did.

Finish separating my websites

I’m almost done with this goal. The content has been separated but the new portfolio isn’t up yet. I hope to be uploading the new portfolio at the same time as this blog but I can’t be sure of that. I can start uploading the portfolio as soon as I have finished updating a couple of “projects” under the school category. After that the portfolio will be filled with recent work and fully functional. (and quite possibly even pretty).

Clean all my servers (both soft and hardware)

I went above and beyond on this one. I even cleaned my regular pc (soft + hard) and my laptop (soft only). Cleaning my regular (dev) pc wasn’t by choice however. A lightning strike wiped out some electrical stuff and one of the things that got fried was my motherboard. While replacing my motherboard I decided to clean the entire pc since it was open already. After installing a new motherboard I had thought I would have to reinstall Windows. This however was not the case! Windows 8 and Windows 10 both successfully booted after changing my motherboard! This is the first time I have seen any Windows version do that and I was happy to see they’ve improved. I still decided to wipe the system however and replace the ssd Windows was on. A couple of days later I also wiped the linux sdd and replaced that with my old Windows ssd (which was purchased later and thus much faster).

As far as servers go… I have cleaned them too ! Not only have I cleaned all my servers I have also cleaned up both my Dropbox and my Google Drive folders.

Finish my portfolio

I kinda made this goal… kinda. Hear me out on this one ! When I wrote my last blog post I had an idea in mind for the portfolio. There was already a sorting mechanism in place and things didn’t look half bad either. However… I decided to change things around and use a different sorting mechanism (mixitup) and also revamped the look of the portfolio. /images/videos are now the first thing you’ll get to see and they will be as wide as the page itself. (some videos , like app showcases, will have black borders though). Directly beneath the media content is the customer and designer information and below that you’ll find a short detailed summary for each aspect of the project. So I decided to chalk this one up as a success because I’ve done more work than I had originally anticipated :).

The other goals

Only a few of the goals I had set I haven’t reached. I have given up on losing weight and because of the extra portfolio work I didn’t really start LSR either. 5 new tutorials is a detable goal. I haven’t exactly written 5 tutorials but I have done some work on mi-soft.nl (and still wrote 2 tutorials). And 1 blogpost… well I’m trying !

Losing some of my stuff

As I mentioned before I have lost some of my hardware to a lightning strike. I didn’t just lose my motherboard however. I also lost my microphone (blue snowball), my keyboard (razer blackwidow), my xbox controller receiver, 8gb’s of ram and some random usb peripherals. Some of the stuff was insured and thus I have gotten some money for it. However I did not simply want to replace the lost stuff, I wanted to upgrade too.

I upgraded my keyboard to a ducky shine with black cherry mx switches. I have to say I’m still getting used to the linear-ness of the switches but so far so good. The big upgrade however is my recording setup. I went from having a single blue snowball to a blue yeti with shock mount, pop filter and boom arm. Initial tests prove to be very promising and there is a definite upgrade in sound quality. Aside from sounding a bit better it looks A LOT better. It almost look’s pro! Here, have a look:

Looking spiffy!

Don't mind the mess, it's the mic we're looking at

Youtube channel split + new intros and outros

Mi-Soft.nl and Rick van Lieshout are now 2 separate Youtube channels. Mi-Soft.nl will only be used to host tutorials. Nothing else. My personal channel (Rick van Lieshout) will host all other things.

I have designed a new outro for all of the Mi-Soft.nl video’s. I made the outro with “Camtasia studio” and “Gimp”. The outro can be found here:

I also bought some intro’s from Flixpress. Both of these can be found here:

New look, new comments, new everything!

Tue, 16 Jun 2015 00:00:00 GMT

Separating the personal stuff

First off, it’s been a while… I know. ☺

I’m trying to separate mi-soft.nl and myself, yet keep them together somehow. I have removed almost all personal content from mi-soft.nl and am currently busy moving it to this “personal website”. The end goal is to have them completely separate apart from the “software” tab. Software that I want to reveal to the public (more so than some scripts I use myself) will still be uploaded to mi-soft.nl and more personal projects (like CSVtoDatabase) on my github.

In order to “keep the sites together” I have given this site the exact same look as mi-soft.nl except for the colours. I have also made a little system that allows users to switch the website’s language. Everything apart from the blog will be Multilingual. Doing all these upgrades however also means upgrading Disqus, which meant losing all my comments. While sad, I feel like it had to be done.

Replacing the “about me” page on mi-soft.nl

I will be replacing the about me page on mi-soft.nl with a “Linux software review” (LSR for short) section. LSR will consist of short video’s (in Dutch) promoting Linux software.

Another maker project

I am currently gathering materials and setting up the software for another cool maker project. It is going to include both the arduino and the raspberry pi. But that is pretty much all the information I want to disclose right now.

School

I have only 2 days of school left this year, (and maybe a ceremony) which means an increase in productivity (hopefully). There are a couple of things I want to do before school starts again:

Lose 20+ pounds
Finish separating my websites
Clean all my servers (both soft and hardware)
Finish my portfolio (in both Dutch and English)
Get started with LSR
Write at least 5 new tutorials on mi-soft.nl
Write at least 2 new blogs. (not counting this one)

Let’s hope I can get all my planned projects done and then some.

A status update

Sun, 16 Nov 2014 00:00:00 GMT

School

In my last blog post I mentioned that I was busy with schoolwork (and all related matters like friends & gaming). That all changed yesterday, I decided not to game as much anymore and focus on something exciting again. I have no idea what kind of exciting thing I’m going to focus myself on, but most likely it will be either Arduino or Raspberry PI (cluster / solo).

Some more great news! We were able to collect 15 points (out of 60 in a year) during this first semester. And I got them all. The most difficult test for me was of course the Dutch language test. Which I barely passed. One of the other “harder” tests was a subject called “UML”, and as some of you might know… I hate, most of, UML.

A new semester has started this week and we’ve already had a taste of what is to come. Sadly what is to come isn’t all that Linux friendly. They have opted to use MsSQL instead of the (much better) MySQL(MariaDB) / PostgeSQL. This means that I, and another classmate, have to install virtualbox with Windows on our laptops. Shouldn’t be a big problem should it? Well no it shouldn’t. But me trying to be smart resulted in me downloading a “stripped down / light” version of Windows. (as if there is such a thing ! ha!) After setting up Windows, and installing office (since we’ll be using Access (another YUK!) I tried installing MsSQL. Turns out however that one of the stripped features was powershell, and for some inexplicable reason MsSQL will not install without powershell. To make matters worse Microsoft decided that they wouldn’t offer an install package through the website anymore since powershell is built-in in the newer versions of Windows. So I had to resort to giving up almost 30GB of hard drive space just to install MsSQL… (and I will probably end up using MySQL anyways)

Numix-icon-theme-square

Last night a friend of mine tried installing the Numix square theme, something that usually doesn’t require my help. Last night however my friend did need some help because the theme just would NOT install correctly. (kept throwing a GPG key error) So after trying to install an older package (which I did not know he removed from his system) I finally decided to google the matter. Turns out the “Square” theme’s actually cost money (who knew?). Turns out that Antergos Linux has some sort of a license to use them but general Linux does not.

That is why the team behind Numix has requested the package to be removed from the aur. And their request was granted. (as it should have been). Last night however we didn’t have the tools handy to be able to make a purchase of the Icon theme, so I sent him my version and he installed that instead. I will however buy a license from them today, since I use the icon theme on pretty much all my machines.

Arduino sneak peak

I acquired some new toys for the arduino, two of them being the “Adafruit mini 8x8 LED matrix”. To use them I had to install 2 libraries, and with those came “Example sketches”. One of those example sketches was a “Robotic eye”, naturally with me having 2 new toys I decided to create a little “face” (just eyes) with the prebuilt sketches. Anyways I thought that looked cool so I made a photo which you can view below:

Google eyes... but matrix style

Me, my blog, and projects.

Tue, 21 Oct 2014 00:00:00 GMT

The blog

So, I’ve already gone past my weekly schedule. Which tends to happen every time… I’m not giving up on the blog though ! I’ve just been far too busy. Busy with what you ask? well… college. Now surely the first period shouldn’t be that hard for someone who can code reasonably well already You’d be right in thinking that, but.. that’s not the only part of college, college also means new friends. New friends mean gaming. So yeah I’ve been gaming a bunch. Sorry.

Projects

For now I’m putting off some of the projects I had originally planned. I simply can’t find the time to write detailed blog posts on them and I don’t want to disappoint you guys with half-baked posts.

I did start a new “project”, meaning I ordered 2 of the most beautiful computers on the planet. Yes, I bought 2 iMac G4’s . Even though I don’t like Mac os, or the designs Apple is currently pushing out, I do really like these older Mac’s. I’ve ordered 2 of them from the internet, the seller is currently looking for a box to send them both in since they are pretty big and heavy. I can only wait and hope they will arrive in good condition.

So I hope you guys appreciate this little update and can forgive my absence. Have a nice day!

Shellshock and why you shouldn't get your panties in a bunch.

Sat, 27 Sep 2014 23:00:00 GMT

On September 24 2014, A Bash vulnerability, known as “Shellshock” was disclosed. This “bug” allows an attacker to run code on your machine, which is bad, very bad.

Now then, what exactly is “Shellshock”. Shellshock is a “bug” (or vulnerability) that can be used to remotely run code on someone’s machine by altering an environment variable. The real danger however lies in programs that can alter these variables without requiring root. Thus enabling anyone with access to your computer to run commands. Imagine someone attacking you twice, the first time they’ll create a user and the second time they’ll set a password for the user. After that they can just log in to your system and destroy the entire thing, or worse still, monitor your every move. Some examples of the services which can alter environment variables without root privileges are:

OpenSSH (which 90% of all Linux distros use)*
DHCP clients
Apache servers with CGI scripts

When are you vulnerable?

This seems to be a point many news sites forget about. When exactly are you vulnerable and how worried should you be. The thing is, in order for you to be vulnerable you’d have to have one of two things:

A server which is open to the world (web,vps, etc.)
An intruder in your network.

Most people won’t have either one of those things. And to be fair, if you can’t trust the people on your own network you’ve got far bigger problems than “Shellshock”. Unless of course you’re using a whole lot of “free-wifi” services, in which case you really don’t know who is on the network.

I’d imagine the number of people having a “server” , or simply a computer which is accessible from outside your home network is way higher. So if you are one of those people, please patch your bash to avoid any trouble. ( I too like to have an easy way to log into my server from anywhere (work / family / vacation) and thus I would be vulnerable.)

How to test whether you’re vulnerable

Testing whether you’re vulnerable is quite easy. You can just use this “attack” on yourself with an “echo” command (which won’t cause any harm). If it works, you’re vulnerable. If it doesn’t you’re not. Here’s how to construct a test command:

env x='() { :;}; echo vulnerable' bash -c 'echo hello'

If you’re not vulnerable you’ll get:

bash: warning: x: ignoring function definition attempt bash: error importing function definition for `x' hello

You might also simply get “hello”. If you are vulnerable however you’ll get:

vulnerable hello

How to patch your system

By now most distributions have updated their repo to include the patched version of bash. If that is the case you can simply do a regular update:

Ubuntu / re-spins

sudo apt-get update && sudo apt-get upgrade

Arch / re-spins

sudo pacman -Syu

If however you’re using a mac you’ll have to do it by hand. Since Apple hasn’t yet released an update nor commented on the matter. Instructions for a mac can be found here.

In summary

So should you worry? Yes and no. If your computer (/server) is exposed to an outside network you should absolutely worry. If it’s not however you are relatively safe. Still, patching is a simple and fast process, so just patch it and get on with your life !

Documentation

For those of you who like to research this “Shellshock” vulnerability some more, here are 2 entries in the NVD: CVE-2014-6271 & CVE-2014-7169.

If you want to see Shellshock in action I recommend watching the following video from Quidsup:

The pi cluster, home automation & kickstarter monitoring

Sat, 27 Sep 2014 00:00:00 GMT

Some of you might be wondering “why doesn’t he write more blogs about the cluster?” and I wouldn’t blame you. I have a pretty good reason for not blogging about the pi cluster (even though I promised sample code). That reason is “home automation”. As some of you might know I’m big into home automation, and have done quite a bit already. But it’s always fun to try something different!

I usually use single or double relay boards which you can control with an arduino or a Raspberry pi. This time however I ordered an 8 relay board instead of a double relay board. I got the board from China at a store called “Sainsmart”, here’s the link to the exact board I got: https://www.sainsmart.com/products/8-channel-5v-relay-module

So what does that have to do with the cluster you ask? Well, I used to have a Raspberry pi to experiment with… But I used that in the cluster to save some money. (here I was, thinking to be smart!) So I had to “disassemble”(taking the top off) the cluster to connect the relay board. In the next blog-post I’ll have a couple of pictures showing off the relay board and the pi cluster.

I donated a pi

While talking on IRC with a friend(Robbie Ferguson) of mine I discovered that he had never used a Raspberry pi before. And to make matters worse he told me he had lost interest in them now. When I asked why he never got one he told me the following:

When they were new I couldn’t get one here in Canada, and now it’s been so long that I’ve lost interest.

While discussing the fact he jokingly told me, if you’ll send me one I’ll try it out. A small problem arose though, me sending a pi to Canada would cost way more than actually buying one over there. So we decided that I’d just send him the money and he would order his own. I can’t wait to see what he does with it. If you guys are interested in seeing what he does with it too you can check out Category5.tv

What Robbie doesn’t know is that I wanted to do something for the show / him anyways. I was originally planning on just donating $20, but giving him the joy that comes from owning a Raspberry pi seemed like a way better plan! Besides, he’s been working really really hard on the new Studio D so he deserves some fun!

Kickstarter monitoring

I have once more decided to back a certain Kickstarter campaign and find myself checking the website a lot. I plan to change this by creating a little script (don’t know which language yet) that will check it for me and report back. More info about this project to come.

Raspberry pi cluster step 5: Expanding the cluster

Fri, 19 Sep 2014 00:00:00 GMT

In today’s post we’ll have a look at expanding the cluster with more nodes! We will have a look at how to clone them and how we can set up a universal login using ssh keys.

Windows users can use the “read” option of the previously installed win32imager to create an img.

Linux

Insert the tested and configured SD card and use lsblk to find out on which /dev/ the SD card resides. Now use dd to make an img of your Raspberry pi master node (note! this can be used as a backup too) with the following command:

sudo dd if=/path/to/sd/card of=/home/`whoami`/raspy.img

After running that command you will have an image of the configured Raspberry pi in your home folder. You will now have to write this image to all SD cards.

After writing all the images to the SD cards you have to change the hostname on all “copies”. To do this put the SD card into one of the pi’s, let it boot , and log in. Then give it the following command:

sudo raspi-config

Next, you’ll have to change the ip on each of the nodes by changing the “address” line after executing the following command:

sudo nano /etc/network/interfaces

After setting up all 4 pi’s and plugging it all in you should see all 4 led’s light up on the pi’s. If this is not the case you might’ve forgotten to change either the ip address or the hostname. (which will lead to a conflict resulting in one of them not having internet access). Mine now looks like this (note the dramatic night picture taken at 10 past 12 am):

Logging in to the nodes with ssh keys

To make working with the cluster a lot easier we will use ssh keys to login in place of the old password. We are going to generate the master node’s key first, to do this enter the following command.

ssh-keygen -t rsa  -C "pi@raspberrypi0"

You’ll be asked where you want to save the key, the default directory is usually fine. After that you will be asked to enter a passphrase. (the passphrase is optional but also recommended in a production environment) After generating the key a “randomart” image will appear. After generating the key you want to execute the following command:

cat ~/.ssh/id_rsa.pub | ssh pi@192.168.1.8 "mkdir .ssh;cat >> .ssh/authorized_keys"

Keep in mind you might have to change the ip address in the previous command. After executing this command for the 2nd node, repeat for the third,fourth,fifth, etc., etc. nodes.

After we have copied the key to all nodes we have to edit the previously created machine file (on master) to include all the ip addresses of the other pi’s. To do this type in the following command:

nano mpi_scripts/machinefile

Simply add all the other pi’s ip addresses to the file (each on a new line).

Testing

Now that we’ve added the pi’s ip’s to the machinefile we can use them from our master node. To test this we can run the following command:

mpiexec -f mpi_scripts/machinefile -n 4 hostname

This should give you 4 hostnames. If it does you’ve set it all up correctly! We can even try and run some C code with them by running the next command:

mpiexec -f mpi_scripts/machinefile -n 4 ~/mpich_build/examples/cpi

Is that correct? O.o

If you have a look at the screenshot above you’ll see that the cluster as a whole (4 pi’s) actually take longer to calulcate pi than a single node would. This is to be expected with something that runs this quickly. The cluster has to communicate with each of the pi’s, which results in a tiny little bit of network overhead. In the case of calculating pi this network overhead turns out to be greater than the time it takes to calculate pi to 15 digits.

Now that we have a complete (working) cluster we are all set to start using it! But first, here’s a video of me power cycling the cluster:

Raspberry pi cluster Step 4: Setting up the master node

Wed, 17 Sep 2014 00:00:00 GMT

After installing raspbian to the sd card it’s time to plug in that SD card and fire up the Raspberry pi. If everything went well you should end up with a screen similar to mine:

raspi-config boot screen

The first thing you should do is select the “Expand Filesystem” option, this will make sure the OS can access the entire SD card. After doing that you might want to change the user password (user = pi). The last thing you want to do is hidden in the “Advanced options”, after going there you should see:

expand filesystem menu options

In here, select the “A2 Hostname” option and rename it to something better. (I appended a 0, indicating this would be the master pi)

After doing all this hit finish, reboot the pi, and log back into the shell. (default user = pi pass= raspberry) The first thing we are going to do is update the pi, in case any updates have come out since the image was uploaded to raspbian.net. We can update the pi by running the following commands:

sudo apt-get update && sudo apt-get upgrade

After updating the pi we need to give the pi a static ip address (so we can find it later). We can do this by following a couple of steps:

sudo nano /etc/network/interfaces
replace iface eth0 inet dhcp with iface eth0 inet static

Directly underneath add the following lines: (remember to change info if required)

address 192.168.1.11
netmask 255.255.255.0
network 192.168.1.1
broadcast 192.168.1.255
gateway 192.168.1.1

Hit ctrl + x to close nano and hit y to save the changes.
reboot the pi

After that we have to install some required software, we can do so with the following command:

sudo apt-get install gfortran

After installing some of the required software we still have to install “mpich3”, do so by running the following commands:

mkdir /home/pi/mpich3
cd mpich3
wget https://www.mpich.org/static/downloads/3.0.4/mpich-3.0.4.tar.gz
tar xzf mpich-3.0.4.tar.gz
sudo mkdir -p /home/rpimpi/mpich3-install
sudo mkdir -p /home/pi/mpich_build
cd /home/pi/mpich_build
sudo /home/pi/mpich3/mpich-3.0.4/configure -prefix=/home/rpimpi/mpich3-install
sudo make && sudo make install

After installing mpich3 we need to edit a file. Execute the following command:

sudo nano /home/pi/.profile

Scroll down all the way and add the following text:

PATH="$PATH:/home/rpimpi/mpich3-install/bin"

After adding that line hit ctrl + x and save your work. Then either restart the shell or reboot the pi.

After you have rebooted the pi you have to create a so-called “machine file”, we will store this file in the folder “mpi_scripts” in our home directory. To do this execute the following commands:

mkdir ~/mpi_scripts
cd ~/mpi_scripts
echo `/sbin/ifconfig eth0 | grep "inet addr" | awk -F: '{print $2}' | awk '{print $1}'` > machinefile

The last command creates the machinefile and populates it with the ip of this (master) node automatically. With all of this setup we can do our first real test with the mpi commands, so try out the following command:

mpiexec -f machinefile -n 1 hostname

The -f is the filename parameter (so you would type the machinefile path behind that). The -n is the number of nodes we want to use (we are working from 1 now). And finally “hostname” is the command.

if all went well you should now see the hostname of the raspberry pi (for me: raspberrypi0). After this we are done setting up the master (for now) and can begin copying the SD card. To shutdown the pi you can execute the following command:

sudo shutdown -h now

Raspberry pi cluster Step 3: Installing the OS

Sun, 14 Sep 2014 00:00:00 GMT

Installing an OS for a Raspberry pi is quite different from installing an OS on any other computer. We will have to write an “image” to the SD card rather than actually “installing” anything. Don’t worry if you’re confused just now, even I don’t know what I’m rambling on about at times!

Linux

First off, if you want to use Windows, skip this bit by clicking here.

First we will have to download the .img from the Raspbian website, we can do so by clicking here. After downloading the image we can use a kernel application known as “dd” to write the file to our SD card. We will first need to find out which /dev/ the SD card is currently using. To do this we can use the following command:

lsblk

This will show us a list of all the connected volumes. You should be able to determine which of the entries is your SD card by looking at the size of each entry. If you are not able to determine which one you need based on the size of the entry however, you can try removing the sd card and re-running the command. The entry that disappears is the SD card.

As you can see below, /dev/sde is my SD card

NAME   MAJ:MIN RM   SIZE
sda      8:0    0 119.2G
├─sda1   8:1    0   243M
├─sda2   8:2    0   7.5G
└─sda3   8:3    0 111.6G
sdb      8:16   0 111.8G
└─sdb1   8:17   0 111.8G
sdc      8:32   0 111.8G
├─sdc1   8:33   0   350M
└─sdc2   8:34   0 111.5G
sde      8:64   1   7.5G
└─sde1   8:65   1   7.5G

Now that we know both the input path (the downloaded iso’s location) and the output path (the /dev/sd* device) we can assemble our dd command as follows:

dd if=/path/to/iso of=/path/to/sd/card

When you execute the dd command the terminal will appear “busy” for the duration of the write. You can either wait patiently until it’s done or check up on its status by entering the command:

sudo kill -USR1 `pidof dd`

Windows

If you ended up here you have chosen to use the much inferior Windows to get Linux on your Raspberry pi’s. Don’t worry though, you can still install the, much superior, Linux images to your SD cards.

The very first thing we have to do is download the Raspbian image, which we can do by clicking here. After downloading the image we will have to download some software to write the image to the SD card. The tool we are going to use is called win32DiskImager, and can be downloaded free of charge right here.

After the tool has been downloaded (and installed) we have to start it by double clicking on the .exe file, which will leave us staring at the following tool:

win32diskimage

Using the image above as a reference:

This is where you have to specify the image location (you can hit the blue box to select a location)
Here you will have to specify which device you want to write to.

After keying in the right values you can hit “write” and wait for the “install” to finish.

First boot

Once you boot up the pi you will get a settings screen, you can go through it if you wish. (change password and so on) If you want to start the GUI you can type:

startx

Raspberry pi cluster Step 2: Assembling the hardware

Wed, 10 Sep 2014 00:00:00 GMT

So today I finally received the required parts to finish up my Raspberry pi cluster and I was extremely excited! To share this excitement with you lot here’s a picture of all the parts I’ll be using:

all the bits!

I began by building the Raspberry pi “tower”, sadly I forgot to take a picture with the DSLR so I only have this one cell image to show you guys:

rising up!

On the picture above you can clearly see how the “tower” is being assembled. I’m basically using the stand-offs that come included to mount the pi to the bottom plates and then instead of attaching a top plate I attach a bottom plate again. I did have to customize the plates a bit to prevent the pi’s from overheating (because there now is pretty much no room between a pi and the top plate).

After mounting all 4, the “tower” now looks like this: (oh, I also added SD cards)

Assembled tower of pis

Then I attempted to glue the tower to my switch… Some complications came up though. First I glued my knife to my finger and then my finger to another finger. Once I had successfully managed to glue the tower onto the switch I needed something to apply pressure to let the glue dry. Since I didn’t have anything heavy handy I had to improvise. And so I did, in true nerd style:

balancing a weight.... a netbook

Yes, those are 2 packs of cards (which weren’t heavy enough) and an old netbook. I went to have dinner while I let the glue dry and when I returned everything was all glued firmly into place.

After adding the cables and the usb switch I am left with my beautiful raspberry pi cluster:

looking great!

In my next blog post, I will cover how to install raspbian on a raspberry pi in general and how I plan on cloning the install across all pi’s

Raspberry pi cluster Step 1: Shopping

Fri, 05 Sep 2014 23:00:00 GMT

So, A raspberry pi cluster. What is it and what does it do?

Well, a raspberry pi is a small $35 dollar Linux computer with an SD card as a hard drive and an USB cable as it’s power source. While they are only $35, they perform really well!

A cluster is basically a group of computers that work together to run a very intensive program. Say for example you needed to calculate 1000 things with a single raspberry pi and every calculation takes 1 second. That would be 1000 seconds! Now if you add a second raspberry pi and you have them share the load they will be twice as fast! Imagine running 4 of them, that would cut it down to 250 seconds! Well that’s exactly what I plan on doing

What software are you going to use?

I am going to use raspbian for the OS and I will be using python (and mpi4py) to program the cluster.

What did you need and how much did it cost

Below you will find a list of items you would need if you were to build a 4 pi cluster from scratch. Prices are in Euro’s because that’s what I pay with (I know, crazy right?)

4 raspberry pis ( €131.80 )
4 ethernet cables ( €3.40 )
4 USB power cables ( €7.20 )
4 sd cards ( €12.72 )
4 raspberry pi cases ( €11.72 )
An Ethernet hub ( €10.95 )
An USB hub ( €2.64 )

This all adds up to ( €180.43 which is $233.76 according to google)

I ordered some of the stuff from a shop in china called ”banggood”. Since they deliver all around the world I will add the links to those items below.

The raspberry pi cases can be found here
The 8gb SD cards can be found here
The USB hub can be found here
The USB power cables can be found here

My brand new blog

Fri, 05 Sep 2014 22:00:00 GMT

Hey all and welcome to my new blog! A couple of questions pop-up right away I’m sure. So let’s get those out of the way right now!

English?!? Is this a joke?

Nope, I have decided to blog in English for a couple of reasons.

It will be easier for me to write articles.
I get to share my posts with a couple of non-Dutch friends.

Why did you start over again?

Yeah, I know… This is like the fourth time I start over isn’t it? Well let me give you some reasons for switching:

Control. I now have full control over how the page looks and performs.
No more Wordpress!!! Many of you will know I have a love/hate relationship with Wordpress. But in the end, the hate won over the love. I am just fed up with the entire system and its problems.
Because I wanted to convert to English. And why not convert platform at the same time?

So what can we expect?

I’ll be doing a lot more technical blogs. (up next: raspberry pi cluster) I also plan on doing some code examples and general rambling. So pretty much the same as any other blog.