Aaron Ogle

A Decade at Rocket.Chat: Looking Back on 10 Years of Building, Breaking, and Growing

Mon, 23 Mar 2026 00:00:00 +0000

It’s hard to believe I’m writing this. After nearly 10 years at Rocket.Chat, I’m saying goodbye. What started as just a guy on the internet submitting pull requests to fix image stretching… Turned into a decade of building across multiple cloud platforms, traveling the world, meeting/working with great people, making friends, leading teams, and learning A Lot.

I want to look back on this journey. Not just the technical milestones, but how I grew as a technical professional, the team evolution, the org changes, the migrations that aged me, and the people who made it all worth it.

The Very Beginning: A Community Contributor Fixing Bugs (2015)

I discovered Rocket.Chat in 2015. Bradley Hilton and I had actually been building our own chat application. It started because he needed a staff chat for his Minecraft server and was tired of using Slack (and definitely didn’t want to pay for it). We couldn’t find anything that fit, so we started building our own. When I found Rocket.Chat it was exactly what we were trying to build. So instead of reinventing the wheel, I started using it, hit rough edges, and started fixing them. My first pull request on August 21st, 2015 was titled “Fix image stretching.” The second one came the same day: “Fix search not coming up.”

Within a week I was submitting PRs for pin/unpin functionality, configuring the site name via environment variables, adding /invite and /leave slash commands, and improving hubot-rocketchat. 33 PRs across 4 repos before the year was out. I wasn’t an employee. I wasn’t getting paid. I was just a guy who liked the product enough to make it better.

In October of 2015, Diego Sampaio built the first easy way to try Rocket.Chat, a proof-of-concept system brought online for OSCON in Amsterdam. You’d enter your email and a subdomain like something.rocket.chat and it would spin up a workspace. Built on Rackspace with Tutum, a single Mongo node, a single worker, and Hipache for load balancing. The address was deploy.rocket.chat. Simple. Scrappy. It worked.

The seeds of everything that would come later were planted right there.

By mid-2016 I was going deeper, I contributed the original Jitsi video conferencing integration in the hotel lobby across from OSCON in Austin. We met Emil Ivov the founder of Jitsi earlier that day in the Atlassian office. Got excited and started building.

Fleetcommand and the Cloud Team (2016 - 2018)

In September 2016, we built the rocketchat-server snap, one of the very first snaps created. This was a direct result of interactions with Mark Shuttleworth from Canonical. A small but cool moment in our open-source story.

By October 2016, it was becoming clear that Rocket.Chat was going to receive its first round of funding (which officially came in November 2016). I started building Fleetcommand, the software that would manage trials, handle Stripe billing, and automate provisioning for our hosted customers. The original vision was a portal at my.rocket.chat for users to manage their workspaces. We explored many options like using Juju by Canonical(after earlier success with snaps), docker swarm, mesos, nomad and several other ideas. But ultimately decided on kubernetes and utilizing its API and specifically the same go packages kubectl leveraged.

In January 2017 we received our Google Cloud credits, and Fleetcommand went live around March 4th, 2017.

A fun story during this time: I had traveled to Brazil on February 23rd, 2017 for our first ever company summit. While there, we received an opportunity for a free booth at Google Next in San Francisco only a few days later, March 8th to 10th. So we flew from Brazil to San Francisco. I was gone for 20 days. My wife was not so sure about this new startup I had joined! 😂

At the booth, we switched the deployment of workspaces from rackspace/tutum to run on Kubernetes on Google Cloud. We were literally refining the process and deploying fixes live from the conference floor. We even rushed a proof of concept that automatically linked users to their Google accounts. Once it was finished, we actually then returned to Brazil and had our summit.

The First Great Migration

For a while we supported Google Cloud alongside the original Tutum deployment. But as the Rackspace credits ran out, we needed to migrate everything over. This was our first zero-downtime migration and we developed the pattern we’d use for every migration after.

The strategy: extend the existing use1-db1 MongoDB replica set by standing up additional replicas within Google Cloud. Once the new replicas were healthy, switch the primary over. Spin up all new workspaces in Kubernetes on Google Cloud. Switch the DNS records. Completed March 26th, 2017.

At this point I was basically a solo operation as the “Cloud Architect.” I was writing Fleetcommand, managing infrastructure, handling deployments, writing documentation. 93 PRs across 23 different repositories in 2017 alone. My fingerprint was across the entire ecosystem.

Becoming a Team

Early 2018 after our summit is when the “cloud” team was born. Bradley, who had been present since the company was formed, joined the cloud team. He was initially focused on the Apps Engine, but his focus pivoted to needing a way to distribute apps, which led to the creation of the Marketplace. Its first commit landed on July 14th, 2018.

In July 2018 we also made our first infrastructure-specific hire. The team grew from 2 to 3.

If you look at my GitHub history from the second half of 2018 there’s a clear pattern: take a service, containerize it, add automated builds with Google Cloud Build, deploy it. On July 31st alone I submitted PRs adding a Dockerfile and auto-deploy to the marketplace-api and adding Prometheus to Fleetcommand. Three days later: “Release initial version” of cloud-portal. Then statuscentral, our status page which we’d built in about 2-3 days in Go inspired by another open-source status page, got Docker and Cloud Build (September 27th). That little project would keep evolving for the next seven years: maintenance windows, a CLI, snapshot ability, incident history, even a Twitter integration so customers could follow along during outages. Then the Push Gateway was rewritten from JavaScript to Go and containerized (November 15th). The JavaScript version had been locking up under load; the process would just stop responding and the only fix was killing the pod. The Go rewrite was night and day. Memory went from around a gigabyte down to megabytes. Go’s built-in concurrency handled the high-throughput push notification traffic without breaking a sweat. The OmniChannel Gateway got a multi-stage Docker build. The Federation Hub got simplified and containerized. One by one, the cloud platform took shape.

By the end of 2018 we had the bones of the cloud platform. I had 142 PRs across 27 repos that year, and the operational burden of managing a growing hosted customer base was growing right alongside it.

The Great Migrations (2019 - 2020)

Google Cloud to AWS (2019)

We operated on Google Cloud until those credits also began to diminish. Amazon approached us with AWS credits, so we kicked off our second major migration. This time the approach was a little bit more sophisticated. Like last time, we established a secure tunnel between the two environments so they could talk to each other, and extended MongoDB replica sets from Google Cloud into AWS. This time we introduced ingress records in the new infrastructure that would proxy traffic back to Google Cloud. We shifted DNS to point to the new AWS ingress, then migrated customers in controlled batches, spinning up workspaces in AWS and flipping their specific ingress records one by one.

I later documented this migration pattern in a blog post: A K8s Migration Concept. The migration was completed on March 1st, 2019. The most satisfying PR? “Remove GCP”, 5,279 lines deleted. Goodbye Google Cloud.

The Bare Metal Leap: OVH (2020)

In 2020, we needed to decrease our operational costs. Oracle offered credits but their infrastructure and consoles were challenging. Our search led us to OVH and bare metal servers.

This was a whole new world. We had to master deploying Kubernetes on bare metal with Rancher, deal with OVH’s unique “vracks” requiring manual IP assignment, set up bastions for access, and learn that their firewall rules only protected against traffic from outside OVH, meaning other OVH customers could bypass them entirely. Storage was another challenge. On managed cloud you just request a volume and it appears. On bare metal there’s nothing. We found Longhorn, which let us carve out persistent volumes from some nodes we provisioned with big raid disk arrays, giving us the storage layer Kubernetes needed.

We executed another zero-downtime migration, completing it on July 3rd, 2020. We refined our process further, exposing many ingress switching functionalities as API calls within Fleetcommand.

OVH also taught us about hardware failures. Nodes just die on bare metal and way more often than we thought. CPU overheating and bootlooping randomly. Water cooling manifold related issues, thermal paste etc. Not things we expected to be having to raise support tickets and deal with. You also learn quickly about capacity planning when there’s no auto-scaling button. You learn to have spares ready. You build close relationships with your vendors.

Throughout all of this, three major migrations, multiple cloud providers, growing customer demands, a small team of 3-4 people kept everything running. 199 PRs in 2020 alone. “Begin OVH adventure” (16,539 lines), “Svc migration to ovh” (18,593 lines). Not glamorous work, but the kind of work that keeps a platform alive.

The Spotify Era: Tribes, Squads and Chapters (2021)

In early 2021, the company adopted a Spotify-style organizational model. Tribes, Squads with Product Managers, Engineering Managers, and Tech Leads. Chapters for disciplines like frontend, backend and infrastructure.

Up until this point I’d been doing everything: writing code, managing infrastructure, and being the engineering manager. Running 1:1s, performance reviews, salary negotiations, hiring. I was listed as the hiring manager for the whole cloud team. It was too much for one person, and something had to give.

The squad reorg changed that. What had been “the cloud team” became a cloud tribe with multiple squads. I moved into the Tech Lead/Architect role across the Cloud tribe, specifically the Cloud Service Squad (Fleetcommand, cloud portal, push gateway, and others), the Marketplace Squad and still was involved in the Infrastructure Squad as a major stakeholder/architect as we did have a team member step up as the main lead for that team.

This was also a period of significant growth. We brought on our first PMs. New engineers joined like Debdut and Daniel, both coming in as interns (and still at Rocket.Chat today). The team was bigger than ever. And crucially, Gabriel Casals came on as an Engineering Manager for the Cloud squads. This was a turning point for me. He was the first non-technical EM I worked closely with, and he showed me what the role should actually look like: handling the people side, the process, the coordination, so I could focus on technical direction. That partnership taught me how a strong EM and a strong tech lead can complement each other instead of stepping on each other’s toes. After years of carrying both loads, it was a huge relief to no longer be responsible for the people side of things.

Floating, Splitting, and beginning of the SRE Transformation (2022 - 2023)

2022 brought major changes. Our PM for the Cloud tribe left in January. Julio Bertelli joined. The company went through a restructuring that impacted 12 people.

I found myself increasingly acting as a floating architect across squads rather than being embedded in one. The Cloud Service Squad, the Cloud Hosting efforts, and the infrastructure team all needed technical direction, and I was the thread connecting them. But it was getting harder. The infrastructure team and the service squad had different priorities, and things were falling through the cracks.

Meanwhile, a new Engineering Manager came in, initially working across QA and security, and eventually taking on the infrastructure team as well. By early 2023 he was EM for both the security team and the infra team. The arrangement meant I could focus on the technical side while he handled the people management.

Around this time, the Marketplace Squad was dissolved and absorbed into the Cloud Services Squad, which was then renamed to the Connectivity Squad under Gabriel Casals as EM, handling registration, licensing, marketplace, and workspace connection services. The Infra team we started calling the SRE team.

The catalyst for the SRE transformation was the launch of Premium and Dedicated offerings on February 2nd, 2023, with a commitment to a 99.9% SLA. That kind of promise changes how you think about everything. It’s not “best effort” anymore. You need monitoring, runbooks, incident response, post-mortems, on-call rotations. You need to think like an SRE team, not just an infrastructure team.

This was probably the most intense period of my career. I was the technical lead spanning both squads, setting direction for the SRE team on the infrastructure side and the Connectivity Squad on the services side, each with their own EM handling the people management. It meant I was the common thread between the two; the person who understood how the infrastructure and the services fit together end-to-end. We were reshaping what had been a purely infrastructure-focused group into a team that thought about reliability as its core mandate.

Writing Things Down: RFCs, RFDs, ADRs, and Post-Mortems

One of the things I’m most proud of across my entire time at Rocket.Chat isn’t code; it’s process. Specifically, the evolution of how we made and documented technical decisions.

It started in November 2018 when I created [RFC 1] Self Hosted Registering with Rocket.Chat Cloud on our internal Discourse forum. The idea was borrowed from the open-source world: before you build something significant, write it down. Explain the problem, propose solutions, invite discussion.

That first RFC kicked off a real discussion, 13 posts from me, Gabriel Engel, Guilherme Gazzo, Rodrigo, and others. RFC 2 (Cloud Multi-Region) followed two days later. Then RFC 3 (Marketplace Licensing) in February 2019. Over the next few years, 39 RFCs were written on Discourse covering everything from marketplace subscriptions to the pros and cons of moving away from Meteor to airgapped workspace registration. I personally authored about a dozen of them, mostly around cloud architecture, marketplace, and workspace connectivity.

The post-mortem practice started around the same time. My first one was the Cloud Workspace Outage of May 2nd, 2018. Then the Community Server Outage. Then the Push Gateway Outage. We wrote them up on the same Discourse forum, creating an internal record of what went wrong and what we learned. The post-mortem practice started on our Discourse forum with around 23 incident write-ups between 2018 and 2022. When we moved to GitHub, we imported those and kept going, ending up with roughly 45 unique post-mortems across both. Cloud disruptions, certificate expirations, database elections, a Game Day DR exercise where we discovered our backup credentials were broken.

By 2021, the RFC process had evolved. We started calling them RFDs (Requests for Discussion), a subtle but meaningful shift. RFCs felt like they were asking for permission. RFDs were asking for engagement. I migrated everything from Discourse into a GitHub repo, 18 documents in a single day, pulling together the existing RFCs about microservices architecture, moving away from Meteor, the identity service, and more into one place where they could live alongside the code.

But having them in GitHub wasn’t enough. ADRs were still scattered across different team folders in GitBook. Someone would share a link to an ADR and you’d never find it again. The problem was discoverability.

So in August 2023 I built rfd-tool, a small Go service that read from the GitHub repo and published to adr.rocket.chat. A searchable, tagged, browsable dashboard of every architectural decision. You could filter by project, by author, by tag. In that single month we pushed through 13 ADRs: License v3, RabbitMQ to NATS, Statistics Enforcement, Cloud Credential Storage, and more. The naming settled on ADR (Architecture Decision Record) as the umbrella term, with the understanding that discussion was always implied.

The tool kept evolving right up to the end. I added support for D2 diagrams, switched the backend from BoltDB to SQLite, added the ability to create ADRs from the web UI, and eventually added support for making specific ADRs public. One of my last PRs at the company was adding GitHub PR links to each ADR to make collaboration easier.

RFC → RFD → ADR. Discourse → GitHub → adr.rocket.chat. The format changed, the tools changed, but the core idea stayed the same from RFC 1 in 2018 to ADR 148 in 2025: write it down, invite discussion, make it discoverable.

Stepping Away and Coming Back (2024 - 2025)

By mid-2024, after nearly nine years, I decided to step away. I transitioned out in July 2024 and stayed on in an advisory capacity, keeping a connection to the team and the platform I’d helped build.

The time away was valuable. I found balance. I picked up hobbies outside of work, something I hadn’t really had in years. But I also realized how much I missed the work itself. Building things that matter. Solving hard problems with good people. I wrote more about that period in my 2024 year in review.

On May 19th, 2025, I officially returned as Head of Infrastructure & Deployment.

This time I took both People and Tech hats deliberately. No split between a technical lead and an engineering manager. I’d seen what happened when those roles were separated and the partnership didn’t gel. Simpler is better.

The SRE Team: Building Something That Would Outlast Me (2025 - 2026)

When I came back, one of the first things I did was write a team charter, a document that made clear the team’s purpose, scope, and strategic objectives. We started an SRE book club. We formalized documentation practices in Confluence. I wrote a lot… Posted priority alignments every week making sure that any adjustments in priority were clearly communicated to the team, every month setting the months priorities. The main goal was to increase alignment so it was very clear what we were working towards. I pushed for post-mortems after every incident, and tried to build a culture where reliability wasn’t just my obsession but the team’s identity.

We launched Launchpad (our opinionated deployment tool), drove SLO definitions across services, and pushed forward on everything from Helm chart improvements to airgapped deployment support.

By The Numbers

Looking back across the data (because of course I had AI build a tool to analyze all of it):


Years	~10.5 (Aug 2015 - Mar 2026)
GitHub PRs	1,520 across 84 repositories
PR Review Comments	2,885 (lower then I expected. But we dog fooded a bit heavy and tilted towards feedback often times inside of Rocket.Chat
Rocket.Chat Messages	359,980
Rooms Active In	1,665
Confluence Pages Created	98 (Note we migrated across 3 different documentation tools over the years so this number is just for confluence since it was adopted in 2024)
Confluence Edits	221
Zero-Downtime Migrations	4 (Rackspace→GCP, GCP→AWS, AWS→OVH, OVH→AWS)
Cloud Providers Operated On	5 (Rackspace, Google Cloud, AWS, Oracle, OVH)
Post-Mortems	~45 unique (This isn’t how many I wrote but total over the years. We started this process in 2018)
RFCs/RFDs/ADRs	148 total, ~30 authored by me
Services & Tools Built/Contributed To	Fleetcommand, cloud-portal, Push Gateway, Marketplace API, marketplace-worker, marketplace-ui, statuscentral, statscollector, omni-gateway, launchcontrol, airlock, Launchpad, billing-service, sponsorship-service, rfd-tool, filestore-migrator, portmaster, Go SDK, FederationHub, and many more
Team Names	Cloud → Cloud Service Squad → SaaS Squad → Connectivity Services → SRE

My peak year in the data was 2020: 199 PRs and 29,097 Rocket.Chat messages. I’d contribute a large part of the total to OVH migration. The year I was building, reviewing, deploying, and firefighting simultaneously. It was a massive learning curve for us. A small team of 3-4 people running an entire cloud platform.

The relationship that shows up most clearly in the data is with Rodrigo Nascimento, the CTO. 8,324 DMs over the years, peaking at 2,407 messages in 2020. That was the builder-manager relationship during the most intensive infrastructure work. He believed in me when I was just a community contributor, and that belief shaped my entire career.

What I’ve Learned

Start by showing up. I got here by fixing a bug. Then another. Then another. Nobody gave me a role. I just kept contributing until the role found me.

Small teams can move mountains. Three to four people executed four zero-downtime migrations across five cloud providers. We didn’t wait for permission or headcount. We just figured it out.

Titles don’t matter as much as you think. I was a “Cloud Architect” with no official title. An “Engineering Manager” who didn’t know his own job description. A “Staff Engineer” who also managed people. Through all of it, the work was the same: understand the system, set direction, build the thing, help the people.

Take the break when you need it. I left burned out and came back with perspective. The hobbies I found during that gap made me a better leader when I returned.

Build things that outlast you. The team charter, the documentation culture, the post-mortem practice, the SRE book club. Those matter more than any code I wrote. Code gets replaced. Culture persists.

Thank You

To everyone I’ve worked with over these 10 years, thank you. There are too many people to name, but a few I want to call out:

Bradley, for jumping into this crazy startup with me. Having a friend along for the ride made it a lot more comfortable to take that leap, and together we kept the cloud alive in those early days. Lucia, for keeping the infrastructure running when it was just us. Joseph, my favorite designer and an absolute joy to collaborate with during the squad era. Daniel, Debdut, and Paul on the SRE team. I’m proud of what we built together.

Gabriel Casals, for showing me what a great EM looks like. You set the example for how to genuinely care for the people on your team. Rodrigo, Gabriel Engel, and Chris Skelly, for the trust at every stage.

Guilherme Gazzo, for the partnership across engineering. Sing Li, for being a constant sounding board from the very beginning, always encouraging me to write more and take the leaps into the unknown that the startup world demands. Julio Bertelli “the Professional,” one of the most talented Go developers I’ve had the chance to work with. Douglas Gubert, for the many Ketchup sessions where we’d talk about anything and everything. And Julio Araujo, the coolest head of security I’ve ever worked with, and one of the most talented security engineers too.

To the Rocket.Chat community, you’re the reason I started. A product good enough that strangers on the internet wanted to make it better. That’s special. And a big part of what makes it special comes from the founders themselves. There’s a warmth that Gabriel Engel and Rodrigo, Diego, Marcelo have always brought to this project from day one, always welcoming, always open. I remember the first time meeting everyone in person and receiving big hugs like we’d known each other for years. That warmth is something you can feel throughout the entire company. It’s invisible on paper but you feel it in every interaction, and honestly it’s what kept me going through all the hard times.

It’s been an incredible ride. From “Fix image stretching” to Head of Infrastructure & Deployment. From 1 repo to 84. From fixing UI bugs to architectural decisions that shaped how tens of thousands of workspaces run.

Time for the next adventure.

- Aaron Ogle

Year In Review: 2024

Tue, 31 Dec 2024 00:00:00 +0000

2024 is now in the books!

In previous years, I wrote internal posts at work to reflect on the year and highlight accomplishments. This exercise was revealing. It was often a great way to see how much we’d achieved that you might lose sight of in the daily grind. This year, I decided to apply the same reflection to my personal life.

Here’s a TL;DR of 2024:

Personal Life

A Year Free From Caffeine
Returned to Martial Arts
Increased focus on Working Out
Lost 12 pounds
Traveled to Brazil and London

Professionally

Emphasized Production Readiness
Grew as a Staff Engineer
Changed jobs after nearly a decade
Explored New Technology
Adopted a Streamlined Tech Setup

Personal Life

2024 was full of ups and downs, but I made meaningful progress in key areas.

Health

Improving my health became a major focus. Thankfully not due to any serious issues, but because I’ve started to realized it’s easier to get and maintain health now than to regain it later.

A Year Free From Caffeine

In October 2023, I noticed I couldn’t function without caffeine. Specifically, I was drinking at least 2-3 Dr. Pepper a day. Worse, if I ran out at home it meant struggling through the day. It felt like an addiction, so I quit cold turkey (that weekend headache sucked).

As of October 2024, I’ve gone an entire year without any caffeinated soda. However, I replaced it with sugary alternatives, which completely negated any health benefits that could be had. This year, I’ve started removing soda altogether to reduce my sugar intake.

Martial Arts

Martial arts have always been a passion i’ve mine. I earned a yellow belt while taking Karate for a PE credit in college. Sadly I didn’t continue because I was a poor college student that didn’t think it was worth going to the dojo and paying for sessions. This year my kids started doing Karate. After I was good and behind them.. 2 belts. I decided to join.

I’ve regained my yellow belt and plan to continue. It’s a great way to completely disconnect. Not to mention the practice for sparring got me moving which was really good for me.

Working Out

I’ve been gradually building a home gym to be able to practice Martial Arts at home as well as lift some weights. The beauty of it being at home is makes it easier to go at weird times. Sometimes even work out with my kids and encourage them to be more physically active as well.

Impact

These efforts have been paying off. So far, i’ve lost 12 pounds in the last four months. Not to mention I feel much better overall. I can even spar without feeling like I’m going to pass out!

Travel

I traveled internationally twice this year:

Brazil: A work trip brought me to Porto Alegre just a week before the airport was severely flooded.
London: I took my family on a trip to London. The kids first trip overseas. They were finally all old enough I felt like they would remember and appreciate it a lot more. Not to mention a lot easier to travel with when they can all walk around on their own.

Work / Professional

2024 brought some changes to my career.

Staff Engineer

At Rocket.Chat over the course of almost 10 years, I held various roles. Engineering Manager, Tech Lead, Team Lead, etc. From building a team, to taking our scope and spliting it up into 3 seperate teams. But Staff Engineer as a level I didn’t really know what it was or what I was supposed to do. So this year I tried to settle in and figure out what the industry considers a Staff Engineer and try to learn what I could.

This article and book with the same title I really recommend reading if you’d like to learn: https://staffeng.com/guides/what-do-staff-engineers-actually-do/

I tried to also spread our support of Staff Engineers so others coming into that role would have more direction. Its a role I really enjoy and realize I do even more so in retrospect.

Production Readiness

In the year’s first half, I focused on a cross team initiative: Production Readiness. I might be missing some great sources of information out there. But I felt like this was a subject very under documented on the web.

This process was a lot of talking to stakeholders and together:

Defining SLAs/SLOs
Identifying critical user experience factors
Api response time
Error rates
Error budgets

Then working backwards from there with the engineering team to identify the risks to being able to fullfill these targets.

Things like:

Building and participating in an on call team to actually be responsible for delivering targets
Implementing new metrics to gain visibilty into something critical
Removing code complexity from endpoints to speed up response times
Ensuring could dynamically scale to meet load
Ensuring could handle dependency failures appropriately
Ensuring alerts were properly set
Ensuring systems like Mongo, Redis, Nats etc were set to a higher SLA

If anyone is looking for good things to blog about.. This would be a good topic to see more information out there. If not I hope to get to it at some point.

Technical Writing

As part of that focus on Staff Engineer it was important to help with technical alignment. This means documenting and writing things down. This has been a passion of mine for a long time, but we seemed to always struggle getting stakeholders to go read our docs written in markdown, keeping the docs together so they were discoverable, and having a common agreed on format and system for these documents.

For a while I had been eyeing Oxide Computer’s Ellusive RFD Process. What really stuck out to me was not just documenting but trying to drive discussions. I really wanted to try it out, but their system at the time was part of a larger CIO toolset they had built.

So at the end of 2023 I wrote a tool that from what I could tell worked a lot like their tool. I very creatively called: RFD Tool

Over the course of 2024 I rolled this out. I also took a bunch of documents I and others had written and imported them into the system. From what I hear the final count at the end of this year was 114 RFDs.

I plan to write more about this system in a future post.

New Job

After nearly 10 years at Rocket.Chat, I transitioned to a new role as a Senior Software Engineer at a new company that helps build out fiber networks in communities and then provides software to allow the members of the community to sign up through a marketplace and chose their ISP, voice services and facilitate the provisioning of the CPE’s hooking them up to their chosen services. This change was important to allow me to reset and perform a rebalance of work/life. But also a good opportunity to learn something new, particularly in the ISP space which i’ve always found interesting.

Some times a change is required to help us realize what we value. I realize I find fulfillment in participating in architecture and direction, not just coding. This is something I’ll be focusing on in the new year.

Technology

Joining a new team meant working with new technologies:

Microservices: I’ve worked with microservices before typically by breaking part of a monolith off to enable more dynamic scaling. I’ve never worked with a microservice first architecture. While microservices can enable scaling and team parallelism, the complexity isn’t always worth it for smaller teams.
GraphQL: Although I see its appeal, GraphQL introduces challenges with network traffic, authorization, and increased boilerplate code necessary. I still prefer well-crafted RESTful APIs. A good article on this
gRPC and Protobufs: Defining services with Protobufs and generating REST APIs, openapi documentation, and gRPC Gateway code is really cool. I plan to explore this more.

This article by tailscale is a good read that has been top of mind: https://tailscale.com/blog/new-internet

Daily Tech Stack

This year my primary daily stack was:

Silverblue, an immutable OSTree OS,
A custom OSTree branch, Triiodide (i3 - 3 iodine atoms is a triiodide ion) that brings i3wm instead of gnome
Dev Containers
Ghostty for a performant terminal experience
A single 32" curved monitor plus a smaller secondary display for chat, video conferencing and music

Plan to write more about my immutable/Container based flow at some point soon.

2025

I don’t really do new year’s resolutions any more. I feel like uou don’t need to wait for a new year to make a change. Plus most people I see that only do resolutions one time a year.. Usually set themselves up for failure by trying to do too many at the same time. How many times did you set a goal to workout and buy a gym membership only to stop going after 2-3 months? I have.. several times.

But, I do plan to continue to focus on health, doing more writing, and focus on work I find fullfilling.

Here’s to another year of growth!

Fedora Iot, Greenboot, Systemd and of course DNS

Sun, 24 Sep 2023 00:00:00 +0000

This post is mostly as a resource for future me. But maybe it’ll come in useful to someone else.

This weekend I was working on provisioning a new NAS/Media box setup after having to do a bit of hardware shuffling. Long story.

I decided to try Fedora iot on a whim. Fedora iot has an interesting project called greenboot. The objective of the project is to define some checks for the state of the machine. When the machine is booting and the machine isn’t configured or working as expected it reboots and tries to correct. From what I understand it does this several times and then if unsuccessful it rolls back to the previous ostree commit. In an edge deployment scenario where you can’t always have your hands on the device and it almost assuredly will break… I can see ostree commits + this to being great to rollback when something gets broken. Much easier than in my previous post:

Adventures of the geek: Remote Password Reset

Anyways.. In this particular case i’m also installing pi-hole on it. This will act as the DNS server in my house. Blocking domains related to tracking, ads etc. But also lets me do other typical internal dns stuff, block list of inappropriate domains etc.

So I started off by just tying to run pi-hole:

sudo podman run -d \
 --name=pihole \
 -e TZ=America/Chicago \
 -e DNSMASQ_LISTENING=all \
 -e WEBPASSWORD=password \
 -v ./etc-pihole:/etc/pihole:Z \
 -v ./etc-dnsmasq.d:/etc/dnsmasq.d:Z \
 -p 53:53/tcp \
 -p 53:53/udp \
 -p 80:80/tcp \
 --restart=unless-stopped \
 docker.io/pihole/pihole:latest

This started throwing erorrs about :53 already being used.

Turns out that systemd-resolved listens on :53 with a stub-listener and by default all queries goto 127.0.0.1:53 and then through systemd-resolved to upstream dns resolvers.

To disable I had to create: /etc/systemd/resolvd.conf.d/stub-listener.conf

With contents:

[Resolve]
DNSStubListener=no

Then of course restart systemd-resolved:

sudo systemctl restart systemd-resolved

This let me start up pi-hole and I went on about my business setting up volumes, moving data around etc.

As part of it a reboot was finally needed. Oops… It wouldn’t boot.. get to login screen and then promptly reboot again. I realized this was greenboot. I let it go a few times but it never recovered.

So I finally intervened and modified the boot arguments from grub and added:

init=/bin/bash

To drop into single user mode.

I poked around and finally found: /usr/lib/greenboot/check/required.d/01_repository_dns_check.sh

This made me realize what had happened. pi-hole wasn’t up at this point and systemd-resolved is no longer running the stub. So this check was failing.

[aaron@super-terribly-named-box ~]$ dig google.com
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to 127.0.0.1#53: connection refused

; <<>> DiG 9.18.17 <<>> google.com
;; global options: +cmd
;; no servers could be reached

I found this issue: https://github.com/fedora-iot/greenboot/issues/113#issuecomment-1689929196

The suggestion is simple. Disable the service. Problem was I couldn’t even get logged in fast enough over ssh or console access to do this.

In single user mode systemd isn’t running of course so finding the unit file to quickly mask it didn’t work. And at the time I didn’t remember how to do it manually with ostree involved anyways.

For the record: /usr/lib/systemd/system/greenboot-healthcheck.service but still the problem of it being read only because of ostree and being in single user mode where rpm-ostree did not seem to work like I was used to. This is a subject i’ll be diving into at a later date.

I looked for a way via boot args to turn off greenboot. They don’t have any. But then I found:

systemd.mask=greenboot-healthcheck.service

Can read a bit more here

By adding this to the boot args it will mask that service from starting during that session.

From here then it was as simple as:

sudo systemctl mask greenboot-healthcheck.service

Now to go on to evaluate greenboot further. Seems like a cool idea.

systemd.mask bootarg saved the day!

How to Set Up Client SSL Certificate Authentication for Rocket.Chat

Tue, 23 May 2023 00:00:00 +0000

Are you looking to add an extra layer of security to access Rocket.Chat? One way to do this is by implementing client SSL certificate authentication. This authentication method requires clients to present a valid SSL certificate to authenticate themselves to the server.

In this post, we will walk through the steps to set up client SSL certificate authentication using Nginx.

Prerequisites

A server running Ubuntu
- Allow HTTP/HTTPS/ssh in security group
- Create a public IP and associate it.
A domain
Domain pointed to the server

Step 1: Install Docker

Lets do a quick and easy install of Docker if you haven’t already.

curl -L https://get.docker.com | sh

Now Lets add your user to the docker group so you don’t have to use sudo before every docker command.

sudo usermod -aG docker $USER

Normally here we’d say logout and back in.. but I don’t want to mess with it so I use:

newgrp docker

Step 2: Install Rocket.Chat

Now that we have Docker installed lets get Rocket.Chat up and running.

Start off with creating a rocketchat directory and grabbing the docker-compose file:

mkdir rocketchat && cd "$_"
curl -L https://go.rocket.chat/i/docker-compose.yml -O

Next lets fire it up.

docker compose up -d

If impatient or only came here to install Rocket.Chat you could access on port 3000 if your firewall is open. But I suspect you came here for seeing how to do client SSL. So lets carry on!

Step 3: Install Nginx

sudo apt install -y nginx

Step 4: Install Certbot

To help us get a valid certificate we are going to use letsencrypt but using a tool called certbot. This will help us make sure to keep the certificate valid as well.

sudo snap install --classic certbot

Now use certbot to generate and plug everything up for letsencrypt and nginx:

sudo certbot --nginx

You’ll be asked to provide a valid email and the domain set.

Step 5: Generate Certificate Authority (CA) Certificates

In order to do client SSL Authentication we’re going to need a CA.

Generate a key for your CA:

openssl genrsa -des3 -out ca.key 4096

Generate a certificate for your CA:

openssl req -new -x509 -days 365 -key ca.key -out ca.crt

Note what you’ve entered for Country, State, Locality, and Organization; you’ll want these to match later when you renew the certificate.
Do not enter a common name (CN) for the certificate.
Email can be omitted.
Note renewing certificate involves running the same command. If you need to remember what options you chose you can run:

openssl x509 -in ca.crt -noout -text

Move CA cert

To: /etc/ssl/private/client-cert-ca.crt

Update Nginx config

We need to add CA cert, turn on client ssl authentication and add location block:

ssl_client_certificate /etc/ssl/private/client-cert-ca.crt;
ssl_verify_client optional;

location / {
 if ($ssl_client_verify != SUCCESS) {
 return 403;
 }

 proxy_pass http://localhost:3000;
 proxy_http_version 1.1;
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection "upgrade";
 proxy_set_header Host $http_host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto https;
 proxy_set_header X-Nginx-Proxy true;
 proxy_redirect off;
 }

Step 6: Issue Client SSL Certificates for users

You can have your users perform most of these steps if you want. But the following are the steps needed to create a certificate to present as client authentication.

Generate key for user

openssl genrsa -des3 -out user.key 4096

Generate a CSR

openssl req -new -key user.key -out user.csr

A number of questions will be asked; answer each one, including the Common Name (CN) and email address. The CSR needs to be sent to the admin (or you if you are doing this for the user).

Sign CSR with CA

As the admin, take the CSR given to you or generated by you and sign the CSR and create valid certificate:

openssl x509 -req -days 365 -in user.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out user.crt

You’ll want to increment the serial number with each signing. Once the certificate expires, a new CSR doesn’t need to be recreated; the same one can be signed, which will create a new certificate tied to that public key.

Return Certificate

The signed certificate (user.crt) can now be sent back to the user along with the CA cert(ca.crt).

To be able to use in browsers and mobile generate a pkcs #12 using the user cert and key along with the Ca:

openssl pkcs12 -export -out user.pfx -inkey user.key -in user.crt -certfile ca.crt

Beware if you intend to install on Mac or iOS, you may need to use an older version of OpenSSL. More info here: https://developer.apple.com/forums/thread/697030?answerId=710429022#710429022

Step 7: Access Rocket.Chat using Client SSL Certificate

To access your application with client certificate authentication:

On iOS:

Add the file on Files app (depending on what you do, iOS will try to install it on the whole OS. e.g. copying from airdrop)
Go to new server screen and try the server URL just to confirm (an error message should show because you haven’t applied the cert yet)
Tap apply your certificate on the bottom of the screen and select it from the Files app
Try again, and it should navigate to the workspace info

On Android:

Install the certificate
Go to new server screen and try the server URL just to confirm (an error message should show because you haven’t applied the cert yet)
Tap apply your certificate, and the OS should prompt you to select a cert
Select it
Tap connect, and it should navigate to the workspace info

On Firefox:

Open Firefox and click on the three horizontal lines in the top-right corner of the window.
Select “Preferences”.
In the left-hand menu, click on “Privacy & Security”.
Scroll down to the “Certificates” section and click on “View Certificates”.
Click on the “Your Certificates” tab.
Click on “Import”.
Browse to the location of your client certificate file and select it.
Enter the password for the certificate if prompted.
Click “OK”.
Your client certificate should now be imported and ready to use in Firefox.
Visit the address and you should be prompted to select the certificate.

Congratulations! You have successfully set up client SSL certificate authentication for Rocket.Chat using Nginx.

Extremely useful Reference: https://fardog.io/blog/2017/12/30/client-side-certificate-authentication-with-nginx/

Adventures of the Geek: Remote Password Reset

Thu, 16 Jun 2022 00:00:00 +0000

Our adventure begins on a Monday evening. I received a call from a family member that lives a bit over 8 hours away.

I powered on my laptop today and it won’t take my password

Unfortunately the laptop runs Windows 10. I have made it a point to not use windows and haven’t for last several years. Primarily using linux or osx. Leaving me at a bit of a disadvantage.

So then I did what any self respecting computer guy does when need to find out the options. I duckduckgo’d it!

Options to do Password Reset

Found a few options

Password Reset via Security Questions

Apparently these weren’t set. Looks like it was introduced on Windows 10 Version 1803. Not sure if we originally installed before then or exactly what happened.. but didn’t have this.. so wasn’t an option.

Password Reset via Password Reset Disk or USB

We hadn’t prepared one. Awesome… After this is solved, making one is a must.

Password Reset via Microsoft Account

This was a local account.. not an option.

Password Reset via Safe Mode

Tried this..

I honestly don’t know what happened here. I tried to navigate doing this based on screenshots on the internet. Relying on my remote eyes and hands to describe what was happening.

It didn’t work.

Password Reset via Offline Windows Password & Registry Editor

More info about it can be found here: https://pogostick.net/~pnh/ntpasswd/

This one had a lot of potential. Looked like it would do exactly what I needed.

Requested my helpful remote hands find a flash drive and put in an older laptop in the house. Used teamviewer to connect. It was already installed this isn’t the first time i’ve needed to help out with something.

From there I downloaded a super lightweight tool called Rufus and put the ISO on the flash drive.

With a little luck and a little trial and error. We managed to boot from the USB. But kept getting “Kernel panic not syncing”. Even after trying all suggested work arounds.

This took several hours of verbally instructing these options to try, rebooting trying again.. Flashing an older version.

All with the same results.

We had spent several hours by this point. The idea was even suggested that would make unscheduled trip to visit us. Given the price of gas though and how rough it was for them to travel.. I really didn’t want it to come to this.

We decided to call it a night. Went to bed thinking how much easier it would be if I could just use my own eyes and hands. I’m a problem solver. I knew that if I was able to see it with my own eyes and use my own hands.. The whole try, process and iterate can happen a lot faster!

Being able to use my own eyes and hands

Sounds a bit cliché.. But it hit me in the shower Tuesday morning.

I’ve made livecd’s before… What if I could have a livecd like the one we had tried.. but based on a more versital distro like Ubuntu. Ubuntu typically boots up on just works about any hardware.

While i’m at it, what if it connected back to me so I could run commands my self? With linux at my hands.. I knew we could get it resolved.

Connect back to me?

Turns out this is actually extremely easy.

With ssh you can expose a local port remotely with a command like this:

ssh -i ~/path-to-ssh-key -R 4022:localhost:22 -N user@my-vps

It uses an ssh key and expose port 22 on the remote host on port 4022.

Creating your own livecd?

I’ll save you from some of the discovery. There are many ways to create linux livecds. A few of even with a GUI.

But, I wanted a way to do it quickly and repeatable with out having to use a GUI.

I’ve made some before manually but not very repeatable as it was always manually editing the squashfs and chrooting in and poking around.

I found this guide great example of loading a livecd as a docker image and then using docker to modify it

https://slai.github.io/posts/customising-ubuntu-live-isos-with-docker/

Perfect!

Lets make it!

The important aspects of the livecd would be:

Lighter weight - Ubuntu Server would be the base.
Automatically connect to the internet
Automatically run the ssh tunnel back to me
Automatically reconnect the ssh tunnel if disconnected
Have some preinstalled tools
- ntfs - i’d need to mount that partition
- ssh server - of course
- chntpw - most importantly in this case the tool we discovered thanks to the previous day that could modify the registry
Baked ssh keys for bi-directional communciation

Nice to have:

Don’t start up the installer it would be confusing
Customize boot menu so it doesn’t say “Install Ubuntu Server”
Include some text above the login prompt to communicate to the user

For more technical details see this github repo: https://github.com/geekgonecrazy/remote-recover-livecd

I set about building after my work day ended.

Attack of the remote desktop tooling

It came time to load the iso up on the USB drive.

Teamviewer showed its true colors.

They apparently decided since I needed to remote in two days in a row.. that I was exhibiting signs of commercial use.

Meaning apparently they start being extremely aggressive. My first session of the evening was barely open and then abruptly ended. I tried connecting again and it said something like: You can’t reconnect again so soon try again at: 17:55.

I waited until then and it showed another saying try again at 17:56. I guess a clock is off somewhere or the error time frame was wrong? It ended up being at least 5 minutes before could connect.

Here begun that pattern of wait 5 minutes.. Scramble to work for 1 minute before getting the session ripped away from me.

Extremely frustrating!

I did debate paying them. But appears they don’t sell monthly.. so I’d have to shell out $400+ just to ocassionally use it to help out family. Nope.. Sorry.

I went shopping for another tool. Even if I had to pay for it surely there is something better out there.

I posted about this frustration on mastodon

Big thanks to @amolith@nixnet.social for the suggestion

Rustdesk was perfect!

30 minutes or so later… Thanks teamviewer.. I had rustdesk installed and was able to move forward.

If you are using teamviewer I highly recommend switching to RustDesk. You can even self host the server portion if you want.

From there used Rufus to put on the USB and then had them plug ethernet into laptop along with the fresh USB.

We booted.. and it didn’t connect back up.

We had to call it a night. This ended night 2.

One more try

Wednesday came and I quickly realized it was netplan.. I think because I had removed the installer squashfs not thinking much about it.. It must have been initalizing a netplan file.

At the end of the work day I made some changes and built the iso again. Quick rustdesk and rufus later…

USB inserted and there it was!

I finally had access and the process could finally begin of resetting the password!

10 minutes later the password was reset.

That was a lot

Yes it was! 3 days to reset a password. Just a bit crazy..

But now I have a flash drive on site that if this ever happens again.. I can spin up my vps and have them insert the flash drive.

So all in all.. Another tool for the tool belt. Definitely might be closer to earning the Crazy in Geek Gone Crazy. :smile:

You left me hanging how did you reset?

Ok if you want the last 10 minutes.. they went something like this.

I mounted the hard drive with a quick:

mkdir /mnt/laptop
mount /dev/sda2 /mnt/laptop

Using fdisk -l to determine the drive and partition.

Then:

cd /mnt/laptop/Windows/System32/config

Here is where all of the registry hive files are for windows.

Now we use the chntpwd util we pre-installed

chntpw –l SAM

From here you follow the prompts to reset the password. Or in my case I went the path of just blanking the password out.

Rocket.Chat and the Matrix Protocol

Mon, 30 May 2022 00:00:00 +0000

Rocket.Chat now has federation support via the Matrix protocol.

This announcement is an exciting one to me. I’ve been working with Rocket.Chat since I found it almost 7 years ago. One of the things I recall from early on is the vision of being as ubiquitious as email. One of the beauties of email is not everyone is on the same domain or let alone even the same kind of mail server. Just all speaking the same language.

Rocket.Chat being able to talk the Matrix Protocol to me is a start of it achieving that.

It does this by registering as an AppService with a Matrix homeserver. This means Rocket.Chat is able to create users on the homeserver and then manage them. So when you opt to federate a room and invite a remote user using Matrix. Rocket.Chat creates your user on the homeserver and controls it as if they are one and the same user.

tl;dr You can now talk to anyone speaking the Matrix Protocol from Rocket.Chat.

How to setup Rocket.Chat with a Matrix homeserver

Matrix support in Rocket.Chat is currently considered alpha. Expect things to break
This guide also uses Dendrite which is beta

Video guide should you prefer that format:

Compute Requirements

If your Rocket.Chat workspace will only support a few users you can generally get away with a machine like:

2GB of RAM
2CPU
20GB of Storage

If you start using it more heavily you’ll probably want to increase to a couple of CPU and more ram. See the Rocket.Chat Documentation for more details.

In this case i’ve selected this digitalocean droplet:

I’ve also selected Ubuntu 20.04 as the distro.

Domain Requirements

You’ll need a domain you control. In this case i’m going to use geekgone.dev.

For federation generally the idea is that you are accessed much like you are by email.
So if I was to be reached at aaron [at] geekgone.dev then I would also be reached at @aaron:geekgone.dev.

Either way you need to decide what domain will you be reached at? We’ll call this the federated domain. Then you’ll also need to expose matrix and Rocket.Chat on their perspective domains.

In my case i’m going to use:

Federated Domain: geekgone.dev
Rocket.Chat Domain: chat.geekgone.dev
Matrix Domain: matrix.geekgone.dev

Go ahead and point those all at the machine you’ve provisioned.

Install some pre-reqs

We need a reverse proxy. I’m going to use nginx:

sudo apt update && sudo apt install -y nginx

Now we need docker:

curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh

Now for ease we’ll add our self to the docker group and assume it:

sudo usermod -a -G docker ubuntu
newgrp docker

Finally we need certbot. This is one of the quickest and easiest way to install it currently:

sudo snap install --classic certbot

Configure Lets Encrypt Certificates

certbot certonly --nginx -d geekgone.dev,chat.geekgone.dev,matrix.geekgone.dev

Setup Nginx Rules

Edit /etc/nginx/sites-available/default

server {
 listen 80 ;
 listen [::]:80 ;

 if ($host = geekgone.dev) {
 return 301 https://$host$request_uri;
 }

 if ($host = chat.geekgone.dev) {
 return 301 https://$host$request_uri;
 }

 if ($host = matrix.geekgone.dev) {
 return 301 https://$host$request_uri;
 }

 server_name matrix.geekgone.dev chat.geekgone.dev geekgone.dev;
 return 404;
}

server {
 listen 443 ssl;
 ssl_certificate /etc/letsencrypt/live/geekgone.dev/fullchain.pem;
 ssl_certificate_key /etc/letsencrypt/live/geekgone.dev/privkey.pem;
 include /etc/letsencrypt/options-ssl-nginx.conf;
 ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

 root /var/www/html;

 server_name geekgone.dev;

 location /.well-known/matrix/ {
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 proxy_pass http://localhost:8008;
 proxy_read_timeout 90;
 }
}

server {
 listen 443 ssl;
 ssl_certificate /etc/letsencrypt/live/geekgone.dev/fullchain.pem;
 ssl_certificate_key /etc/letsencrypt/live/geekgone.dev/privkey.pem;
 include /etc/letsencrypt/options-ssl-nginx.conf;
 ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

 root /var/www/html;

 server_name chat.geekgone.dev;

 location / {
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 proxy_pass http://localhost:3000;
 proxy_read_timeout 90;
 }
}

server {
 listen 443 ssl;
 ssl_certificate /etc/letsencrypt/live/geekgone.dev/fullchain.pem;
 ssl_certificate_key /etc/letsencrypt/live/geekgone.dev/privkey.pem;
 include /etc/letsencrypt/options-ssl-nginx.conf;
 ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

 root /var/www/html;

 server_name matrix.geekgone.dev;

 location / {
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 proxy_pass http://localhost:8008;
 proxy_read_timeout 90;
 }
}

Do a nginx -t just to make sure everything is ok.

Then:

systemctl reload nginx

Install Rocket.Chat and Dendrite

I’ve commited a docker-compose and config file in a public repo. So will start off with grabbing that:

git clone https://github.com/geekgonecrazy/rocketchat-matrix-federation-example.git matrix
cd matrix

Lets bring Rocket.Chat up:

docker compose up -d rocketchat mongo mongo-init-replica

Goto your Rocket.Chat install and finish the initial setup. In my case it was available at https://chat.geekgone.dev

Now goto: Admin->Settings->Federation and expand the Matrix Bridge section

In the Homeserver URL section use:

http://dendrite:8008

In the Homeserver Domain use your federation domain. In my case:

geekgone.dev

Now flip to enabled and click save changes.

Grab the Homeserver Token and the AppService Token and put them some place safe for a moment.

Rocket.Chat is ready. Now lets go back to the terminal and look at dendrite.

We need to edit the dendrite config file. Edit: config/dendrite.yaml

Replace geekgone.dev in this section with your federation domain:

global:
 # The domain name of this homeserver.
 server_name: geekgone.dev

Replace matrix.geekgone.dev in this section with your matrix domain:

 # The server name to delegate server-server communications to, with optional port
 # e.g. localhost:443
 well_known_server_name: "matrix.geekgone.dev:443"

Now we need to edit: config/rocketchat-registration.yaml Replace: hs_token value with the Homeserver Token you got from Rocket.Chat Replace: as_token value with the AppService Token you got from Rocket.Chat

Finally now we can start matrix up:

docker compose up -d dendrite

Lets try talking to another Matrix Homeserver

Now that we have dendrite up and Rocket.Chat up. Lets go into Rocket.Chat and try sending a message out to a user on another server out there speaking the Matrix Protocol. Can be another Rocket.Chat server, dendrite, synapse etc.

In this case i’m going to use an account from matrix.org @geekgonedev-demo:matrix.org

Create a channel in Rocket.Chat and then do: /bridge invite @geekgonedev-demo:matrix.org

The client you have connected on the other server you will receive an invite. Once accepted you can now type a message on either side and if all went well you should receive the message on the other side!

Note: if you invite another Rocket.Chat server it currently will auto accept

I’ll leave the server up for a couple of weeks if you want to try reaching me at @aaron:geekgone.dev

Happy chatting!

Rover Project: Update 3

Sun, 04 Jul 2021 00:03:00 +0000

Finally a more eventful update!

Last time I worked with building parts in CAD I was using solidworks or AutoCAD. I have no desire to buy either one of those right now plus I need something that will run on linux. So started investing some time into learning my way around FreeCAD.

Creating first iteration of the wheel

50mm radius for a 10cm/100mm diameter wheel

3mm thick

Based on the specs of the motor:

Features:
Rated voltage: DC 3V
Rated speed: 15000 RPM
Motor size: 38 x 20 x 15 mm/1.5" x 0.8" x 0.6"
Shaft diameter: 2 mm/0.08"
Shaft length: 10 mm/0.4"
Material: metal

The hole is 1mm radius for a 2mm diameter hole

Print time was going to take 2 hours and 9 minutes. So time to reduce some material because we don’t want to wait that long

This takes it down to 1 hour 21 minutes

After lots of wrestling with bed leveling and things not sticking..

While this printed we started looking at how we want the body to look. Designing a caster for the 3 wheeled robot we realized was a little harder then we really wanted. Plus we were inspired by a rover… so we are leaning towards a 4 wheel design with a box to house the brain in… and surface area on top for mounting things in the future

source

The wheel finished printing!

So… we had to try it!

Woot!

Queue the other wheels!

While we waited did a bit of soldering and put antennas on the 433mhz chips

Then set about figuring out basic programs and getting everything hooked up to breadboards.

Got the L239D H-Bridge working doing speed control.

Then after a long period of debugging… finally got the 433mhz transmitter and receiver talking. Ended up getting it working with the virtualwire library.

Next step get the chassis designed and printed!

Rover Project: Update 2

Sun, 04 Jul 2021 00:02:00 +0000

So the robot parts sat in my closet for a few months. I kind of forgot to order the parts.

Recently realized I really need to disconnect more fully from work in my off hours. When your hobby includes coding and in same language as you write for work.. Its a bit harder to keep your brain from making that jump to thinking about work.

So i’ve been on the look out for new hobbies.

I don’t recall where, but I saw someone making something with a 3D printer. Made me wonder how much they were now days. After some research I realized some where actually extremely affordable. After even more research.. I was able to determine that at least a couple of those affordable printers actually were decent.

The possibilities to prototype things seem endless. Not to mention.. I actually spent the first 2 years of my post highschool schooling at a VoTech school learning CAD. So this is perfect! Those 12+ year old skill can actually be put to use. I say that. But really i’ve identified lots of areas that those skills actually have helped me out in how I approach problems. But still.. To actually to use CAD.

So I bought a 3d printer.

I’ve already printed kids items and been praised as it being a great decision by my kids. Even my wife had me print several things for her baking projects.

The purchase of the 3D printer has me inspired to push forward with the robot project. Being able to draw up in CAD and print the wheels and chassis will help greatly in moving this foward.

Time to pick up where we left off.

When it got shelved we had realized we had the wrong motors and controllers. So first we need to correct that.

We needed a controller. From reading around looks like this is called an H-Bridge.

I found a couple of examples using the L293D seemed like a low cost chip that didn’t need a lot of additional components to work with. Also found this great write up about the chip set:

https://maker.pro/custom/projects/all-you-need-to-know-about-l293d

So picked up a few of those from Amazon.

Now we’ve got to find some DC motors. This might be a point i’ll have to iterate on. But as long as I can power them and they will move I think we will be good. Ordered a pack of DC Motor 1.5V-3V 15000RPM.

https://www.amazon.com/gp/product/B07JYM8H18/ref=ppx_yo_dt_b_asin_title_o01_s01?ie=UTF8&psc=1

I’m not 100% positive on these.. But looks like operates between 1.5-6V. Also claims 3 hours running on 2 x 1.5V AA batteries. So with 4 and other components maybe we can get at least 30 minutes?

A couple of articles I found on working with this H-Bridge:

Now to wait on these parts to arrive. Plus some tools needed to build out my electronics work area. Has been pretty barebones since Harvey. Time to fix that :)

Rover Project: Update 1

Sun, 04 Jul 2021 00:01:00 +0000

Parts came in!

Finally got time to work on it again. We wired everything up.

We gathered up some cardboard. We found something to trace and cut out 4 wheels. (you’ll notice this is defintely a deviation from the original sketch. But this was insisted on once the additional motors were spotted).

We also put together the transmitter on another breadboard

Now if you do electronics seriously at all you’ll probably notice straight away two different issues. I was tempted to omit it and just have the build log with final parts.. But this is part of the journey. I think it was important for kids to go through as well. Not everything works first time around. In fact probably doesn’t most of the time.

Issue 1: Those are stepper motors and drivers. 🙈 Yeah.. I don’t even know how I did that. I did place the order late.. but I even have some articles stashed away specifically for working with those drivers. So its not like I didn’t research the parts. We’ll chock it up to lesson learned though.

Issue 2: Those 433Mhz Transmitter / Receivers don’t have even their basic included antenna’s soldered on. Reading later.. with out an antenna range is like a few centemeters.

Needless to say we discovered both of these issues after several hours. I pretty quickly wrote up the arduino code for both. I wrote the code that would send a simple signal that would include a signal for both the left and the right side tank style.

My Plan was:

Forward: 01 FF 01FF
Backward: 02 FF 02 FF
Forward Left: 01 FF 01 99
Forward Right: 01 99 01 FF

Pattern being: {Left Motor Polarity} {Left Motor Speed} {Right Motor Polarity} {Right Motor Speed}

With the speed being 0-255 represented in hex. No CRC or anything at first.

This didn’t work.

So figured maybe I was being too clever and switch to transmitting strings and probably 10x variations.

So after still not even a hum from the motors.. I figured it must be just something wrong with the 433Mhz pair. I wanted to at least validate the motors worked. So I found an example program for those motors show them working. It was here and getting this to work… I realized these aren’t really the right motors for the wheels.

So we shelved it until I could go and order the right motors.

It wasn’t until later I actually realized I some how completely spazzed on putting antenna’s on and their range was only centimeters with out one.

Rover Project

Sun, 04 Jul 2021 00:00:00 +0000

My family is very STEM oriented. I grew up learning hands on at home. Taking things apart. Questioning how things worked and eventually learning how to actually fix those things I took apart. As much as possible we’re trying to encourage this for our kids.

They are pretty fortunate as they have good examples of this on both sides of the family. So they come by their curiousity about the world around them very naturally. Now days its not surprising to get asked questions about things even we don’t know or remember the answer to.

Not going to pretend some times the constant questions doesn’t get a little annoying. But just have to stop and appreciate the curiousity and help them learn more about the things they are actually curious about.

Anyways we often watch Rocket Launches together. Now days thanks to the likes of SpaceX, Rocket Labs and even the old school ULA and others.. we frequently get to watch something launch.

About a year ago on 07/30/2020 we watched the Mars Perseverance mission take off. This kicked off a flurry of questions.

What is it launching? What is the rover? What does it do? How will it land? How does it work?

Then after spending some time showing them and answering. I was left with one statement:

I wish kids like us could make robots

I like making things.. and I used to help a highschool robotics team build their robot for competition every year. I really missed it. So really this was a win-win-win:

I get to teach my kids how to approach engineering problems as best I know how.
They are really excited
I get to work on building something too

So off began the process!

Brainstorming

So we started right off with brainstorming. During brainstorming we came up with a quick sketch of some that it could look like:

A couple of things to note:

There is an arm
In the left corner there is a tablet controlling

We’re considering these stretch goals. We agreed that the first priority was getting a basic robot built.

But through brainstorming we identified the main pieces that will be needed:

2 motors to drive the wheels
2 wheels
A base to act as chassis and to put the electronics on
Transmitter / Receiver
Bread boards - I didn’t have any more.
Batteries
A brain - some sort of microcontroller

We also identified that we would need some buttons and a way to control. So we needed some buttons.

We weren’t too certain on the construction of it. We figured maybe CD’s would make ok wheels. We’ve seen robots with CD wheels. Also we have plenty of cardboard. Might work as a prototype.

I tried to involve the kids as much as possible on the research behind the parts to choose. But this part quickly bored them. So I went off and spent a few hours that evening locating all of the parts.

Landed on using the Ardunio Nano. This is mostly because info for arduino is a lot easier to come by. I don’t want to waste too much time on the code portion. I want to be able to explain and the kids be able to follow along with the code and how it works.

For the batteries I’m a noob on batteries. I didn’t want to go off and order some rechargable battery and it be wrong for our circuit. From what I could tell for the Arduino nano everything seemed to be 1.5V compatible so just picked up battery pack to put AA batteries in.

Motors… I’m writing this in retrospect… so I don’t really know why I ended up purchasing stepper motors and the controllers. Its of course not a waste… But it did delay the project.

Parts Ordered:

Now we waited on the parts to arrive.

OVH Serial over LAN (SOL)

Mon, 07 Sep 2020 00:00:00 +0000

Its been a while since I’ve worked with physical servers and typically when something happened I could pretty quickly get physical access to the box if things went pear shaped.

But when you buy or lease servers from a provider like OVH. Those servers might be sitting half way around the world. So if something goes wrong you need to some how virtually be in front of the keyboard.

OVH has a few different ways to access your servers console to interact with the machine directly.

From a Java applet (KVM)

This way while works across all of the servers.. requires Java Runtime.. Yuck. Not to mention it isn’t necessarily going to work on every machine.

Via your web browser (KVM)

This way is by far the best. But seems for some reason, i’m not completely sure why.. its not available for all of their servers. So its not really a consistent option

Serial over LAN (SOL)

This way is probably the most easy to work with. Can access over ssh from any device that can ssh. Which is pretty much anywhere, so very appealing.

So I really wanted to be able to get Serial over LAN working.

The Problem

Out of the box when you connect all you get is:

 Could not chdir to home directory /home/ipmi: No such file or directory
OVH
IPMI Serial Over Lan gateway
IPMI SOL on [Omitted ;-)].us
ipmiutil ver 2.93
isol ver 2.93
Opening lan connection to node 10.245.29.163 ...
Connecting to node 10.245.29.163
-- BMC version 2.00, IPMI version 2.0
Opening lanplus connection to node 10.245.29.163 ...
[SOL session is running, use '~.' to end, '~?' for help.]

Hitting enter here should drop you to a login prompt to login to the console. But nothing happens. Turns out this is because out of the box on OVH the linux kernel isn’t outputting or opening a tty on the serial port.

Solution

To make the linux kernel output console output and open a tty on the serial port as needed by Serial Over Lan you need to ssh into your server and modify your grub settings to instruct the kernel to open a tty on that serial port. Also to tell it the baud rate to use. The arch wiki is useful here.

From what I can tell this is typically /dev/ttyS1. But if it doesn’t work for you try /dev/ttyS0.

In the bios you can confirm the baud/bit rate. But in my case it was: 115200 and looking around that seems to be the most common.

First hop into: /etc/default/grub

and add:

 GRUB_CMDLINE_LINUX_DEFAULT="nomodeset console=tty0 console=ttyS1,115200"
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --unit=1 --speed=115200 --word=8 --parity=no --stop=1"

This will both set the default linux command args, but also will let you interract with grub over the serial console as well.

To make sure takes effect:

 sudo update-grub

If you want to verify take a peak in /boot/grub/grub.cfg and see if the kernel options are there.

If you would like to also be able to interact with bios and everything after POST you can also hop into bios via one of the KVM solutions and set this setting to enabled:

I want to thank the author of this blog post for setting me on the right track: http://divideoverflow.com/2016/07/Configuring_IPMI_on_OVH_server/

Tinkerbell or iPXE boot on OVH

Mon, 07 Sep 2020 00:00:00 +0000

Recently have had to work with a number of physical servers that need to be treated more cattle like than a bunch of pets. I need to be able to quickly and easily provision servers to fit a specific role with out having to go through a complicated setup process.

For this type of thing PXE or iPXE comes to the rescue. Allowing you to boot over the network. Simply rebooting the machine and then having it pick up its instructions and be provisioned like needed.

Tinkerbell

Because of this need a project called tinkerbell has come to my attention. Project website is: https://tinkerbell.org. Seems packet uses this or a variation of this inside of their infrastructure.

Tinkerbell is made up of a few components:

tink-server

This provides the state. It is made up of:

templates

Templates have a series of actions that are docker images to be ran.

hardware

This is a json representation of the machine, metadata, nics with their interface definition ip/mac address.

workflows

Workflows are essentially just a job. A match up of hardware and template for it to run. The state is updated as the job runs.

boots

This provides dhcp and the ipxe script to the server so it can boot. When it receives a request from the server for an ip, it talks to tink-server to find a record of that hardware. If it finds it then it returns it the ip given to it in the hardware definition and returns the ipxe script based on info from that hardware template.

osie

When boots gives the server an image to boot it gives it the osie image. When this image boots up. It has tink-worker. Tink worker talks back to tink-server to fetch any workflows meant for this device. It then runs these workflows in series of docker containers. So you can package all of your scripts/dependencies in a nice neat docker container.

hegel

Hegel seems to mostly just be a tiny little service sitting over tink-server to return to the server its metadata. Pretty much like you are probably familar with if you’ve used AWS. The metadata service.

Setting up tinkerbell

The easiest way to get started on your local machine is to goto: https://tinkerbell.org/docs/setup/local-with-vagrant/ and get started with their vagrant stack.

Of course if you are going to be running in OVH.. This won’t really be an option.

Pre-reqs

So first off you need a couple of things:

A place to run tinkerbell. I personally had a proxmox setup for little VM’s on a couple of baremetal machines. But if I didn’t I would run a sandbox VM on the public cloud.
A machine you want to boot over the network
A vRack. You’re going to want a private network to conduct your experiment on.
Your machines connected to that vRack. Make sure you don’t turn on dhcp on that vRack doing public cloud.
An ip range selected. Personally I tend to over provision: 10.1.0.0/16

Install tinkerbell machine

Their examples use ubuntu 18.04 so I also chose ubuntu 18.04. 20.04 seems to also work just fine.

Once you have that we need to ssh in and start configuring tinkerbell.

 git clone https://github.com/tinkerbell/sandbox.git

Now we need to hop in to the sandbox folder to start configuring:

 cd sandbox

Now we need to generate the environment variables used for all of the rest of the configuration.

Identify the interface attached to the private interface. You can use ip addr and find the interface with no ip. Mine was ens18

Then you need to run:

 ./generate-envrc.sh ens18 > envrc

Then you need to edit the file output unless your network choice is 192.168.1.1/24 in which case you can leave alone

If not you’ll need to change the following lines to two ips you wish to use:

 # Decide on a subnet for provisioning. Tinkerbell should "own" this
# network space. Its subnet should be just large enough to be able
# to provision your hardware.
export TINKERBELL_CIDR=16
# Host IP is used by provisioner to expose different services such as
# tink, boots, etc.
#
# The host IP should the first IP in the range, and the Nginx IP
# should be the second address.
export TINKERBELL_HOST_IP=10.0.5.1
# NGINX IP is used by provisioner to serve files required for iPXE boot
export TINKERBELL_NGINX_IP=10.0.5.2

Now you need to load the environment variables into the current shell so we can continue executing.

 source ./envrc

Now you need to run:

 ./setup.sh

Once that finishes you will then be able to startup the components needed:

 cd deploy
docker-compose up -d

Note: If you happen to come back and want to interact with the docker-compose in the future make sure to first load the envrc file. If in the deploy folder can just do: source ../envrc

Wala the stack is up and running!

Tweak boots service

I hope to be able to replace this section. But at the time of this writing.. the boots service uses a command in the ipxe script called “params” which can be found here: https://github.com/tinkerbell/boots/blob/1e1d60ac32bac18d9b3f1c07611cd3b3613ecec7/ipxe/script.go#L34

In the version of ipxe my servers are running the param command causes things to error out. From what I can tell this just means that ipxe was built with out PARAM_CMD build option. But since these are OVH servers there isn’t really anything I can do to ensure its added.

You can track this issue here: https://github.com/tinkerbell/boots/issues/78

To make it work on OVH first follow pre-reqs listed here: https://github.com/tinkerbell/boots/tree/1e1d60ac32bac18d9b3f1c07611cd3b3613ecec7#local-setup you’ll need go installed from package manager as well.

cd ~
git clone https://github.com/tinkerbell/boots.git
curl https://clbin.com/12XhH -o /tmp/ovh-boots-ipxe.patch
make
docker build -t quay.io/tinkerbell/boots:local-ovh-params-fix .

Then modify: docker-compose.yaml in sandbox/deploy/docker-compose.yaml and change tag to local-ovh-params-fix

Then in the sandbox/deploy folder run: source ../envrc; docker-compose up -d

Prepare OVH machine

In OVH by default they have an ipxe server listening on the public interface, and intercept all requests. So we need to setup a script through them to bootstrap things so we can boot from our LAN.

You’ll need API access from here: https://api.us.ovhcloud.com/console

Go down to: POST /me/ipxeScript and insert:

#!ipxe
ifclose net0
dhcp net1
set iface net1
chain --autofree http://10.0.5.1/auto.ipxe || exit

This script will:

close net0 the public interface.
get dhcp on net1
set net1 as the iface
chain and try to boot the ipxe script from boots. If it gives 404 then exit.

Then grab the name of machine you wish to use and goto GET/dedicated/server/{serviceName}/boot

Set bootType=ipxeCustomerScript

You’ll get an array back. If you aren’t sure which one is which you can call: GET/dedicated/server/{serviceName}/boot/{bootId}

This will tell you the name of the script so you can make sure it matches

Now we need to set our server up to boot this script: PUT /dedicated/server/{serviceName}

Add info to tinkerbell

Go back to your sandbox/deploy folder and run: source ../envrc; docker-compose exec tink-cli sh

Now you have the tink cli available to you.

You need to create and inject a couple of files

hardware.json

{ "id":"0fba0bf8-3772-4b4a-ab9f-6ebe93b90a94",
"metadata":{
"facility":{
"facility_code":"ewr1",
"plan_slug":"c2.medium.x86",
"plan_version_slug":""
},
"instance":{},
"state":"provisioning"
},
"network":{
"interfaces":[
{
"dhcp":{
"arch":"x86_64",
"ip":{
"address":"10.0.6.3",
"gateway":"10.0.0.1",
"netmask":"255.255.0.0"
},
"mac":"d0:50:99:d7:63:20",
"uefi":true
},
"netboot":{
"allow_pxe":true,
"allow_workflow":true
}
}
]
}
}

You’ll need to swap out the ip and mac address.

Add this hardware with: tink hardware push --file hardware.json

hello-world.tmpl

version: "0.1"
name: hello_world_workflow
global_timeout: 600
tasks:
- name: "hello world"
worker: "{{.device_1}}"
actions:
- name: "hello_world"
image: hello-world
timeout: 60

Add the template with: tink template create -n hello-world -p hello-world.tmpl

It’ll spit out an id.

Now we need to create a job to run this template. So we need to create a workflow: tink workflow create -t {template_id} -r '{"device_1":"d0:50:99:d7:63:20"}'

Swap out the mac address for the same address used in your hardware.json above.

Try it

Now for the fun part. Connect to KVM console for your machine and then reboot it.

If all went well you should see it boot up

Decision Fatigue in 2020

Fri, 28 Aug 2020 00:00:00 +0000

This last couple of weeks I finally took some time off of work to try and reset. I’m not always the greatest at taking time off. I take an extra day or two combined with the weekend for a mini-vacation around the Holidays to see family and it’s often the minimum time needed. This year with everything going on really had to force my self to step back and try to reset with a staycation.

It’s all too easy now days to get overwhelmed and stressed out. A couple of key contributors now days:

Decision Fatigue
World Circumstances

Decision Fatigue

There are many articles written about this topic that do a much better job than I could to describe this. A couple that a quick DuckDuckGo uncovered:

https://www.healthline.com/health/decision-fatigue

https://blog.rescuetime.com/decision-fatigue/

But in short.. if you are constantly having to make decision after decision. There is a point when it just gets mentally taxing to have to keep making decisions.

I personally loop into this what I guess you would call Context Switching Fatigue. It boils down to some of the same things. But basically if you are constantly having to switch between tasks it can also lead to fatigue.

World Circumstances

It’s super easy to look at everything happening in the world and worry about things you can’t control.

Covid19 - We watch as the number of reported cases climb.. people can’t agree on how it’s reported and it’s politicized. With so much conflicting reporting… We are faced with more of decision fatigue. What info is correct?
Protesting and Riots - same thing.. mixed reporting. While the initial reason for protesting feels like a no brainer… everyone is taking extreme views on how to solve it. We see a lot of reports on who is doing what… and here we are stuck with making sense of everything.
Social Media chaos and unrest - a longer topic.. but everyone seems to be looking for a cause. They just pick a side in a fight with little to no research and shame anyone with an opposing view. You don’t have to scroll far in your feed to see someone post something political.. then in the comments you see anyone that opposes the view point or even question it.. dog piled on. Peaceful debating is long gone.
Deaths - as if it’s not a hard enough year.. people still get older and still die from varying causes
Hurricanes - In my part of the world it’s in the middle of hurricane season.

Resetting and Handling

Reducing decision fatigue doesn’t have a silver bullet. But one way I’m handling it on my staycation is taking it easy and making as little decisions as possible. Spent the first week blasting through my honey-do-list and generally cleaning. Then just taking it easy, avoiding news and social media(except Hurricane Laura updates).

But long term my approach:

Cut down / Cut out Social Media - it’s filled with a lot of toxicity.
Avoid too much News - I admit I don’t do this as well as I’d like. If you spend too much time reading news, you will be forced to spend more time reading both sides and making a decision.
Simple and structured morning routine - for me this includes Bullet Journaling.
Organization - being organized helps decision fatigue a lot.
Value your own time. Just because someone DM’s you asking for something doesn’t mean you need to stop what you are doing now and respond. In fact busy mode / do not disturb is your best friend here. Not even seeing it prevents you from having to make a decision.

Don’t be afraid to take time off to reset! It’s hard to take time off. But it’s important to take care of your self!

A k8s Migration Concept

Fri, 01 May 2020 00:00:00 +0000

Surprisingly I don’t have a lot of kubernetes related content on my blog yet. This might just because I’m too busy using it to talk about it. But i’m a heavy user of it. To me it just makes orchestrating work loads easier.

One thing in general I don’t see a lot of talk about is migrating between infrastructure. Especially don’t see much for kubernetes.

So I wanted to share a high level concept set of steps that i’ve followed a couple of times to migrate between a couple of kubernetes clusters with minimal to no downtime.

Establishing a link between environments

The first step in this process was establishing a link between the old environment and the new environment. Most major cloud providers have VPC peering so can set this up pretty easily. So far all have been IPSEC. A few tweaks to the firewall later you can talk across.

Spin up new DB nodes

In my case it was some mongo nodes. Once they were up, joined them to the replicaset on the other side. One important thing here is they were set with 0 priority to prevent them from some how becoming a primary before we were ready

Proxy Endpoint

Once the DB is ready and ready to migrate. Established an endpoint from the new environment across to the old environments ingress provider. On caveat here is did actually have to spin up a second internal ingress provider that would handle proxying to load inside the cluster. But did not handle TLS termination. This is what we pointed the endpoint to.

Create ingress records

Now that the other side is ready to accept traffic over the peering and route the traffic to the appropriate place… Need to actually tell the new environment what routes to listen for and where to send them. So all of them just point to the proxy endpoint.

At this point you can make a request like:

curl --verbose --header 'Host: www.example.com' 'http://10.1.1.36:8080/url'

DNS

Now that requests are verified to go across you can swap over the DNS from old environment to the new environment. Making sure to take note of the TTL on the record. I actually decreased the TTL a couple of days prior to prevent having to wait as long.

So the traffic will flow into new environment and the proxy over to old environment

Migrate

Once give the time for the TTL time to have fully ran out. Could start migrating.

For us this process looked like:

Deploy Deployment
Deploy SVC
Swap ingress to point to service
Verify traffic hits
Spin down resource in old environment

DB Primary Switch

We waited until about half of the deployments had switched and then adjusted priority in our DB cluster so that only nodes in the new environment had priority and now all in old environment had none and then made our primary in the old environment step down.

Important to make sure your application can actually detect and handle this. Otherwise it won’t work.

Finishing up

Once it was all over only thing left to do was to clean up.

To me the magic here is in being able to change the DNS and traffic still be working with some minimal latency increase.

Things can definitely go wrong. You have to plan for them. This is also why giving your self a maintenance window is also extremely important.

Also if you know anything about working with the kubernetes api.. consider writing something to do this.

Quarantainment

Fri, 01 May 2020 00:00:00 +0000

I don’t think its likely a surprise to anyone if I say I’m typically an introvert. I stay at and around home a lot in general. But when its no longer by choice.. kinda takes the fun out of it.

We identified early on that we needed to keep our selves busy and entertained.

Started out the quarantine with me teaching my wife to play chess. So pulled out the ‘ol Mario chess set. Yes its awesome!

I’ve wanted to learn to play go for a long time. Pretty much ever since seeing AlphaGo compete against the top players. So I figured what better time to learn? Picked Goban and stones or Yunzi as they are apparently called.

Proceeded to play a couple of very noob games of go.

The same board doubles as a chess board on the other side so have been alternating between this board and the other for playing chess.

Even taught my kids. They now can all correctly identify the pieces and how the pieces can move.

Picked up a few more various card games and board games to add to our collection. We already regularly enjoy family game nights.

Have spent a fair amount of time in my backyard. Got some patio furniture and a Hammock.

I’m pretty sure amazon and UPS hate us.

I also bought some controllers and loaded a raspberry pi up with retropie.

So far its working. We have stayed fairly sane and we are coming out the other side a bit better then we went in.

Respecting Privacy in Analytics

Wed, 29 Apr 2020 00:00:00 +0000

There is a reason why people tend to add analytics to sites. I’d say most people add for completely honest reasoning. They just want to be able to see how their site is doing and if anyone actually shows up.

But its really important to go down the path with caution. If you dont carefully select and think through your analytics… You end up spying on your users likely without even realizing it.

Let’s imagine you read the exciting sales pitch from AssociatedIn on how adding their analytics will let you see the types of users or businesses visit your site. Seems innocent enough right?

Lets put on our privacy detective hat!

So you turn it on and they start giving you this data.. How are they able to give you this info? The only way they would know anything about your users if they send something to them selves right? So of course they do this. But the only way them sending the page to themselves would be useful if they included something about the user. So they have to some how identify you or to see if unique.

Maybe they see you are logged in to their service? Ouch.. But ok ones easy.

Maybe they fingerprint you by grabbing an assortment of clues the browser gives away? Could be.. To make it even easier on themselves maybe they drop a cookie in the browser so next time that user visits a page they will know it’d the same user. Oh it happens to also send to any other site using their analytics?

This just went from analytics straight on over in to tracking territory. Now by getting some analytics about the type of business your users run.. You have traded it for your user being tracked.

The tracking company is then aggregating this data, building models about users, maybe selling it? Or maybe they aren’t so blatenly nefarious.. maybe they are just offering a paid plan offering advanced abilities. Your users data that you are giving them now let’s them make money.

Seemed innocent though right?

Super important to think about the type of information we collect about our users. Is that bit of data worth allowing a tracker to invade your users privacy?

Ok so what should I use?

Great question! First off self hosting an open source analytics tool can be a great way to know exactly what data you collect and how its collected. Also you ensure that data isn’t then being aggregated to track your users across the web.

I’m still on the hunt for good options but will update this list as I find out about them. But here are a couple:

Matomo

https://matomo.org is one recommended by many people in the open source community. Full featured alternative to the likes of Google Analytics.

Fatham

Recently came across this one. I’m actually using it for my site and inspired this topic. Feels great for use cases like mine.

Its kind of confusing.. But apparently their first version is/was open source. Then they rewrote it and this version is not open source but available on their SaaS offering.

But the open source version can be found here: https://github.com/usefathom/fathom

What I like about this is I can see exactly what is transmitted and what is stored.

Transmitted: here Saved: here

Here is a sample of my brief moment on hackernews yesterday. But what you see there is it. Super simple, but does the job.

Think about privacy the next time you start adding analytics to your website. Think about what you are making your users give up in exchange. Do the right thing 🙂

De-Googling: My Progress

Tue, 28 Apr 2020 00:00:00 +0000

I wrote a bit about my reasoning to De-Google here. Now I want to write a bit about my progress, the struggles and what I ended up with.

Search

Or Google as many think of it. This one I have toyed with off and on with over the years. But a while back probably around a year ago.. I was listening to a podcast. I think either Ask Noah Show or one of the Jupiter Broadcast Linux podcasts and was inspired to challenge my self to switch and see how it went.

I switched to:

Duckduckgo

It was one that I had tried direct comparison searches with before.. Every time I would feel it wasn’t as good. But I went ahead and switched it as my default on my phone.

At first I noticed what felt like results not showing. Felt like I was digging a lot. The same sort of thing I see everyone else complain about and turn away. But I decided to stick it out.

Before long I stopped noticing. I also almost always find the result on the first page. No different than Google.

What I’ve come to realize is I’m no longer depending on what Google knows about me to help me find what I want and instead I have become better at searching.

Google knows you, it knows the typed of things you would look for. It farms data on lots of people just like you. So of course its able to fill in the gap.

I’ll take training my brain my privacy any day!

Browser

I almost forgot to add. I actually feel kind of bad.. I help move a lot of people to Chrome from other browsers. 😔

I have to give Google major props for Chrome. The web was getting a bit stagnant. With Chrome it feels like they gave the web the kick in the pants we needed to evolve.

But it’s still Google and my browsing habits is my data. So I switched to:

Firefox

Firefox has gotten a lot better! Like night and day difference. Its fast, lighter on memory, looks a bit better and perhaps most importantly…

Its maintaining some diversity. Most browsers now use chromium or at least blink under the hood. Competition is good. Firefox’s phenoix like rise from the the ashes is proof enough of this.

I keep chromium and brave around as a backup for times when Firefox doesn’t cut it.

Email

I drug my feet for the longest time on this. I had a grandfathered in Google apps free account for my domains email. So I’m already in better shape then most.

I started off in a similar way to search. I first started using a non-gmail account on my phone. Desktop I used Thunderbird or Mail.app on osx. This started to train my self to not really care about the gmail features.

Then after really dragging my feet some more… I eyed protonmail.. But didn’t really feel like the encryption they provided and plan options ever really was that great of a deal.

I switched to:

Self-hosted with mail-in-a-box

This made it so simple! Simple enough for me to spend half an hour to set it up a $5 droplet on DigitalOcean and have it receiving email for several domains and a couple of new ones 🙂

Took another few hours to fail miserably with the terrible Google Takeout export of my email. Then force Thunderbird to fully download all of my mail and then push it all back up. Moral of story just use IMAP and you can move your mail anywhere.

I’m not 100% satisfied with this. I think in the future I will use some sort of docker based solution or roll my own. So I can self host a few more things a bit easier. But I got through the hardest part… I got my data back.

Phone

One of my biggest tie-ins to Google has actually been mobile. I’ve had a nexus or pixel for as long as I can remember. I tried Samsung and absolutely hated the skin they threw on it.

I tried ROMs like Cyanogenmod but it was always so buggy.. My phone is the one device I need to just work.

I tried just switching apps but I’d find my self right back into Google Assistant, Google Now etc…

So I switched to what only an insane person would:

iPhone

I probably just lost 99% of you 🙈

But doing this in one fell swoop I cut off a lot of temptation and random little Google apps.

This also let me see how the other side is. To be honest not that different.. I haven’t really missed anything.

But I think this really helped cut my reliance on Google and I feel a lot less reliant on any particular phone OS.

I’m keeping my eye on pinephone. Very exciting work being done. I’ll be buying one to play with before too long.

Photos

All photos used to backup automatically in full resolution to Google Photos. By having original pixel I got this for free.

Switched to:

iCloud

Yeah… I’m still looking for a good place to store this. But at least I’m paying for this service and am not the product.

I’m open to suggestions on self hosted. I’ve already pulled down my 2-300GB Google Photos archive.

Contacts/Calendar

Another byproduct of using Android my Google account was default.

Switched to:

iCloud

I’m thinking about using nextcloud.

Docs

I never really used Google Drive or any of the gsuite.

Switched to:

Hackmd

I like writing markdown. Downside… typing on mobile doesn’t work very well. Especially copy and pasting or moving cursor.

Nextcloud might fill this gap. But I don’t feel a void to fill right now.

Google Hangouts

I think I’ve used every incarnation of messenger Google has had and killed off. I had all of my family on Google Hangouts in group conversations.

Switched to:

Yeah I know.. One evil for another. I was traveling and one of the services I could use for free and worked best in Brazil was WhatsApp. Moving my family, in-laws, etc off is battle for another day.

I would like to get them all using a self hosted Rocket.Chat workspace. One day!

Edit to clarify: This was replacing my need for extended family communication groups. My wife’s family my family and extended family thats pretty much it. For other communications I use Rocket.Chat for team collab, signal, keybase chat or sms(which if they do happen to have iMessage does convert). I definitely do not trust facebook any more than Google.

Music

Used Google Music a lot and actually really liked its recommendations. But even before my De-Googling exercise I noticed that they started pushing YouTube Music. Rumor is it will be replace with YouTube Music. My guess is to leverage all of the music brought on by artists with vevo.

Switched to:

Spotify

Its one of the few services I don’t mind paying for. It does an okay job at playing music I might be interested in.

Todo

Looking through my phone at the Google Apps installed.

Youtube is the only app/service I visit any more for personal use. I’ve tried using invidious.. But I don’t know how much this really helps. Also if I self host then the data is then tied to me… so kinda defeats the purposes.

When ever I do find my self having to access Google I do so in a separate Firefox Container.

Overall I feel like I’ve done a great job removing my reliance on Google! I have a feeling I will be self hosting a lot more in the future. Maybe more on that another time. 😉

De-Googling

Mon, 27 Apr 2020 00:00:00 +0000

This is not by any means a new idea. A quick search around the web will return many results of people doing this. My reasons are likely similar to many others. Privacy being the most common shared reason.

Ok that’s it I spoiled it! Now you don’t need to read on 😜 But I’ll explain anyways.

Privacy

Google gives their applications and services away for free. This is extreeemly attractive! We humans are cheap! That’s not totally fair.. But I think in regard to software it holds true.

We will buy the best hardware in the world and then we hunt for the cheapest software we can find. But this is a rabbit hole.. I won’t go down today.

What we often forget(my self included) is to stop and ask your self a question:

How are they able to pay people to make this if they are giving it away for free?

Don’t know? Let me help you out. You are paying with your data!

Of course there are exceptions that I won’t go into here. But for the most part this holds true.

Google specifically is an ad company. This is how they make money. “But I thought Google was just a search engine?” Nope they historically make most money on ads. Here is one example of many out there talking about this. So your data feeds their ad machine.

Creeped out yet? Me too..

Google Grave yard

A less common reason.. I’ll try not to sound too bitter here. Bear with me. But Google has left in its wake a trail of bodies.

A few that I used to use:

Google Wave - so cool
Google Reader - still don’t have one I like as much
100 different messaging products - I’m exaggerating… but only slightly
Many more

Its enough that there is a dedicated website just to products Google has killed:

https://killedbygoogle.com

If they can’t figure out how to get their ad clicks or get bored (seems to be the case of messengers) to the chopping block they go!

Diversity

It goes a bit beyond Google for me. Its more like a way of life. As the old saying goes:

Dont put all of your eggs in one basket

What if a company went out of business and took all of your data with them? This has happened.

Diversifying the companies you use for services you can’t host your self is like hedging your bets.

Thats it folks! Hopefully my perspective is different enough to be interesting. 😜 I might follow this up with how I have de-googled.

Hacking in the Physical World

Sun, 26 Apr 2020 00:00:00 +0000

Hacking is often a term used to mean malicious behavior. But I believe hacking is a mindset. It’s about trying to be creative.. Taking things and making them work in a different way. It’s about not letting your self be your own limiter. Just because you’ve never done something before… doesn’t mean you shouldn’t try.

I don’t consider my self good at carpentry. Or using screws, hammers, power tools etc.. I’m also sure ~~some~~ most of the work makes professionals cringe just like I do at code by a novice.

But when a project comes around that needs done.. I think the only way to learn is to do.

When ever I can I like to help on projects where someone knows what they are doing. This way I can slowly build up some tips and tricks.

I’ve always been a tinkerer with a very curious mind. My parents love to tell the story of me as a young kid ripping the thermostat off of the wall so that I could see how it worked.

Over the last few years I feel I’ve gotten a bit more comfortable with power tools. Hurricane Harvey hit me and my in-laws houses. Fortunately for us we were renting. But my in-laws have spent the last few years gutting and remodeling their house. A lot of opportunities to learn there and they have gotten to make the house like they want it. Look for the silver linings right?

Anyways.. Few years back we bought a house and I inherited an amazing fence . I’ve done repairs in the past replacing pieces that butt up against my neighbors yard. It was an urgent repair to keep the neighbors aggressive acting dog from barging through the very weak pieces. The side has always leaned out a bit towards the road.

What needs to happen is I need to pull all planks off and fix it properly. But until I have resources to do so I have attempted to straighten some and strengthen it.

First thing was obvious.. some of the bottom 2x4s weren’t even connected. 🙈

The second major issue is the top 2x4s were layed flat and super bowed.

I loosened some pieces around the beam leaving the section unhooked from the post and then since mostly doing by my self made a make shift pulley out of rope and stake in the ground.

Once they were as straight as I could.. cut a few spare 2x4s put them in as square as I could and then removed the bowed pieces.

Anyways… it’s not pretty. But it will last a little longer until I can do it right.

Hopefully interesting in some way! If not we’ll try again. I’ve got several more projects needing done 🙂

100 Day writing challenge

Sat, 25 Apr 2020 00:00:00 +0000

Hello its me again your friendly neighborhood Geek Gone Crazy.😜 Its been a bit less time since my last post than normal. Recently I came across a challenge on mastodon to do a 100 day writing challenge. Inspired by the 100 days of code.

The idea being much like 100 days of code it encourages a person to sit down to write more often. Of course by doing this you get better at it.

As you can see from my posting history here.. I don’t post a lot. But I do enjoy writing, I do a fair amount of writing for work and have always wanted to post a bit more. So.. here goes nothing!

Here’s to hopefully a few more posts!