blog.fidelramos.net - Software

Vibe Coding Escapades: Venice.ai KDE widget

2026-06-25T00:00:00+00:00

The Problem

I’ve been using Venice.ai (referral link if you want to sign up) for quite some time now for all my artificial intelligence needs. I love it because, unlike most AI platforms, it’s private:

When you use Venice’s own models (open-source models running on their own servers), they don’t keep a log of the prompts, meaning your usage is private. Most providers keep logs of everything, and even worse are those that identify you personally (say, through your Google account) so they can link all your conversations to create profiles with a very dangerous level of detail.
When using external models (GPT, Claude, Gemini, etc.), Venice acts as an anonymizing proxy, meaning the final model doesn’t know who’s making the request. Obviously, they receive the full prompt and will log it, but as long as you don’t include personal information in it, they shouldn’t be able to identify you.

Another thing I like is that they have a token system through which I enjoy a Pro account (with 100 VVV), and I also have credits to spend every day (1 DIEM gives you $1 in credits every day, forever).

The problem was that sometimes I’d be coding in Opencode and suddenly see a message saying I’d run out of credits. I wanted a widget on my desktop (KDE) so I could see at a glance how much I had left to spend for the day. In the past, this would have been a project buried in the dustbin of history, but after discovering whatcable-linux, I realized that creating these widgets was well within my reach.

Venice.ai KDE widget over a retro sun

Implementation

I started a new project with Deepseek V4 Pro on Opencode. As I mentioned in the previous article it had just been released and promised results similar to Claude Opus at a fraction of the price, so I wanted to test drive it on real projects.

The first step was to generate an AGENTS.md file with the project’s objective and structure. Then I asked it to document the Venice API we were going to use. So far, so good.

The next step was to generate a static widget that would display the information I needed, but without interacting with any APIs for the time being. This is where Deepseek failed miserably: the QML files (the language specific to the Qt library used by KDE) it generated wouldn’t even compile; the widget wouldn’t load at all. After several attempts, I had to give up and switch to Claude Opus 4.7.

Opus discovered that the problem was that the widget was being cached by KDE, so even if I installed a new version, the old one kept loading. Deepseek had probably fixed the problem, but I was still seeing the old cached version.

Opus found a way to restart Plasma so that it would load the new version of the widget, and with that, it compiled and displayed correctly. From there, the process became much more iterative.

Next step: connect to the actual API. To simplify things, I asked it to read the token from an environment variable, and Opus got it right on the first try. I could already see my real data! As a personal project, I could have stopped there, it was already useful and did what I needed, but I wanted to polish and publish it in case others wanted to use it, so I iterated a bit more.

The next step was to store the token securely, making sure it wasn't stored in plain text anywhere. I asked Opus to explore some options:

I decided to go with secret-tool because, although KWallet is KDE’s native system, it has long been integrated with libsecret (and therefore with secret-tool). Opus once-shotted it (minus a startup bug).

The rest involved iterating on the user interface (gotta love those transparent widgets), tweaking colors and positions a bit, and adding some configuration options; nothing major worth mentioning.

Result

After spending the weekend tinkering, the result was venice-kde-widget.

Venice.ai KDE widget

I use it every day, and it’s been working like a charm. :)

Lessons

The initial problem (that the widget was cached by KDE Plasma) made me realize that many models require more specific instructions than others; they’re less creative. If I had asked them to consider alternative reasons for why it kept failing, they might have figured it out, but I myself didn’t think of the possibility that it was cached. Opus was the only one proactive enough to think, “Let’s confirm whether the code that’s actually running is the one we have in the repository.” It was smarter than me, all things considered.

That proactivity is something Simon Willison wrote about in relation to Fable 5. This experience suggests that this applies to other models as well. In other words, Claude Opus 4.7 was more proactive than Deepseek V4 Pro. I’m curious to know if proactivity can be measured across models, and if there are any studies on the subject.

Another lesson is that for a project using such a niche technology (KDE widgets in QML), most models won’t yield great results, they likely lack training. Perhaps the results would have been better if I had told them to look at the code of other widgets to use as a reference. Opus ended up doing just that for some problems he couldn’t solve: he looked at the other widgets installed on the system to analyze their code. It probably would have been a good idea to search for similar widgets and request an analysis before implementing. Or even better fine-tune a coding model on KDE widgets, but that would be way out of scope.

The results from this project, along with those from pelican-copy-code in the previous article, have led me to significantly reduce my use of Deepseek V4 Pro; it’s cheaper, but it’s not worth it if it leads me down dead ends with code that doesn’t work and needs to be fixed.

I continued using Kimi 2.6 as my default model, mainly because it’s private on Venice.ai. That changed with GLM 5.2, but that’s another story.

Self-hosted Tailscale, Part 4: Tailscale-Only Web Services

2026-06-04T00:00:00+00:00

I've been running a home server with a bunch of self-hosted services (Immich, Jellyfin, Nextcloud...) behind a single Caddy instance using caddy-docker-proxy (documented in this post). Some of them have genuine public uses, e.g. sharing a folder in Nextcloud or an album in Immich, or connecting to Jellyfin through another device (like a Smart TV). However the overwhelming majority of my interaction with those services is personal, from my own devices, and exposing them to the public internet is a considerable security risk, as they are entry points to my internal systems.

The solution now that I have Tailscale running is to make as many services as practicable accessible only through the tailnet. To achieve this I will run two Caddy instances in parallel: one bound to the public interface, for public-facing services; another bound to the Tailscale interface, for tailnet-only ones. It wasn't immediately obvious how to do that with caddy-docker-proxy, but I got it working without dirty hacks.

This post is the write-up of the setup I landed on, and the pile of gotchas I hit getting there.

The architecture

Two separate Caddy instances in Docker, both using caddy-docker-proxy:

caddy-public: bound to the LAN IP, reached from the Internet via the router's port forward on 80/443. Handles public services.
caddy-tailscale: bound to the host's Tailscale IP (100.x.y.z), reachable only from devices on my tailnet. Handles private services.

Each Caddy watches Docker labels with a different prefix, so services opt in to one or the other (or both, rarely) by using the appropriate label prefix and joining the appropriate Docker network.

Private services use certificates issued via DNS-01 challenge through Gandi (my DNS hosting provider). Public services keep using normal certs via HTTP-01.

Here's the high-level picture:

Internet ──► router:443 ──► host:LAN_IP:443 ──► caddy-public    ──► headscale
Tailnet  ──► host:TAILSCALE_IP:443          ──► caddy-tailscale ──► immich, jellyfin, nextcloud, etc.

Why two Caddys and not one with an IP allowlist?

I considered this. A single Caddy with a remote_ip matcher restricting certain hostnames to 100.64.0.0/10 would have worked. But:

Port 443 would still be reachable from the internet for those hostnames. I'd be relying on Layer 7 filtering to keep attackers out of services that shouldn't be WAN-exposed at all.
Private services need DNS-01 certs (because Let's Encrypt can't reach a 100.x address for HTTP-01), public services don't.
One forgotten Docker label and an internal service is on the public internet.

Two separate Caddys gives me network-layer isolation: the private services literally aren't listening on a WAN-reachable socket. That's a much stronger guarantee than "listening but configured to reject".

The Docker Compose setup

Here's the essential shape (with some env vars for clarity):

services:
  caddy-public:
    build:
      context: .
      dockerfile: Dockerfile.caddy
    image: caddy-docker-proxy-gandi:local
    container_name: caddy-public
    restart: unless-stopped
    environment:
      - CADDY_INGRESS_NETWORKS=caddy_public
    ports:
      - "${LAN_IP:?LAN_IP must be set}:80:80"
      - "${LAN_IP:?LAN_IP must be set}:443:443"
    labels:
      caddy.email: ${CADDY_EMAIL}
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ${CADDY_FILES_PATH}/public:/data
    networks:
      - caddy_public
    dns:
      - 1.1.1.1
      - 9.9.9.9

  caddy-tailscale:
    build:
      context: .
      dockerfile: Dockerfile.caddy
    image: caddy-docker-proxy-gandi:local
    container_name: caddy-tailscale
    restart: unless-stopped
    environment:
      - CADDY_INGRESS_NETWORKS=caddy_tailscale
      - CADDY_DOCKER_LABEL_PREFIX=caddy_ts
      - GANDI_BEARER_TOKEN=${GANDI_BEARER_TOKEN}
    ports:
      - "${TAILSCALE_IP:?TAILSCALE_IP must be set}:80:80"
      - "${TAILSCALE_IP:?TAILSCALE_IP must be set}:443:443"
    labels:
      caddy_ts.email: ${CADDY_EMAIL}
      caddy_ts.acme_dns: "gandi {env.GANDI_BEARER_TOKEN}"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ${CADDY_FILES_PATH}/tailscale:/data
    networks:
      - caddy_tailscale
    dns:
      - 1.1.1.1
      - 9.9.9.9

networks:
  caddy_public:
    name: caddy_public
  caddy_tailscale:
    name: caddy_tailscale

A few things worth calling out:

Separate label namespaces. CADDY_DOCKER_LABEL_PREFIX=caddy_ts tells the tailnet Caddy to read labels starting with caddy_ts. The public Caddy uses the default caddy. Services opt in by prefix, no possibility of cross-contamination.
CADDY_INGRESS_NETWORKS: Each Caddy is scoped to discover only containers on its own network. Without this, each Caddy would try to proxy every labeled container it could see via the Docker socket, and they'd fight.
Separate /data volumes. The two Caddys have their own certificates storage. Sharing would work but separate is cleaner and removes an edge case.
Binding to specific IPs, not 0.0.0.0. This one cost me an embarrassing amount of debugging time (see below).
The :? guards on env vars. Compose silently falls back to 0.0.0.0 if the host-IP part of a port spec is empty. The :? syntax makes it fail loudly instead. After getting burned by this twice, I will always use this kind of guard when setting up services that must be bound to a specific IP.

Building a Caddy image with the Gandi DNS plugin

I use Gandi as my DNS provider, and luckily Caddy has a plugin for it. However the lucaslorentz/caddy-docker-proxy image doesn't include any DNS plugins, and they have to be compiled in, you can't add them at runtime. I built a small custom image by creating a Dockerfile.caddy file:

ARG CADDY_VERSION=2

FROM caddy:${CADDY_VERSION}-builder AS builder

RUN xcaddy build \
--with github.com/lucaslorentz/caddy-docker-proxy/v2 \
--with github.com/caddy-dns/gandi

FROM caddy:${CADDY_VERSION}-alpine

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

CMD ["caddy", "docker-proxy"]

Then docker compose build caddy-tailscale would trigger the image build.

This pattern is common and documented to make custom Caddy images.

A key detail is the final CMD. caddy-docker-proxy runs in a different mode than the normal Caddy server, and you have to override it explicitly.

For the Gandi side, I created a Personal Access Token (not a legacy API key) with "Manage domain technical configurations" permission. I stuffed it into .env as GANDI_BEARER_TOKEN.

Moving services to the tailscale Caddy

With DNS-01 working, now it's time to move my web services to the tailnet.

The changes are easy: put the services in the caddy_tailscale network, and change the caddy labels to caddy_ts.

For example my Immich service (only the relevant parts):

immich-server:
  labels:
    caddy_ts: immich.ts.fidelramos.net
    caddy_ts.reverse_proxy: "{{upstreams 2283}}"
  networks:
    - caddy_tailscale
    - immich

Beautiful, isn't it?

DNS setup

There are no public DNS records. Instead I rely on Blocky's customDNS (see Part 2) to create DNS records that will only be visible to my tailnet devices.

In Blocky's configuration:

customDNS:
  customTTL: 1h
  filterUnmappedTypes: true
  rewrite:
    ts.fidelramos.net: tail   # allows 'X.ts.fidelramos.net' as alias for 'X.tail'
  mapping:
    # Friendly names for services on tailnet
    immich.tail: 100.64.0.1
    jellyfin.tail: 100.64.0.1
    nextcloud.tail: 100.64.0.1

This requires manually managing the entries and IPs, but this being a humble home network they don't change that often and it's a reasonable compromise.

Verification

After docker-compose up -d I check the logs of the containers that everything looks good. Both caddy instances spin up correctly. caddy-tailscale issues new certs through DNS-01 challenge.

In the host I verify that each caddy listens on their bound IP:

$ ss -tlnp | grep -E ':80|:443'
LISTEN 0 4096 192.168.1.50:80   0.0.0.0:*
LISTEN 0 4096 192.168.1.50:443  0.0.0.0:*
LISTEN 0 4096 100.64.0.1:80     0.0.0.0:*
LISTEN 0 4096 100.64.0.1:443    0.0.0.0:*

The router only forwards to the LAN IP, so the tailnet listener is invisible to the Internet.

Public services remain reachable at https://<service>.fidelramos.net from anywhere. Private services are reachable at https://immich.ts.fidelramos.net only from devices on my tailnet.

New private services need a DNS record in Blocky and a couple labels and a network attachment in Docker compose to be live.

The gotchas I hit so you won't have to

This is the part I wish someone had written down for me, they took significant time to figure out.

Compose silently falls back to `0.0.0.0`

If you write ${TAILSCALE_IP}:80:80 and TAILSCALE_IP is empty, Docker binds 0.0.0.0:80. Both Caddys then fight for the same port and one fails to start with port is already allocated.

I used ${TAILSCALE_IP:?...} to make it fail loudly if the environment variable is not defined.

Solving this was quick enough, but then I found the next gotcha...

Failed containers leave zombies

When a container fails to bind its ports, Docker leaves it in a weird half-state: NetworkMode is set correctly, but the container is actually attached to no network. docker inspect shows "Networks": {}, and ip addr inside the container shows only lo. A subsequent docker compose up -d sees the container "exists" and just starts it without fixing anything. You end up with a running container that has no connectivity at all.

The fix was to start with a clean slate, docker compose down, then docker compose up -d.

Binding `0.0.0.0:80` conflicts with `100.x.y.z:80`

Even if both IPs are "valid", Docker treats 0.0.0.0 as a superset. I had caddy-public binding 0.0.0.0:80 and caddy-tailscale binding 100.64.0.1:80 and got intermittent startup failures depending on order.

The fix is to pin both to specific IPs (LAN IP for public, Tailscale IP for tailnet).

No overlap, no race.

`caddy-docker-proxy` needs the service's network

The Caddy's CADDY_INGRESS_NETWORKS filter and the actual routing both require the proxied service to be on the same Docker network as the Caddy. A service can be on additional networks (for its own database, etc.), but it needs at least the Caddy's ingress network, or it's invisible.

Would I do it again?

Yes. The initial setup was surprisingly painful because there were multiple failures stacking on top of each other, but the resulting architecture is clean and cheap to extend.

Every new service I add goes through the same dead-simple decision:

Public: caddy.* labels, caddy_public network, DNS record in Gandi.
Private: caddy_ts.* labels, caddy_tailscale network, DNS record in Blocky.

No firewall rules to keep in sync; no per-service auth middleware; no "I'll just expose it temporarily" that becomes permanent.

The short lesson from the whole exercise: Tailscale is excellent at making "just me" services reachable from my devices, but integrating it with a Docker-based reverse proxy has more sharp edges than either technology has on its own. Binding your listeners to specific IPs, failing loudly on missing env vars, and giving containers explicit DNS resolvers are the three habits that would have saved me most of the pain.

Future improvement: limited access for other people

With this setup I lost the ability to share an Immich album with my family or friends, or to share a Nextcloud folder by sending a link. These are genuine helpful use cases that I would like to restore in some way.

For my family I will study getting them access to my tailnet, and then using ACLs to restrict their devices to only the services they need.

For file sharing I'm researching options like Filestash or FileBrowser Quantum, which are tailored services for this particular use case. What I want is to make the full admin UI accessible only through tailnet, and make their public sharing endpoints Internet-routable.

Whatever solutions I land on I will likely write blog posts about them.

Vibe Coding Escapades: pelican-copy-code plugin

2026-05-26T00:00:00+00:00

The Problem

I realized that this blog uses a lot of code blocks, but it’s not easy to copy them because there’s no Copy button on each block.

I looked for a plugin for Pelican that would do this, convinced that there must be one already out there, but I was surprised to find nothing!

It seemed like an ideal project to solve quickly with AI. I could fix my issue and create a useful project for the community.

Execution

My goal was to create a new project publishable to PyPI, with these requirements:

Compatible with stable Pelican.
Installable with pip and featuring auto-discovery. In other words, it must be a next-generation Pelican plugin; it doesn’t need to be added to the Pelican configuration (pelicanconf.py), it installs with pip and runs right away.
Static JavaScript and CSS files, without npm or similar.
No external JavaScript or CSS dependencies, only standard APIs.
Should not require changes to the templates. Once activated, the plugin automatically inserts the button into all blocks.
No DOM manipulation; the button is inserted when the Pelican site is generated.
Configurable.

I used OpenCode (in a sandbox, I describe it in this other article), initially with Kimi 2.6, then with Deepseek V4 Pro, and later with Claude Opus 4.7. I’ll discuss the details in the Lessons section.

Development took place in fits and starts over the course of an afternoon. While I was doing other things, I’d go to the computer, ask it to prepare the next change in planning mode, then come back, quickly review it, and give the go-ahead to implement it.

I tested the changes locally on this blog. I simply installed the plugin with uv in editable mode: uv add --editable ~/Code/pelican-copy-code

It’s a tiny project, so I didn’t bother generating or maintaining tests, linting, or GitHub actions.

Results

The plugin is pelican-copy-code, published on Github and PyPI.

It can be installed directly as intended; it’s just a pip install pelican-copy-code away.

Instead of a screenshot, here’s a code block so you can try it out live, just hover over it and click “Copy”:

def swallow_airspeed(laden=False):
   return 24 if not laden else RuntimeError("African or European?")

class BlackKnight:
   def __init__(self):
       self.flesh_wounds = ["'Tis but a scratch!"]
   def is_defeated(self):
       return len(self.flesh_wounds) >= 4

print(swallow_airspeed(laden="coconut"))

Lessons

At first, I used Kimi 2.6 because it’s my default model on Venice.ai. It’s cheap, fast, and private (Venice.ai doesn’t keep logs of conversations in its own models).

I tried switching to Deepseek V4 Pro because it was new and they said it was better for programming, comparable to Claude Opus 4.5. I didn’t notice much of a difference; for such a small project, either one would probably have worked well.

It is true that Kimi 2.6 made some mistakes in its initial implementation; for example, some of the configuration settings I asked it to include had no effect, that is, the code never used them. It was easy to fix in separate sessions, but it opened my eyes to how important the testing phase is.

I also ended up doing some manual refactoring that I probably could have avoided if I had been more specific in the instructions in AGENTS.md. For example, keeping configuration variables in alphabetical order should have been a style requirement in AGENTS.md. The agent also placed some configuration variables outside the dictionary. A style error that a human probably wouldn’t have made.

When it came time to test the plugin for the first time on a real site, it turned out that it wasn't working; it threw an error at build time due to incorrect use of signals and generators. I asked Deepseek V4 Pro to fix it but it couldn’t figure it out with a couple attempts. That’s when I switched to Claude Opus 4.7, which found the problem on the first try and fixed it correctly. I then remembered the advice I’d read here and there about using a smart model as a planner and another cheaper model to implement the changes. You can tell Opus is the most advanced model, but it’s also much more expensive than Kimi or Deepseek.

When I implemented translation support, I did it with Deepseek V4 Pro, and it worked well. However, it duplicated code (the strings “Copy” and “Copied!” were duplicated across two different files). I asked it to refactor to remove the duplication, but the proposal wasn’t good at all, it eliminated the duplication but added more poorly styled code. Once again, I switched to Opus 4.7, which proposed a much better solution that actually simplified the code.

Vibe Coding Escapades

2026-05-24T00:00:00+00:00

I'm going to start a new series of articles on how I'm using AI to develop software or solve technical problems.

The articles will follow this structure:

Problem
Execution
Results
Lessons and findings

In this article, I don't want to delve into ethical, environmental, or political-economic considerations regarding AI, but rather into my experience as a developer. Opinions and studies on the impact of AI on software developers' productivity vary widely. Some say it can multiply productivity, others that while it may seem to improve it, it actually decreases it. After these months of personal experience, I'm inclined to think that they are especially effective for those with deep technical knowledge.

I got my first computer at age 5 (almost 40 years ago now!), which is when I started learning computer science and programming. I've been using GNU/Linux since 1998. I completed my higher education in Software Engineering. I've worked as a software engineer for 15 years. And with AI, I feel 10 times more productive.

Note that I say “I feel”, because I don't have objective metrics to say just how much more productive I actually am. Studies suggest that the improvement is around 20%, but statistics are usually based on large projects and companies with complex workflows, which isn't the case for me. When working on new or small projects, I think AIs are very efficient. In my case, there's no doubt that these projects wouldn't be a reality if I didn't have the help of AIs.

My main problem in recent years has been time.
Time to learn something new.
Time to devote hours to a single task.
There are too many things to do, and I'm too busy with life to do it all. I think this is a common experience, especially among parents.

That's why my ideas would sit for years as TODO items in my notes. Now I'm getting into a routine where I glance at that list, pick one item, and successfully complete it in minutes or hours, in spare moments over the weekend. This is what I plan to document in this series of articles.

Having said all that, I don't want to ignore the voices warning of the negative effects of the use (or abuse) of AI. Personally, I hope it doesn't affect me negatively, because I use it relatively little, but I'll try to be introspective about my mental well-being.

I also read about many programmers who hate AI, to the point of quitting their careers, because what they enjoy is writing the code themselves. I feel their pain when they have jobs where they're forced to use AI and are pressured to produce more and faster. Their day-to-day has changed rapidly, and whereas they used to spend most of their time writing code, now that's been reversed, and they have to focus on generating, reviewing, correcting, and fine-tuning, which is much less fun and mentally stressful and exhausting.

In my case, writing the code is secondary; I care more about the end result than the process. That's why I'm happy to take the shortcuts that AI offers me, as long as I can maintain control and get a faster result with the quality I'm looking for.

Self-hosted Tailscale, Part 3: Syncthing over Tailscale

2026-05-18T00:00:00+00:00

I've been using Syncthing for years to keep files in sync between my laptop, desktop, phone, tablet and home server. It is an amazing piece of software and has been very reliable, with only the occasional conflict, but I have never lost any data. On the contrary, Syncthing file versioning has saved me more than once.

Syncthing has its own discovery and NAT traversal solutions, so devices that are behind firewalls can see each other and still connect. However that depends on external servers to handle the connections, and they could keep metadata. Even if they don't, they might degrade performance.

Now that I have a tailnet interconnecting all my devices, there's no reason to deal with any of that. Every peer already has a stable address reachable from every other peer. Syncthing can just use it, it only needs some configuration.

This post is about reconfiguring Syncthing to rely on the tailnet exclusively, turning off all the WAN-oriented machinery it no longer needs.

Table of Contents

Goals
Why do this
Reconfiguring Syncthing
Verifying
Encountered pitfalls
What's next

Goals

Syncthing peers talk to each other only over the tailnet.
No reliance on Syncthing's public global discovery servers.
No reliance on Syncthing's relay network.
No need for NAT traversal, it's handled by tailscale.
No open ports in the router, no need for UPnP.
Everything Just Works™ when I'm on the go.

Why do this

A few small wins that add up:

Privacy. Syncthing's global discovery phones home to announce (device ID → public IP). With the tailnet, that broadcast is pointless and can be disabled.
Reliability. Relays are slow and occasionally flaky. Direct tailnet connections are fast and consistent, and Tailscale's DERP already provides relay fallback at the network layer if a direct path can't be established.
Simplicity. Each peer's address becomes a fixed tailnet IP (or MagicDNS name). No more wondering why two devices aren't finding each other.
Security. Syncthing listens on TCP/UDP 22000 and UDP 21027 by default. If those ports are exposed on the WAN, they're an exposed service to the Internet. Turning everything off except the tailnet path closes that door, reducing the attack surface.

Reconfiguring Syncthing

Syncthing ships with a bunch of features to make it work across the internet with no configuration:

Setting	Default	Purpose	On tailnet
Sync Protocol Listen Addresses	`default`	Where to accept incoming syncs	Device's tailnet IP
Enable NAT traversal	On	UPnP / NAT-PMP to open ports in router	Off
Global Discovery	On	Announce device ID → IP on public discovery servers	Off
Local Discovery	On	LAN multicast to find peers	Off
Enable Relaying	On	Fallback traffic via public relay servers	Off

Global and local discovery, NAT traversal, and relaying are all redundant once every peer has a stable tailnet address.

I changed the settings on every computer via the Syncthing web UI (Actions → Settings → Connections):

Syncthing web settings

And in Android devices in the settings of the Syncthing app (inside Syncthing Options):

Syncthing Android settings

You might need to restart Syncthing after saving.

About Listen Addresses

Sadly Syncthing doesn't support binding to a specific interface, i.e. this won't work:

tcp://%tailscale0:22000, quic://%tailscale0:22000

Binding to the tailnet IP directly does work:

tcp://100.64.0.1:22000, quic://100.64.0.1:22000

The listener is then tied to a specific IP that could potentially change if the device ever re-registers, but that will be unlikely enough for me that it's a reasonable compromise.

An arguably less secure alternative is to leave the listen as default, which will bind to all interfaces, and then rely on the device firewall to block connections not through the tailnet.

Configure peers by tailnet address

With global discovery off, peers can no longer find each other automatically. The solution is to update each device's configuration to point explicitly at the peer's tailnet address:

Device → Edit → Addresses:

tcp://homeserver.ts.fidelramos.net:22000

MagicDNS hostnames work fine here because Syncthing resolves the address on every connection attempt, not just at startup (unlike listen addresses). If MagicDNS is flaky on a particular client, you can use the raw tailnet IP (tcp://100.64.0.1:22000).

I do this for every pair of devices in both directions.

Firewall

Since Syncthing should only be reachable over the tailnet, I remove existing rules that allowed Syncthing connections on my public network.

In my case I use firewalld, I removed rules from the public and home zones and let trusted (which is associated to the tailscale0 interface) accept everything automatically:

# Clean up any previously-allowed Syncthing ports
$ sudo firewall-cmd --permanent --zone=public --remove-port=22000/tcp
$ sudo firewall-cmd --permanent --zone=public --remove-port=22000/udp
$ sudo firewall-cmd --permanent --zone=public --remove-port=21027/udp
$ sudo firewall-cmd --permanent --zone=home --remove-port=22000/tcp
$ sudo firewall-cmd --permanent --zone=home --remove-port=22000/udp
$ sudo firewall-cmd --permanent --zone=home --remove-port=21027/udp
$ sudo firewall-cmd --reload

Check:

sudo firewall-cmd --zone=public --list-all
sudo firewall-cmd --zone=home --list-all

Neither zone should list port 22000 anywhere.

trusted zone doesn't need anything listed because it's permissive by default.

For Docker-based Syncthing deployments, it's worth mentioning that a configuration like ports: ["22000:22000"] bypasses firewalld. Either bind to the tailnet IP (ports: ["100.64.0.1:22000:22000/tcp"]) or use host networking. Native installs are simpler here.

Router

I used to have port TCP/UDP 22000 forwarded to my home server, and other ports for other devices in my home network. Not anymore:

Remove port 22000 (TCP and UDP) from the router's port-forwarding rules.
Remove any inbound allow rules for it.

Fewer exposed port on the WAN is always a good thing.

Verifying

The Syncthing web UI is my first check. On each device:

The peer shows as Connected.
The connection detail shows something like tcp://100.64.0.2:22000, i.e. a tailnet address, not a public one.
No "relay" indicator.

From the command line on the home server:

$ ss -tulpn | grep -E '22000|21027'
udp   UNCONN 0      0               100.64.0.4:22000      0.0.0.0:*    users:(("syncthing",pid=1526,fd=113))
tcp   LISTEN 0      4096            100.64.0.4:22000      0.0.0.0:*    users:(("syncthing",pid=1526,fd=110))

Syncthing is still listening on 22000, but on the tailnet IP.

That's fine, firewalld ensures only the tailnet can reach it.

# Connection to public IP fails
$ nc -zv 83.241.182.82 22000
nc: connect to 83.241.182.82 port 22000 (tcp) failed: Connection refused

# Connection to LAN IP fails
$ nc -zv 192.168.1.10 22000
nc: connect to 83.241.182.82 port 22000 (tcp) failed: Connection refused

# Connection to tailnet IP works
$ nc -zv 100.64.0.1 22000
Connection to 100.64.0.1 22000 port [tcp/snapenetio] succeeded!

With Tailscale back on, the same check via tailnet address should succeed.

Encountered pitfalls

Listen address `%tailscale0` syntax

As mentioned above, don't try to bind Syncthing to an interface name. Use the device's tailnet IP, or leave as default (but configure firewall properly).

Peers can't find each other after disabling discovery

Obvious in hindsight, but it caught me once: if a peer's address is still set to dynamic with all discovery off, it has no way to find anyone. Set explicit addresses for every peer.

MagicDNS fails at startup on a laggy client

If Syncthing tries to reach e.g. homeserver.ts.fidelramos.net before Tailscale is up on a device (e.g. on a phone that was rebooted), the connection fails with a DNS error. It retries a few seconds later and succeeds. There is nothing to fix, it's just noise in the logs.

Using IP addresses instead of MagicDNS names avoids the retry but loses the readability and resilience if a device ever reregisters on Headscale.

Syncthing on a containerized deployment

If you run Syncthing in Docker, the ports: section in your compose file needs to not expose 22000 to the WAN. My recommendation: run it on the host instead, or use network_mode: host, or bind explicitly to the tailnet IP. Docker's port publishing defaults to 0.0.0.0 and bypasses firewalld.

Discovery announcement caching

After I turned off global discovery, other Syncthing users who had previously discovered my device via the public discovery server would still have the stale public IP cached for a while. This isn't a leak, it's just that they'd try connecting to the old IP and fail. Purely cosmetic, and it goes away on its own within a day or so. Nothing to do unless you're sharing folders with people outside your tailnet (and in that case, you shouldn't have disabled global discovery in the first place).

What's next

Syncthing now runs entirely inside the tailnet, with no WAN exposure and no reliance on Syncthing's public infrastructure. I have noticed that devices connect faster, but I haven't run any benchmarks on transfer speeds.

In Part 4 I'll do the same thing for my web services: I will move most of my apps from a public app.fidelramos.net domain to a tailnet-only app.ts.fidelramos.net, using a split caddy-docker-proxy solution and TLS will still Just Work™.

How I Sandbox my AI Agents

2026-05-14T00:00:00+00:00

I've been using Opencode as my AI coding agent. Very unlike me, I don't have a long list of reasons why I picked it among the many alternatives. I just wanted to get started with an agent system that checked my main requirements:

Must be fully open-source.
Popular, meaning it's probably going to last. I don't want to spend time learning to use a program only for it to disappear in a few months.
Extensible and configurable.
Support for many AI providers, including my current choice, Venice.ai (here's a referral link for those interested!)

One of the things I don't like about Opencode is that it doesn't use sandboxing, and that's not an oversight but a conscious design decision. Yes, Opencode does ask before an agent tries access a file outside the project, but the Opencode software itself is not restricted and has system access.

My solution is simple: I wrap Opencode with Firejail, which I'm already using for most other software in my computers. This way I restrict Opencode not to have access to what it's not meant to (like my SSH keys!) at the kernel level.

Sandboxed Opencode cannot access my SSH keys

Firejail setup

Firejail merged an Opencode profile but is not in a stable release as of the time of this writing. For now I use my own firejail profile in ~/.config/firejail/opencode.profile:

# Firejail profile for opencode
# Description: An open source AI coding agent.
# This file is overwritten after every install/update
quiet
# Persistent local customizations
include opencode.local
# Persistent global definitions
include globals.local

# allow executables in HOME, see disable-exec.inc
ignore noexec ${HOME}

# blacklisted by disable-programs.inc
noblacklist ${HOME}/.cache/opencode
noblacklist ${HOME}/.config/opencode
noblacklist ${HOME}/.local/share/opencode

# Allows files commonly used by IDEs
include allow-common-devel.inc

# Disable Wayland
blacklist ${RUNUSER}/wayland-*
# Disable RUNUSER (cli only; supersedes Disable Wayland)
blacklist ${RUNUSER}
# Remove the next blacklist if your system has no /usr/libexec dir,
# otherwise try to add it.
blacklist /usr/libexec

# disable-*.inc includes
include disable-proc.inc
include disable-write-mnt.inc
include disable-x11.inc
include disable-xdg.inc

mkdir ${HOME}/.cache/opencode
whitelist ${HOME}/.cache/opencode
mkdir ${HOME}/.config/opencode
whitelist ${HOME}/.config/opencode
mkdir ${HOME}/.local/share/opencode
whitelist ${HOME}/.local/share/opencode

# Commands that reduce access to resources.
caps.drop all
##caps.keep CAPS
##hostname NAME
# CLI only
ipc-namespace
machine-id
no3d
nodvd
nogroups
noinput
nonewprivs
noprinters
noroot
nosound
notv
nou2f
novideo
protocol unix,inet,inet6
seccomp
#tracelog

disable-mnt
private-cache
private-dev
private-etc alternatives,ca-certificates,crypto-policies,dconf,fonts,ld.so.cache,ld.so.preload,machine-id,pki,resolv.conf,ssl
private-tmp

dbus-user none
dbus-system none

env NO_BROWSER=true
restrict-namespaces

This profile uses a whitelist model, so Opencode will only have access to those specific directories and not even see the rest. That means when invoking Opencode I need to whitelist the project directory I want it to work on. I do that like this:

~/Code/someproject$ firejail --profile=opencode --whitelist=(pwd) /usr/bin/opencode

Because writing all that would quickly become a bore, I have opencode set up as an abbr, so it will expand to the above.

An abbr is a fish shell construct that works like an alias, but it's editable upon expansion. This is very useful in case I want to edit the command, like in this case I might need to whitelist some extra path.

Results and Next steps

I have been using this setup for a month now, and it's been solid once I nailed down the configuration for the Firejail profile. I can now use coding agents with the assurance that they won't exfiltrate some private data, or that a bug in Opencode could affect my files or my system.

Next I want to extend this setup so I can use Opencode with a local LLM model through llama.cpp, but without Internet access. I want this to work on highly personal projects where I wouldn't trust even Venice.ai's private models, like my homelab configuration.

I have also developed my own research agents on Opencode, to track long-lived research projects as a set of Markdown files that serve as memory. It uses an extension of the previous Firejail profile, plus extra Opencode configuration and prompts for the researcher agent and searcher subagents. I keep refining it but it's already proving useful, so I will probably end up writing a post about it.

Self-hosted Tailscale, Part 2: Ad-blocking DNS

2026-05-11T00:00:00+00:00

In Part 1 I set up Headscale and had every tailnet client use Cloudflare DNS. That works, but it's leaving value on the table: every device that connects to my Tailscale gets whatever DNS servers I set, so I might as well run my own DNS server inside the private network and get ad-blocking, tracker-blocking, and internal name resolution everywhere I go.

This post swaps Cloudflare for Blocky, a lightweight DNS proxy with built-in blocklist support.

I chose Blocky over Pi-hole or AdGuard Home because it's simpler, configured in a single YAML file. It doesn't try to be a full web product, just a DNS server that does its job and stays out of the way.

You could easily replace Blocky with some other DNS server or proxy of your choice by keeping the Headscale and Tailscale bits.

Table of Contents

Goals
Why not just Android's Private DNS?
Architecture
Blocky in Docker Compose
Blocky configuration
Sanity check from the host:
Changes to Headscale
Why this is arguably better than device-level DoT
Encountered pitfalls
What's next

Goals

Block ads and trackers at the DNS level on every tailnet device.
Keep DNS encrypted on the wire (DoT upstream), even though traffic between my devices and Blocky is already protected by WireGuard.
Have internal name resolution, so for example homeserver.ts.fidelramos.net still resolves.
No device-level DNS configuration. Everything is pushed by Headscale and clients pick it up automatically.

Why not just Android's Private DNS?

Android has a built-in "Private DNS" feature that does DoT to any public resolver. I was previously using a private AdGuard DNS's server (xxxxxxxx.d.adguard-dns.com). I tried using it alongside Tailscale, but it doesn't work: as soon as Tailscale pushes any DNS settings to the client, Android routes DNS through the tunnel via Tailscale's internal resolver (100.100.100.100), which doesn't speak DoT. Android's Private DNS probe fails with "cannot connect" and DNS resolution breaks device-wide.

You can disable "Use Tailscale DNS" in the app to get Private DNS back, but then you lose MagicDNS.

Forced to choose, I went the other way: run my own DNS on the tailnet and let Headscale push it to all clients.

Architecture

  Device on tailnet
        │
        │ plain DNS (over WireGuard tunnel)
        ▼
  Blocky  ──▶  blocklists (local)
        │
        │ DNS-over-TLS (DoT) to upstream
        ▼
  Cloudflare / Quad9 / Google DNS / etc.

Clients talk plain DNS to Blocky's tailnet IP. "Plain" is fine because the whole path is inside a WireGuard tunnel.
Blocky does the filtering against blocklists it maintains locally.
For anything that isn't blocked, Blocky forwards upstream over DoT.
Blocky also keeps mappings for local tailnet domains to tailnet IPs.

Blocky in Docker Compose

I run Blocky in my existing Docker Compose project that runs everything else in my home server.

compose.yml:

services:
  blocky:
    image: spx01/blocky:latest
    container_name: blocky
    network_mode: host
    restart: unless-stopped
    volumes:
      - ${BLOCKY_ROOT}/config.yml:/app/config.yml:ro
      - /var/log/blocky:/logs:rw
    environment:
      - TZ=${TZ}
    healthcheck:
      test: ["CMD", "/app/blocky", "healthcheck"]
      interval: 1m
      timeout: 10s
      retries: 3

Highlights:

network_mode: host. This is a bit lazy on my part, but opening the Docker port only to the tailnet IP is trickier than one would expect. Blocky will be configured in the next section to only bind to the tailnet IP, so I don't think it's a big security risk. I'm trusting Blocky not to bind to other interfaces or ports in the system.
Config file is read-only inside the container. It's safer, and reminds me that the source of truth is my versioned config on disk.
I had to chown 100 /var/log/blocky, as that is the UID that the blocky container uses. Otherwise I'd be getting permission errors in Blocky's Docker logs.

Blocky configuration

This is my configuration with internal names redacted. Check out the Blocky docs for full details.

/var/opt/blocky/config/config.yml

# =================================================================
# Blocky config — Tailscale/headscale global resolver
# =================================================================

# ---------------------------------------------------------------
# Upstream resolvers (encrypted via DoT)
# ---------------------------------------------------------------
upstreams:
  init:
    strategy: blocking
  groups:
    default:
      - tcp-tls:dns.quad9.net:853
      - tcp-tls:one.one.one.one:853

# Explicit IPs for upstream resolution at startup
# (avoids chicken-and-egg when Blocky is itself the system resolver)
bootstrapDns:
  - upstream: tcp-tls:1.1.1.1:853
    ips:
      - 1.1.1.1
      - 1.0.0.1
  - upstream: tcp-tls:9.9.9.9:853
    ips:
      - 9.9.9.9
      - 149.112.112.112

# ---------------------------------------------------------------
# Client identification (static, since no DHCP on tailnet)
# Add a new entry here whenever you add a tailscale node
# ---------------------------------------------------------------
clientLookup:
  clients:
    homeserver:
      - 100.64.0.1
    laptop:
      - 100.64.0.2
    phone:
      - 100.64.0.3
    desktop:
      - 100.64.0.4

# ---------------------------------------------------------------
# Custom local DNS records for your tailnet nodes
# Add a new entry here whenever you add a tailscale node
# ---------------------------------------------------------------
customDNS:
  customTTL: 1h
  filterUnmappedTypes: true
  rewrite:
    ts.fidelramos.net: tail   # allows 'X.ts.fidelramos.net' as shortcut for 'X.tail'
  mapping:
    homeserver.tail: 100.64.0.1
    laptop.tail: 100.64.0.2
    phone.tail: 100.64.0.3
    desktop.tail: 100.64.0.4
    # Friendly names for services on tailnet (for Part 4)
    immich.tail: 100.64.0.1
    jellyfin.tail: 100.64.0.1
    nextcloud.tail: 100.64.0.1

# ---------------------------------------------------------------
# Blocking
# ---------------------------------------------------------------
blocking:
  denylists:
    ads:
      - https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt
      - https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
      - https://adaway.org/hosts.txt
      - https://v.firebog.net/hosts/AdguardDNS.txt
      - https://v.firebog.net/hosts/Easyprivacy.txt
    malware:
      - https://urlhaus.abuse.ch/downloads/hostfile/
      - https://v.firebog.net/hosts/Prigent-Malware.txt
    tracking:
      - https://v.firebog.net/hosts/Easyprivacy.txt
    smart-tv:
      - https://perflyst.github.io/PiHoleBlocklist/SmartTV.txt
    kids:
      - https://blocklistproject.github.io/Lists/porn.txt
      - https://blocklistproject.github.io/Lists/gambling.txt

  allowlists:
    ads:
      - |
        # Common false positives
        clients4.google.com
        clients2.google.com
        googleads.g.doubleclick.net

  clientGroupsBlock:
    default:
      - ads
      - malware
      - tracking
    # Per-client overrides by name (from clientLookup above)
    # phone:
    #   - ads
    #   - malware
    #   - tracking
    #   - smart-tv
    # kids-tablet:
    #   - ads
    #   - malware
    #   - kids

  loading:
    refreshPeriod: 24h
    downloads:
      timeout: 60s
      attempts: 3
      cooldown: 10s
    strategy: blocking
    maxErrorsPerSource: 5
  blockType: zeroIp
  blockTTL: 1m

# ---------------------------------------------------------------
# Caching
# ---------------------------------------------------------------
caching:
  minTime: 5m
  maxTime: 30m
  maxItemsCount: 0
  prefetching: true
  prefetchExpires: 2h
  prefetchThreshold: 5
  cacheTimeNegative: 30m

# ---------------------------------------------------------------
# Ports & binding
# Bind ONLY to tailnet IP so Blocky isn't exposed on LAN/WAN
# ---------------------------------------------------------------
ports:
  dns: 100.64.0.1:53
  tls: 100.64.0.1:853     # optional DoT for tailnet clients
  http: 127.0.0.1:4000    # metrics / API (local only)

# ---------------------------------------------------------------
# Logging & metrics
# ---------------------------------------------------------------
log:
  level: info
  format: text
  timestamp: true
  privacy: false

queryLog:
  type: csv-client
  target: /logs
  logRetentionDays: 7
  creationAttempts: 3
  creationCooldown: 2s

prometheus:
  enable: false
  path: /metrics

# ---------------------------------------------------------------
# Misc
# ---------------------------------------------------------------
# Block leaks of private-use TLDs to public resolvers
specialUseDomains:
  rfc6762-appendixG: true

At a high level:

Upstreams over DoT: Cloudflare and Quad9 as parallel resolvers, using DNS-over-TLS so the traffic coming from Blocky is encrypted.
Custom DNS entries: a few internal hostnames I want to resolve tailnet-wide without touching Headscale's extra_records (I'll cover that in Part 4).
Blocking: a mix of general lists and a couple of targeted blocklists I care about. I will revisit these lists, but they seemed like a good starting point.
Caching: enabled, with sensible defaults.
Prometheus metrics: off for now, I don't have a metrics collector right now, will do that in my new home server.

Start it up:

$ sudo docker compose up -d
$ sudo docker logs -f blocky

Sanity check from the host:

DNS resolves:

$ dig @100.64.0.1 example.com
; <<>> DiG 9.20.22 <<>> @100.64.0.1 example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57184
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;example.com.                   IN      A

;; ANSWER SECTION:
example.com.            300     IN      A       172.66.147.243
example.com.            300     IN      A       104.20.23.154

;; Query time: 24 msec
;; SERVER: 100.64.0.1#53(100.64.0.1) (UDP)
;; WHEN: Sun May 10 16:30:57 WEST 2026
;; MSG SIZE  rcvd: 72

Blocked domain (returns 0.0.0.0):

$ dig @100.64.0.1 doubleclick.net
; <<>> DiG 9.20.22 <<>> @100.64.0.1 doubleclick.net
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45469
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;doubleclick.net.               IN      A

;; ANSWER SECTION:
doubleclick.net.        60      IN      A       0.0.0.0

;; Query time: 1 msec
;; SERVER: 100.64.0.1#53(100.64.0.1) (UDP)
;; WHEN: Sun May 10 16:32:50 WEST 2026
;; MSG SIZE  rcvd: 49

Blocky only binds to tailnet IP, no DNS port exposed to the Internet:

$ sudo ss -ulnp | grep :53
UNCONN 0      0          100.64.0.1:53         0.0.0.0:*    users:(("blocky",pid=2984612,fd=15))

$ sudo ss -tlnp | grep -E ':53|:853'
LISTEN 0      4096      100.64.0.1:53         0.0.0.0:*    users:(("blocky",pid=2984612,fd=16))
LISTEN 0      4096      100.64.0.1:853        0.0.0.0:*    users:(("blocky",pid=2984612,fd=14))

Changes to Headscale

Now for the Tailscale magic: I tell Headscale to push Blocky's tailnet IP as the global resolver to every client. Edit /var/opt/headscale/config/config.yaml, its dns block changes as follows:

dns:
  magic_dns: false
  override_local_dns: true          # push our DNS to clients, replacing their system DNS
  base_domain: ts.fidelramos.net
  nameservers:
    global:
      - 100.64.0.1                  # Blocky on the tailnet
  search_domains: []

Restart Headscale:

$ sudo docker compose restart headscale

Each client has to pull the new settings:

Linux: sudo tailscale up --login-server=https://headscale.fidelramos.net --accept-dns=true
Android: toggle the Tailscale VPN switch off and back on. Also disable Android's Private DNS (Settings → Network → Private DNS → Off), since it would conflict.

Verify from a Linux client:

$ resolvectl status | grep -A2 tailscale0
# Should show 100.64.0.1 as the DNS server on tailscale0

$ dig example.com
# Should resolve via Blocky

Why this is arguably better than device-level DoT

The encryption story ends up equivalent or stronger:

Device → Blocky: plain DNS, but inside a WireGuard tunnel. Your ISP sees a UDP stream to your home server, but nothing else.
Blocky → upstream DNS: encrypted DNS-over-TLS. Your upstream sees the query from your host server, but not which device made it.

And I get things I couldn't get with per-device DoT:

Autoconfiguration, just by connecting through Tailscale.
Centralized blocklists. Change one file, every device benefits.
Per-client query logs in one place. Blocky can log queries by client IP.
Internal DNS entries, I'll use them in Part 4.
Works identically on every platform, no Android Private DNS dance.
No need to change router configuration. My current ISP router doesn't allow me to override the DNS, so this is a neat way of setting DNS on all my devices.

Encountered pitfalls

Android "cannot connect" with Private DNS set

Already mentioned in Part 1, but worth repeating because it bites everyone: with Tailscale's DNS active, Android's Private DNS setting must be Off or Automatic. Setting it to a strict DoT hostname breaks DNS entirely on the device while Tailscale is connected.

DNS resolution breaks on the host itself

If your home server uses 127.0.0.53 (systemd-resolved) and you then ask it to use Blocky for everything, and Blocky is in a container that depends on DNS to start... you can paint yourself into a corner.

Two ways out:

Leave the host's /etc/resolv.conf pointing at an external resolver (e.g. Cloudflare) and only push Blocky to tailnet clients. This is what I do — the host itself doesn't use Blocky, only the devices that connect through Tailscale.
Or, pin Blocky's image so it's always locally available and let Docker start it before anything else needs DNS.

Clients don't pick up the new DNS

Headscale pushes DNS config on every map update, but some clients (especially Linux with NetworkManager) need a nudge. Running tailscale up again with --accept-dns=true forces a re-sync. On Android, toggling off then on is usually enough.

Have to disable MagicDNS

If magic_dns had been left enabled in Headscale configuration it would have been in conflict with Blocky DNS, as both would try to use the same port 53.

I decided to disable MagicDNS and keep the mapping in Blocky's config of domains and IPs. It's a manual process, but my tailnet is small and won't change much, so I'm OK with it if it means a simpler setup.

An alternative solution would be to run Headscale MagicDNS in a different port, then have Blocky use it.

What's next

With Blocky in place, every device on my tailnet gets ad-blocking and filtered DNS, under any Internet connection anywhere in the world.

In Part 3 I'll reconfigure Syncthing to use the tailnet exclusively, disabling all its WAN-oriented discovery and relay features now that they're unnecessary.

Self-hosted Tailscale, Part 1: Headscale and clients

2026-05-05T00:00:00+00:00

I had been hearing a lot of people raving about Tailscale as a solution for interconnecting devices, or in other words for creating your own mesh VPN. It does seem great on paper: easy to set up, fast and lightweight, based on an open protocol (WireGuard), works everywhere, solves the NAT problem... However I was put off by handing a company my data, even if only metadata, and requiring personal information for sign-up. Is there no sovereign solution? Well, there is! And I will tell you step by step how to set it up.

I found the official article on how Tailscale works incredibly valuable. If you don't know the concepts of data plane, control plane, hub-and-spoke vs. mesh networks, or NAT traversal I recommend you give it a read. After all, our goal will be to self-host our own control plane.

This is the first post in a series about setting up a self-hosted Tailscale-compatible mesh VPN on my home server, with no cloud dependencies and no third-party accounts.

Goals

A mesh VPN between my devices (phone, tablet, laptop, desktop, home server) so I can reach any of my devices from any other device from anywhere in a secure way.
No third-party account required for login. No Google, no Apple, no GitHub, no Microsoft.
No third-party servers involved in coordination. My devices talk to my server, and that's it.
Works on Linux and Android (the platforms I actually use), using open-source clients. It's worth noting that other clients are closed-source, but I don't care because I don't use those platforms.
Easy to keep updated, both clients and server.

Why not just use Tailscale's hosted service?

Tailscale is a great product and I want to be fair about this: the data plane is end-to-end encrypted with WireGuard, so Tailscale Inc. does not see any actual traffic.

But:

They still see metadata. Every client contacts the control plane to fetch the network map. That means Tailscale would see the public IP of every device I own, every time it came online, from wherever I was. That's a lot of location history in one company's logs that I would rather avoid if I can help it. And I can, so I will.
Identity depends on a third party. The free tier requires signing in with Google, GitHub, Microsoft, Apple, or a similar OIDC provider. If any of those suspends my account, I'd lose access to my own network.
It's someone else's service. If Tailscale has an outage, changes pricing, pivots their product, or gets acquired, my setup would be impacted.

Architecture

The core of the architecture is based on Headscale, an open-source re-implementation of the Tailscale coordination server. It removes all three previous concerns. The official Tailscale clients work with it unchanged.

Architecture of a Tailnet with self-hosted Headscale

Headscale runs in Docker Compose on my home server, behind Caddy for HTTPS.
Tailscale clients run on each device, configured to use my Headscale instance as the control server. Each client tells the server about their network situation, and get connections to the other nodes in return.
DNS during Part 1 uses Cloudflare. Part 2 will automatically replace the DNS on all Tailscale clients with a self-hosted tracker-blocking ad-busting Blocky DNS server.
Clients establish direct peer-to-peer WireGuard tunnels with each other; Tailscale's public DERP relays are used only as a fallback.

Prerequisites

A domain I control (fidelramos.net).
A way for my home server to be reachable from the internet on ports 80 and 443. I use dynamic DNS because my ISP gives me a dynamic public IP. My server periodically updates an A record pointing headscale.fidelramos.net to my current IP. If you have a static IP, even better.
The router forwards TCP ports 80 and 443, and UDP port 41641 to my home server. My server sits in the router's DMZ so this is automatic.
My server, running Debian with Docker and Docker Compose installed and set up.
Caddy with caddy-docker-proxy as a reverse proxy. I wrote about that setup in this earlier post. It's the simplest way I have found to have an HTTPS-enabled reverse proxy. It really sparks joy.

Headscale in Docker Compose

I have a compose project with other services, including a caddy service with caddy-docker-proxy as described in that post.

Add a headscale service to compose.yml:

services:
  headscale:
    image: headscale/headscale:stable
    container_name: headscale
    restart: unless-stopped
    command: serve
    volumes:
      - ${HEADSCALE_ROOT}/config:/etc/headscale
      - ${HEADSCALE_ROOT}/data:/var/lib/headscale
    networks:
      - caddy
    labels:
      caddy: headscale.fidelramos.net
      caddy.reverse_proxy: "{{upstreams 8080}}"

A few highlights:

image: headscale/headscale:stable tracks the latest stable release. I combine it with WUD for controlled updates (I will write a future post about it). You can of course do a manual docker compose pull.
No ports: section. Headscale is only reached internally via Caddy.
I set my HEADSCALE_ROOT at /var/opt/headscale, but you can use whichever directory makes sense for you.
The two labels are all caddy-docker-proxy needs to route headscale.fidelramos.net to the container's port 8080.

Headscale configuration

I put it in /var/opt/headscale/config/config.yaml, but you can put it wherever works for you, just adjust the volume mapping in the compose configuration.

server_url: https://headscale.fidelramos.net
listen_addr: 0.0.0.0:8080
metrics_listen_addr: 0.0.0.0:9090
grpc_listen_addr: 0.0.0.0:50443
grpc_allow_insecure: false

noise:
  private_key_path: /var/lib/headscale/noise_private.key

prefixes:
  v4: 100.64.0.0/10
  v6: fd7a:115c:a1e0::/48
  allocation: sequential

derp:
  server:
    enabled: false
  urls:
    - https://controlplane.tailscale.com/derpmap/default
  auto_update_enabled: true
  update_frequency: 24h

disable_check_updates: true
ephemeral_node_inactivity_timeout: 30m

database:
  type: sqlite
  sqlite:
    path: /var/lib/headscale/db.sqlite

log:
  level: info
  format: text

dns:
  magic_dns: true
  override_local_dns: true
  base_domain: ts.fidelramos.net
  nameservers:
    global:
      - 1.1.1.1
      - 1.0.0.1
  search_domains: []

unix_socket: /var/run/headscale/headscale.sock
unix_socket_permission: "0770"

policy:
  mode: file
  path: ""

The bits worth understanding:

server_url is the public HTTPS URL for the control plane. It must be reachable by clients.
base_domain is the suffix used by MagicDNS for tailnet hostnames (e.g. homeserver.ts.fidelramos.net). It must not be equal to or a parent of the server_url host, otherwise clients get confused.
dns.nameservers.global pushes a DNS server to every client. For now I'm using Cloudflare; in Part 2 this becomes my self-hosted Blocky instance.
DERP is disabled for embedded server mode and uses Tailscale's public DERP map as fallback relays. Using their DERPs doesn't leak traffic (it's relayed WireGuard, still end-to-end encrypted) but it's the one non-self-hosted piece, a reasonable compromise.

DNS records

Only one: an A record for headscale.fidelramos.net pointing to my home server's public IP (via DDNS). That's what clients connect to and what Let's Encrypt will verify via HTTP-01 through Caddy.

ts.fidelramos.net (the base_domain) does not get a public DNS record. It's resolved entirely inside the tailnet by Tailscale's MagicDNS. This means that only my Tailscale clients will be able to access these domains, and they not be published anywhere.

First run

At the Docker compose project:

sudo docker compose up -d
sudo docker logs -f headscale

Once the logs show the API listening and Caddy has obtained a certificate, I confirm the control plane is reachable:

curl https://headscale.fidelramos.net/health

Should return {"status":"pass"}.

Creating a user and pre-auth keys

Headscale uses users as namespaces that own devices. I create one for myself:

sudo docker compose exec headscale headscale users create fidel

Then generate a pre-auth key each time I need to register a new device:

sudo docker compose exec headscale headscale preauthkeys create --user 1 --expiration 1h

The key is only valid for an hour. Long enough to register a device, short enough to not worry about leaking it.

Tailscale on Linux

Debian:

CODENAME=$(lsb_release --codename --short)
curl -fsSL https://pkgs.tailscale.com/stable/debian/$CODENAME.noarmor.gpg | \
  sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg >/dev/null
curl -fsSL https://pkgs.tailscale.com/stable/debian/$CODENAME.tailscale-keyring.list | \
  sudo tee /etc/apt/sources.list.d/tailscale.list
sudo apt update
sudo apt install -y tailscale
sudo systemctl enable --now tailscaled

Fedora:

sudo dnf config-manager addrepo --from-repofile=https://pkgs.tailscale.com/stable/fedora/tailscale.repo
sudo dnf install tailscale
sudo systemctl enable --now tailscaled

Arch Linux and derivatives:

pacman -S tailscale
sudo systemctl enable --now tailscaled

For all systems, once Tailscale is enabled, register with Headscale by using a preauth-key:

sudo tailscale up \
  --login-server=https://headscale.fidelramos.net \
  --auth-key=<preauth-key> \
  --hostname=$(hostname)

Verify:

# on client
tailscale status
# on server
sudo docker compose exec headscale headscale nodes list

The node should appear in both lists, with a 100.64.0.x address.

Tailscale on Android

The Android app talks to public Tailscale by default and there's no obvious setting to change it. I consider this obfuscation a dark pattern, understandable as they want to run a successful business, but still infuriating having to search for how to do it despite knowing the option was there somewhere.

The trick is the alternate server option, which is hidden:

Install the Tailscale app from F-Droid, the Play Store or, my chosen way, via Obtanium using https://pkgs.tailscale.com/stable/tailscale-android-universal-latest.apk as source.
Launch the app and cancel the attempt to log in to Tailscale.
Once you get to the screen with the Connect button, tap the cog for the settings menu, then tap on Accounts.
In the top-right menu on that screen, choose Use an alternate server.
Enter your Headscale server URL (in my case https://headscale.fidelramos.net).
Sign in. A browser page will open with instructions on what to run in the headscale server to register this client.
As instructed, run the command in the headscale server:

sudo docker compose exec headscale headscale nodes register \
  --user 1 \
  --key nodekey:abcdef...

If you want a visual guide, I found this video with step-by-step instructions:

After registration, toggle the connect switch in the app. tailscale status on a Linux device on the tailnet should show the phone, and it should appear in Headscale's node list as mentioned in the Linux client section.

I will mention one more Android-specific thing: Tailscale gets added as a VPN service, and at least in my devices it got marked as both always-on and to block connections without VPN. I recommend leaving always-on enabled, but disable blocking connections without VPN. As Tailscale is a VPN only between your devices, if it goes down you don't get deanonymized like with a traffic-forwarding VPN, you just lose access to the VPN peers.

Android VPN Settings

Other clients

I only use Linux and Android, so that's what I've covered and tested. For other platforms the Tailscale clients work the same way, just point them at the alternate server and register. The Headscale project maintains up-to-date instructions for macOS, iOS, and Windows which I won't duplicate here.

Encountered pitfalls

A few things that tripped me up along the way, collected here so you don't have to.

Wrong key

If you pass the wrong key to headscale nodes register you would get the following error:

Cannot register node: node not found in registration cache

The expected value is the nodekey:… string from the client's browser URL, not a pre-auth key. Pre-auth keys are used on the client side with tailscale up --auth-key=..., not on the server.

The registration cache is in-memory and expires after ~15 minutes, so if you take a lunch break between the client login and the server command, re-trigger the login on the client to repopulate it.

Android device appears as `invalid-...` node in headscale

Headscale rejects hostnames that aren't valid DNS labels (spaces, unicode, etc.). Android sends the human-readable device name, which often fails this check (e.g. "Pixel 8 Pro" in my case). Headscale then automatically renames the node to invalid-<hash>.

You can rename it in the headscale server after registration:

sudo docker compose exec headscale headscale nodes rename -i <node-id> pixel8pro

Purely cosmetic, but MagicDNS won't work for the original name until you do this.

Android "Not connected" with no error

If the app shows the Headscale account but sits in "Not connected" when you toggle the switch, a few things to check in order:

Check sudo docker compose logs -f headscale while toggling. You should see the client hitting /ts2021. If nothing appears, it's not reaching the server. Could be a TLS problem (check caddy's logs), DNS, firewall...
From the phone's browser, https://headscale.fidelramos.net/health must succeed.
Check that DERP loaded: sudo docker compose exec headscale headscale debug derp should list regions. If empty, the container can't reach controlplane.tailscale.com for the map, fix the container DNS or Internet egress first.
Android Private DNS: this is worth a section of its own.

Android Private DNS interferes with Tailscale bootstrap

I was using AdGuard DNS as a custom Private DNS, but it broke when Tailscale connected, leaving me without a working DNS setup, so most Internet connections wouldn't work.

I saw two errors:

In Android network settings, Private DNS would say couldn't connect.
Shortly after setting the custom Private DNS, a notification would pop up: Network has no internet access. Private DNS server cannot be accessed.

Luckily I was able to sidestep this issue by self-hosting my own ad-blocking DNS, and having all my Tailscale clients use it. This was even better as I would be in full control and wouldn't have to pay for AdGuard DNS any longer.

This solution will be presented in the upcoming Part 2 of this series.

MagicDNS not resolving on a specific device

After changing the Headscale DNS config, clients need to pull new settings.

On Linux, sudo tailscale up --login-server=... --accept-dns=true re-syncs.

On Android, toggle the VPN switch off and on.

Clock skew

Tailscale's noise handshake fails silently on significant clock drift. If nothing else explains a connection issue, check that NTP is working on both ends. On Android, set date/time to "automatic" in the system settings.

What's next

This setup was already functional: my devices could reach each other through my own control server, with no Tailscale nor other third-party account nor traffic. I could access apps that previously were only accessible through the LAN from anywhere.

In Part 2 I'll swap the currently configured Cloudflare DNS for a self-hosted Blocky instance to get ads and tracker blocking and internal name resolution on every tailnet device.

Pre-establishing SSH master connections for Ansible, or using OnlyKey with Ansible without losing your mind

2026-04-07T00:00:00+00:00

I use an OnlyKey as a hardware security device for a variety of purposes:

Common passwords are one keypress away, they get typed automatically.
As 2FA in my password manager (KeepassXC), by applying HMAC on the password.
SSH connections.
GPG encryption and signing.

OnlyKey

SSH connections are handled through the onlykey-agent, and it works well enough. However with Ansible it becomes a nightmare as it asks for dozens upon dozens of keypresses:

Part of an Ansible session with onlykey-agent

This pain was making me use Ansible less and less for my automations, so I resolved to find a fix. The goal was to have one OnlyKey press per host per script run, while keeping the behavior explicit and reasonably safe.

I'm happy to say that I managed to find a workaround, which I'm documenting here in hope that it's useful to other people. It might help not just OnlyKey users, but anybody with an SSH setup that requires manual intervention at connection time, and for some reason Ansible SSH connection reuse is not working.

The problem

Ansible often opens more than one SSH session per host during a run. Even with sane settings, you may still see repeated connections for tasks, fact gathering, privilege escalation, and other operations.

With a normal software SSH key, that's mostly invisible. You would be leaving performance on the table with all those network round-trips and extra encryption going on, but it'd work.

With onlykey-agent, each fresh SSH authentication requires physical confirmation. That makes repeated connections very tiring very quickly.

What I observed

The high-level fix is obvious: reuse SSH connections.

And the thing is that Ansible supports reusing connections by way of OpenSSH's connection multiplexing.

In a nutshell, OpenSSH can establish a connection as master, leave a socket open, and have new connections to the same host/user/port automatically reuse it.

However when I tried to set it up with Ansible it did not work, I kept getting the same prompts on OnlyKey.

When I looked at the control path directory I saw the sockets were being created, but they disappeared briefly after.

Then I tried SSH multiplexing outside Ansible, and it worked.

I set up ControlMaster in ~/.ssh/config, and once I connected to a host with onlykey-agent I could open a new connection without another OnlyKey prompt.

However once I closed the connection the socket disappeared.

My best guess is that Ansible is closing the connections somehow, but I haven't found exactly why or how.

The solution

Given that SSH connections kept the socket open as long as they were in use, I devised a solution that would work consistently:

Configure SSH multiplexing normally.
Configure Ansible to use the same control path.
Run Ansible inside a wrapper script that:
1. Reads the host list from the Ansible inventory.
2. Opens one SSH master per host.
3. Keeps each master alive with a remote sleep.
4. Runs ansible-playbook with all original arguments unchanged.
5. Kills only the SSH master processes it started when it exits.

SSH configuration

First, make sure your SSH config enables multiplexing and uses a stable control path.

Add this to ~/.ssh/config:

Host *
    ControlMaster auto
    ControlPersist 10m
    ControlPath ~/.ssh/sockets/%h-%r-%p

Then create the socket directory:

mkdir -p ~/.ssh/sockets
chmod 0700 ~/.ssh/sockets

I also like to configure users and ports in the SSH configuration, e.g.:

Host blog.fidelramos.net
    User fidel
    Port 22022

Ansible configuration

Use an ansible.cfg like this:

[defaults]
transport = ssh

[ssh_connection]
ssh_args = -o ControlMaster=auto -o ServerAliveInterval=30 -o ServerAliveCountMax=3
control_path_dir = ~/.ssh/sockets
control_path = %(directory)s/%%h-%%r-%%p
pipelining = True

transport = ssh forces OpenSSH rather than other transports that won't use SSH multiplexing the same way. This should be the default behavior in recent Ansible versions.
control_path_dir and control_path make Ansible use the same socket naming pattern as SSH.
pipelining = True reduces some SSH overhead and remote temp-file churn. This is a good idea even without multiplexing to reduce the number of connections and speed-up Ansible execution.

The wrapper script

Here is the full script.

It intentionally does not try to manage per-host usernames or ports itself. Those are applied by the SSH config, and Ansible reuses that too.

It passes all arguments through to ansible-playbook exactly as given. See usage examples below.

I also published the script as a Gist.

`ansible-ssh-masters.sh`

#!/usr/bin/env bash
# ansible-ssh-masters.sh
#
# Establish one SSH master connection per host, keep it alive with a
# remote sleep, run ansible-playbook with all original arguments, then
# clean up only the masters this script started.
#
# Useful when SSH authentication has a per-connection human cost:
# hardware security keys, passphrase prompts, FIDO2/U2F, etc.
#
# Usage (timeout in seconds):
#   ANSIBLE_SSH_MASTERS_TIMEOUT=3600 ./ansible-ssh-masters.sh playbook.yml [args...]

set -euo pipefail

TIMEOUT_SECONDS="${ANSIBLE_SSH_MASTERS_TIMEOUT:-3600}"
SOCKET_DIR="${HOME}/.ssh/sockets"

PIDS=()
STARTED_HOSTS=()
ARGS=("$@")
INVENTORY="inventory"

GREEN='\033[0;32m'
YELLOW='\033[0;33m'
RED='\033[0;31m'
NC='\033[0m'

log()  { echo -e "${GREEN}[+]${NC} $*"; }
warn() { echo -e "${YELLOW}[!]${NC} $*"; }
err()  { echo -e "${RED}[-]${NC} $*" >&2; }

parse_inventory_arg() {
    local i
    for ((i=0; i<${#ARGS[@]}; i++)); do
        case "${ARGS[$i]}" in
            -i|--inventory)
                if (( i + 1 >= ${#ARGS[@]} )); then
                    err "Missing value after ${ARGS[$i]}"
                    exit 1
                fi
                INVENTORY="${ARGS[$((i + 1))]}"
                ;;
            --inventory=*)
                INVENTORY="${ARGS[$i]#--inventory=}"
                ;;
        esac
    done
}

cleanup() {
    echo
    log "Closing master connections..."

    local i pid host

    for i in "${!PIDS[@]}"; do
        pid="${PIDS[$i]}"
        host="${STARTED_HOSTS[$i]}"

        if kill -0 "$pid" 2>/dev/null; then
            if kill "$pid" 2>/dev/null; then
                wait "$pid" 2>/dev/null || true
                log "Closed master for ${host} (pid ${pid})"
            else
                err "Failed to kill master for ${host} (pid ${pid})"
            fi
        else
            warn "Master for ${host} was already gone (pid ${pid})"
        fi
    done

    log "Done."
}

trap cleanup EXIT INT TERM

parse_inventory_arg

mkdir -p "$SOCKET_DIR"
chmod 0700 "$SOCKET_DIR"

# Remove stale sockets (no process behind them)
find "$SOCKET_DIR" -maxdepth 1 -type s ! -exec fuser -s {} \; -delete 2>/dev/null || true

if ! ansible-inventory -i "$INVENTORY" --graph >/dev/null 2>&1; then
    err "Inventory could not be parsed: $INVENTORY"
    exit 1
fi

mapfile -t HOSTS < <(
    ansible-inventory -i "$INVENTORY" --graph |
    awk '
        /@/ { next }
        {
            line = $0
            gsub(/^[[:space:]|]+/, "", line)
            sub(/^--/, "", line)
            if (line != "") {
                print line
            }
        }
    ' |
    sort -u
)

if (( ${#HOSTS[@]} == 0 )); then
    err "No hosts found in inventory: $INVENTORY"
    exit 1
fi

log "Found ${#HOSTS[@]} hosts in inventory: $INVENTORY"
log "Per-host connection lifetime: ${TIMEOUT_SECONDS} seconds"
log "Establishing master connections (authentication may be required)..."
echo

count=0
for host in "${HOSTS[@]}"; do
    ((count+=1))

    if ssh -O check -o "ControlPath=${SOCKET_DIR}/%h-%r-%p" "$host" >/dev/null 2>&1; then
        log "[$count/${#HOSTS[@]}] ${host}: reusing existing master"
        continue
    fi

    warn "[$count/${#HOSTS[@]}] ${host}: connecting..."

    ssh \
        -o ControlMaster=yes \
        -o "ControlPath=${SOCKET_DIR}/%h-%r-%p" \
        -o ServerAliveInterval=30 \
        -o ServerAliveCountMax=3 \
        -o ForwardAgent=no \
        -o ForwardX11=no \
        -o ClearAllForwardings=yes \
        "$host" "sleep ${TIMEOUT_SECONDS}" &

    pid=$!

    for _ in $(seq 1 15); do
        if ssh -O check -o "ControlPath=${SOCKET_DIR}/%h-%r-%p" "$host" >/dev/null 2>&1; then
            break
        fi
        sleep 1
    done

    if ssh -O check -o "ControlPath=${SOCKET_DIR}/%h-%r-%p" "$host" >/dev/null 2>&1; then
        log "[$count/${#HOSTS[@]}] ${host}: master established (pid ${pid})"
        PIDS+=("$pid")
        STARTED_HOSTS+=("$host")
    else
        err "[$count/${#HOSTS[@]}] ${host}: failed to establish master"
        if kill -0 "$pid" 2>/dev/null; then
            if kill "$pid" 2>/dev/null; then
                wait "$pid" 2>/dev/null || true
            else
                err "[$count/${#HOSTS[@]}] ${host}: failed to stop failed SSH process (pid ${pid})"
            fi
        fi
    fi
    echo
done

echo
log "Running: ansible-playbook $*"

set +e
ansible-playbook "$@"
ANSIBLE_EXIT=$?
set -e

exit "$ANSIBLE_EXIT"

Make it executable:

chmod +x ansible-ssh-masters.sh

Screenshot

ansible-ssh-masters in action

Usage

Just replace ansible-playbook with ansible-ssh-masters.sh:

./ansible-ssh-masters.sh playbook.yml

In my case I prepend onlykey-agent with the right key:

onlykey-agent fidelramos.net -- ./ansible-ssh-masters.sh playbook.yml

You can also start an onlykey-agent subshell, then run ansible-ssh-masters.sh inside:

onlykey-agent fidelramos.net -s
./ansible-ssh-masters.sh -i inventory playbook.yml

You can pass any number of arguments, or a different inventory file or expression, they get passed as-is to ansible-playbook:

./ansible-ssh-masters.sh -i production.ini playbook.yml --limit web --tags deploy -v

To change how long the pre-opened master sessions live, set the timeout in seconds:

ANSIBLE_ONLYKEY_TIMEOUT=7200 ./ansible-ssh-masters.sh long-playbook.yml

Key design points of the script

A few design choices are deliberate here.

1. Close connections on completion or exit

With the use of trap the script makes sure connections are closed as soon as they are not needed anymore, or if the script gets interrupted (Ctrl-C, killed or closed shell for example).

2. Cleans up what it started

The script kills only the SSH master processes it created.

It does not sweep the whole socket directory. That matters if you have other SSH sessions or tools using the same multiplexing directory.

3. Flexible inventory parsing

The script detects the use of -i/--inventory argument and it parses it with ansible-inventory to get the hosts where the playbook will execute. That means any of the --inventory formats will work, it's not limited to an inventory file.

4. Uses the SSH configuration as source of truth

Instead of relying on parsing configuration it transparently uses the SSH configuration.

The script does not try to guess usernames, ports, or host aliases from Ansible.

That all stays in SSH config where it belongs.

Security notes

This setup is definitely a usability improvement, but it does diminish security while the script is running.

The main tradeoff

After you approve a host once with OnlyKey, that SSH master connection stays open for the configured lifetime or until the script exits.

That means:

Later SSH sessions to that host can reuse the master.
They do not need another OnlyKey press while that master is still alive.

That is the whole point of the solution, but it also means your local session becomes more important to protect during that window.

Other software in the computer with access to the SSH sockets directory could gain direct access to the remote host, which is kinda the point of using a hardware security device in the first place.

Things this setup does to limit risk

Sockets live in ~/.ssh/sockets.
That directory is set to 0700.
Agent forwarding is disabled.
X11 forwarding is disabled.
The script cleans up automatically on exit.
The timeout is explicit and finite in case automatic cleanup fails for some reason.

Also personally I use Firejail to sandbox most of the software running on my computer. My Firejail setup blacklists access to ~/.ssh for most programs, so I feel a bit safer with this approach.

Why automatic cleanup stays enabled

The script always closes all masters after the script ends (via trap).

This is a design decision, and the script would be even simpler by just leaving the connections open until they time out. This would also mean that further Ansible invocations would not need any further OnlyKey presses.

Personally I don't want that, and I would not recommend it as the default as it further reduces security. The shorter that connections stay open the less likely they can be taken over.

Final thoughts

I feel sad that I couldn't get to the bottom of why the Ansible connection reuse was not working properly, but I couldn't allocate more time to that research.

I do feel happy to have found this workaround, which is a reasonable sweet spot between convenience and security. If this hadn't worked I was feeling close to going back to regular SSH keys, which would have reduced the security of my servers considerably.

If like me you have a cumbersome SSH connection scheme I hope this script helps you enjoy Ansible more.

If you made it this far and have an idea on how to fix it The Proper Way™, please leave a comment!

Announcing Reflex, a Pelican theme

2026-02-16T00:00:00+00:00

I'm happy to announce the public availability of my Pelican theme: Reflex. It is published on PyPI, which should be the best way for most users to install it.

When I first started this blog I decided on Pelican because I had lots of experience with Python, so it would make it easier for me to get involved and contribute. I then looked at all the Pelican themes and settled on the Flex theme by Alexandre Vizenzi because it checked all my requirements:

Minimalistic and beautiful.
Responsive layout.
Archives, categories, tags support.
Pygments support for code highlighting.
Light/Dark modes support.

As it was inevitable I eventually hit some missing features and small tweaks I wanted to implement. That is the beauty of open-source, and I started contributing to Flex. However some of my desired changes were too deep, and they didn't fit in Flex as the project was in maintenance mode. After some thinking I decided to fork the theme to introduce my more breaking changes, and that is when Reflex was born.

The main differences with Flex for now:

Installable via PyPI (so pip install pelican-theme-reflex works).
Style for Table of Contents created by the toc markdown extension.
Styles for figure and figcaption HTML elements.
Shynet tracking support (see my article on self-hosted comments).
In-repo documentation instead of Github wiki.
Display language flags for alternative article languages.
X social icon.
Improvements to developer experience: gulp watch support for instant changes, pre-commit hooks, AGENTS.md for AI development, revamped GitHub Actions workflows.
Updated Pygments styles and FontAwesome.
Accessibility improvements.
Updated Isso comments integration, with support for highlighting comments by the author and reply notifications by default.

I don't have a long roadmap planned for Reflex, I will mostly implement changes that I find useful for this blog. I'm open to new integrations and ideas, so collaborators are welcome!

For known bugs and planned features take a look at the open issues. For any questions feel free to open a discussion.

If you use Reflex feel free to add your site to the list of Reflex users, I would love to hear from you!

Update Feb 19th 2016: Reflex 3.1.0 was released with a few extra improvements.

Self-hosted Home, Part 1: Design and Planning

2026-01-25T00:00:00+00:00

Those who have been paying attention to the blog know that I bought a house and have been planning its full renovation. As a computer nerd what excites me the most is the opportunity to implement my dream smart home, networking and home server.

I have a clear idea of what I want to achieve: an open‑source homelab that replaces closed cloud ecosystems with local‑first services. Smart home automation, security cameras and video doorbell, contacts and calendars, media players, photo collection, documents, and more, all running on premises and fully under my control. Do note that I already have a homelab, where some of these services have been running for years already. This new project will replace it completely, it will be bigger and better! Do read that article if you have the chance as it gives a rationale of what moved me to start self-hosting services and the historical context of how I got here.

In this series of posts I will go into all the technical details, starting with the design I arrived at after months of research.

The setup is completely personal and opinionated, tailored to my particular needs and taste, so don't take it as a recommendation. I do hope that it's a fun read and you learn something cool!

My goals are:

Local‑first: Smart home, cameras, media, and personal data should work without any cloud dependency. Internet access could go down and everything would keep working locally. Also I can trust my data won't leave my home.
Strong isolation: IoT and cameras are treated as hostile. They get no Internet egress and limited access to internal services. Cloud connectivity is to be avoided whenever possible.
Standards‑based and open-source: ONVIF, RTSP, IPv6, WireGuard, ZFS, Proxmox, TrueNAS, IPFire, Home Assistant, Frigate, Nextcloud, Jellyfin, Immich... avoid proprietary black boxes.
Cost: While the whole setup won't be cheap by any means, I will look for refurbished server‑grade hardware and favor low-power devices. The system will have a lot of responsibilities so I need quality reliable hardware.
Future‑proof: 10 GbE-ready via Cat6a, core‑rich servers, room for expansion, IPv6 everywhere.

1. Overview

The homelab will consist of multiple components mounted on a 19-inch 42U rack in a dedicated room in the basement of the house.

Core networking and security
- I want to be in control of the home network: prevent Internet access for smart home and cameras and limit access to internal services.
- A dedicated server running IPFire will provide routing, firewalling, IDS and VPN.
- UniFi switch and WiFi access points (APs).
- VLANs for strict segmentation (LAN, IoT, cameras, guests, management).
Application server
- I will migrate the self-hosted services running in my current home server, and expand them.
- Proxmox VE on a server‑grade box (Dell PowerEdge class).
- Mix of VMs and LXC containers for services.
Storage
- I have considerable storage needs as I keep a large photo collection and a local media library.
- TrueNAS running as a VM on Proxmox, backed by a dedicated disk pool.
- ZFS for redundancy, snapshots, and replication.
Cameras and doorbell
- Dahua cameras and doorbell with ONVIF/RTSP/IPv6 and 2‑way audio.
- Dedicated PoE switch.
- Frigate for NVR‑like recording and object detection.
- Coral TPU for efficient inference.
Power resilience
- Solar panels and batteries will keep at least core infrastructure running.
- An UPS will provide critical backup to keep the network and cameras running and to allow graceful shutdown of the app server.

For those who enjoy a little diagram:

Diagram of the rack

Self‑hosted services

Planned services include:

TrueNAS, detailed below in the Storage section.
Home Assistant for smart home automation and integration of all platforms and data.
Frigate NVR for cameras, object detection and doorbell communication (also integrated into Home Assistant).
Nextcloud as a personal cloud (contacts, calendar, files, notes, etc.).
Immich as photo library.
Jellyfin for media playback.
Music Assistant for music playback.
Paperless-ngx to manage documents.
AdGuard Home for ad blocking house-wide.
Auxiliary services: monitoring, uptime checks, logging, backup orchestration, NTP...

I mean to give access to most of these services to my family. That would amortize the setup and running costs, while bringing more privacy into their lives.

2. Networking

Networking is the foundation for everything else, it is key to the performance, security and comfort of the system.

I'm taking the opportunity that a full-house renovation brings to install Cat6a Ethernet cables throughout the house. Every room will get at least one RJ-45 port in a wall socket, and some of them multiple ones, depending on expected usage. Cat6a cables guarantee future upgrades to 10 GbE by replacing the switch, NICs and APs, the wiring would already be in place. Initially the connections will be at 2.5 GbE, with the exception of two 10 GbE for the two desktop computers.

All cables will run through the walls to the server room, to rack-mounted patch panels.

For router and firewall I will install IPFire in a dedicated server, it will be the core of the setup. A UniFi switch and multiple UniFi WiFi access points will aggregate the traffic. There will be a secondary switch for the security cameras, it will be discussed in the Cameras and Doorbell section.

IPFire router/firewall

IPFire is an open-source router and firewalling software based on GNU/Linux. I will put it on a small, low‑power but robust rack server with these (ideal) requirements:

At least 2 SFP+ NICs (one for WAN, another for LAN, optionally another NIC for dedicated MGMT or DMZ).
Enough CPU to handle up to 10 Gbps WAN speed.
Dual SSD or NVMe drives to set them up in mirror for redundancy.
Dual PSUs for power redundancy.
As low power as possible, especially on idle.

I'm still researching options for this server. IPFire offers its own appliances, but they seem to cost too much for what they offer, so I'm looking at rack servers.

The best fit I have found so far is the Supermicro X10SDV-7TP8F, which is a system-on-chip motherboard with an Intel Xeon D-1587 CPU. It checks most requirements: enough compute power for 10 Gbps WAN; low power consumption (10W idle, 45W TDP); embedded 2x 10 GbE SFP+ NIC. It's missing dual PSUs and dual drives, but nothing is perfect.

The IPFire box will connect directly to the ISP router (in bridge mode) or a dedicated ONT.

It will handle these responsibilities:

Terminate the WAN connection (PPPoE or DHCP, depending on ISP).
Provide stateful firewalling for both IPv4 and IPv6.
Perform DHCP and DNS for internal networks.
Handle WireGuard VPN tunnels for remote access to the home network.
IDS (Intrusion Detection System).

I want IPv6 to be a first‑class citizen: IPFire will obtain a delegated prefix from the ISP and split it across the internal VLANs. This, coupled with a deny-by-default policy, will make it easier to open services to the outside in a controlled and safe manner without the hassles and performance impact of IPv4 NAT.

UniFi switching and WiFi

Downstream of IPFire there will be a UniFi Pro HD 24 PoE switch which will:

Aggregate all wired segments (rooms, rack, camera uplink, WiFi APs).
Provide PoE for the APs and potentially other devices.
Implement VLAN tagging and firewall‑like segmentation at Layer 2/3 in cooperation with IPFire.

I searched far and wide for rack-mounted switches, but nothing came close spec-wise to this UniFi. It's got 4x 10 GbE SFP+ ports for connecting the IPFire and application servers, 2x GbE RJ-45 ports for my desktop computers, and 22x 2.5 GbE RJ-45 ports with PoE. Most other 24-port switches I found have only 1G ports, and no 10 GbE. Having 2.5 GbE from the get go will give great performance network-wide for a long time.

UniFi Pro HD 24 PoE switch

For WiFi APs I considered Grandstream APs, which are reportedly of good quality, but I discarded them because they don't support the 6 GHz band, which is the novelty of the WiFi 7 standard. I eventually settled on UniFi WiFi APs (most likely the U7 Pro), because they work with a single Ethernet cable (2.5 GbE for the U7 Pro, but in the future I could upgrade to an AP with 10 GbE), and they can be managed together with the UniFi switch.

The APs will allow for seamless roaming around the house, and will have multiple SSIDs:

Main SSID: For trusted devices (family laptops, phones, tablets).
IoT SSID: For wireless smart devices, mapped to the IoT VLAN.
Guest SSID: Isolated from internal services, Internet‑only.

UniFi U7 Pro

Proxmox will run a UniFi Network Server in a VM, that will allow for configuration of the switch and APs without cloud connectivity. I'm really trusting Ubiquiti not to screw me up with future changes that require cloud connectivity, I would be very mad if they do. This risk is why I didn't go with UniFi cameras for example. Worse comes to worst I would only need to replace the switch and APs, it's a measured risk.

VLAN design and trust model

The network will be divided into trust zones:

Internal (Trusted): Family computers, smartphones and tablets.
IoT: Smart home devices (sockets, switches, appliances, TV, air conditioners...).
Cameras: Security cameras and the doorbell.
Guest: For visitors; Internet only, no access to LAN/IoT/Cameras.
Management: Personal computer, Proxmox, switches, router, APs, out‑of‑band management (IPMI, iDRAC).

High‑level routing/firewall rules:

Internal: Can access services: Home Assistant, Frigate, NAS shares, etc..
IoT: Connection to Home Assistant; no direct Internet access, except for a handful of whitelisted endpoints if absolutely required; no access to personal data.
Cameras: Connection to Frigate; no Internet access at all, don't want my camera data exfiltrated; no access to personal data.
Guests: Internet access only; no access to any internal services.
Management: Only reachable from a small set of trusted admin devices.

Firewall and routing rules will be enforced jointly by IPFire and the UniFi switch, ensuring that camera and IoT traffic never reaches the Internet, and guests never reach anything internal.

WireGuard VPN

For secure external access I will set up Wireguard on IPFire:

Each device (laptop, phone, tablet) gets its own public/private key pair and virtual IP.
WireGuard peers land in a remote access subnet that has controlled access to LAN and management.
This will avoid opening internal services to the Internet, instead providing a single, auditable path: WireGuard → IPFire → internal services.
WireGuard is limited in features, but because of that its attack surface is much smaller than other VPNs.

This should allow full remote control of Home Assistant, camera dashboards, and administration tasks without exposing those services directly to the Internet.

3. Application Server

The core of the homelab will run on Proxmox VE, installed on a single, server‑grade box with proper remote management and redundancy.

Proxmox is a secure and robust Linux-based OS that simplifies running virtualized or containerized services. This provides a number of benefits:

Open-source: It may seem obvious, but it avoids vendor lock-in.
Mix VMs and containers: Both can be managed through the same interface.
Security through isolation: If a service gets compromised, the attacker would not gain access to the whole system, but only to the data and network it has access to.
Portability and testing: I could have a staging Proxmox instance, for example on my desktop computer, as a testbed for new services. When they are ready they can be migrated to the production Proxmox instance.
Safer upgrades: Snapshotting VMs is an easy way of testing a software upgrade and rolling back if issues are found.
Backups: Besides ZFS snapshots, Proxmox has its own open-source backup server, for incremental deduplicated backups.
Availability and clustering: I won't use it myself, because I feel it's overkill for my use case, but it's possible to have redundancy and automatic failover of Proxmox instances.

Proxmox and licensing considerations

Proxmox is licensed per CPU socket. In order to reduce ongoing costs I'm looking for single-CPU servers, but with enough cores to run all planned services, and have room to grow into the future.

Modern single‑socket server CPUs can have hundreds of cores, which is more than enough for a homelab.

Server hardware options

The target form factor is a refurbished rack server, with these characteristics:

Power-efficient CPU with at least 24 cores, ideally 32 or more.
At least 128 GB of ECC RAM. ECC is critical here to avoid silent data corruption. DDR5 RAM would be ideal, but with the ongoing memory shortage I will settle with DDR4.
Space for at least 8 HDDs, ideally 10 or 12 to have room to grow if needed.
Support for dual NVMe storage, or at the very least SSDs.
At least one 10 GbE SFP+ NIC.
Dual redundant PSUs.
IPMI-compatible remote management.

The Dell PowerEdge series has a wide range of models and are easy to find refurbished, so they set my baseline:

Intel‑based
- R730: Old but very common and cheap. Dual‑socket Intel Xeon, up to 3 TB DDR4, 10 GbE through optional NDC (Network Daughter Card), no NVMe, iDRAC enterprise‑grade management.
- R740 / R750: Newer generations with more efficient CPUs, more cores, NVMe, better PCIe (Gen 3/4 depending on model), better iDRAC...
AMD‑based
- R7415: Single-socket AMD EPYC (1st generation); excellent core density and memory bandwidth at lower power and cost than equivalent Intel parts, and designed as single‑socket platforms.
- R7515: Single-socket 2nd and 3rd generation AMD EPYC, up to 64 cores.
- R7525: Like R7515 but with dual socket. Not a deal-breaker because Proxmox only charges per CPU socket in use, so it can be an option if priced fairly.

The single‑socket AMD EPYC platforms seem particularly attractive for Proxmox because:

They deliver higher core counts than Intel and more memory bandwidth in a single socket, avoiding per‑socket licensing overhead.
EPYC platforms are known for strong virtualization performance and high I/O density.

Dell PowerEdge R7515

A Dell PowerEdge R7415 or R7515 looks like the best fit, but I will compare to similar offers from other vendors:

Supermicro AS-2014S-TR: mostly similar, but it has more PCIe lanes and more drive bay with space for 12 HDDs, so it would allow for expansion of the NAS if necessary.
HPE ProLiant DL385 Gen10 Plus: Dual socket; up to 28 drives. Overkill.
Lenovo ThinkSystem SR665: Supports single-socket AMD EPYC CPUs with the "P" suffix. They are reportedly hard to find refurbished, single-socket even more so, so it's not my first option.

One thing that makes me reluctant to go HPE is that they don't offer free downloads of BIOS or firmware upgrades without a support contract, which is prohibitively expensive for a home user. I was already bitten by that with my current HPE Microserver Gen10, so I will probably go with Dell or Supermicro for the next one, as they have free downloads without a contract.

My final choice will depend on local availability and pricing of refurbished units, but I'm hoping to nail one of those sweet Dell PowerEdge with an AMD EPYC.

Redundancy and reliability

The Proxmox server is treated as the primary infrastructure node, it's not a toy. While my smart home is designed so that the house will keep working even if Home Assistant goes down, given that all my personal services will be running on this hardware means it is of utmost importance that it stays up and healthy.

With this in mind, these are the steps I will be taking to guarantee the availability of the application server:

Dual PSUs: Connected to separate outlets, so a single PSU or electrical circuit failure does not bring the server down.
UPS: Connected to at least one UPS, for emergency power backup and graceful shutdown. More details in the Power and Resilience section below.
Boot / VM pool: 2x NVMe drives in a mirrored ZFS pool (often described loosely as "RAID1 on ZFS"). This provides redundancy for the Proxmox host, services, and configuration, so if one drive fails the system will keep on going (and the faulty drive could be hot-swapped with a new one and ZFS would rebuild the pool online).
Dual network ports: Use at least two NICs: one for management; one for VM traffic and storage. I could also implement failover for the network links by using interface bonding, that would mean 2x for management, 2x for traffic.
Remote management: IPMI, iDRAC, iLO, or equivalent to allow out‑of‑band management, BIOS access, and remote power control.
Physical console for emergency access: I will have a cheap console (monitor and keyboard) around to directly connect to the server if necessary.

4. Storage

I have large storage needs, so I know from the get go that I want a NAS, with good speed, redundancy and reliability.

Why not Synology?

I initially considered a Synology NAS. Particularly the RS1221+ seemed to be exactly what I was looking for.

Synology RS1221+

Synology products are popular for good reason: they are easy to manage and have a rich ecosystem. However, recent policy and business decisions have made them less attractive, and particularly deal breakers for me:

Increasingly restrictive stance on third‑party drives, with models that:
- Warn, complain, or degrade the UX when non‑Synology drives are used.
- In some cases, limit support or functionality based on drive vendor.
A growing "appliance lock‑in" direction that goes against the open, standards‑first philosophy of this homelab.

Given these concerns, I made the decision to avoid Synology and instead use open platforms and commodity hardware where I would be in full control.

Enter TrueNAS

TrueNAS is probably the best known open-source OS specialized as a NAS. It's robust, which is the number one feature you want for your storage.

It only works with ZFS, which is what I want to use for my data layer, and its UI simplifies management, monitoring and alerts. As much as I like technology and getting my hands dirty I will be busy setting everything up so I will take ease of setup and maintenance whenever I can, without losing control of course.

I considered openmediavault (OMV) because it's a direct competitor, also open-source and Linux-based. I discarded it because it is more flexible (for example ZFS is optional), it is meant for people looking to run apps in their NAS, but I don't need to because I will have a separate app server for that purpose.

So I made plans for setting up a separate server for TrueNAS, which given my storage requirements would need another beefy box (at least 64 GB of RAM to begin with, and another 10 GbE NIC). Like everything in life, it's got pros and cons:

Pros:
- Clear separation of concerns: storage on one box, compute on another.
- Good fault isolation: if the compute node fails, storage remains up.
Cons:
- Higher upfront cost: another server chassis, more PSUs, more NICs.
- Higher ongoing power usage.
- For a family homelab, this can be overkill.

There were two alternatives that I had read about:

Proxmox has ZFS support, so it could double as the NAS.
- Manual ZFS configuration, one more thing to learn.
- No out-of-the-box monitoring or alerting.
- Direct hardware access, no virtualization performance loss.
Run a TrueNAS VM inside Proxmox:
- TrueNAS runs as a VM with direct access to the HDDs via passthrough of the PCIe HBA.
- The hard drives are dedicated to TrueNAS; Proxmox does not touch them.
- ZFS is used at the TrueNAS layer for redundancy, snapshots, and replication.
- Performance hit is <5% according to some user reports.
- TrueNAS will enjoy the high-availability of the application server (dual PSUs, dual NVMe, redundant networking...)

After a pros & cons analysis I'm decided to go with the TrueNAS VM as the best compromise for maintainability, reduced hardware count, power draw, upfront cost and reliability. Having Proxmox handle the HDDs might work, but I would lose more time and lose on TrueNAS built-in monitoring and alerts.

Storage layout

At a high level:

IPFire (fast):
- Network logs (firewall hits, IDS reports, VPN connections, etc.).
Proxmox pool (very fast, redundant):
- Two NVMe SSDs in ZFS mirror for:
  - Proxmox host.
  - VMs and containers.
  - Configuration, databases and metadata.
- System will continue working if one drive fails.
TrueNAS pool (slow, redundant):
- Multiple HDDs in a ZFS pool.
- I'm planning initially for 8 HDDs in RAIDZ2, ideally with room for expansion up to 12.
- Fault tolerant up to 2 drives simultaneously.
- Stores:
  - Immich's photo library.
  - Jellyfin media.
  - Music Assistant music.
  - Nextcloud data.
  - Paperless documents.
  - Camera recordings.
  - Proxmox backup.
Snapshots and replication:
- TrueNAS will periodically snapshot datasets.
- Critical data snapshots (Nextcloud, Immich, Paperless, some Frigate recordings) will be replicated to a remote backup (off‑site).

This combination provides both local redundancy and off‑site backups, which is essential: RAID/ZFS is not a backup.

5. Cameras and Doorbell

Cameras are a key part of this homelab, but also a major security and privacy concern. I want to have cameras both outside and inside the house, but I have to make absolutely certain that nobody is snooping through them. I also want to have a video doorbell that will ring on my smartphone even if I'm away from home, but it has to work without cloud connectivity.

I will first have an installer run in-wall Ethernet cables for the cameras, so they will be connected and powered through PoE. There is a wide range of PoE cameras and doorbells available.

Initial plan: UniFi or Reolink?

At first I was debating whether to go with UniFi cameras, because they are good quality (if pricey), their software and apps are best-in-class and I am already planning on getting into their ecosystem with the switch and APs. However Ubiquiti has tried to require a cloud account for UniFi Protect in the past, so I cannot trust that their systems will continue to work in a local-only fashion, which is one of my stated goals.

Having discarded UniFi cameras, my other option was Reolink for its good quality-price ratio and perfect Home Assistant integration. The Reolink RLN16-410 has 16x PoE ports, so cameras would connect and record directly to it. That would simplify the setup and integrate with Home Assistant.

However after some research I ended up discovering that Reolink has no IPv6 support at all, and no roadmap to add it. Complete lack of IPv6 support is especially problematic in a network where IPv6 is a first‑class citizen. Over time, services and ISPs will rely more on IPv6, and running an IPv4‑only camera ecosystem creates unnecessary friction and technical debt. Without any kind of promise that IPv6 will be supported in the future I would risk having to replace all cameras (and the NVR!) at some point, so I went looking for alternatives again.

Frigate and Dahua cameras

My final solution is to stop relying on closed systems and instead have the excellent Frigate NVR software managing the cameras directly. Frigate will run as a VM in the Proxmox server.

Looking for a camera maker that respects standards and has IPv6 support I found Dahua:

Standards support:
- ONVIF for discovery and basic control.
- RTSP for video and audio streams.
- IPv6 support.
2‑way audio capabilities that work with standards‑compliant tools.
A broad ecosystem and documentation from the self‑hosting community.

There will be:

12 Dahua cameras, mix of indoor/outdoor.
1 Dahua doorbell with 2‑way audio.

All cameras and the doorbell will be connected via PoE over Cat6a cabling, ensuring good signal and also making them harder to tamper and interfere with.

Frigate will store recordings on TrueNAS (via NFS or SMB), so they automatically benefit from its hard disk redundancy, ZFS snapshots and backup routines.

Network design for cameras

All cameras will connect to a dedicated unmanaged PoE-enabled layer-2 Ethernet switch in the rack. This switch will uplink to a single port of the UniFi switch, that way it can tag all cameras in a dedicated VLAN:

Receive addresses only on the Camera VLAN (both IPv4 and IPv6).
IPFire firewall will disallow any Internet egress.
Can be accessed only from:
- Frigate (for video ingest and 2-way communication).
- A very small set of admin systems (for configuration).

The main downside of this approach is that the secondary unmanaged switch can only be used for connecting the cameras, because it will be the UniFi router upstream that will handle the VLAN, but it should be cheap enough that having some unused ports won't hurt.

Object detection

Frigate NVR has built‑in object detection:

Frigate runs as a VM (or container) on Proxmox.
It ingests RTSP streams from the Dahua cameras and doorbell.
It performs motion detection and object detection (people, cars, etc.).
It manages event‑based recording, snapshots, and retention policies.

For efficient inference, Frigate uses a Coral TPU accelerator:

A USB or PCIe Coral TPU will be plugged into the Proxmox server.
Proxmox will passthrough the device to the Frigate VM.
As a specialized hardware device, the TPU has low power consumption but is capable of handling multiple video streams in real time.
It's far more efficient and cost‑effective than running GPU inference for this scale of deployment.

Video doorbell

The video doorbell can be viewed as a normal camera, but I want to be able to talk with whoever calls. Frigate can handle 2‑way audio with the Dahua doorbell, enabling local‑only, privacy‑preserving communication without any vendor cloud.

Remember that all cameras will be accessible only through the local network, but my devices will connect through the Wireguard VPN as if they are at home.

When somebody rings the bell that will trigger an event in Home Assistant, and I will be able to do any number of automations, from sending a notification to our phones, playing a chime in the house if we are present, playing a recorded message or even having an AI chat with the visitor.

6. Power and Resilience

Power planning has two separate parts:

Reducing power consumption, both during normal day-to-day and in an emergency (e.g. a blackout).
Securing data and graceful degradation of services in case of power loss.

On power consumption

The house will have a full solar installation:

Plenty of solar panels that should cover the house most days of the year.
Enough batteries to cover the house most nights of the year.
The grid will probably be used only in cloudy winter days, it's not like I'm planning to be completely independent.
An off-grid emergency circuit will allow the batteries and panels to provide electricity in case of a grid blackout.

With this system in place, the rack should be properly powered under most situations. However I can imagine some cases where it might come up short:

Multiple overcast days in Winter could drain the batteries fully. I will not know the performance for sure until we move in and I start collecting stats. This means I cannot rely on just the solar batteries to power the rack.
A prolonged grid blackout (like the 2025 one) could be problematic, especially in Winter.

As protection against these edge cases my plan is to install an UPS in the rack, and program the services for the eventuality.

Power emergency: UPS and shutdown strategy

Critical components will be connected to an UPS:

IPFire router.
Main UniFi switch.
PoE camera switch.
Proxmox server.

I have two goals:

Keep network + cameras + core services running for as long as possible during an outage.
Ensure the Proxmox server shuts down cleanly before the UPS battery is depleted, to keep data safe.

The shutdown flow would look like this:

UPS communicates via USB or SNMP to a monitoring agent (either on Proxmox or a small sidecar device).
On UPS battery activation (and the same if there is a grid blackout and the system relies on the solar batteries) trigger an energy save mode across all systems and notify me.
On low battery threshold:
- Proxmox initiates graceful shutdown of non-critical VMs in order of priority (e.g. Jellyfin, Immich and Nextcloud first).
On critical battery threshold:
- Proxmox shuts down all VMs to avoid data loss.
- Proxmox host shuts down.
Networking and cameras remain up as long as the UPS can sustain them.

This protects data integrity while maintaining some level of surveillance and remote troubleshooting capability.

7. Backups and Recovery

Redundancy is not backup. There has to be an off-site copy of the critical data in case catastrophe strikes the home:

Regular backups of Proxmox VMs (configuration + disk images) to the NAS.
Replication of the Proxmox backups to an off-site location.
Backup of critical datasets from the NAS to the off-site location:
- Photo collection (Immich), with metadata database.
- Nextcloud files and data (calendar, contacts...).
- Paperless documents.
- Frigate event clips (at least metadata and critical recordings, e.g. object detections or alerts).

Periodic restore tests will be part of the operational model: a backup is only as good as the last successful restore test. The plan is to test restoring at least one VM and one dataset periodically to ensure the tooling and procedures actually work.

8. Operations, Monitoring, and Security Practices

Maintainability is a key concern for my systems. I really enjoy tinkering with all this stuff, it's a hobby really, but once setup I expect it to run hassle-free for months. That is the experience with my current homelab, and what I want out of the new one.

The operational practices I'm planning to keep the system maintainable:

Monitoring:
- Basic uptime checks for core services, for example with Uptime Kuma, with real-time notifications.
- Resource monitoring for CPU/RAM/disk/network, with threshold alerts to be aware of bottlenecks.
- SMART and ZFS scrub alerts for storage.
- IPFire intrusion alerts.
Update strategy:
- Apply updates in a controlled fashion:
  - Stable versions only.
  - Take Proxmox and VM snapshots before critical upgrades.
  - Update less critical services first, then core components.
Security:
- Deny-by-default firewall.
- IoT and Cameras VLANs with no Internet egress.
- As few port‑forwarded services exposed to the Internet as possible, and those with minimal permissions and data access.
- VPN only for remote access to Home Assistant, Frigate and any other house services which are not be exposed to the Internet.
- Minimal attack surface on IPFire and Proxmox:
  - Only necessary ports open.
  - Admin only through trusted devices (desktop and laptop computers).
  - Strong keys and passwords.
  - MFA where supported.
  - Up-to-date software.

9. Next Steps

This post captures the target design I arrived at after months of research. The implementation will proceed in stages, which will no doubt end up overlapping:

Procure router/firewall server, install IPFire and start the base routing/firewall setup.
Procure application server and install Proxmox VE.
Procure and install rack, patch panels, and switches once the dedicated server room at the house is ready.
Install servers and connect everything; connect all network cables; connect devices to UPS for power; start servers.
Procure and install HDDs in the application server; set up the TrueNAS VM on Proxmox.
Set up monitoring, logging, alerts.
Bring up Home Assistant, integrate core smart devices, and start defining automations.
Deploy Frigate and integrate all Dahua cameras and the doorbell.
Deploy Nextcloud, Immich, Jellyfin, Paperless, pointing them to TrueNAS storage.
Supporting services (monitoring, logging, alerts, etc.).

Each of these steps will generate its own set of notes, configuration snippets, and "wish I had known this earlier" insights. This is why this post is labeled as "Part 1", because I intend to fully document how this dream homelab evolves from design to reality.

Subcribe for updates!

Got any ideas, suggestions or feedback? Leave me a comment below!

Periodic script to back up installed OS packages

2025-12-08T00:00:00+00:00

A while ago I switched to yadm for managing my dotfiles. One of its features I have been enjoying is its bootstrap support. I have been adding idempotent setup operations

My latest addition is a script that creates backup files with all installed OS packages. In case of disaster if I have to reinstall the OS I can just feed the package manager the latest backup list for that host. I'm working on a different set up for my data backup, that will be the subject of a future post.

On to the script, it is simple and tailored for my systems. It only supports APT (I use Debian for my headless servers) and pacman (I use Arch Linux on my desktop), but adding DNF or other package managers should be easy enough. I wrote it in fish shell, because it is my preferred shell now.

You can see the commit in my dotfiles for the full changes, but I will be reproducing them below as of the time of writing this article.

The main script I put in ~/.local/bin/backup_installed_packages_list.fish:

#!/usr/bin/env fish

function get_hostname
    if type -q hostname
        hostname
    else if type -q hostnamectl
        hostnamectl --static
    else if test -f /etc/hostname
        read -l host_from_file < /etc/hostname
        echo $host_from_file | string trim
    else
        echo "Error: cannot reliably get hostname" >&2
        exit 2
    end
end

set current_hostname (get_hostname)
set current_date (date +%Y-%m-%d)
set backup_dir "$HOME/Backups/$current_hostname/packages"
set temp_file (mktemp)

function save_if_changed -a input_file -a suffix
    set -l target_file "$backup_dir/$current_date$suffix.txt"

    # regex matches YYYY-MM-DD followed optionally by suffix, ending in .txt
    set -l pattern
    if test -z "$suffix"
        set pattern '^\d{4}-\d{2}-\d{2}\.txt$'
    else
        set pattern "^\d{4}-\d{2}-\d{2}$suffix\.txt\$"
    end

    # find the most recent backup file that matches the specific pattern
    set -l latest_backup_name (ls -1 $backup_dir 2>/dev/null | string match -r $pattern | sort | tail -n 1)

    if test -n "$latest_backup_name"
        set -l latest_backup_path "$backup_dir/$latest_backup_name"

        if cmp -s $input_file $latest_backup_path
            echo "No changes in package list compared to $latest_backup_path. Skipping."
            rm $input_file
            return
        end
    end

    # if we reach here, either no backup exists or content is different
    mv $input_file $target_file
    echo "Installed packages backup saved: $target_file"
    notify-send \
        --urgency=normal \
        --wait \
        --icon=backup \
        "New backup of installed OS packages: $target_file"
end

mkdir -p $backup_dir

if type -q apt
    # https://www.debian.org/doc/manuals/debian-reference/ch10.en.html#_backup_and_recovery_policy
    dpkg --get-selections > $temp_file
    save_if_changed $temp_file ""
else if type -q pacman
    # https://wiki.archlinux.org/title/Migrate_installation_to_new_hardware#List_of_installed_packages
    pacman -Qqen > $temp_file
    save_if_changed $temp_file ""
    pacman -Qqem > $temp_file
    save_if_changed $temp_file "_aur"
else
    echo "Error: Neither apt nor pacman found." >&2
    rm $temp_file
    exit 1
end

I made the get_hostname helper because my Arch Linux setup didn't have the hostname command available, only hostnamectl.

notify-send is nice to get desktop notifications, I have been using it frequently in my new scripts, especially those running in the background like this one. Note that it requires libnotify to be installed (libnotify-bin in Debian-based installations). I install it as a common package in my yadm bootstrap configuration.

I then define systemd service and timer units, in ~/.config/systemd/user/backup-installed-packages.service and ~/.config/systemd/user/backup-installed-packages.timer:

[Unit]
Description=Backup list of installed OS packages
After=graphical-session.target

[Service]
Type=oneshot
ExecStart=%h/.local/bin/backup_installed_packages_list.fish
StandardOutput=journal
StandardError=journal

[Unit]
Description=Timer to backup list of installed OS packages

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

Then just enable the timer:

$ systemctl --user daemon-reload
$ systemctl --user enable --now backup-installed-packages.timer

I enable the systemd timer in my yadm bootstrap file.

Responsible education in computing

2025-11-09T00:00:00+00:00

My oldest daughter is turning 7 soon, and I have promised her a gift: her first computer. I'm secretly hoping that she will develop an interest in computing similar to my own. Not because I want to reproduce my childhood in her (although, of course, I would like to share some of my hobbies and passions, like any father would), but because I believe it is important to be educated in how computers and software works—this is digital literacy.

Think about how digital our lives are nowadays. Government, banking, shopping, gaming, socializing, work, movies, shows, videos, music, dating—it all happens increasingly online. Digital privacy can be as important as physical privacy, and you should care about it, but that is difficult if you don't understand how these systems work, or what alternatives exist. Then there are online threats like scams, phishing, viruses, trojans, malware, adware, spyware, ransomware, and security vulnerabilities like zero-day exploits and rootkits. There's also identity theft, social engineering, data breaches, and the need for tools like password managers, firewalls, and encryption.

Computers were the future when I was young, but for my kids, they are the present. What will their future look like? My crystal ball is under repair, but I can identify some key technologies likely to shape our societies:

AI in its different forms
Robotics
Cryptocurrencies
3D printing
AR & VR
Solar/wind power, energy self-reliance

I believe as parents we have to prepare our children for the world they live and will likely live in, not the world we grew up in. That is why I will research, plan and develop responsible ways to expose my kids to these and other technologies. I see computing as the base for everything else, just as learning mathematics is important for so many other disciplines, which is why it's my first step in this journey.

Some personal history

When I was 4 years old (almost 5) I asked the Three Wise Men for a Nintendo NES console, because those videogames looked really cool. However the wise men had other plans for me, and instead brought me my first computer, a ZX Sinclair Spectrum+. I was a bit disappointed at first when I opened my present and it wasn't the game console I'd asked for, but instead a black block with a keyboard of square keys. My disappointment quickly dissolved into ecstasy as soon as my father set it up on the TV, and the first game loaded. I was hooked.

Fidel playing on his ZX Spectrum+

I played every game I could get my hands on. I can still hear the sound of games loading from the cassette player. Some of my favorite games of that era were: Manic Miner, Jet Pac, Sacred Armor of Antiriad, Pipe Mania, Gaunglet, Arkanoid, Chuckie Egg and Thanathos to name a few. It was also a golden age for Spanish videogames, companies such as Opera Soft, Topo Soft and Dinamic made gems that were very popular in Spain and abroad. I remember playing La Abadía del Crimen, Abu Simbel Profanation and Fernando Martín Basket Master among others. These games were challenging, far from the easy dopamine hit of modern mobile or pay-to-win games. They improved my reflexes and concentration, introduced me to the English language and taught me perseverance.

But in the end the videogames were just a gateway, and the real destination that would shape my life was programming. My father has always been a science fiction buff. He'd read so many sci-fi novels that he knew that computers were going to be big in the future. He’d gotten a couple of books for learning BASIC—the programming language supported by the ZX Spectrum (along with assembly, of course)—along with the computer. He was planning on learning himself, but as soon as I saw him typing with the keyboard I got interested, he showed me how to do it and then my curiosity did the rest. I tried copying the sample programs, some of which were many pages long, and it was a revelation seeing that somehow this special text was telling the computer what to do. Then I tried making some modifications, and marveled at how the behavior changed when I tweaked this or that variable. I didn't write any complicated programs (I was only 5-8 years old, mind you), but the seed had been planted.

A few years later, when it was clear that I was into this computer thing for good, my father bought me my first IBM PC: a sensational AMD 286 @12 MHz with 1 MB of RAM, a 5¼-inch floppy disk drive and a 40 MB HDD (at a time when most home computers didn't have an HDD at all!) This opened a whole new world of possibilities. I learned MS-DOS, my first operating system. I learned about AUTOEXEC.BAT and CONFIG.SYS and how to tweak them to scratch those precious KB of available RAM to run my games. I learned to use hex editors to hack saved games for extra money or resources. Eventually I got myself a copy of Borland Turbo C and a C course, and self-taught me C, and later some x86 assembly. My 12-year old mind had to struggle a bit with conditionals, loops and especially pointers, but it all clicked. I could call myself a coder.

With this background I studied Software Engineering at college, and I was the best student in campus. I started using Linux in 1999, then exclusively since 2003. I got a job as a programmer in 2007, right after graduating. And in 2012 I got a great job in Silicon Valley, easily the biggest jump in my career.

Fidel on his PC

I don't want to end this nostalgia trip without reminiscing on some of the many many games that I played with my PC. I'm not going to argue that every game is a piece of art, but I think they are a more positive experience than most adults would credit them for. There is concern nowadays about the harm that screens can do to children, but from my experience the problem isn't the screens themselves, but what you do with them. I've frequently been praised for my good English, which I learned mostly by playing videogames with a dictionary handy. I also learned a lot from games. You easily pick up knowledge and skills when you are a child, and most games offer educational content, brain teasers, or improve reflexes or fine motor skills.

Anyway, here is an inevitably incomplete list of my favorite games from the 1990s:

Adventure games taught me puzzle-solving, humor and how to appreciate narratives, scripts and world building. Titles like Monkey Island 1 & 2, LOOM, Indiana Jones and the Fate of Atlantis, Day of the Tentacle, Sam and Max Hit the Road, Legend of Kyrandia 1 & 2, Simon the Sorcerer, Space Quest 1-6, King's Quest 1-6, Discworld, Full Throttle, The Dig and so many others.
I was big into RPGs, or Role-Playing Games, because I also played a lot of tabletop RPGs. The Eye of the Beholder series, Dungeon Master 2, Ultima Underworld, Fallout 1 & 2, Anvil of Dawn, Wizardry VII, The Elder Scrolls, Dark Sun...
Strategy: Civilization I & II, Colonization, Sim City, Sim City 2000, Sim Ant, Sim Earth, Master of Orion 1 & 2, Master of Magic, Star Control 2, Dune 2, Stronghold, Warcraft, Starcraft, Age of Empires, UFO Enemy Unknown, X-COM Terror from the Deep, Command & Conquer, Theme Park, Theme Hospital, The Settlers, Heroes of Might & Magic, Mad TV...
Puzzle: Gobliiins 2 & 3, Lemmings 1 & 2, The Incredible Machine, Chip's Challenge, Blockout, Tetris...
Arcade: Pac-Man, Doom 2, Duke Nukem 3D, Golden Axe, Raptor, Tyrian, Arkanoid, Descent 1 & 2, Grand Theft Auto, Pinball Fantasies, Tomb Raider, One Must Fall 2097, Cannon Fodder 1 & 2...
Platform: Bubble Bobble, Prince of Persia, Commander Keen, Rick Dangerous 2, Rayman...
Racing: Lotus Turbo Challenge, WipeOut, Carmageddon, Destruction Derby, IndyCar Racing, Test Drive...
Sports: Advantage Tennis, Skate or Die, Ski or Die, Summer Games, Winter Games, 4D Sports Boxing, NBA Jam, PC Futbol, FIFA 94-98, Sensible Soccer, NBA Live, NHL Hockey...
Flight simulators: Microsoft Flight Simulator 4 and 5, F-19 Stealth Fighter, F-15 Strike Eagle II, X-Wing, Tie Fighter, Strike Commander, Privateer 1 & 2, Comanche Maximum Overkill, Secret Weapons of the Luftwaffe...

Looking back, I believe the key was to spend time at the computer. Do that long enough, and you'll end up learning computing.

It's like this quote by Terry Crews:

Go to the gym, don't even work out. Just GO. Because the habit of going to the gym is more important than the work out. Because it doesn't matter what you do. You can have fun - but as long as you're having fun, you continue to do it.

A proper computer

The most important tool to learn computing is, of course, a computer. A proper computer. What do I mean by proper? To put it simply, it's one that gives the user full control:

Install a different OS.
Install any software you want (as long as it's OS and hardware compatible, of course).
Play any media (again, limited by hardware).
Program software without restriction, access the hardware in full.

This definition rules out all popular smartphones and tablets, both Android and iOS. Not because of the touchscreens—notice that isn't among the requirements. The point is to be the master of your own domain.

iOS has been closed from the very beginning. Its users trade sovereignty for comfort. You can install the apps that Apple lets you install, and that is it. They are convenient devices with great quality and polished interfaces and user experience—I won't dispute that—but their users are at the mercy of Apple. Programming apps is also restricted, requiring a Mac computer to code and an Apple Developer account ($99/year) if you want to distribute.

Android started as open-source, Linux at its core, but has since been corrupted by Google. Virtually every Android device is now closed due to Google Play Services, a system component with root (i.e., full) access that, while not being part of the base Android operating system, is a required dependency for most Android applications to work. Google also wants to require developers to register.

If you are still not convinced, others have argued better than me for why it is important to have control over our computing devices:

OK, OK, so no smartphones and no tablets. What then? Are "regular" computers safe? Well for the most part yes, although there are also worrying trends that I don't want to get into right now. They remain the bulwark of computing freedom, and it's paramount that they remain that way.

Choices

There are more choices for a computer available nowadays than ever before, so I'll be opinionated here.

As a free-software advocate, I first discarded all closed-source systems. That means no Windows or macOS. I believe it's important for children to learn the principles behind free software and to value not only its ethics but also the practicalities. Using free software puts us in control of our digital lives, and as those lives grow larger, that control becomes more important.

So some of the myriad GNU/Linux distributions will do the trick (more on that later). Thus my first requirement for the hardware is perfect Linux compatibility. This isn't a difficult task these days, though it wasn't so 20 years ago.

For my children, I don't want to give them a laptop. I'd rather have a desktop setup so they don't try to bring the computer everywhere. This will make computer time more purposeful and easier to control.

For desktop computers there is a wide range of options, from full-size towers to mini PCs and keyboard computers. I knew I wanted something small and affordable to begin with, around 200€.

As I researched, the keyboard computer form factor made me nostalgic for my ZX Spectrum+. I instantly fell in love with the Raspberry Pi 400, but I was hoping for a bit more horsepower. I looked for similar alternatives—found the Orange Pi 800, PineTab2, CutiePi, Juno Tab 2 and RaspPad 3—but wasn't sold on any them for different reasons. Then Raspberry Pi announced the Pi 500, which looked great, except it lacked NVMe support. I decided to play the waiting game, and was duly rewarded: the Raspberry Pi 500+ was released recently. It has NVMe, 16 GB of RAM and a mechanical keyboard!

Raspberry Pi 500+ vs. ZX Spectrum+

Next steps

I just got my hands on the Raspberry Pi 500+, and I'm incredibly excited! I'd love to show it to my daughter right away, but I'll wait until her birthday. I'll prepare it to give her the best first experience possible, hoping for the magic to happen.

The Raspberry Pi 500+ comes preinstalled with Raspbian OS, which is a good option, but I need to evaluate it for a 7-year-old’s needs:

No root account (for now).
No Internet access, and heavily filtered later on.
Educational software.
Creative software (paint, music, video, printing...)
Programming environments.
Age-appropriate videogames.

So my next step is to research child-friendly GNU/Linux distributions. I expect requiring Raspberry Pi compatibility will narrow down the search. I don’t think I'll find a distribution that perfectly fits my requirements, but I just want to find a good base setup.

One of my takeaways from all this is that, given how excited as I am about educating children in computing, it might be my calling. I'm sure other parents would like to teach their children responsible computing but might not have the time or resources. I plan on documenting my efforts and create resources to help and guide them—this article is the first step.

Stay tuned for updates!

Now with self-hosted comments

2025-02-10T00:00:00+00:00

A few days ago I received an email from a reader and fellow blogger, to let me know that he was following my blog and also gave me a shout out about my New Year's resolution to write at least one article every month of 2025, and that I was already running behind.

I love getting messages like that, because if one person takes the trouble to email me I'm sure there are 99 others who have thought about it but haven't done so. That's why after receiving the email I decided to give priority to something I had been planning for a long time: adding comments to the blog. Spoiler alert: goal achieved.

I wanted it to be something fast, so instead of looking for the perfect option I was going to optimize for the one that would take me the least time to implement. That meant starting by trying out the options already built into the Reflex theme I use on the blog (fork of Flex):

Disqus: probably the most popular commenting platform on the web. It is very convenient and powerful, but totally closed and most likely spies and sells our data.
Isso: self-hosted Disqus-style software.

Those who know me will think that I would have jumped head first for Isso, but honestly my first attempt was with Disqus, because it would have been the quickest thing to do: open an account, put a password in the Pelican options and go. Well, my plan quickly run into a wall because it turns out that Disqus no longer has a free plan! I don't mind paying for worthwhile services, but not for comments on this humble blog, so I forgot about Disqus and started looking at the details of Isso.

I liked what I saw about Isso:

Self-hosted on my own domain.
Programmed in Python: easy installation in a virtualenv and it would be particularly easy for me to collaborate with the project if needed.
SQLite as database, no need for a MariaDB or Postgres server. This was one of the main reasons why I chose Shynet for blog analytics.

The only thing I didn't like about Isso is that development seems a bit stalled, there are several PRs that have been waiting for months for merge. But it doesn't seem to be a dead project, I'm hopeful that there will continue to be new versions at least to keep it up to date in terms of compatibility and security.

By the way, I ended up discovering Remark42 as an alternative, after having deployed Isso on my server. It looks very good both in terms of features and development status, but personally I would have continued to choose Isso because it is already integrated into my blog theme.

Technical details

I installed Isso on the same server that hosts this blog, a humble VPS with 2 GB of RAM. I didn't need but to follow the official documentation, but in case anyone is interested this was my process:

Install isso and a WSGI server (I favor gunicorn):

# create new user for isolation
adduser --disabled-password isso

# install isso
su - isso
virtualenv venv
source venv/bin/activate
pip install isso gunicorn

Create server configuration file in /home/isso/isso.cfg:

# https://isso-comments.de/docs/reference/server-config/

[general]
dbpath = /home/isso/comments.db
host = https://blog.fidelramos.net/
notify = smtp

[server]
listen = http://localhost:8081/

[smtp]
host = localhost
port = 25
security = none
to = <notification_email@example.com>
from = isso@blog.fidelramos.net
timeout = 5

[hash]
salt = Eech7co8Ohloopo9Ol6baimi
algorithm = pbkdf2:10000:6:sha256

[admin]
enabled = true
password = <PASSWORD>

Set up reverse proxy through Caddy by adding to /etc/caddy/Caddyfile:

isso.fidelramos.net {
    reverse_proxy localhost:8081
    encode zstd gzip
}

Add a systemd service unit at /etc/systemd/system/isso.service to run isso. Isso documentation includes other init systems.

[Unit]
Description=isso commenting system
Documentation=man:isso(8)

# Require the filesystems containing the following directories to be mounted
RequiresMountsFor=/home/isso

ConditionPathIsDirectory=/home/isso
ConditionPathIsReadWrite=/home/isso

ConditionFileIsExecutable=/home/isso/venv/bin/gunicorn

# Start only if there is a conf file
ConditionPathExistsGlob=/home/isso/isso.cfg

[Service]

Environment=ISSO_SETTINGS="/home/isso/isso.cfg"
ExecStart=/home/isso/venv/bin/gunicorn -b localhost:8081 -w 4 --preload -n gunicorn-isso --log-file /home/isso/isso.log isso.run

#UMask=0007
Restart=on-failure
TimeoutSec=1
User=isso

LimitNOFILE=16384
LimitNPROC=16384
LimitLOCKS=16384

# ensures that the service process and all its children can never gain new
# privileges.
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

Add A and/or AAAA records to your DNS (isso.fidelramos.net in my case) pointing to the server.

Enable and start isso, and reload caddy:

systemctl enable isso
systemctl start isso
systemctl reload caddy

Now I can access isso.fidelramos.net/admin.

Add configuration to pelicanconf.py:

ISSO_URL = "//isso.fidelramos.net"
ISSO_OPTIONS = {
    "vote": "false",
}

Deploy the blog. Comments are live! 🎉

The future

I'm looking forward to seeing how much use the comments will get. I'm sure they will encourage me to write more.

Spam worries me, because Isso has no automatic filtering system. I don't want to anticipate solving a problem before it manifests itself, but we'll see how long the peace lasts....

Did you make it this far? You can't leave without leaving a comment!

My homelab setup

2023-12-29T00:00:00+00:00

Welcome to my homelab!

If you don't know what a homelab is, it refers to running servers from your home. What is a server? It's basically a computer that is always on, connected to the network, often with no display or input. For a longer explanation take a look at this article.

The homelab movement crosses paths with self-hosting (running software services for your own use), but they are not the same thing. You can have a homelab not dedicated to self-hosting services. You can self-host using computers not in your home (e.g. on a hosting provider). But I believe there is a considerable overlap of people interested in both, as is my case.

My setup is quite involved and some people will be reading this article and think “this is way too complicated, I cannot do any of this”. And they would be right! BUT, the good news is that you don't have to do things the same way I do. There is a whole gradient of possibilities with varying degrees of difficulty, effort, reward, control and price, and it's normal to combine them:

Use paid hosted services, i.e. you pay and get access to a software that somebody else manages. You don't get as much control but it's simpler, data might be encrypted and the provider doesn't have the incentive to sell your data (because you already pay for the service). Some of these services are actually open-source software, and you can later migrate to your own instance if you so desire. As an example, I use Protonmail as my email provider, because setting up your own email is ever more complicated to get right.
Use hosting services that make it easy to deploy apps. Same as the previous point, I would prioritize open-source services that you could migrate to your own instance later on. Some examples are Sandstorm and YunoHost. I personally used OwnCube before migrating to my own self-hosted Nextcloud instance.
Run your own services, but on off-premises hardware (i.e. with a hosting provider, or in “the cloud”). There is a wide range of hosting possibilities, from dedicated servers (i.e. a whole real computer) to shared hosting where you share a computerf with other users (separated by some virtualization or containerization techniques), and others. You can read Wikipedia on Internet hosting for more details. Personally I use Vultr to host some services I want to be always online and publicly accessible to the Internet.
Dedicate a computer at home as a server. You can start cheap and small, like a Raspberry Pi 5 for $80. Or you can use some spare computer you don't need anymore, like an old laptop. You can evolve your homelab as you understand your needs better.

There are countless resources on self-hosting and homelabs, if anything the difficulty lies in knowing where to start. My recommendation is to ask people you trust about what to focus on to avoid getting overwhelmed.

Nevertheless, here are some resources:

Self-Hosting Guide is a thorough overview.
Awesome Selfhosted has a long list of self-hostable applications.
selfhosted at Lemmy.World is a good community of self-hosters, who migrated from the selfhosted subreddit.

No matter what you choose to do I encourage you to read and educate yourself about digital privacy and free software. I believe it's time well spent because our lives are increasingly more digital, not knowing about these topics leave you open to abuse.

Lastly I want to thank my friend Harmeet for his encouragement to write this article. It probably wouldn't have happened without him!

Table of Contents

Some history
Goals
Summary
Home server
- Storage
- Network
- UPS
- Applications
  - Caddy
  - Nextcloud
  - Transmission
  - Mumble
HTPC (Home-Theater PC)
VPS
Replication
Backups
Uptime monitoring
The future

Some history

I'm old in Internet terms. Ancient.

I was browsing the World Wide Web and participating in Usenet in 1995. In 1999 I tried out GNU/Linux for the first time. By 2003 I had eradicated Windows from all my computers. The free software bug really bit me deep.

During this time I saw Google's birth as the best search engine of its time, hands down. They had the best developers and made some amazing products. I really believed in Google's “Don't be evil” slogan, and was an enthusiastic user for years. I got a Gmail account back when it required an invite to get in. I put my events in Google Calendar, my photos in Google Photos, my (shared) documents in Google Drive. I got an Android phone because it was open source. I even used Google+ and Google Wave actively.

Fast-forward to 2013 and Edward Snowden's revelations shatter my confidence to pieces. Google was not to be trusted, nor any of the big Silicon Valley companies. I resolved to trace a plan to take back control of my digital life. That plan took years to unfold, but every day it passes I'm more glad that I put in the effort.

This article will not be a chronological story, but only the latest version of my setup.

Goals

My main goal should be clear after reading the story above: take control of my data. I don't want others to abuse it, sell it to innumerable third parties that will analyze it, profile it, use it to target me with ads or misinformation campaigns, train AI models with it... I also don't want to see my accounts closed one day for no good reason, my valuable data gone like tears in the rain. Having my data at home gives me peace of mind.

A secondary benefit of having all my data locally available is that I get to do stuff with it. Things like directly sharing with people, without size limits or accounts nor anybody else knowing about it.

With the advent of powerful open-source AI models I'm excited about the possibilities of having my data readily available:

Train LLMs with all my text, so I can query it for knowledge or ask it to write in my style.
Fine-tune Stable Diffusion models using my photos so I can generate images portraying me, my family and friends.
Use coding LLMs to analyze and write code.
Train voice models with my own voice, I could make podcasts out of these blog posts for example.
Who knows what the future holds! The field is moving at breakneck speed.

My guiding principles are simplicity and stability. The less moving pieces the better. I favor software that is light on resources, properly maintained and that respects backwards compatibility or has clear ways to upgrade. I have limited time so I don't want things to break often.

Using free software is paramount so I can understand what is going on, diagnose problems and fix them, and also that I can trust the software is not doing something I don't want it to.

Summary

My current setup is composed of 3 systems, not counting regular personal computers (laptop, desktop, mobile, tablet...):

A home server for data storage and personal apps.
A HTPC (Home Theater PC) media player.
A VPS for public apps/websites.

I will go over each of them in detail.

Home server

The heart of my setup is an HPE ProLiant MicroServer Gen10, bought in 2018. It has a modest dual-core AMD Opteron X3216 CPU and “only” 8 GB of RAM, but that has proven plenty for my needs, which mostly boil down to storage. It costed 375€ (about $400) back in the day.

The main selling point for me was having a sufficiently powerful GPU, because at first it doubled as an HTPC. Nowadays I have a dedicated HTPC media player, so my pick would have been different, but I'm not replacing it yet.

The server has been happily running Debian GNU/Linux stable all this time. I have been using Debian for over 2 decades and it feels right for servers. If I had more free time I would love to dive into NixOS or Guix, they make for rock-solid systems due to their reproducibility and ability to roll-back to any previous system configuration.

Storage

The server has a 256 GB SSD that I use for the OS and software, and then 4x 6 TB HDDs for data hosting. Storage keeps getting better and cheaper all the time. I paid $189 for each HDD, but nowadays you can a 10 or 12 TB drive for the same price.

Reliability is very important to me, as these hard drives will hold my personal data. I picked the top performer in Backblaze's drive reliability report and all 4 drives have been performing well since 2018.

I use SnapRAID to have a software RAID 5, by dedicating one HDD for parity. That leaves the server with 18 TB of available storage. A cron job runs a SnapRAID sync every night and emails on success or failure.

If a drive fails I might lose up to 24 hours of data, but it shouldn't be fatal as the data should be available somewhere else (e.g. my photos are also in my computer). I prefer the software RAID as I can recover files or undo unwanted changes before the next sync runs. This has proven useful more than once.

The 3 non-parity HDDs are combined into a single logical filesystem using MergerFS. It has worked fine all these years, set and forget.

All drives are LUKS-encrypted. Because the server is headless I set it up so it can be decrypted on boot by connecting an USB drive.

Network

The server is exposed to the Internet because it's going to run services I want to use from anywhere. I configured my home router to give the server a static IP address, and set it as the DMZ.

I don't have a dedicated IP address in my home internet connection, so I use a Python script that updates the necessary DNS records by using my DNS provider's APIs. A cron job executes the script every 10 minutes.

I use Firewalld to manage the firewall. I was using UFW before but it didn't play well with Docker: it will pierce the firewall with the default configuration.

I allow external SSH connections, but only with public key encryption, never passwords.

I use Samba for file access.

UPS

I use an Eaton 5E 850 IUSB in front of the home server, the router and the fiber modem. This gives me a few minutes of runtime if the power goes down. I paid 62€ for it in 2018. A good UPS is expensive, this one is not.

NUT monitors the UPS and gracefully shuts down the server if its battery drops below a certain threshold.

Applications

The home server runs applications that fulfill these criteria:

Used by me and maybe family/friends.
Need access to private data.
Heavier on CPU/RAM/storage than I'd like to put on the VPS.

All applications run with Docker compose.

Caddy

Caddy is a webserver focusing on ease of use. It improves my life by having much simpler configuration with better defaults and automated TLS certificates. Yes, you read that right: Caddy will automatically obtain and renew a TLS certificate for every domain in its configuration. No certbot, no cron, nothing but bliss.

My setup in Docker compose uses the caddy-docker-proxy image. It automates generating a Caddy configuration by looking at the labels of the containers. It's simple, flexible and effective.

Previously I was using the popular Nginx (and I'm old enough to have used Apache). See this article on my migration from nginx-proxy to caddy-docker-proxy.

Nextcloud

Nextcloud is “a safe home for all your data”. It's a platform that not only hosts and syncs files across multiple devices, but it offers a plethora of applications (both first and third-party) to do all kinds of things.

The apps I currently use include:

Files
Calendar and Contacts (synced with my phone using DAVx5)
Photos (need to try Memories, it's supposed to be much better)
Talk
News (RSS reader)
Polls (Doodle replacement)
Bookmarks
Appointments (Calendly replacement)

You can browse the full list of Nextcloud apps.

My setup currently uses the nextcloud image, but this is suboptimal because it runs an Apache instance internally. Given that I'm already running Caddy in front it's a complete waste of resources, but not so much to make it urgent.

I'm assessing migrating to the official all-in-one image. I will write a blog post about it when it's done.

Transmission

Transmission is a BitTorrent client that can run as a daemon, i.e. it runs in the server and I connect to it from my laptop or my phone.

I use the linuxserver/docker-transmission Docker image.

Mumble

Mumble is an “Open Source, Low Latency, High Quality Voice Chat”. I use it mostly for gaming where low-latency really makes a difference.

I use the mumble-docker image.

HTPC (Home-Theater PC)

Originally the home server doubled as media player: it was in the living room and it has a good-enough graphics card to play videos. However after moving to a new house the server is in a different room, so I had to get some kind of HTPC to play my media.

This move happened during the COVID-19 chip shortage so my preferred option was not available. I couldn't wait so I ended up getting a Raspberry Pi 4 Model B, which was in stock.

Nevertheless the Raspberry Pi performs well enough, decodes H.264 and H.265 in hardware, can do audio passthrough to my home theater, and is definitely a solid choice with a big community and great compatibility.

It currently runs Kodi in LibreELEC.

It accesses the media files in the home server through a read-only Samba share.

VPS

I have a small VPS (Virtual Private Server) at Vultr that I dedicate to public content, like this blog and my personal website. It's meant to run services that are open to the Internet, and that should be always up. This is not part of the homelab of course, just self-hosting, but I thought it'd be valuable to include here.

I use Caddy webserver for all static file serving and reverse proxying. Its benefits were already highlighted above so I will not repeat them here.

Besides web hosting the VPS also runs a Shynet instance that tracks analytics in a privacy-respecting way. I got it to work with a SQLite DB, which simplifies deployment and maintenance. I also use Podman to run it. I prefer Podman over Docker because it's lighter (no daemon running) and it's rootless without extra hacks.

If you want to try out Vultr consider using my referral link for some free credits.

Replication

To synchronize data between devices I use Syncthing. It's a beautiful piece of software, fast and end-to-end encrypted. I highly recommend it over something like Dropbox, which is sharing your files with OpenAI.

You might be thinking why would I use a separate app for file syncing when I already got Nextcloud. When I started using Nextcloud in 2018 its file syncing was very buggy for me, which is why I looked at more robust alternatives. From what I hear Nextcloud file syncing has gotten much better, but now I'm happy with my Syncthing setup. If it ain't broke...

I won't go into the details of my whole setup, but these are some Syncthing tips & tricks I have been discovering and applying:

The home server acts as a central always-online node. For example I synchronize the photos taken in my smartphone with my laptop, I also share the same directory with the home server so the phone can sync to it even if the laptop is offline. Then when the laptop comes online it can download from the server and/or the phone if available.
Syncthing supports configuring multiple addresses per device. Most of the time my devices are in my local network so I set local network addresses first, then DNS names and then dynamic as a fallback.
Use QUIC protocol if possible, especially on mobile.
Judicious use of file versioning. I'm particularly liberal versioning files in the server, because it's got plenty of storage. I favor staggered file versioning most of the time.

Backups

I use restic to make weekly backups of all important data in all servers. Important is anything that cannot be downloaded again from the Internet. It has worked well all these years, I only had to use it once and it performed as expected. My current backup is 2,5 TB big and it takes 8 hours to run, a few more hours if restic prune runs.

The script that cron runs weekly is published here.

If I had to set up the backups again today I would look into Borg. They have a nice desktop client called Vorta which I have been using for backing up my laptop.

Uptime monitoring

I use UptimeRobot to monitor the uptime of my services. If they are down I get a nice email about it.

At the time of this writing they give 50 monitors with a 5 minute period for free, which is more than enough for my personal use.

If you want to try it out consider using my referral link, I'd appreciate it.

The future

There are many things I'd like to do, and too little time to do them:

Monitoring and alerting for both the home server and the VPS.
Pihole in the home server.
Update Nextcloud setup to use Nextcloud All-in-One.
Wireguard VPN to tunnel to the home network from anywhere.
Paperless NGX in the home server.
Jellyfin in the home server for media access for the family.
Set up fail2ban for as many services as possible.
Set up game emulators in the media player.

Rest assured there will be new articles as these items get knocked down.

Migrating to fish shell

2023-12-07T00:00:00+00:00

I'm a heavy shell user. As a programmer I enjoy the power and flexibility of the shell, so I always keep multiple terminals open (just one keystroke away thanks to Yakuake).

I have been using Bash for as long as I have been a GNU/Linux user, that's 24 years as of the time of this writing. Back then I didn't think twice about my shell of choice, it was the default and it was much better than what I was used to in Windows and MS-DOS.

With time I discovered that there were other shells and that they were better than Bash. I gave zsh a good try, with oh-my-zsh of course, but it didn't stick. I was too used to Bash and didn't want the hassle of installing a non-default shell.

Fast-forward to early 2023, I heard news that fish shell was going to rewrite the full project in Rust. Full rewrites are dangerous, but sometimes they are the way to go. I believe this is the case here: the project is of manageable size so the rewrite shouldn't take that long; it will bring a breath of fresh air and new contributors.

A few months later, the rewrite is almost done. This is great news, it tells me the project is healthy with an enthusiastic community and maintainers. Reading through the comments on Hacker News I see a lot of people loving fish shell, so I decide to read up more about it.

As I start seriously considering switching to fish as my main shell one thing really jumps at me: fish is not POSIX-compatible. Do I really want to learn a new syntax? Rewrite my existing functions and scripts? I take a look at their documentation and I really like what I see: modern sane syntax. Scripts are actually clean and readable. Yes, learning new syntax and updating some configuration will be some effort, but I can enjoy it for the rest of my life. The words "sunk cost fallacy" cross my mind, and I decide to make the jump.

In this article I will highlight the features that sold me on fish, go over details on how I ported my Bash configuration, mention which plugins I use and share some lessons I learned along the way and the gotchas that bit me.

Features

I'm not going to go through all of fish features, their tutorial does a good job at that.

In particular I'm not going to mention the syntax, but it's so much cleaner and readable, it's day and night compared with a POSIX shell like Bash. For example check the if, for and switch syntax. I also encourage you to compare the Bash and fish versions of some of my custom functions side-by-side (more on that in their dedicated section below).

Here I will focus on features that set fish apart and sold me on it.

Autocomplete

You get a lot out of fish without writing any configuration. For example the out-of-the-box autocomplete experience is amazing. I encourage you to give it a try by just installing fish and running it, no configuration necessary!

Autocompleting commands:

fish autocompletion for commands

Autocompleting arguments, with help extracted from man pages!

fish autocompletion for arguments

Multiline, autoindent, syntax highlighting

Another great feature is the multiline support in interactive mode, with automatic indent and syntax highlightning:

Multiline edit with autoindent and syntax highlightning

When bringing back a multiline command from history I can press Ctrl+e to move around it and edit it in the shell, or Alt+e to open it in my editor. Oh and fish_indent also automatically formats my fish scripts in Emacs when using fish-mode. Nice!

It's worth mentioning that the syntax highlighting is not just the colors of functions, variables, strings, etc., but it also marks errors in red. Quoting the documentation:

Detected errors include:

Non-existing commands.

Reading from or appending to a non-existing file.

Incorrect use of output redirects

Mismatched parenthesis

Colors, themes and prompts

Just run fish_config and you will get a nice web UI to interactively choose theme, change individual colors, change the prompt, browse through all defined functions, browse variables, search the history and remove individual records, and see and edit keybindings.

fish_config web interface

Don't like the web? fish_config theme show will display available themes, with a preview:

fish_config theme show

fish_config theme choose <theme> will set the theme you want in the current session. fish_config theme save will save the current selection as universal variables, applied on next startup.

Analogously fish_config prompt show will display different available prompts, fish_config prompt choose <prompt> will apply to current session, and fish_config prompt save will save the current configuration as universal variables.

See the full documentation for fish_config.

And the list goes on

History search with pagination and individual word search.
Directory history.
Expandable programmable abbreviations.
Private mode which doesn't display past history nor records any.
Advanced argument parsing.
Debugging support.
Event handlers.

Porting my Bash configuration

My full configuration is at my dotfiles repository. Fish configuration is in .config/fish/. It respects the XDG Base Directory Specification, which is another thing I like about fish shell.

My Bash configuration was nothing terribly complicated, mostly some aliases, exports and functions, plus redefining the prompt, enabling autocompletion...

Porting to fish meant learning the new syntax and features. Most of the time reading the fine fish documentation was enough to get me on my way. Special kudos for their fish for Bash users doc.

My porting process can be broken down like this:

Port environment exports.
Port aliases.
Port functions.
Add plugins.
Adjust my init_dotfiles script.
Script to update vendored plugins.

Porting environment exports

This was fairly straightforward, just used set -gx VAR value in ~/.config/fish/conf.d/exports.fish.

Some exports I like to do conditionally, so the configuration works on a variety of systems that might not have my favored tools (e.g. prefer most over less, or exa/eza over ls).

if type -q most
    set -gx PAGER most
    set -gx MANPAGER most
else if type -q less
    set -gx PAGER less
    # don’t clear the screen after quitting a manual page
    set -gx MANPAGER 'less -X'
end

I will probably end up moving many of these global exports to universal variables installed by init_dotfiles script. More on that below.

Porting aliases

Initially I ported my Bash aliases using fish alias and didn't think twice about it. Turns out that fish aliases become functions under the hood, i.e. fish alias is just a wrapper for creating functions.

I discovered that fish has something called abbreviations, which get expanded into their full form after pressing Space or Enter. This has a number of benefits:

Faster than functions.
Commands appear in full in history.
Can edit after expansion.
Can optionally be expanded anywhere in the prompt, not just at the beginning.

I rewrote all my aliases into abbreviations, as they were all command expansions, appearing at the beginning of the prompt.

A fish abbreviation can be configured to appear anywhere in the prompt by using --position anywhere. I'm not using this feature yet but will be on the lookout to see where it will come in handy.

Oh and if you want to keep fish from expanding an abbreviation just press Ctrl+Space.

Porting functions

I have a few Bash functions for some repetitive tasks. I could have extracted them into separate scripts with #!/bin/bash shebangs and placed somewhere in $PATH to keep executing them same as before. But I wanted to give fish a test ride, so I decided to rewrite them. And boy was I glad I did! Just this little practice made me feel comfortable with fish syntax and I learned some valuable lessons by reading the documentation.

These are my ported functions as of the time of this writing:

cbr2cbz: to transform digital comic books in CBR format to CBZ.
mkd: create a directory (with arbitrary level of nesting) and cd into it.
mts2mkv: transform MTS video files into MKV. Some of my digital cameras create MTS videos.
nds: get a NodeJS shell inside a Podman container. I don't trust npm installing gigabytes of unvetted dependencies in my systems.
pdf2cbz: transform PDF into CBZ, meant for digital comic books. Internally uses pdfextractimages function.
pdfextractimages: extract all JPEG and PNG images from a PDF.
splitflac: takes music in the form of a single big FLAC file + a CUE file and creates separate FLAC files for each track, with proper tags.
urldecode and urlencode: simple convenience functions to encode and decode URLs.

I'm still a fledgling fish user, so some bits might not be idiomatic. If you want to help me improve them, pull requests are welcome!

Plugins

One of the things I like about fish is that the default experience is already great with zero configuration. That means if I ever found myself in a fresh or strange environment it wouldn't differ much from my custom setup. But also that I would spend less time customizing. I already spend too much time bikeshedding my Emacs. 😁

Nonetheless there are some great plugins that are worth installing. awsm.fish has a curated list of plugins and other fish resources.

In my case I narrowed it down to three plugins, as I like to keep things simple:

pure prompt

I considered the three main prompt plugins for fish:

I liked tide the most, but to minimize dependencies and simplify setup for now I settled on pure.

I will probably review this decision in the future.

fzf.fish

fzf is seriously amazing, a project that I can recommend hands down. I have been using it with Bash for a while now, and it's the first integration I look for.

fzf.fish is the best fzf plugin for fish. It allows you to:

Search the current directory (recursively), with preview of the files as you scroll.
Search in git log, with preview of the commits.
Search git status, with diff of the changes of each file.
Search shell history, improving upon fish's already good search.
Search processes, showing details in the preview area.
Search variables.

I'm purposefully not including screenshots of fzf.fish as its README already includes a video and screenshots, do take a look at them.

foreign-env

foreign-env is a little plugin that runs a Bash script and imports defined variables in fish. It calls Bash internally, so it's not just a parser.

I use it to source /etc/profile so I can make fish my default shell, more details about that in the Gotchas section below.

Vendoring third-party plugins

Plugin managers like fisher or oh-my-fish are a convenient way of installing plugins, but the problem is that they download and run unsigned code. This is something that I avoid doing as much as possible, as it's a big security hole.

For my fish configuration I decided to vendor them, i.e. include a full copy inside my configuration. This way I have better assurances about the code that my computers will be running. This is particularly important for the shell, even more when I aim to make it my global default.

I put the plugins code in a dedicated vendor directory. Then my init_dotfiles script copies the vendor files into the right locations. I could have done without the vendor directory altogether, but I think keeping them separate is more respectful of the original creators.

The second part is to keep the plugins up to date. I wrote the update_vendor.fish script that I will run from time to time. It downloads the latest version of the plugins I use and puts the files in my vendor directory. If there are any changes I can review them before committing them to my repository.

Lessons learned

In the process of moving my Bash configuration to fish and reading through their documentation I learned some things I think are useful and not immediately obvious. I will share them here in hope that they may be helpful to other people looking to migrate to fish.

Lesson 1: function autoloading

Fish autoloads functions when needed. That means even if you have hundreds of files in your ~/.config/fish/functions directory, they won't all be eagerly loaded when a shell starts, but only when you invoke a function with a name that matches the file name.

A function file can still define multiple functions, but it will only get loaded when the exact name of the file is invoked in a shell.

Lesson 2: error handling with and/or chaining

and and or combiners are a neat way of handling command successes and errors. For example:

tar xvf file.tgz
and echo "Extracted!"
or echo "Error!" && return 1

However the fish documentation advises not to overuse chaining with and and or, as it can lead to wrong code paths being taken. Instead it recommends using if even if a bit more verbose:

if tar xvzf file.tgz
    echo "Extracted!"
else
    echo "Error!"
    return 1
end

Lesson 3: string built-in

The string builtin is powerful, ergonomic and readable, worth learning and using.

string match has both simple globbing syntax or Perl-compatible regexes (I wonder if the syntax will change to Rust's regexes after the rewrite). It can capture groups, ignore case, do partial or entire string match... An example from my pdfextractimages function:

# check that $pdffile ends in .pdf
else if not string match -q -i -e '*.pdf' $pdffile
    echo "$pdffile doesn't seem to be a PDF file"
    return 3

string split is a nice replacement for cut:

> string split . example.com
example
com

# one single split on the right
> string split -r -m1 / /usr/local/bin/fish
/usr/local/bin
fish

string replace is featureful and clean, can do simple string replacement or regex, including capturing groups:

# simple substitution blue -> red, first appearance only!
> string replace blue red 'blue is my favorite blue'
red is my favorite blue

# substitute all appearances with -a
> string replace -a blue red 'blue is my favorite blue'
red is my favorite red

# replace .pdf extension with .cbz using a regex
> string replace -r '.pdf$' .cbz somedocument.pdf
somedocument.cbz

I found this idiom with string replace --filter ... where it first attempts to match input with the given pattern. I pair that with and and or for error control. For example in my cbr2cbz function I replace the .cbr extension with .cbz and at the same time detect if the input is invalid:

set cbzfile (string replace -f -r '.cbr$' .cbz $cbrfile)
or echo "$cbrfile doesn't seem to be a .cbr file" && return 2

One last example is that string escape and string unescape support multiple styles, so I was able to refactor my urlencode and urldecode functions to drop their Python dependency.

Lesson 4: globbing

An important difference with Bash and other shells is globbing behavior.

An empty glob results in an error in fish when used on a command, but not with set, for or count. This is often a good thing to avoid gotchas.

For example this code was failing sometimes because the glob was empty (no images to zip):

zip -9 --move --junk-paths --quiet $cbzfile $tmpdir/*.{jpg,JPG,jpeg,JPEG,png,PNG}

I rewrote it to use set:

set images $tmpdir/*.{jpg,JPG,jpeg,JPEG,png,PNG}
if test (count $images) -eq 0
    echo "No images extracted, stopping"
    return 7
end

zip -9 --move --junk-paths --quiet $cbzfile $images

Also of note is that fish deprecated the ? single-character glob, and will disable it in the future. I like it because it makes working with URLs in the shell easier. In the same vein I like that fish only treats & as the special character to run a program in background if it's followed by a non-separating character. No more having wget, curl or yt-dlp run as a background job because I forgot to quote the URL.

Lesson 5: variable defaults

There is no way in fish to set a default value for a variable, something that can come in handy when defining functions.

Currently I use code like this:

set output $argv[2]
# default to current directory
test -z $output; and set output .

Lesson 6: array variables and slices

Bash supports arrays, both index-based and associative. But the syntax is ugly and hard to remember. Access elements with ${array[i]}, instead of just $array[i]. Count the number of elements with ${#array[@]} (notice the #?). I rarely used arrays in Bash, and when I did I had to constantly look at references.

fish has lists, which are functionally the same, but the syntax is what you would expect coming from other programming languages:

> set fruit apple orange banana mango pineapple

# indexes are 1-based
> echo $fruit[1]
apple

# negative indexing
❯ echo $fruit[-1]
pineapple

# slices
❯ echo $fruit[1..3]
apple orange banana

# reverse slices
❯ echo $fruit[3..1]
banana orange apple

# multiple indexes in one expression
❯ echo $fruit[3 5 2]
banana pineapple orange

# multiple slices in one expression, even with repetitions
❯ echo $fruit[1..3 2..4]
apple orange banana orange banana mango

# number of elements
❯ count $fruit
5

Do note that path variables (those ending with PATH) are treated especially: the elements are joined with colons when exported. This is for compatibility with other shells and programs that might not support arrays.

Lesson 7: keybindings

Do read on fish keybindings, there are some hidden gems. Some highlights:

Alt+← moves to the previous directory in the directory history, Alt+→ moves forward.
Alt+↑ and Alt+↓ search through history for tokens containing the token under cursor. If for example you are looking for PDF files used in the history, you can type .pdf and then Alt+↑ will search for other tokens with that component.
Alt+H opens the man page for the current command. I don't know about you but I find myself opening new shells to look up man pages for the command I'm trying to use all the time. This and fish autocomplete are an invaluable help.
Alt+S prepends or removes sudo from the current command.
Alt+O opens the file under cursor in the $PAGER.
Alt+E or Alt+V open the current prompt in the $EDITOR. Useful for editing multi-line prompts comfortably.

There is support for both Emacs style and Vi style modes.

fish makes it easy to add new bindings.

Gotchas

Sourcing /etc/profile

One common issue when running fish as default shell (hint: chsh) is that it doesn't run /etc/profile and /etc/profile.d/* scripts, as they are not in fish syntax. For me this manifested in a blank screen when loading my graphical environment. I had to get to the console, login as root and change my user's default shell back to Bash.

A workaround is to leave Bash as default shell and run fish only interactively. One way of doing this is to add this to the end of ~/.bashrc (as proposed here):

[ "$PS1" -a -x /bin/fish ] && exec fish

This loads up Bash, but replaces the process with fish. It's inefficient but probably negligible with today's computers. Still it leaves a bad taste in my mouth.

The Archlinux Wiki on fish documents this way of sourcing /etc/profile in login shells by using Bash and then replacing it with a fish process:

# add to ~/.config/fish/config.fish
if status is-login
    exec bash -c "test -e /etc/profile && source /etc/profile;\
    exec fish"
end

However this was not working for me, my desktop environment was still not loading.

My current solution is to use the foreign-env plugin to load the profile:

# add to ~/.config/fish/config.fish
if status is-login && test -e /etc/profile
    fenv source /etc/profile
end

This works for me in Kubuntu 22.04 LTS, I now have fish as my default shell, login and interactive.

Closing words

It's only been a few days with fish as my main driver, but I have been pleasantly surprised by its features. Time will tell whether I find any bumps along the road, but I get the feeling that if anything I will regret not making the switch sooner.

One thing that was bugging me is whether this change would be productive or worth it. It's still early to answer that question, but I can tell you that it was fun and joyful, like a good vacation. And sometimes that makes it worth it.

Looking forward to that Rust rewrite!

Thank you fish team!

Updates

2023-12-19: Correct description of foreign-env plugin, it does run Bash unless originally stated. Add Alt+S keybinding.

Replacing Nginx with Caddy in Docker Compose

2023-11-09T00:00:00+00:00

The main server in my homelab runs a bunch of services, but the heart of it is a Docker Compose configuration, several of them exposed to the Internet via a reverse proxy webserver.

Before I was using three Docker images to run Nginx, proxy traffic to other services and have it generate Letsencrypt certificates for the necessary domains:

nginx with the actual Nginx server running.
nginx-proxy to generate Nginx configurations.
docker-letsencrypt-nginx-proxy-companion: to handle LetsEncrypt certificate provisioning and renewal.

The relevant configuration in compose.yml looked like this:

services:
  nginx:
    image: nginx
    labels:
        com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy: "true"
        UFW_MANAGED: "true"
    container_name: ${NGINX_WEB:-nginx}
    restart: always
    ports:
      - "${DOCKER_HTTP:-80}:80"
      - "${DOCKER_HTTPS:-443}:443"
    volumes:
      - ${NGINX_FILES_PATH}/conf.d:/etc/nginx/conf.d
      - ${NGINX_FILES_PATH}/vhost.d:/etc/nginx/vhost.d
      - ${NGINX_FILES_PATH}/html:/usr/share/nginx/html
      - ${NGINX_FILES_PATH}/certs:/etc/nginx/certs:ro
      - ${NGINX_FILES_PATH}/htpasswd:/etc/nginx/htpasswd:ro
    logging:
      driver: ${NGINX_WEB_LOG_DRIVER:-json-file}
      options:
        max-size: ${NGINX_WEB_LOG_MAX_SIZE:-4m}
        max-file: ${NGINX_WEB_LOG_MAX_FILE:-10}

  nginx-gen:
    image: nginxproxy/docker-gen
    command: -notify-sighup ${NGINX_WEB:-nginx} -watch -wait 5s:30s /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf
    container_name: ${DOCKER_GEN:-nginx-gen}
    restart: always
    volumes:
      - ${NGINX_FILES_PATH}/conf.d:/etc/nginx/conf.d
      - ${NGINX_FILES_PATH}/vhost.d:/etc/nginx/vhost.d
      - ${NGINX_FILES_PATH}/html:/usr/share/nginx/html
      - ${NGINX_FILES_PATH}/certs:/etc/nginx/certs:ro
      - ${NGINX_FILES_PATH}/htpasswd:/etc/nginx/htpasswd:ro
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - ./nginx.tmpl:/etc/docker-gen/templates/nginx.tmpl:ro
    logging:
      driver: ${NGINX_GEN_LOG_DRIVER:-json-file}
      options:
        max-size: ${NGINX_GEN_LOG_MAX_SIZE:-2m}
        max-file: ${NGINX_GEN_LOG_MAX_FILE:-10}

  nginx-letsencrypt:
    image: nginxproxy/acme-companion
    container_name: ${LETSENCRYPT_CONTAINER:-nginx-letsencrypt}
    restart: always
    volumes:
      - ${NGINX_FILES_PATH}/conf.d:/etc/nginx/conf.d
      - ${NGINX_FILES_PATH}/vhost.d:/etc/nginx/vhost.d
      - ${NGINX_FILES_PATH}/html:/usr/share/nginx/html
      - ${NGINX_FILES_PATH}/certs:/etc/nginx/certs:rw
      - /var/opt/acme.sh:/etc/acme.sh
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      NGINX_DOCKER_GEN_CONTAINER: ${DOCKER_GEN:-nginx-gen}
      NGINX_PROXY_CONTAINER: ${NGINX_WEB:-nginx}
    logging:
      driver: ${NGINX_LETSENCRYPT_LOG_DRIVER:-json-file}
      options:
        max-size: ${NGINX_LETSENCRYPT_LOG_MAX_SIZE:-2m}
        max-file: ${NGINX_LETSENCRYPT_LOG_MAX_FILE:-10}

Other services to proxy defined VIRTUAL_HOST and LETSENCRYPT_HOST environment variables. For example this was my Nextcloud service configuration (I will only show the relevant parts):

services:
  nextcloud:
    image: nextcloud:latest
    container_name: nextcloud
    environment:
      VIRTUAL_HOST: ${NEXTCLOUD_HOST}
      LETSENCRYPT_HOST: ${NEXTCLOUD_HOST}
      LETSENCRYPT_EMAIL: ${LETSENCRYPT_EMAIL}

Besides the Docker compose services configuration, I also required a Nginx configuration template (that's the nginx.tmpl that you can see in the configuration), and other per-service tweaks (like increasing the maximum request body size for my Nextcloud instance).

I had already replaced the Nginx webserver that hosts this blog to Caddy, so I wanted to do the same with the home server. I decided to use caddy-docker-proxy because it was similar to nginx-proxy, so it would simplify the migration. What I didn't expect is how much simpler it would become.

The above configuration was replaced by:

services:
  caddy:
    image: lucaslorentz/caddy-docker-proxy:ci-alpine
    restart: unless-stopped
    ports:
      - "${DOCKER_HTTP:-80}:80"
      - "${DOCKER_HTTPS:-443}:443"
    networks:
      - default
      - webproxy
    labels: # Global options
      caddy.email: ${CADDY_EMAIL}
      UFW_MANAGED: "true"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      # this volume is needed to keep the certificates
      # otherwise, new ones will be re-issued upon restart
      - ${CADDY_FILES_PATH}:/data

Proxying a service is now a matter of just adding some labels:

services:
  nextcloud:
    image: nextcloud:latest
    container_name: nextcloud
    labels:
      caddy: ${NEXTCLOUD_HOST}
      caddy.reverse_proxy: "{{upstreams 80}}"
      caddy.request_body.max_size: ${NEXTCLOUD_MAX_UPLOAD}

And that was it!

Caddy automatically manages the TLS certificates, so no need for any extra services. It doesn't get any easier than that.

I was also able to remove the Nginx config files, instead replacing them with a couple labels in the containers that required some special configuration.

Before I close off this article I want to highlight how well nginx-proxy and its acme-companion had been working for years with minimal maintenance. Their stability and backwards compatibility had been rock solid, kudos and a big thanks to their creator!

Automating Python code quality

2023-06-23T00:00:00+00:00

In this article I explain what I mean by code quality and how it benefits developers.

In the first half I discuss general concepts and workflows that apply to most software projects. Even if you are not writing Python code you might learn something from it.

In the second half I offer a step-by-step guide to setting up tooling to improve the code quality of Python projects. I focus on tools I have used and favor after 15 years of professional experience with Python. I list some available alternatives to each of my proposals.

Code quality

All coders develop a (subjective) sense of nice or ugly code, and it gets stronger the more experienced we get with a particular technology.

Sometimes it's hard to put into words, but it's usually a combination of:

Formatting rules.
Hard errors, e.g. referencing an undeclared variable or dereferencing a null pointer.
Soft errors or warnings, e.g. having an unused import or variable.
Language idioms.
Comments.
Documentation.

All these have one aspect in common: they are not part of the strict definition of the programming language, so developers are free to do things in different ways. Whenever there is room to do things differently, people will do it in any way possible, and fight for their reasons to do it their way.

I define code quality as the set of metrics that apply directly over the source code artifacts (i.e. the source files). These metrics are a subset of non-functional requirements. Different projects will care more or less about each metric, and might ignore some of them.

Why care about code quality?

Even if end users of the software might not directly notice, higher code quality has clear benefits:

Reduced count of bugs per line of code (LoC).
Consistent formatting leads to more readable code, which improves maintainability.
Better code introspection by having richer type information. Explicit function parameter types, jumping to definitions, browsing documentation...
Happier developers. This is more important than it sounds, because current developers will be more productive and likelier to stay longer with the project, and it also makes it more attractive to new recruits. "Quality of your projects" ranks #2 in developer happiness, and "quality of development tools" is #4, according to Zenhub's 2022 report on developer happiness.

But nothing is free, and the downside is obvious: time and effort have to be invested to keep code quality high.

Fortunately we have a way out: automation.

Why automate code quality?

Manually keeping code quality high has a constant level of effort. By automating these processes, most of the effort is paid upfront when setting up the tooling.

These tools have some maintenance cost, like upgrading versions or onboarding new developers, but for big or long-running software projects the benefits should outweigh the costs.

The benefits stack:

Detect errors earlier in the software life cycle (ideally as the developer is writing the code).
Less or no time spent formatting code. Even better: less mental overhead as you stop paying attention to formatting and focus more on what the code is doing.
No time spent discussing with other developers about how exactly to format the code. And besides time, also avoid negative feelings that can arise in those discussions.
Happier developers. Yes, I mentioned this before, but if a high-quality codebase makes programmers happy, an automated one even more so.

Categories under consideration

These are the specific code quality categories that will automated for Python in the second half of the article.

Linting

Linting is the detection of errors and warnings of all kind by static code analysis, i.e. parsing code without actually executing it.

Linting hints cover a wide range of issues, from "referenced an undeclared variable" to "unused import" or "potentially insecure string interpolation". Some linting warnings can quickly get pedantic and unhelpful, but the hard errors are definitely good to know about as early as possible, before your code runs for real.

In a compiled language (e.g. C, C++, Rust, Go...) the compiler catches the errors so the program will not even build until they are fixed. In an interpreted language such as Python or Javascript many of these errors happen at runtime. That is where a linter shines by detecting them before the program runs.

Formatting

Code format is every stylistic aspect that can be changed without making the code fail. Different developers have different preferences on how to style code, so much so that there are discussions that span decades:

Tabs or spaces?
How many spaces per indent level? 2? 4? 8?
Single quotes (') or double quotes (")?
Opening brace at the function declaration/flow-control statement or at the following line?
How to format function definitions? Parameters on same or separate lines? One parameter per line or up to the maximum line width?
How many blank lines between module declarations?
How to sort imports?
Etcetera.

A codebase with consistent formatting makes it faster to read and understand the code, which is why many developers care about formatting. The problem is that opinions vary, and different developers want to format things in different ways. These discussions are not just a time sink but also they can mine morale by pitting developers against each other, sometimes splitting them into winners and losers.

When formatting is automated those discussions go away. At most there would be initial discussion on the exact configuration to use for the formatting tool, but once set it will rarely change, as that would cause a reformatting of the whole codebase.

Another common counter-argument by some developers is that they are too attached to their way of formatting and no tool matches it exactly. If you think like that I encourage you to give automated formatting a serious try. For me the pure joy of auto-formatting gets over any personal preferences I might have. After a while I wondered why I even cared in the first place.

Type-hinting

This is specific to Python and Javascript/Typescript (as the most popular languages with optional type-hinting, I'm not saying there are no others!).

Both Python and Javascript are dynamically typed languages, which means that the type of a given variable can change at runtime, as opposed to statically typed languages, where all types are set and checked at compilation time and cannot change at runtime.

Python is strongly typed, so every variable has a specific type at any point in time during the execution of the program. On the other hand Javascript is weakly typed, as variables can change type depending on the context sometimes (e.g. '1' + 1 will cause a cast of 1 into a string for a surprising final result of '11'). Typescript is strongly typed which is a good reason to prefer it over Javascript.

As with almost everything in programming there are pros and cons to each approach, and discussing them goes beyond the goal of this article.

In the case of Python it wasn't until PEP 484 and Python 3.5 that it became possible to annotate the code with the expected type of variables, arguments and function returns. Many years later now there are tools to not only check the code according to the type hints, but also to do other cool stuff like runtime data validation, like Pydantic does.

Type-hinting would be worth a full blog post about its benefits, but for this article I will focus on type checks, which can detect a variety of bugs that linting alone cannot.

What about tests?

First of all: of course you should have tests! However they are out of scope for this blog post as they can hardly be fully automated, in the sense of setting them up once and they will cover future code without any additional work. Also tests are usually application-specific, so they are not directly shareable between projects like the proposed tools in the article are.

What about code coverage?

Code coverage, i.e. counting how much code is covered by tests and how much isn't, is also a common code quality metric. I am not including it in this article for two reasons:

It wraps the unit test runner, which is outside of the scope of this article.
Code coverage arguably contains a perverse incentive for developers: to avoid adding code that would be hard to test. For example detecting error cases implies extra code and specific test cases, and therefore extra effort to maintain the code coverage, so a developer might think of not handling some errors and let them bubble up the stack trace.

Code coverage can be a good quality metric for some projects and teams. Pay special attention so it doesn't actually become a detriment to code quality.

If you are interested, the most common way of measuring code coverage in Python is through the coverage package.

Python code quality tools

flake8

flake8 is a linter, it will check for a wide variety of issues ranging from errors (e.g. mention to an undeclared variable) to warnings (e.g. unused import) to styling (e.g. long lines or excessive whitespace).

This is a good starting configuration, place it in a .flake8 file in the project root:

[flake8]
max-line-length = 90
extend-ignore = E501, E203, W503
per-file-ignores =
    settings*.py:E402,F403,F405
exclude =
    .git,
    __pycache__,
    .tox,
    .eggs,
    *.egg,
    .venv

max-line-length should match whatever you configure in Black (see next section). I like 90, but feel free to use less or more. Black's default is 88. I wouldn't recommend going over 100, as long lines can be harder to read in diff tools such as Github's split view:

Example of split diff with long lines

per-file-ignores is useful to ignore some errors or warnings in some files. In this example I'm ignoring the warning about using import * in Django settings files, as that is a common pattern. Feel free to tailor it to your specific case.

Some alternatives to flake8:

pylint: a configurable linter with plugin support.
pyflakes: a simpler linter that ignores styling. Faster as it parses source files instead of importing them. No per-project configurability. No plugins. It can be a good choice for projects using automated formatting tools like Black and isort and that don't use plugins.

Black

Black is a Python code formatter. It will take any Python file (as long as it doesn't have syntax errors) and change it to conform to Black's code style: split long lines; use double quotes in strings; remove spurious whitespace and empty lines; and a lot more. In short, Black will make files PEP 8 compatible.

Many programmers are very opinionated about style, and don't want a tool to enforce a style they don't like. However automating code formatting not only saves a ton of time but it also frees us from not having to discuss styling with other developers. If you are like me, once you let go of whatever specific style you prefer you will feel the bliss and never go back. Saving a Python file in the editor and seeing it autoformat instantaneously still feels magical.

So to use Black, just install it with pip then add this to pyproject.toml file in the project root:

[tool.black]
line-length = 90
target-version = ['py310']
extend-exclude = '''
(
  migrations   # Django DB migrations
)
'''

Configuration is straightforward and uses good defaults. You could run Black without any configuration at all and it would work great, just be aware that line-length defaults to 88. I like explicit configuration so I recommend adding a configuration file even if minimal.

Notice the use of extend-exclude instead of exclude. Black's exclude setting has a sane default that will exclude most files that you would usually want (.git and .hg, build, dist, etc.). So unless there are files in exclude that you actually want to include then better stick to extend-exclude to append to the default exclude values.

When introducing Black into an existing codebase you can use this trick to avoid ruining git blame.

Black documentation also includes a guide on configuring other tools to be compatible with black.

When running Black in CI without pre-commit use --check option so files are not actually reformatted, it will just give a pass/fail result with the faulty files.

Some alternatives to Black:

yapf: created by Google. Still considered beta as of the time of this writing, while Black released its first stable version on Jan 29, 2022.
autopep8: uses pycodestyle. It can correct deprecated code, so it can be useful for migrating codebases from Python 2 to 3.x.

isort

isort will automatically sort all imports in your Python files following a given configuration, or as they say in their home page: isort your imports, so you don't have to.

The imports will be split into groups and sorted alphabetically. The groups are:

Python standard library.
Third-party dependencies.
Current project dependencies.

Configuration goes into the same pyproject.toml file that Black uses. This is an starting point:

[tool.isort]
profile = "black"
line_length = 90
multi_line_output = 3
skip_gitignore = true
skip_glob = ["**/migrations/*", "**/settings/*"]
src_paths = ["<your_code_dir>"]

The key is to use profile = "black" to make it compatible with how Black expects imports to be sorted. This gets us the best of both tools: Black will format the files and isort will sort the imports, without interfering with each other. isort has many other profiles available.

Of course make line_length match the same setting from Black and flake8.

In skip_globs you can put files that shouldn't be automatically processed by isort. In the example I'm excluding Django DB migrations, as they are automatically generated, and setting files, that in some projects have bad-style imports, like star imports.

isort has a --check flag same as Black, use it in CI if running without pre-commit.

mypy

Type hints are a relatively new addition to the Python language, starting with PEP 484 in Python 3.5. Since Python has always been a dynamically-typed language, any typing extension is something optional that developers can choose whether to use or not.

Mypy is a static type checker for Python. It will parse the type hints and determine whether the program is correct or whether there are inconsistencies. For example if a variable can be a string or null and you try to invoke a string method on it, mypy will throw an error. At that point you can check the code and control for the null case appropriately, thus avoiding a potential runtime bug.

Type checking being optional in Python, it actually requires quite a bit of effort from the developer to add the type hints. It will reduce the number of bugs and make the code easier to read and understand, but it's not clear whether this effort is worth it from a productivity standpoint. I think it's worth it for long-running projects, especially those starting from scratch. There are also type-aware libraries like Pydantic that make type hints useful for other use cases, and combined with type checking can make it worthwhile for more projects.

So on to setting up mypy, it has extensive configuration. It supports either a mypy.ini file or it can use pyproject.toml. I recommend using pyproject.toml as it's becoming the standard file for all Python tooling, I consider other configuration files as deprecated.

This is an example mypy configuration in pyproject.toml:

[tool.mypy]
mypy_path = "./src"
follow_imports = "silent"
strict_optional = true
warn_redundant_casts = true
warn_unused_ignores = true
disallow_any_generics = true
check_untyped_defs = true
no_implicit_reexport = true

[mypy-pyproj.*]
# use for dependencies that don't have typing stubs
ignore_missing_imports = True

[tool.pytest.ini_options]
addopts = [
    "--import-mode=importlib",
]
pythonpath = "src"

For external dependencies that don't have type hints you should check and install the type stubs, they usually have the types- prefix. They will provide type hints that mypy can use. For packages with no type hints and no stubs available you can use the ignore_missing_imports option as in the example above to silence mypy errors.

For larger codebases you want to check out the mypy daemon to speed up checking.

Do check mypy documentation on adding to an existing codebase if that is your case.

Alternatives to mypy:

pyright: A Microsoft project. Combines a type checker with a language server. pyright is open-source but there is also pylance which is closed-source but leverages pyright.
pyre: a type checker with a focus on performance. Claims to be "performant on large codebases with millions of lines of Python". Created by Facebook.

Other tools to consider

bandit checks for potential security bugs. It integrates with pre-commit. There is a flake8-bandit plugin.

flake8-bugbear is a flake8 plugin that adds additional checks for some Python pitfalls, i.e. code that can have unexpected results.

pydocstyle automatically formats docstrings. If you are using flake8 check out the flake8-docstrings plugin.

Integrations

The tools I have presented can be invoked in a CLI, but that is not how you want to use them most of the time. Remember, the goal is automation.

This section is split into 3 parts:

IDE (Integrated Development Environment), i.e. your code editor.
CI (Continuous Integration).
pre-commit hooks.

IDE

If you are reading this article and are already a programmer, chances are you are already using an IDE of some kind. Whichever IDE it is, you should learn how to set up all these tools in your IDE. If your code editor doesn't allow for running all the necessary tools then it might be time to look for another one. It's a bit out of scope for the article, but talking IDEs is always fun, so I will leave just some thoughts and recommendations for whoever wants to listen.

Before jumping onto specific IDE recommendations, I want to mention the Language Server Protocol, or LSP. Modern IDEs don't implement features like auto-complete, jump to definition, find references, refactor operations, etc. They communicate with a language server that offers those operations. For example for Python there is python-lsp-server, which has a variety of plugins to integrate with different tools. This way your editor doesn't need to worry about using Black for formatting or flake8 for linting, just to communicate with a LSP server that supports them.

About IDE recommendations, I personally use Emacs and don't think I will ever change. The learning curve is like no other, but for that same reason it can take you higher than any other editor (flamewar!). To connect to a LSP server you want to use either lsp-mode or eglot (which is part of Emacs itself as of Emacs 29). If you are curious about my Emacs configuration it's published here.

A lot of people like Vim. It is a great editor, fast and available in virtually every UNIX-like operating system in the form of the venerable vi. I used it for years until I decided to give Emacs a try after hearing so much about it. I have heard that Neovim is amazing, definitely check it out if you are into Vim.

Nowadays Visual Studio Code, or VSCode, is the popular new kid on the block. And for good reason: the developer experience is great, easy to set up and add plugins to do all kinds of stuff. I dislike it because it's closed-source software, so I would bet it's a matter of time until Microsoft pulls the rug under its users in some nasty way, like it's happened time and time again with closed-source software. At least there is VSCodium which is a free-software version with the proprietary bits removed, like Chromium is to Chrome, but who knows how long it will last.

If you are a young developer, or someone willing to put in the effort, I recommend choosing between Emacs and Vim/Neovim and committing to it for a few months. You won't regret it, they will last you a lifetime.

Continuous Integration (CI)

It's important to notice that all these tools and checks on a codebase can be undone by an uncollaborative or uncaring developer. For this reason it's a good idea (some would say a requirement) to enforce the checks, keeping developers from merging any breaking changes. Usually this is done by a system that runs the necessary checks on the code before it is merged to the main branch. That system is called a Continuous Integration system, or CI.

The workflow for getting changes into the project then becomes:

Developer commits changes to a branch and creates a Pull Request (PR) in the code hosting service (e.g. Github or Gitlab).
Code hosting notifies the CI tool, which starts a new job to check the branch at its current commit.
Developer cannot merge until the job finishes successfully (usually there are special permissions for some users to override this and merge anyway).
On a successful job, developer can merge the changes. If new changes are added another CI job will be triggered and flow will restart.
On errors the developer can look at the job log to figure out what is wrong, fix it and push the new changes. This will trigger a new CI job and restart the flow.

There are dozens of CI platforms out there, but I will list a few that I have worked with:

pre-commit

When introducing CI-enforced checks it becomes painful for developers to push some changes and then seeing an error several minutes later. The feedback loop between errors and their fix should be as short as possible for maximum developer productivity (and happiness!). For this reason it's a good idea to not just allow the developers to run the checks locally, but to make it easy and fast to do so.

pre-commit is a framework for managing and maintaining multi-language pre-commit hooks. git hooks are just programs that can be set up in git repositories to be executed when git triggers certain actions. As pre-commit's name suggests it commonly runs before a commit is created, but it can be set up to run before git push or at some other point if necessary.

Benefits of pre-commit:

Same checks run by developers and CI, reducing the chances of the developer encountering CI errors later in the development pipeline. This is the more valuable the slower the CI checks are, as it reduces the feedback loop for developers.
Quick and easy to add third-party hooks and keep them updated (pre-commit autoupdate).
Opt-in for developers. They can either not use pre-commit at all (by just not installing the hooks) or skip the checks for specific commits (by using git commit -n). But then don't complain about CI failing! ;-)

Cons of pre-commit:

Yet another dependency/tool to manage.
pre-commit installs hooks in a separate environment, which can be wasteful if you are already installing some tools in the project's virtualenv.

This is an example pre-commit configuration that integrates my recommended tools, just put it in .pre-commit-config.yaml:

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
  rev: v4.4.0
  hooks:
  - id: check-json
  - id: check-toml
  - id: check-yaml
  - id: check-merge-conflict
  - id: debug-statements
- repo: https://github.com/PyCQA/isort
  rev: 5.12.0
  hooks:
  - id: isort
    args: [--filter-files, src/]
- repo: https://github.com/psf/black
  rev: 23.3.0
  hooks:
  - id: black
    exclude: migrations
    args: [--check,--config=pyproject.toml]
- repo: https://github.com/PyCQA/flake8
  rev: 6.0.0
  hooks:
  - id: flake8
    args: [--config=pyproject.toml]
- repo: local
  hooks:
    - id: mypy
      name: mypy
      entry: "sh mypy_run.sh"
      language: system
      # run if any Python file is changed
      types: [python]
      # mypy will always check all files
      pass_filenames: false
      # use require_serial so that script is only called once per commit
      require_serial: true

pre-commit autoupdate finds new versions of the pre-commit hooks you use and automatically updates the .pre-commit-config.yaml file to use them.

Mypy and pre-commit

Making mypy work in pre-commit can be challenging, because pre-commit runs on its own environment, so it won't see any of the installed dependencies for the project. This is the reason why the official mypy hook for pre-commit has limited usefulness in my opinion.

My usual hack is to define a shell script that activates the project's virtualenv and invokes mypy, and have pre-commit run it whenever a Python file changes. This is a sample mypy_run.sh file as referenced in the .pre-commit-config.yaml example from above:

./venv/bin/mypy src/

If you know of a better way of running mypy with pre-commit I would love to hear it, please send me a message.

Conclusion

I first gave a short generic introduction to code quality and why it's important for software projects.

Then I presented ready-to-use configurations using my tools of choice for Python programming, and instructions on how to automate the necessary checks to maximize developer productivity and happiness.

Privacy-respecting self-hosted Web Analytics

2022-05-27T00:00:00+00:00

It all started with a desire: I wanted some basic analytics for this blog to know how it's doing, which articles are popular and which are the main referrers. But therein lied the conundrum: I am a privacy zealot, very much against extensive online tracking (just look at the browser extensions I use), so using a third-party service that collected all my visitors' data was against the question.

I knew there were open-source web analytics projects out there, like Piwik which has been around for a long time, but never had the time or need to look into them seriously. Until now.

Please note that this article is by no means an extensive list. There are a lot of other options out there, just check out this big list. Each software has its own pros and cons, but I hope this article can help you choose and make the jump towards self-hosted analytics, or at least to abandon Google Analytics in favor of an open-source-based SAAS option.

Table of Contents

My requirements
Matomo (formerly Piwik)
Plausible
Shynet
Open Web Analytics (OWA)
Conclusions and my winner

My requirements

I researched options for my specific use case:

Must be free software (free as in freedom, not as in free beer) and self-hosted.
Simple deployment, the fewer moving parts the better. SQLite if possible so no DB server will be needed. No Redis/Memcached if it can be avoided. Option to deploy without Docker would be nice.
Reasonable resource usage. My sites don't usually get a lot of traffic, so I would like to reuse the 2 GB VPS that hosts this blog.
Technology that I can easily hack is a plus. I will favor Python projects, but YMMV.
Lightweight and not too intrusive. I basically just want to know which articles are seen when, plus referrers, no need for advanced analytics.
Actively maintained.

Matomo (formerly Piwik)

Home page

GPLv3. Paid SAAS cloud service with free self-hosting option.
Installation docs.
- Requires a webserver (Apache, Nginx, etc.)
- Requires MySQL.
- Configuration of Matomo is all done in-app.
Matomo self-hosted requirements:
- <100k page views/month: 1 server for both app and DB. 2 CPU, 2 GB RAM, 50GB SSD disk.
- ~1M page views/month: 1 server for both app and DB. 4 CPU, 8 GB RAM, 250GB SSD disk.
- ~10M page views/month: 2 servers for app and DB. 8 CPUs, 16 GB RAM, 400GB SSD disk
- ~100M page views/month: 3 or more servers for app, DB and load balancer. 16 CPUs, 32 GB RAM, 1 TB SSD disk.
PHP for backend. VueJS and TypeScript for frontend. Github repository and developer guide.
Very comprehensive set of collected analytics. Wordpress plugin available.
About 10 commits per week, multiple contributors.

Plausible

Home page

AGPLv3. Paid SAAS cloud service with free self-hosting option, but the latter lags behind in features (2 releases per year).
Official installation documentation. tldr; needs docker-compose, PostgreSQL and Clickhouse.
From Plausible's requirements documentation:

The server must have a CPU with x86_64 architecture and support for SSE 4.2 instructions. We recommend using a minimum of 4GB of RAM but the requirements will depend on your site traffic.
Elixir/Phoenix backend. React and Tailwind frontend. Github repository and developer guide.
<1 KiB tracking script. No cookies. I couldn't find a concrete list of tracked metrics, but it seems less extensive than Matomo or OWA.
Weekly activity, usually condensed in long-form pull requests.

Shynet

Home page

Apache License Version 2.0. Self-hosted only.
Easy deployment with Docker or docker-compose, otherwise undocumented manual deployment. Supports any DB that Django supports, including SQLite. Cache server optional.
Low resource consumption, basically just a Python process running Django, ~100 MiB of RAM when using SQLite.
Python/Django backend. Uses its own TailwindCSS plugin for frontend. Easy to build from source (Docker image). Github repository and developer guide
<1 KiB tracking script. No cookies. 1x1 pixel tracking fallback. Well-defined list of captured metrics, pretty basic.
Not a lot of new feature development, but dependencies are kept up to date and the main developer is very responsive to issues and pull requests (see conclusions).

Open Web Analytics (OWA)

Home page

GPLv2. Self-hosted only.
From OWA's installation documentation and technical requirements:
- Only PHP projects supported.
- In-app install wizard.
- Only Apache 2.x officially supported. Nginx works but needs manual configuration.
- MySQL with strict mode off.
Unclear as to the exact requirements.
PHP backend. Webpack for frontend assets. Multiple repositories: base repo, Wordpress plugin repo, SDK for PHP repo.
Very detailed tracking, including heatmaps and "Domstream session recordings", but only for PHP apps. Dedicated integration plugins for WordPress and MediaWiki.
Actively maintained, but apparently by a single developer.

Conclusions and my winner

Looking at the highlighted requirements my choice for this humble blog was easy: Shynet won hands down, it checked every box. For bigger sites Plausible and Matomo both seem to be good options. I would say Plausible has a better foundational technology (it launched in 2019 after all) but Matomo has been around for longer (2007), it's more battle-tested and seems to collect more data. Plausible has a comparison page between Plausible and Matomo. OWA might be a good option for PHP and Wordpress sites.

Shynet's hackability, point 4 in my requirements list, quickly came into the spotlight because SQLite support was broken. Fortunately I was able to fix it and deploy it into the very server where you have been reading this article. This is why I love free software.

Another issue with the deployment was that the server didn't have Docker installed and I wanted to keep it that way, so I used the opportunity to learn about Podman. I instantly fell in love with it and its ability to run rootless containers, which is a boon to security. Only issue I encountered was having to change the permissions of the mounted data volume, otherwise Shynet's Django process was unable to write the SQLite DB as files were owned by root inside the container. This article explains it well, tldr; need to use podman unshare.

To end this article, here is a screenshot of Shynet's data for this blog (isn't it great to be able to share knowing that nobody's privacy is at stake?):

blog.fidelramos.net on my Shynet instance

Web browser add-ons I use

2022-02-23T00:00:00+00:00

This article lists the web browser extensions I use and why. I will keep it updated from time to time.

Currently I use Brave as my main web browser, and is the one I recommend to everyone. It offers a great experience by default, with features such as a built-in ad blocker, built-in Tor integration, vertical and grouped tabs, a decent cryptocurrency wallet, IPFS support, built-in video calls. The out-of-the-box experience is better than the alternatives in my opinion, especially for regular users who don't know or don't want to tweak their software.

I previously used Firefox. I appreciate that it is the only major browser with its own rendering engine (Gecko) that is not based on the hegemoneous WebKit that is the base of virtually all the other web browsers. As a web developer I know how dangerous it is to have a monoculture around a single web browser engine, that is what we suffered in the late 90s and early 00s with Internet Explorer.

I try to be careful with the browser extensions I install, as they have deep access to browser data. This means a malicious or compromised add-on can quickly become a nightmare if it manages for example to steal the credentials for my email account (although 2FA should mitigate that risk). For this reason I carefully weigh the usefulness of the add-ons versus that risk. The ones I use are usually popular, with thousands of users and often editor-picked.

Without further delay, these are the add-ons I'm currently using on Brave:

KeepassXC-Browser: I use KeepassXC to hold my passwords and secrets, this is its official browser extension which enables form autocomplete. It works pretty well although not as well as competitors such as Lastpass or 1Password, but I wouldn't trust them with my data.

In Firefox I used to use these extensions:

uBlock Origin: the best ad blocker there is. It uses little CPU and RAM, usually saves more than it consumes, i.e. web pages load faster without all the crap they usually have (banners, tracking scripts, etc.). Especially on mobile it can be very noticeable. I used to use a bunch of other privacy-related extensions (NoScript, ClearURLs, PrivacyBadger, HTTPSEverywhere and DecentralEyes), but after reading this article on Arkenfox I learned that they are superfluous if uBlock Origin is properly configured.
Firefox Multi-Account Container: while Firefox has built-in support for containers, this extensions is needed to manage them. It's an official add-on from the Firefox team itself, so I fully trust it. What are containers? They are a way of categorizing tabs, in order to isolate the information that is accessible. For example I set up my bank's website to open in a Banking container, so a potential attacker that makes my browser run code with a security exploit in a different tab wouldn't have access to it unless it was opened in the same Banking container. I also use it to isolate logged-in tracking, for example I have a Google container where I'm signed-in to Google, but in other containers I'm logged-out. I use a bunch of other containers: Gov for government websites, Personal for my self-hosted web apps, Shopping whenever I'm buying something...
Skip Redirect: detects and skips redirects, to avoid unnecessary tracking. For example if you look closely you will notice that when clicking on a Google result the URL points to Google, which then redirects you to the target. This way Google knows which results you visit. This extension extracts the target and goes to it directly, thus avoiding tracking.

Updates:

2022-11-19: Replaced Metamask with Tally Ho; added Sidebery.
2023-01-20: Added Skip Redirect; removed NoScript, ClearURLs, PrivacyBadger, HTTPSEverywhere, DecentralEyes.
2023-04-20: Updated Tally Ho to Taho, they rebranded in Feb 21, 2023.
2026-03-31: I now use Brave as my main web browser.

Nextcloud CardDAV and CalDAV discovery with DNS records

2022-02-17T00:00:00+00:00

I use a self-hosted Nextcloud instance to hold a lot of my personal data, and that includes my contacts and calendars. Recently I had to reinstall DAVx5 on my phone, and I was surprised that the Nextcloud account was failing to be added.

The DAVx5 error logs showed that it was making requests to the /.well-known/caldav and /.well-known/carddav URLs, as described in RFC 6764. Nextcloud was returning proper 301 HTTP responses, checked in a web browser, that redirected to the actual WebDAV endpoint, so it looks like the problem was that DAVx5 wasn't properly following them.

Browsing the DAVx5 documentation on service discovery they mention that the DAV standard also supports using SRV and TXT DNS records pointing to the endpoints to be used for CardDAV and CalDAV, and I decide to give it a go as it sounded easier than further debugging DAVx5 or digging into Nextcloud's and/or the webserver's configuration.

I added the following DNS records to fidelramos.net:

_caldavs._tcp 10800 IN SRV 0 1 443 cloud.fidelramos.net.
_caldavs._tcp 10800 IN TXT "path=/remote.php/dav/"
_carddavs._tcp 10800 IN SRV 0 1 443 cloud.fidelramos.net.
_carddavs._tcp 10800 IN TXT "path=/remote.php/dav/"

The path I find by following /.well-known/caldav and /.well-known/carddav URLs in my Nextcloud instance.

Once the DNS records were in place DAVx5 connected successfully, but I had to use fidelramos.net as the base URL, instead of cloud.fidelramos.net as before.

Unlocking a LUKS-encrypted partition on boot with an USB drive

2019-10-13T00:00:00+00:00

The use case I wanted to solve was this: I have a headless server with a LUKS software-encrypted hard drive, and I want to be able to reboot it without having to input the password on a keyboard. The solution I implemented is to create a LUKS keyfile on a USB drive, so if it is plugged on boot the keyfile will be used instead of the password.

Before we begin, a few words on the security of this approach:

Whoever gets access to the USB drive will be able to decrypt the target device. Consider the security requirements of the USB drive: could be put on a safe box, hidden away or you could always carry it with you.
It's best if the USB drive is only used for this purpose, as any other system where the drive is plugged in can potentially read and leak the encryption key.
This is clearly reducing security to gain in usability. Sometimes that is the balance that we are looking for, but oftentimes it's not. It's up to you to decide the value of what you are protecting, potential attack scenarios and how much security it needs.

OK, I'm assumming you have a formatted USB drive that you have mounted on the computer with the LUKS device you want to unlock on boot, e.g. on /mnt. Then create a key file with random information:

# dd if=/dev/urandom of=/mnt/key bs=4096 count=1

Add the generated key to LUKS, so it can be used to decrypt the root device:

# cryptsetup luksAddKey /dev/sda3 /mnt/key

/dev/sda3 should be the encrypted partition that holds your root filesystem.

Next we will update /etc/crypttab, which defines encrypted volumes that can be handled by LUKS. Your crypttab file should have an entry similar to this one:

cryptroot UUID=452ac6ac-8bbb-484f-b508-a11a5585e031 none luks

We need to update this line to use the passdev script provided by cryptsetup. This script handles waiting for a device to become available, then mounting it and reading the key from a file. It's a good idea to refer to the USB drive using a device that won't change unexpectedly. I'm using /dev/disk/by-label/<LABEL>, which is a symlink to the device based on the partition label. After updating /etc/crypttab it should look like this:

cryptroot UUID=452ac6ac-8bbb-484f-b508-a11a5585e031 /dev/disk/by-label/FIDELRAMOS.NET:/key:20 luks,keyscript=/lib/cryptsetup/scripts/passdev

Leave the original mapper name (cryptroot in this example) and UUID of your encrypted root, otherwise things will break as the mapper name is referenced in /etc/fstab. Just change the none and add the keyscript to the options.

Last step is to regenerate the initramfs image so that it includes the passdev script, otherwise the boot process would fail:

# update-initramfs -k all -u

And that's it! When rebooting the system it should mount the USB drive automatically and decrypt the root partition using the created key file. If the USB drive is not present it should fall back to reading the passphrase from keyboard in 20 seconds.

blog.fidelramos.net - Software

Vibe Coding Escapades: Venice.ai KDE widget

The Problem

Implementation

Result

Lessons

Self-hosted Tailscale, Part 4: Tailscale-Only Web Services

The architecture

Why two Caddys and not one with an IP allowlist?

The Docker Compose setup

Building a Caddy image with the Gandi DNS plugin

Moving services to the tailscale Caddy

DNS setup

Verification

The gotchas I hit so you won't have to

Compose silently falls back to 0.0.0.0

Failed containers leave zombies

Binding 0.0.0.0:80 conflicts with 100.x.y.z:80

caddy-docker-proxy needs the service's network

Would I do it again?

Future improvement: limited access for other people

Vibe Coding Escapades: pelican-copy-code plugin

The Problem

Execution

Results

Lessons

Vibe Coding Escapades

Self-hosted Tailscale, Part 3: Syncthing over Tailscale

Goals

Why do this

Reconfiguring Syncthing

About Listen Addresses

Configure peers by tailnet address

Firewall

Router

Verifying

Encountered pitfalls

Listen address %tailscale0 syntax

Peers can't find each other after disabling discovery

MagicDNS fails at startup on a laggy client

Syncthing on a containerized deployment

Discovery announcement caching

What's next

How I Sandbox my AI Agents

Firejail setup

Results and Next steps

Self-hosted Tailscale, Part 2: Ad-blocking DNS

Goals

Why not just Android's Private DNS?

Architecture

Blocky in Docker Compose

Blocky configuration

Sanity check from the host:

Changes to Headscale

Why this is arguably better than device-level DoT

Encountered pitfalls

Android "cannot connect" with Private DNS set

DNS resolution breaks on the host itself

Clients don't pick up the new DNS

Have to disable MagicDNS

What's next

Self-hosted Tailscale, Part 1: Headscale and clients

Goals

Why not just use Tailscale's hosted service?

Architecture

Prerequisites

Headscale in Docker Compose

Headscale configuration

DNS records

First run

Creating a user and pre-auth keys

Tailscale on Linux

Tailscale on Android

Other clients

Encountered pitfalls

Wrong key

Android device appears as invalid-... node in headscale

Android "Not connected" with no error

Android Private DNS interferes with Tailscale bootstrap

MagicDNS not resolving on a specific device

Compose silently falls back to `0.0.0.0`

Binding `0.0.0.0:80` conflicts with `100.x.y.z:80`

`caddy-docker-proxy` needs the service's network

Listen address `%tailscale0` syntax

Android device appears as `invalid-...` node in headscale

`ansible-ssh-masters.sh`