Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

A production deployment

This chapter sketches a real fleet managed with Navi. It is here to show what the workflow looks like at scale, rather than on the single node the tutorial builds. The names below are placeholders; the shape is what matters.

The fleet is heterogeneous and lives in one flake:

  • Around thirty GCP virtual machines, grouped across several tenants and environments. Many have no public address and are reached through an Identity-Aware Proxy tunnel.
  • On-prem hardware running on encrypted ZFS, unlocked remotely as part of a deploy.

Within each tenant the machines fall into tiers — an edge tier that terminates DNS, a gateway, ingress nodes, the application nodes behind them, and an observability tier. The same tier shape repeats across tenants and across the staging and prod environments.

From that single description, operations that would otherwise be multi-step runbooks reduce to one command each.

The NixOS configuration it deploys

Every command below operates on ordinary NixOS configuration. The fleet lives in a flake, under the navi output. nixpkgs comes from a flake input rather than the channel, so evaluation stays pure. Each node is a NixOS module — the same one you would write without Navi — plus a small deployment block that tells Navi how to reach it and which slices it belongs to. The tags are what the @ selectors match:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.11";
    navi.url = "github:cafkafk/navi";
  };

  outputs = { nixpkgs, ... }: {
    navi = {
      meta.nixpkgs = import nixpkgs { system = "x86_64-linux"; };

      # Merged into every node.
      defaults = { pkgs, ... }: {
        services.openssh.enable = true;
        system.stateVersion = "24.11";
      };

      "staging-<tenant>-ingress-1" = { ... }: {
        deployment.tags = [ "ingress" "staging" "<tenant>" ];

        services.nginx.enable = true;
      };
    };
  };
}

The remaining snippets are entries inside that navi output. A cloud node adds its provider, so Navi can reach it through the proxy tunnel when it has no public address, and names the provisioner that creates it:

"staging-<tenant>-app-1" = { ... }: {
  deployment.tags = [ "app" "staging" "<tenant>" ];
  deployment.provisioner = "<tenant>";
  deployment.providers.gcp = {
    project = "<project-id>";
    zone = "europe-west3-a";
    iap = true;              # reach it over Identity-Aware Proxy
  };
  deployment.buildOnTarget = true;
};

An on-prem node declares how its encrypted disk is unlocked, which is what navi disk-unlock drives. The password command runs locally; the remote command runs in the node's initrd:

"rack-01" = { ... }: {
  deployment.tags = [ "on-prem" ];
  deployment.unlock = {
    enable = true;
    passwordCommand = "pass show hosts/rack-01/zfs";
    remoteCommand = "zfs load-key -a && killall zfs";
  };
};

The tier shape repeats across tenants and environments, so in practice the nodes are generated rather than written out one by one. A small function stamps the environment, tenant, and tier into each node's name and tags, and merges in that tier's shared module:

mkNode = { env, tenant, tier, n }: {
  name = "${env}-${tenant}-${tier}-${toString n}";
  value = {
    imports = [ ./tiers/${tier}.nix ];
    deployment.tags = [ tier env tenant ];
    deployment.provisioner = tenant;
  };
};

Feeding a list of those through builtins.listToAttrs is what produces the thirty-odd nodes. Because every tag falls out of one naming convention, the selectors in the sections below — @ingress, @staging, <tenant>-* — each address exactly the slice you expect.

Selecting slices of the fleet

Nodes follow a naming convention and carry tags, so --on can slice the fleet any way the work demands — by tenant, by environment, by tier, or by a single host. A glob widens the selection; a tag prefixed with @ matches a role that spans tenants:

navi apply --on '<tenant>-*'        # one tenant, every tier
navi apply --on @staging            # one environment, every tenant
navi apply --on @ingress            # one tier, everywhere it appears
navi apply --on '<tenant>-app-1'    # a single node

The selector is the same across provision, install, apply, build, and ssh, so a slice that means something to you means the same thing to every command.

Standing up a tenant

Creating every machine for a tenant, installing NixOS onto them, and switching them to their configuration is one command. The machines do not exist when it starts and are serving their configuration when it finishes:

navi provision --on '<tenant>-*'

This replaces a sequence of Terraform, nixos-anywhere, and Colmena runs. When a tenant needs its infrastructure rebuilt but not reinstalled, scope it down:

navi provision --on '<tenant>-*' --skip-install --reprovision

Some classes of node are expensive to bring up and are provisioned one at a time rather than all at once. Listing them and walking the list keeps each one isolated, so a failure stops at the node it happened on:

for p in (navi provision --list)
    navi provision $p
end

Rolling out a change in order

A change rarely lands on the whole fleet at once. The edge and gateway tiers go first, then the nodes behind them, so traffic keeps flowing through a node that is already on the new configuration. Selecting more nodes widens each step without changing its shape:

navi apply --on @gateway,@ingress switch -v
navi apply --on @app switch -v

Nodes whose change touches the boot path reboot as part of the rollout, and Navi waits for each to come back before reporting success:

navi apply --on @ingress switch -v --reboot

If activation fails on a node, the previous generation stays active, so a bad rollout does not take the tier down with it.

Reaching a single node

When one node misbehaves, open a session on it with the same connection details Navi deploys with, or run a single command and return. This works through the proxy tunnel for machines with no public address:

navi ssh <tenant>-edge-1
navi ssh <tenant>-edge-1 -- systemctl restart bind

Bringing up encrypted hardware

The on-prem machines boot into an encrypted ZFS volume and stall in initrd until it is unlocked. Unlocking one remotely lets it finish booting and rejoin the fleet:

navi disk-unlock <node>

Watching the fleet

For day-to-day operation across this many nodes, the terminal interface shows the fleet by tenant, environment, or tier, with live logs and task state:

navi tui

The point

None of these commands is doing something a pile of separate tools could not do. What changes is that they come from one description and one tool. The fleet has a single source of truth, and scaling it up is a matter of selecting more nodes, not adding more steps.