Setting up a Nebula cluster

Since I have several computers across three different locations, I was looking for a way to ensure that they could all talk to each other without setting up any port forwarding on my router (which only works for a network you yourself control). The solution I came across was Nebula.

Essentially, Nebula is a tool that allows you to create a virtual private network that connects all of your machines together, with each machine having their own dedicated IP address. This works regardless of whether the machines are hidden behind a home network or a public network. In this post, I will give a brief overview on how I configured it for my machines.

Networking model

I only have one node that has a dedicated public IP address. Therefore, this node will act as the master, and all other nodes will be the slaves.

Setting up Nebula

Install nebula on all the nodes. Choose a node that will act as a certificate authority; I recommend you choose the master node. Create a directory, such as $HOME/.nebula, and run the following command inside the directory:

# nebula-cert ca -name "Etrem Certificate Authority"

This creates the files ca.crt and ca.key, which will be used to sign certificates for each node.

In order to allow nodes to connect to each other, you must designate one or more nodes to serve as lighthouses. Essentially, the nodes search for these lighthouses to get information about how to reach the other nodes. Therefore, lighthouses must have a dedicated IP address. In our case, the master node will suffice as the only lighthouse.

For each node, master or slave, generate a certificate pair signed by the certificate authority:

# nebula-cert sign -name master -ip 10.42.0.1/24
# nebula-cert sign -name slave1 -ip 10.42.0.2/24
# nebula-cert sign -name slave2 -ip 10.42.0.3/24
# (etc.)

Take the resulting NODE_NAME.key and NODE_NAME.crt files that were created in the previous step, and copy them onto their respective hosts, as well as the ca.crt file generated by the certificate authority. Place them in the folder /etc/nebula/.

On the master node, create the configuration file at /etc/nebula/config.yml and paste the following, filling in for the NODE_NAME and STATIC_IP_ADDRESS:

pki:
  ca: /etc/nebula/ca.crt
  cert: /etc/nebula/NODE_NAME.crt
  key: /etc/nebula/NODE_NAME.key
static_host_map:
  "10.42.0.1": ["STATIC_IP_ADDRESS:4242"]
  # You may add additional hosts that have a static public IP address
lighthouse:
  am_lighthouse: true
  serve_dns: false
  interval: 60
listen:
  host: 0.0.0.0
  port: 4242
relay:
  am_relay: true
  use_relays: true
punchy:
  punch: true
tun:
  dev: nebula1
  unsafe_routes: []
  routes: []
logging:
  level: info
firewall:
  inbound:
    - port: any
      proto: any
      host: any
  outbound:
    - port: any
      proto: any
      host: any

On regular nodes, the file will be mostly the same, except for the following changes:

lighthouse:
  am_lighthouse: false
  hosts:
    - 10.42.0.1
relay:
  relays: ["10.42.0.1"]
  am_relay: false
  use_relays: true

You can tinker with these settings and figure out what works best for you.

Setting up custom DNS resolution

On the master node, install dnsmasq, and assign static IP addresses to each of your nodes, as well as a private domain name, by adding the following lines to /etc/dnsmasq.conf, replacing the domain names however you wish:

address=/master.neb/10.42.0.1
address=/slave1.neb/10.42.0.2
address=/slave2.neb/10.42.0.3
(etc.)

Enable the dnsmasq daemon on the master node. On each slave node, create the following file at /etc/systemd/resolved.conf.d/nebula.conf:

[Resolve]
DNS=10.42.0.1
Domains=~neb

After this, enable the systemd-resolved daemon on every node, slave or master. This allows each node to resolve private domain names ending in .neb

Setting up SSH keys

In order to allow one to log into any node from any other node, we must set up the SSH keys. In the master node, generate an ssh key that will be used to sign all others:

# ssh-keygen -f id_ed25519_nebula_ca_sign

This will create a private key and a public key. Copy the public key into each node at /etc/ssh/nebula-ca.pub. Now we will generate the ssh key at each node. Run the following command at each node separately:

# ssh-keygen -f id_ed25519_nebula_auth

This creates an ssh-key for that node. One by one, copy the public key onto the master node, and sign it:

# ssh-keygen -s id_ed25519_nebula_ca_sign -I user@nebula -V +52w id_ed25519_nebula_auth.pub

This creates a file called id_ed25519_nebula_auth-cert.pub. Copy this file to its respective node. Then add this line to each node’s /etc/ssh/sshd_config:

TrustedUserCAKeys /etc/ssh/nebula-ca.pub

And finally, add the following to your ~/.ssh/config:

Host    *.neb
    User    username
    IdentityFile    ~/.ssh/id_ed25519_nebula_auth
    CertificateFile ~/.ssh/id_ed25519_nebula_auth-cert.pub

Now each node has been configured to accept your SSH keys. Test your new Nebula cluster to make sure it works.

Time synchronization

One issue you may face, especially in older computers, is that the system clock drifts off from real time, and this messes up with the Nebula protocol. To solve this, simply enable the ntpdate daemon.