A deep dive into using Tailscale with Docker

Summary

The video script discusses the integration of Tailscale with Docker for creating secure, interconnected containerized environments. It explains the benefits of running Tailscale in a container, such as ACL control, reverse proxy replacement, and the ability to connect services across different locations. The script covers how to add a container to a Tailnet using both an auth key and an OAuth client, highlighting the differences in API access, lifespan, and tagging between the two methods. It also delves into Docker's networking with Linux kernel namespaces and how Tailscale's 'serve' and 'funnel' can be used to expose applications on the public internet securely. The video concludes with a practical example of setting up a self-hosted recipe manager on the Tailnet with HTTPS support, demonstrating the ease of managing complex container networking through Tailscale.

Takeaways

• 📦 **Containers in Production**: Using containers in production has become the norm, unlike 10 years ago when it was unusual.
• 🌟 **Tailscale and Docker**: The video discusses the integration of Tailscale with Docker, including adding containers to a Tailnet and exposing applications to the public internet.
• 🔒 **Security and Control**: Tailscale allows for control access via ACLs and can replace reverse proxies, enhancing security without complex configurations.
• 🛠️ **Technical Possibilities**: Tailscale enables connecting services across different environments, such as a home GPU workload with cloud services like AWS or GCP.
• 🚧 **Simplicity of Networking**: Tailscale's encrypted WireGuard-based tunnels eliminate the need for port forwarding, complex firewall rules, or dynamic DNS.
• 🐳 **Official Docker Image**: Tailscale provides an official Docker image available on Docker Hub and GitHub, with customizable parameters through environment variables.
• 🔑 **Authentication Methods**: Two programmatic methods for adding containers to a Tailnet are discussed: using an auth key and an OAuth client, each with its own advantages and use cases.
• ⏱️ **Key Expiry**: Auth keys have a maximum lifespan of 90 days, but this does not mean that the authenticated nodes expire; they are separate from the node keys used by WireGuard.
• 🏷️ **Tagging and Ownership**: When adding a node to a Tailnet, it must have an owner. Auth keys default to the user who generated the key, while OAuth clients use a tag for ownership.
• 🤝 **Docker Networking**: Docker's network mode allows merging of network namespaces, effectively allowing containers to share the same networking context.
• 🌐 **Exposing Applications**: Tailscale serve and funnel can be used to expose applications running in containers to the Tailnet or the public internet with TLS support from Let's Encrypt.

Q & A

What was considered unusual about running a container in production about 10 years ago?

About 10 years ago, running a container in production was considered unusual because it was not a common practice at the time. Nowadays, not running a container in production is more unusual as it has become a standard approach in the industry.

What are the benefits of adding a container to your Tailnet?

Adding a container to your Tailnet allows you to control access via ACLs, replace reverse proxies, and access any other service on your Tailnet from these containers. It enables seamless connections between services, even those located in different environments like your home and the cloud.

How does Tailscale simplify the process of connecting services?

Tailscale simplifies the process by using encrypted WireGuard-based tunnels, eliminating the need for complex port forwarding, firewall rules, or dynamic DNS configurations.

What are the two primary methods for adding a container to your Tailnet?

The two primary methods for adding a container to your Tailnet are using an auth key and using an OAuth client.

What is the main difference between an auth key and an OAuth client in terms of API access?

An auth key grants full API level access to any client that authenticates using it, whereas an OAuth client limits API access via scoping, allowing for more granular control over what actions can be performed.

What is the maximum lifespan of an auth key in Tailscale?

An auth key in Tailscale has a maximum lifespan of 90 days.

How does the use of tags differ between an auth key and an OAuth client when adding a node to a Tailnet?

With an auth key, the node is added to the Tailnet as the user who generated the key. With an OAuth client, the node is owned by the tag assigned at secret creation time, and the client assumes the identity of a tag owner.

How does Docker's network mode work when adding a container to a Tailnet?

Docker's network mode allows you to group two containers together under the same Linux kernel networking namespace, effectively merging their network contexts.

What is the purpose of Tailscale Serve?

Tailscale Serve acts like a reverse proxy, allowing you to provide a configuration file that directs web requests to specific ports on your Tailnet. It can also automatically request and use TLS certificates from Let's Encrypt for secure connections.

What is the difference between a self-hosted application using Tailscale Serve and Tailscale Funnel?

Tailscale Serve allows you to proxy requests internally to your Tailnet, while Tailscale Funnel can expose the application across the internet, allowing external access with no additional configuration required beyond enabling the feature.

How does mounting a directory instead of a specific file in Docker influence the container's operation?

Mounting a directory instead of a specific file allows Docker to properly detect changes to the files within that directory using fsnotify, which is important for dynamic configurations like those used by Tailscale Serve.

What is the significance of using environment variables like 'ts serve config' in a Docker Compose file?

Environment variables like 'ts serve config' are used to pass configuration information to the Tailscale container, allowing for dynamic and flexible configuration without hardcoding values into the Compose file.

Outlines

00:00

🚀 Introduction to Tailscale and Docker

The video begins with a discussion on the prevalence of container usage in production environments over the past decade. It introduces Tailscale and Docker, explaining how to integrate a container into a Tailnet (Tailscale network). The video aims to cover the reasons for running Tailscale in a container, the use of Tailscale serve and funnel to expose applications, and provides chapter markers for easy navigation. It also touches on the benefits of using Tailscale, such as controlling access via ACLs, replacing reverse proxies, and enabling direct connections between services across different locations through encrypted WireGuard-based tunnels.

05:02

🔑 Methods for Adding a Container to Tailnet

The paragraph explains two primary methods for adding a container to a Tailnet: using an auth key and using an OAuth client. It discusses the differences in API access, with an auth key providing full API access while an OAuth client scopes API access. It also covers the lifespan and expiry of auth keys, which last a maximum of 90 days but do not affect the authentication of nodes post-creation. The paragraph further explores the concept of tagging in Tailscale, where nodes must have an owner, and how this is managed through auth keys and OAuth clients.

10:07

🤖 Adding a Container with an Auth Key

This section provides a step-by-step guide on adding a container to a Tailnet using an auth key. It details the process of using the Tailscale Docker image, setting up a Docker Compose file with specific parameters like the container name, hostname, and environment variables. The importance of the state directory for persisting the container's state is emphasized, along with the need for specific capabilities like 'net admin' and 'sys module' for networking. The paragraph concludes with generating an auth key from the Tailscale admin console and using it in the Docker Compose file.

15:09

⏱️ Expiry and Ephemeral Settings for Auth Keys

The paragraph discusses the misconceptions around the expiry of auth keys and how they differ from the underlying WireGuard keys. It explains that while auth keys have a 90-day lifespan, the actual keys used for node authentication do not expire simultaneously. The concept of ephemeral nodes, which automatically remove themselves from the Tailnet after going offline, is introduced. The paragraph also covers how to tag auth keys and the implications of using single-use keys versus reusable ones.

20:13

🏷️ OAuth Clients and Tag Ownership

The focus shifts to OAuth clients, contrasting them with auth keys. The paragraph outlines the process of generating an OAuth client secret in the Tailscale admin console, assigning tags, and using these secrets to add nodes to the Tailnet. It emphasizes that with OAuth clients, nodes are owned by the tag assigned at the time of secret creation, which is different from the user ownership associated with auth keys. The paragraph also covers how to modify Docker Compose files to use OAuth clients and the implications of disabled expiry for nodes added via OAuth.

25:19

🤓 Docker Networking and Namespaces

This section delves into the technical aspects of Docker networking, specifically Linux kernel namespaces. It explains how each container gets its own networking namespace and how the 'network mode' in Docker Compose allows grouping containers under the same namespace. The use of 'nsenter' to inspect a container's networking context is demonstrated, and the paragraph shows how to merge network namespaces to allow containers to share the same networking context.

30:25

🍞 Self-Hosted Recipe Manager with Tailscale Serve

The video concludes with a practical example of deploying a self-hosted recipe manager app called Meely on the Tailnet using Docker. It details the use of a Docker Compose file, the significance of the application running on port 9000, and how Tailscale serve can be used to proxy web requests to the application. The paragraph also discusses the configuration of TLS certificates from Let's Encrypt for secure connections and the use of environment variables for dynamic configuration. Lastly, it touches on the potential scalability of this approach across multiple containers and operating systems.

🌐 Exposing Applications with Tailscale Funnel

The final paragraph demonstrates how to use Tailscale funnel to expose the self-hosted recipe app to the public internet, allowing external access with HTTPS. It explains the process of enabling the funnel feature and the considerations for security when exposing applications. The paragraph also invites viewers to share their experiences with Tailscale and Docker and provides links to additional resources for further learning.

Keywords

💡Container

A container in the context of the video refers to a lightweight, standalone, and executable software package that includes everything needed to run a piece of software, including the code, runtime, system tools, system libraries, and settings. Containers are a fundamental technology in modern computing and are used extensively in the video to discuss how they can be integrated with Tailscale for networking purposes.

💡Tailscale

Tailscale is a zero-config VPN (Virtual Private Network) solution that allows users to create secure networks in minutes. It is based on the WireGuard protocol and is used in the video to demonstrate how it can be used to connect containers across different environments, such as local machines and cloud services, to form a unified network.

💡Docker

Docker is a platform that allows developers to develop, ship, and run applications in containers. It is used in the video to illustrate how containers can be added to a Tailscale network and how Docker can be used to manage these containers effectively.

💡ACLs (Access Control Lists)

ACLs are used to control access to network resources by specifying which users or system processes are granted access to objects, as well as what operations are allowed on them. In the video, ACLs are mentioned in the context of controlling access to a Tailscale network when a container is added to it.

💡Reverse Proxy

A reverse proxy is a server that retrieves resources on behalf of a client from one or more servers. It is used in the video to discuss how Tailscale can replace the need for a reverse proxy by allowing direct access to services on a Tailscale network from containers.

💡Auth Key

An Auth Key in the context of Tailscale is a token used to authenticate a device to the Tailscale network. It is discussed in the video as one of the methods to add a container to a Tailscale network, with an emphasis on its single-use and expiration properties.

💡OAuth Client

An OAuth client in the video refers to a method of adding a container to a Tailscale network that involves creating a client secret associated with specific tags and permissions. It is highlighted as an alternative to using an Auth Key, offering different security and management features.

💡WireGuard

WireGuard is a secure network tunnel protocol that is used by Tailscale to create encrypted connections between devices. It is mentioned in the video to explain the underlying technology that enables secure communication within a Tailscale network.

💡Docker Compose

Docker Compose is a tool for defining and running multi-container Docker applications. It is used in the video to demonstrate how to configure and start multiple containers, including those with Tailscale integration, using a YAML file.

💡TLS (Transport Layer Security)

TLS is a protocol used to provide privacy and data integrity between communicating applications and their users. In the video, TLS is discussed in the context of securing web applications running in containers on the Tailscale network using certificates from Let's Encrypt.

💡Funnel

In the context of the video, a Funnel in Tailscale is a feature that allows specific applications to be exposed to the public internet securely. It is mentioned as a way to extend access to a self-hosted application on the Tailscale network to users outside the network.

Highlights

Containers in production have become the norm over the past decade, shifting from being an unusual practice to an expected one.

Tailscale and Docker are discussed, focusing on how to add a container to a Tailnet and the considerations for running Tailscale in a container.

Tailscale can replace reverse proxies and control access via ACLs, offering a simpler alternative to complex configurations.

Tailscale enables direct connections between services on different platforms, such as cloud and local environments, using encrypted WireGuard-based tunnels.

Tailscale provides an official Docker image available on Docker Hub and GitHub Container Registry, with configurable environment variables.

Two primary programmatic methods for adding a container to a Tailnet are introduced: using an auth key and using an OAuth client.

Auth keys grant full API access, whereas OAuth clients limit API access through scoping, which is beneficial for security and auditing.

Auth keys have a maximum lifespan of 90 days, but the authenticated nodes do not automatically expire with the key.

OAuth clients offer the advantage of never expiring, requiring a strategy for regular token rotation.

Tagging differences are highlighted, with auth keys defaulting to the user who generated the key, and OAuth clients assuming the identity of a tag owner.

A step-by-step guide is provided for adding a container to a Tailnet using an auth key, including generating an auth key and configuring a Docker Compose file.

The use of Docker's network mode is explained, showing how it merges network namespaces of different containers for a unified networking context.

Tailscale Serve is introduced, a tool that can act as a reverse proxy to route web requests to the appropriate services within the Tailnet.

A demonstration of using Tailscale Serve to configure a self-hosted recipe manager app, Meely, to be accessible on the Tailnet with a valid HTTPS certificate.

The ability to use Tailscale Funnels to expose applications to the public internet securely is discussed, with a focus on its potential risks and benefits.

The transcript concludes with a call to action for Docker and Tailscale users to share their experiences and a summary of the provided resources for further learning.

Transcripts

About 10 years ago you were pretty unusual if you were running a container in production,

whereas these days you're unusual if you're not. In today's video we're going to talk about

tailscale and docker, we're going to cover how to add a container to your tailnet, why you might

want to consider running tailscale in a container in the first place, as well as how to use tailscale

serve and funnel to expose those tailscale applications from your tailnet directly to the

public internet. Today's video will be quite a long one so i've put chapter markers down there

for you to skip around and find the bit of the video that you need, so with that let's get started.

Well the first question that probably comes to mind is why would i even want to put tailscale

in a container in the first place? Well by putting a container directly onto your tailnet,

that's our term for a tailscale network by the way, you can not only control access via ACLs

but you can also replace reverse proxies. Yes for those of you that have been waiting to learn

reverse proxies for long enough i've got good news you don't have to learn them ever you can

just completely skip that step and you can also access any other service on your tailnet from

these containers as well. So you can have something running in your basement in your house connected

to something in the cloud, take for example a GPU workload some AI workload or something like that,

you've got something running in AWS or GCP and you don't want to pay their GPU prices but you've got

a GPU sat in your gaming computer like right here and you want a way to connect the two together

well using tailscale you can do just that, think of the possibilities and this all happens through

tailscale's encrypted wire guard based tunnels, you don't have to mess around with port forwarding or

complex firewall rules or dynamic DNS or any of that it all becomes a thing of the past.

tailscale provides an official docker image that you can find over on docker hub as well as github

container registry, this exposes several parameters through what are called environment variables

there'll be a full list of those variables exposed by the container down below. There are two primary

methods for adding a container to your tailnet, well three if we include logging into the container

manually, running tailscale up, copying the resulting printout, logging into the browser manually

and redoing that every single time you bring up a container. The two methods we're going to cover

today are both what are called programmatic methods so these are ones that you can repeatedly do

without any manual intervention. The first option is called an auth key and the second is called an

oauth client and as for which one is right for you, well it kind of depends i'm afraid, it's just one

of those classic situations where it really does depend on what you're going to be doing with it.

So let's dig into some of the differences between the two, both of these methods actually support

most of the same things there's just some nuances that you've got to be aware of when choosing

between the two. So let's first of all take a look at api access, an auth key grants full api

level access to any client that authenticates using it, this is in contrast to an oauth client

which limits api access via scoping. So this for example means if i have an oauth client secret

that is scoped to only allow people to modify the dns or scoped to only allow people to read and

write devices on my tailnet or only scope to read audit logs that's all they can do or any

combination of those different things as well by the way. Whereas with an auth key because it

grants full api access to anybody that has that secret they can do whatever they want and so if

auditors are in your regular vernacular it's likely that this alone will be enough of a feature for you

that rules out auth keys in favor of oauth clients. The next thing to look at is expiry and lifespan.

An auth key has a maximum lifespan of 90 days, now a lot of people think that this means that

anything that's authenticated using that auth key also automatically expires after 90 days,

this isn't the case. It's worth bearing in mind that auth keys are separate from the node keys,

so under the hood what happens when you authenticate a client to your tailnet

is node keys are generated, so this is underneath it's the wire guard key exchange magic that makes

tail scale work, that is a completely separate expiry process from the authentication token

that you use to initiate that process. A lot of people use oauth clients simply because they

never expire, now in practice what this means is you're going to have to come up with some way to

regularly rotate those tokens so that any security posture that you have or any risk profiles you

have are met or if you just simply want a set it and forget it type solution and an oauth client

might be right for you but there are plenty of situations where you might want to give someone

temporary access to add something to your tailnet and then have that auth key automatically revoked.

The next important difference is the tagging, so when you add a node to your tailnet it must be

owned by somebody and when you use an auth key that identity by default at least with an auth key

is assumed to be the person who clicked the generate auth key button in the UI. You can

add tags to an auth key if you would like but with an oauth client that secret doesn't actually itself

add the node to the tailnet it has to assume the identity of a tag owner instead. This is because

a node on a tailnet must have an owner whether that's a user or a tag and when you use an auth

key that node is added to your tailnet as the user who generated the key. With an oauth client though

the node is owned by the tag assigned at secret creation time and this is why in our ACLs you will

see tags listed as tag owners. The oauth client is assuming ownership of a specific resource on

your tailnet using the tag so the tag ownership is assumed using the tag assigned to the client

secret at the time of creation. So let's walk through adding a container to our tailnet using

an auth key. I'm using the visual studio code tail scale extension here to access a Ubuntu

virtual machine with docker pre-installed or docker compose ready to go and I have here a

sample docker compose file there'll be a link to a github repo down below with all of these resources

for you as well as the blog post that accompanies this video with all the materials explaining

you know all the details behind this this file. So let's walk through the compose file that we've

got here. So first of all on line five we have the tail scale image this is the official tail

scale docker image from docker hub. Next up we have the container name so this refers to the

name that docker gives the container on the Ubuntu host. Next up we have the host name now this is

important because this is the name that tail scale will will add to this container to the tailnet

with the name of so whenever we want to use magic dns to refer to this container we will use auth

key hyphen test you can make this value whatever you like by the way it doesn't have to match the

container name or the service name. Next up we're going to specify our environment variables so here

we're specifying our auth key this is basically like a password remember so just you know treat

it with care and then the next thing we have here is the state directory now the state directory is

really important and it's coupled by the way with this line here of volume so var lib tail scale is

the directory that the container is going to use to persist all of the state of you know that what

state when i logged in and authenticated to the tailnet what was my wire guard key exchange

underneath like all of that kind of stuff gets stored in this directory and the reason it's

important is because if the container restarts or gets recreated or any other kind of major event in

the container's lifespan the state gets persisted in this directory and so it doesn't have to

re-authenticate to the tailnet every single time or get out of sync with what's going on. Next up

we have the ton device this is a networking necessity and below it we have a couple of

capabilities that we've added to the container as well so net admin and sys module these three or

four lines here mean that we don't need to give this container a privileged status within the

kernel so it's a much more secure way of doing it because we're being very explicit with the

permissions that we're giving this container with these lines here and then restart unless stopped

totally optional you don't need this bit unless you want to. Next up we have the actual web service

that we're trying to run here now this is just a very basic nginx web container it's not even got

a real website in it it's just we're just going to load the nginx test page on port 80 and then

finally we have the network mode now this is the magic source that connects everything together

and makes this whole thing work so network mode service colon ts auth key test this is a docker

compose specific incantation so ts auth key test here actually refers to the name of the service

up on line four up here now if you're doing this with docker run command for whatever reason

you can actually replace the name of this with container and then the name of the container so

if you know if this was uh ts one two three you would you would do ts one two three like this

and those two things would have to match but i'm going to use service because i'm in the docker

compose ecosystem so i'm quite happy with doing it like this now what i'm going to do here oh i

should probably generate my auth key before i do compose up shouldn't i so let's jump across to

our tailscale admin console and go up here to settings and then under settings we're going to

want to jump into the keys section then we're going to click on generate auth key and then we're just

going to give it a name the name can be whatever you like so i'm going to call this docker auth

key test it can be whatever you like it's not important it's going to be a single use key so

we're not going to do any reusability or anything like that reusable keys are they're fine but

they're quite risky so if if you don't treat them with care and you have a key that lasts for 90

days you've effectively given anybody that finds that string of characters root access to your

tailnet you know they can add and remove devices there is no scoping remember for an auth key so

anybody that has this key can do whatever they like to your tailnet programmatically so just

bear that in mind and treat it with care next up we've got the expiration i'm going to set this to

90 days now there is a bit of a misconception around expiration a lot of folks seem to think

that if the container gets to 90 days and you authenticated using an auth key that expires

after 90 days that container is going to drop off your tailnet and completely need a new auth key

bang on 90 days it's not quite the case the keys we use to authenticate the node to the tailnet

are different from the wire guard keys that get exchanged behind the scenes later on those keys

are good for about four months or so and you can disable key expiry if you want to in the

tailscare admin console as well so theoretically you could use an auth key with a one day expiry

set your key expiry to disabled and that container would stay on a tailnet until the end of time

another setting that's worth exploring is ephemeral so if you're doing something with cicd

continuous integration maybe you have a github action that wants to talk to something off site

for example it's not part of the github ecosystem those jobs are typically short-lived so you don't

necessarily want those containers on your tailnet forever so an ephemeral node will automatically

remove itself after it goes offline i don't want this in this case but it can be very useful in

certain situations and now we come to tags we talked about tags a little bit earlier but this

is where you would set a tag as it pertains to a specific auth key and you can set multiple tags

here if you want to i don't really want any tags on this node i want this node to be authenticated

as my user so i'm just going to leave this empty i'm going to click on generate key and then i'm

going to copy this to my clipboard go back to my docker compose file and just enter the value that

we had on our clipboard click save now i can do my docker compose up i'm going to create a couple

of containers you can see there's a whole bunch of stuff scrolling by here the auth key has appeared

the containers appeared in my tailnet you can see that in the vs code dashboard just there

if i go back to my admin console notice that my single use key has actually automatically

invalidated itself remember it was a single use key so it automatically puts itself in the recently

invalidated auth keys section down here and then on my machines page we can see that if i look at

auth key test i now have a container that's in here so if i go to http auth key hyphen test which

is the dns name that we set in our host name remember just here auth key hyphen test we can

actually refer to this container by its host name by its magic dns name and resolve the url that's

running the web page now nginx is running on port 80 underneath so it's just a plain http request

and we're actually connecting we'll dig into the specifics a little bit later on but we're

actually connecting to port 80 of this container which is which has attached itself to the interface

of your tail scale container but that's how we get started with an auth key let's now look at oauth

clients let's start by looking at the differences between an oauth client and an auth key so i've

got a second docker compose yaml file here which you'll notice is pretty similar to the docker

compose on the left so on the left we have the auth key and on the right we have the oauth client

you'll notice that we're actually even using the same environment variable okay we've added an

extra one with ts extra args this is where we specify the container tag that the auth

client uses the oauth client uses when it brings up the container so when the container starts

it's running something called container boot this does a tail scale up command under the scenes

and it recognizes that we've provided an oauth client secret it then uses that secret to generate

an auth key under the covers with the specific tag that we've associated here because if you remember

when you add a node to a tail net it has to be owned by somebody and tags and users are all sort

of in this melange of ownership and so when we add the node with the tag container we're

authenticating that node as if it belongs to that specific tag owner generating an oauth client

secret's quite straightforward so again we're going to jump back to our tailscale admin console

go to settings go to oauth clients over here on the left click generate oauth client i'm going

to select devices right and notice that the read permission is automatically set i'm just going to

write test one two three for the description it's totally optional by the way and under tags here

tag container i'll just show you real quick where i set that up in my acl so i just have a tag

container here which is owned by the group auto group admin remember there'll be a link down below

to all of the details like there'll be a git repository with my acl file just so you can copy

and paste if you want to as well as the linked blog post so if we go back to oauth clients over

here we've got the tag container and then we just click generate client the client id isn't

particularly important to us here but the client secret treat it like a password once again same as

with an auth key i'm going to go back to vs code and just copy in my value here i'm going to click

save and then just going to change into the correct directory and then do a docker compose up

this is going to add a second container to my tailnet now using the name ts oauth test actually

not quite in line seven it would just be oauth test so if we go over here to my machines page we

can see that oauth test is there also note that expiry is disabled nodes that are added to your

tailnet using an oauth client do not expire so that's that's a big difference between an auth key

and an oauth client authenticated node and by default nodes are also added as ephemeral so in

order to make a node non-ephemeral we'll have to stop the container remove the state directory that

we have so uh ts oauth test by the way that is this directory here remember where the state

gets stored when the node authenticates to the tailnet we'll remove that directory

as sudo and then we need to just add on to the end of our auth key just here ephemeral equals false

also need to make sure it's deleted from your tailnet as well in your admin console

then bring the save the compose file and then bring the node back up again with docker compose up and

if we go back to our admin console we will see that the oauth node appears expiry is still disabled

but the node is no longer ephemeral so it's a really minor difference but just being able to

add that ephemeral equals false on the end means the nodes have a lot more permanence on your

tailnet so depending on your use case that may or may not be useful for you and again just to verify

what's going on here i want to do instead of um doing it in the web browser this time i want to do

it in a terminal session so i'm going to do a curl http oauth test and you saw the web request flow

by in the terminal underneath and then we get the the nginx landing page as well so there you go we've

now added a node to our tailnet using an auth key and also an oauth client now i promised you a bit

of a deeper dive as to how the docker networking is actually connecting everything together under

the hood so we're going to dive a little bit into linux kernel namespacing i promise it's not as

intimidating as it sounds so when we create a container what we're doing effectively is creating

a new namespace in the linux kernel with a whole bunch of different things that control where

different resources live one of those namespaces that we create is a networking namespace so each

container has its own networking context when we add network mode to a specific container we can

actually group two containers together under the same namespace so let me show you that in action

so what i have over here is a docker compose file it's basically the same as the one we had before

it's got an nginx web server in it as well as the tailscale container now notice down here on line

22 that i've actually commented out network mode so when we bring these containers up what's actually

going to happen under the hood is it's going to create two networking namespaces in the linux

kernel and we can just check that these two containers are both running with a docker ps

hyphen a let's suppose i want to find which ports this nginx container is listening on

internally well the first thing i might want to do is grab a docker exec docker exec spawns a

process inside the container a shell process and lets us attach to that assuming that that binary

exists within the container so what i want to do here is do a docker exec then the name of the

container in question which is nginx PID test one and then i'm going to do netstat minus ton alp

ah we run into our first issue and this is because images like nginx are designed to be web servers

they're not designed to be diagnostic tools and so a lot of containers strip out unnecessary binaries

but using namespaces we can actually hop into the networking namespace or hop into the context

of the networking namespace using a tool called nsenter now this tool it's really pretty cool

actually but what we can do is we can provide a bunch of different contexts in here for example

we're going to use the -n for net context and we need to grab the PID or the process id

for the docker container that we want to inspect so to do that we're going to use the docker inspect

command and i'm going to look for the PID of the container nginx PID test one which we can see here

is 53.1.6.6 then i'm going to use nsenter i'm going to run it as sudo because i need root

privileges to change into a different namespace then i'm going to give it the target of 53.1.6.6

because we're targeting the process id that owns or is part of that namespace that owns the network

namespace and then i'm going to run the command in the networking context netstat ton alp which

if you recall is exactly the same command i tried to run before with the docker exec that didn't

work and what we can see here is that within our nginx container that we've got a couple of commands

running on ipv4 port 80 here as well as ipv6 on port 80 as well so let's just stop and think about

what happened for a second i'm on my linux host and i changed into a different namespace using

namespace enter as the command and then i ran an arbitrary command as if i was in the context of

the container and so when we talk about containers that's what we're talking about is different

context different processes sliced up within the side the linux kernel in memory space but where

things start to get really interesting is when we start trying to stack the containers together

using network mode so over here in my docker compose file you can see that network mode is

currently commented out what this means is that we're going to be creating two different network

namespaces for these different containers one each so let's do a docker compose up and then use a

couple of commands to examine what's going on with the namespaces underneath so i've got an all-in-one

command here which is going to use ns enter once more and it's going to grab the PID you see it's

going to use docker inspector as we did before to inspect the namespace for ts nginx test this is the

tail scale container not the nginx web server and you can see that my namespace here the network

namespace for just this specific container only contains things pertaining to tail scale d so that

must mean that this is the tail scale container and if i do the same thing again here and i'm going

to load instead of ts nginx test i'm going to investigate nginx PID test one we can see that

we've got nginx listening in its namespace inside its container so these two containers right now

both have separate namespaces within the linux kernel with regards to networking another way we

can check this is to use docker's built-in network tools so if we do docker network ls this is going

to show us all of the different docker networks that get created and by default docker compose

creates a new network per stack so what we have here is a stack called 0 3 nginx example default

so if i do a docker network inspect of that network what we can see and i'll just pipe it

through to jq so it looks pretty and what we can see here is there are two containers currently on

that docker network they've both got their own ip address from the docker bridge so this one's 0 3

this one's 0 2 you can see the different names of the containers here as well and what this shows

us is that these two containers are currently operating as we would expect as isolated units

now let's go ahead and uncomment network mode i'm going to recreate these containers and just

watch what happens i'm going to run the same command again docker network inspect and suddenly

there is only one container showing up we can see it's ts nginx test well this is exactly what we'd

expect because the nginx PID test one container we've given it access to the network namespace

of this ts nginx test container so actually what's happened is we've merged them together

and we can prove this by looking inside the container's networking namespace so let's first

examine the nginx PID test one namespace and see what's going on and then let's examine the actual

tail scale container upstream of that container what we've done is we've merged these two

namespaces together in the kernel and you can see we've got the same processes running in both places

so we've taken those two separate namespaces and merged them together by using the network mode in

the docker compose file so let's put all of this new knowledge together and put a self-hosted

recipe manager onto our tail net i've got an app here called meely and another docker compose file

which looks again suspiciously similar to the ones that we've been looking at throughout this video

we've got our client auth key here we've got our tags there's a new one here called ts serve config

which will come on to throughout this section and then we've mounted a couple of extra volumes we've

got the config directory which is where our tail scale serve configuration lives and then the actual

application itself so we've got meely as an application here using docker volumes to persist

the data so let's bring this container up and it'll take a couple of seconds to start it uses

gu unicorn under the hood and we can wait for it to come up on port 9000 this is important because

what we were doing before with nginx was on port 80 so we didn't have to do anything to make web

requests work we just they just worked because it was on port 80 this one though is on port 9000 so

if i go across here and go to meely port 9000 we can see that's fine and it works just fine but

it's on plain http there's no you know tls certificate or anything like that and i don't

really want to have to remember port numbers and so this is where tail scale serve and in a minute

tail scale funnel come into the equation tail scale serve lets us provide a configuration file

that will tell basically a bit like a reverse proxy that will tell web requests where to go and

so let's take a look at one i've got over here a sample json file we can see at the top these first

few lines refer to proxying TCP/HTTPS traffic we we want this this is a good thing next up we've

got the web section this is what is basically acting like a reverse proxy so when i receive

a request of meely dot velociraptor hyphen noodlefish dot ts dot net on port 443 proxy

that request under the hood to localhost port 9000 now remember that this is actually running inside

the tail scale container but because we've done network mode and added that other container into

the namespace of the tail scale container in the linux kernel it's actually listening on port 9000

inside the tail scale container so that's how the localhost part here works and then finally

i've got allow funnel set to false for right now funnel would allow me to expose this application

across the internet with no further configuration required other than just changing this value to

true so if you had friends and family in another country that weren't on your tail net for some

reason changing this value from false to true would let them access this application as if they

were on your tail net now remember that means it's also exposed to anybody on the internet so use this

option with care the upshot of this configuration file is i can now go to meely dot velociraptor

hyphen noodlefish dot ts dot net and i don't have to worry about typing import numbers or anything

like that what you might actually notice in the terminal window at the bottom is that when i make

this request it was actually requesting a certificate from let's encrypt so this website

is exposed only on my tail net right now it's not on the public internet but it's backed with

a proper tls certificate from let's encrypt so if i look at the certificate here you can see

it's a let's encrypt certificate and and the whole purpose behind tls really is to verify that you

are actually talking to the web server that you think you're talking to it verifies the ownership

of that domain and so let's get logged in and have a little poke around and see what's what

just type in my password here and oh yeah i feel like i'm in the mood to bake some bread don't you

the easiest loaf you'll ever bake well that's fantastic we've got a self-hosted recipe app

on our tailnet but if you noticed if you were keen-eyed hard coding things into a configuration

file is never going to be fun in the long term to manage at least and so what we've actually done is

we've implemented under the hood a way to substitute the values in this file using this

ts cert domain variable just here so if you put this into your configuration file when tail scale

starts up it will actually substitute the value of this variable for the domain of your container's

tailnet and we can verify this by hopping over to the container host itself so if i do a docker psa

and then look for a docker exec command in here so docker exec 0 3 b 1 this is going to give me a

shell inside the tail scale container itself what i can do is do a tail scale serve status this is

going to show us that whenever we receive a request on the root of the domain so the slash is the root

of the domain it's going to proxy localhost port 9000 now if you're not a fan of typing out

configuration files yourself like i said you can do a dash json on the end and it will actually

print out the whole file for you in exactly the right format if you need it to feed into the ts

serve config environment variable that i showed you earlier it's really important for this

configuration file that you mount a directory and not the specific file itself if you mount the file

directly it breaks the way that fs notify tells docker that the file has changed we haven't done

that here we've just mounted a directory so we're okay but just notice what happens when i change

false to true for example it creates a whole bunch of configuration on the tail scale side to now

expose this application using tail scale funnel so anybody on the public internet could now access

this application et voila we have a self-hosted recipes app running natively on our tail net with

a valid https certificate available both internally to tail net devices only and if you turned it on

externally to other devices using tail scale funnel this approach scales to multiple containers

and different operating systems as well not only that but you could also let's just think about it

for a second you could run one container in your basement you could run another one in digital

ocean for example and then have another one at your friend's house and have all of these

different devices all talking to each other natively encrypted over tail scale if you're

already a regular user of tailscale on docker let us know down in the comments how you're using it

so we can make it even better in the future don't forget you can find all of the companion resources

for this video in the description box down below so that's the blog post the git repo various

different kbase articles they're all down there for you to use so until next time thank you so

much for watching i've been alex from tailscale