Like many people, after putting together a few docker-compose projects, I hit an error when I tried to bring a project up:

ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

Specifically, this happened when I had 31 bridge networks, and tried to bring up a compose project that would increase this total to 32.

Searching for the issue, I found it was fairly common, and could be resolved by expanding the range of default address pools.

Before I tell you more - Here’s the rundown on how to fix the problem.

TL;DR - How to fix it

The problem is that the default configuration makes each bridge network very big, which means that only 32 bridge networks can fit in the available space.

Add the following to /etc/docker/daemon.json:

{
  "default-address-pools" : [
    {
      "base" : "172.17.0.0/12",
      "size" : 20
    },
    {
      "base" : "192.168.0.0/16",
      "size" : 24
    }
  ]
}

This reduces the capacity of each bridge network by 1/16, and allows you to have 16 times as many bridge networks.

Don’t worry, the smallest of your bridge networks will still be able to fit up to 256 hosts, and the largest can fit 4096 hosts

Read on if you actually care about why this is a thing

I found a few useful resources:

There were a few things I still wanted to know:

  • When no configuration is given, what are the default address pools?
    • Note: I will refer to these as “vanilla” pools from now on, to avoid confusion with the term “default”
  • If I wanted to write a configuration that was identical to the vanilla pools, how would I do that?
  • If I wanted to write a configuration that preserved the vanilla pools, rather than replacing them, and only expanded them /added new pools only, how would I do that?
  • Why does the default configuration only allow a maximum of 31 networks?

Before we start, here are some terms, things I assume you know:

Terms, and assumed knowledge

Assumed Knowledge

  • You understand what an IP address is
  • Ideally, you’re familiar with the following notations:*
    • hex notation,
    • binary notation
    • how to translate between hex & binary.

Example for terms

To help explain these terms, I’m going to give an example which might be familiar:

Consider your modem. If you’ve ever set the wifi password, you’ve probably connected to an ip address like:

192.168.0.1. This is the first address in your Local Area Network.

  • For some context on what 192.168.0.1 means
  • it’s just a sequence of 4 bytes, (aka 32 bits, as 1 byte = 8 bits).
    • The biggest unsigned decimal number you can represent using a byte is 255.
    • To represent 255dec in binary, you would write: 1111 1111. 1111 1111 means every bit in this byte is 1
    • To represent 255dec in hex, you would write FF

You may have noticed that your computer is assigned an address on your LAN, for example:

192.168.0.3

A typical LAN setup usually defines a pool of addresses starting at 192.168.0.1, up to 192.168.0.255.

There are two parts to this:

192.168.0.___
This is the prefix for your local network. Every host in the network has the exact same prefix.
___.___._.3
This is the host id for your computer. The 3 is what sets your computer apart from anything else on the network, for example, your phone.

Terms

network
A block of routable hosts.
An example is your LAN network (192.168.0.[1-255])
netmask
This is usually notated like: 255.255.255.0, 255.255.240.0, or something similar.
For a given network this defines which parts of a given address define the network id.
For your lan network, this would be 255.255.255.0
In hexadecimal, this looks like: FF.FF.FF.00
In binary, this would look like: 1111 1111.1111 1111.11111 1111.0000 0000
That’s because for your LAN network, 192.168.0.___, the first 3 bytes all specify the network prefix
cidr
Classless Inter-Domain Routing
A notation for identifying networks & devices.
This basically looks like: ip.ip.ip.ip/prefix_length,
prefix_len is an integer between 0 & 32 (inclusive).
This defines how much of the ip address is being used to identify the network.
In the case of your local area network, 192.168.0.___:
  • The CIDR would be: 192.168.0.0/24
  • The last 8 bits (1 byte) define a given host
  • The first 24 bits (3 bytes) define the network
A prefix_len of /24 is equivalent to a netmask of FF.FF.FF.00

What are the default address pools when no configuration is given (“vanilla” pools)?

This information is less trivial to find than I expected - For whatever reason, docker does not formally document it anywhere I could find.

The capstonec.com article describes it informally:

The default address pool for local bridge networks on each individual Docker node includes the CIDR ranges 172.17.0.0/16 through 172.31.0.0/16 and 192.168.0.0/20 through 192.168.240.0/20.

This is helpful, but I wanted a more rigorous defintion - I found this in Issue #8663 for the docker documentation

screenshot of docker#8663

Writing an equivalent config to the “vanilla” configuration

If we wanted to write a /etc/docker/daemon.json that expressed the same configuration as the “vanilla”, it would look like this:

{
  "default-address-pools" : [
    {
      "base" : "172.17.0.0/12",
      "size" : 16
    },
    {
      "base" : "192.168.0.0/16",
      "size" : 20
    }
  ]
}

Writing an expanded “vanilla” configuration

Imagine we wanted to write an /etc/docker/daemon.json that included the above address spaces, but added some extra address pools - How would we do that?

Well, the quick answer is: We Cant

Why is that?

Private Address Ranges in IPv4

Well, the Internet Assigned Numbers Authority (IANA) has assigned 3 ranges for private networks

Address ranges to be use by private networks are:

  • Class A: 10.0.0.0 to 10.255.255.255
  • Class B: 172.16.0.0 to 172.31.255.255
  • Class C: 192.168.0.0 to 192.168.255.255

Anything outside of these ranges should be considered public, and therefore could belong to an external host.

Let’s consider our two vanilla address pools:

  • The address range covered by 172.17.0.0/12 (Vanilla pool 1) spans from: 172.16.0.1 -> 172.31.255.254
  • The address range covered by 192.168.0.0/16 (Vanilla pool 2) spans from: 192.168.0.1 -> 192.168.255.254

This covers everything in the IANA’s class B & class C ranges. Chances are it also collides with your LAN’s network, too.

We also know from docker docs #8663 that the vanilla swarm network pool is given by 10.0.0.0/8 - Meaning 10.0.0.1 -> 10.255.255.255.

This occupies the entire class A range. For now, let’s not dive into configuring the swarm address pool, let’s confine ourselves to the class B & class C ranges.

You might be wondering - If we’re only allowed to make 31 networks using these two ranges, surely that means these ranges aren’t very big?

You’d be wrong:

  • 192.168.0.0/16 can contain 65534 hosts,
  • 172.17.0.0/12 can contain 1048574 hosts.

If your use case is anything like mine, you probably have <100 hosts running - Let’s see mine:

sudo docker ps --filter status=running --quiet | wc -l 

26

Let’s see how many networks I have right now:

sudo docker network ls --filter driver=bridge --quiet | wc -l 

31

The problem is not that there isn’t a lot of space, its that the default network size is much bigger than it needs to be.

How to configure docker to allow >500 bridge networks

In short, we need to increase the value of the “size” mask:

--- daemon-old.json	2021-09-12 17:56:03.344023200 +1000
+++ daemon-new.json	2021-09-12 17:52:49.274194000 +1000
@@ -2,11 +2,11 @@
   "default-address-pools" : [
     {
       "base" : "172.17.0.0/12",
-      "size" : 16
+      "size" : 20
     },
     {
       "base" : "192.168.0.0/16",
-      "size" : 20
+      "size" : 24
     }
   ]
 }

The new, improved config:

{
  "default-address-pools" : [
    {
      "base" : "172.17.0.0/12",
      "size" : 20
    },
    {
      "base" : "192.168.0.0/16",
      "size" : 24
    }
  ]
}

Congratulations - you should now be able create around 512 bridge networks.

If you’re wondering why this worked, read on.

Why were we only allowed 31 networks?

The default pools are given by:

{
  "default-address-pools" : [
    {
      "base" : "172.17.0.0/12",
      "size" : 16
    },
    {
      "base" : "192.168.0.0/16",
      "size" : 20
    }
  ]
}

Let’s break down what this means:

Pool defintion 1

Consider the first set,

{
  "base" : "172.17.0.0/12",
  "size" : 16
},
  • Consider “base”:
    • In hex, we can express “base” as: AC.11.00.00
    • /12 defines a netmask of FF.F0.00.00
      • This means for the base, hosts can fall within the range of:
        • __._h.hh.hh: __._0.00.00 -> __._F.FF.FF
  • Consider “size”:
    • “size” defines the prefix/netmask for a given subnet allocated within the “base” pool above
    • A “size” of 16 specifies a netmask of: FF.FF.00.00
      • This means that for this section of the base network, hosts can fall within the range of:
        • __.__.hh.hh : __.__.00.00 -> __.__.FF.FF

    In essence, "size" : 16 indicates that for a given docker bridge network, everything after the 16th bit identifies a unique host in that bridge network.

So basically our defintion, in binary, becomes:

nnnn nnnn.nnnn ????.hhhh hhhh.hhhh hhhh

  • n: these bits are the prefix for the whole network
  • h: For a given subnet of the whole network, these bits uniquely identify a given host

So what do the ? mean?

  • ?: These bits uniquely identify a given subnet under the whole network.*
    • In our example, there are only 4 of these ? bits, or a single hex digit
    • This means there are only 16 unique combinations these ? bits can take
    • 0-9, or A-F.

Pool defintion 2

{
  "base" : "192.168.0.0/16",
  "size" : 20
}
  • Consider “base”:
    • hex: C0.A8.00.00
    • /16:
      • netmask: FF.FF.00.00
      • total range:
        • __.__.hh.hh: __.__.00.00 -> __.__.FF.FF
  • Consider “size”:
    • A “size” of 20 specifies a netmask of: FF.FF.F0.00
      • Total range:
        • __.__._h.hh : __.__._0.00 -> __.__._F.FF
  • Separating the different regions:
    • nnnn nnnn.nnnn nnnn.???? hhhh.hhhh hhhh
    • There are only 4 ? bits here, meaning 16 possible distinct subnets for this range too.

Let’s put this together, by looking at the 31 bridge networks running on my local machine:

Local Bridge Networks before new config:

  Id range
    |
    v
AC.10.39.00/24 vdf
AC.11.00.00/24 bridge
AC.12.00.00/16 gollum
AC.13.00.00/16 ghost
AC.14.00.00/16 nextcloud-proxy
AC.15.00.00/16 media_default
AC.16.00.00/16 diskover-proxy
AC.17.00.00/16 keycloak-proxy
AC.18.00.00/16 kibana
AC.19.00.00/16 ytdl
AC.1A.00.00/16 homer
AC.1B.00.00/16 keycloak
AC.1C.00.00/16 logstash
AC.1D.00.00/16 ssh
AC.1E.00.00/16 proxy_default
AC.1F.00.00/16 traefik


  Id range
      |
      v 
C0.A8.10.00/20 nextcloud
C0.A8.20.00/20 diskover
C0.A8.30.00/20 calibre-web
C0.A8.40.00/20 ldap
C0.A8.50.00/20 elastic
C0.A8.60.00/20 watchtower
C0.A8.70.00/20 robots
C0.A8.80.00/20 lam
C0.A8.90.00/20 samba
C0.A8.A0.00/20 unmanic
C0.A8.B0.00/20 gotify
C0.A8.C0.00/20 netdata
C0.A8.D0.00/20 gitlab-runner
C0.A8.E0.00/20 huginn
C0.A8.F0.00/20 huginn-proxy

As we can see, only one digit changes:

  • AC.1_.__ pool: The 3rd digit
  • C0.A8.F_ pool: The 5th digit

Local Bridge Networks after new config:

      |
      v
AC.10.00.00/20 bridge
AC.10.10.00/20 ghost
AC.10.20.00/20 elastic
AC.10.39.00/24 vdf
AC.10.40.00/20 kibana
AC.10.50.00/20 unmanic
AC.10.60.00/20 diskover
AC.10.70.00/20 keycloak
AC.10.80.00/20 calibre-web
AC.10.90.00/20 ldap
AC.10.A0.00/20 nextcloud
AC.10.B0.00/20 media_default
AC.10.C0.00/20 gollum
AC.10.D0.00/20 watchtower
AC.10.E0.00/20 robots
AC.10.F0.00/20 homer
   ^
   |
Digit Changes

Only these digits change
      |
      v
AC.11.00.00/20 gotify
AC.11.10.00/20 huginn
AC.11.20.00/20 huginn-proxy
AC.11.30.00/20 lam
AC.11.40.00/20 ytdl
AC.11.50.00/20 netdata
AC.11.60.00/20 diskover-proxy
AC.11.70.00/20 keycloak-proxy
AC.11.80.00/20 nextcloud-proxy
AC.11.90.00/20 samba
AC.11.A0.00/20 gitlab-runner
AC.11.B0.00/20 ssh
AC.11.C0.00/20 logstash
AC.11.D0.00/20 proxy_default
AC.11.E0.00/20 sshportal_default
AC.1F.00.00/16 traefik