How To Change MTU Size of Docker’s docker_gwbridge Network
Overview:
This guide shows how to change the MTU size of Docker’s docker_gwbridge network. It addresses the issue of the same curl command or request working on the host system but not in the containers.
Page Contents
Foreword
Sometimes, your application running in Docker containers may fail to make requests to external resources, for example, external or third-party APIs. This can be caused by many things including the MTU (Maximum Transmission Unit – a measurement in bytes of the largest data packets that a network interface can accept) size of a Docker bridge network not matching that of the default host network interface.
I encountered a similar challenge with one of my applications running in a Docker Swarm cluster. I did tests using the curl command on the host system, I could access the remote resource but the same command was failing/hanging in all my Docker containers running in the Swarm.
In Docker Swarm mode networking, you need to keep in mind that:
- The docker_gwbridge is the default bridge network. It connects the overlay networks within the Swarm to an individual Docker daemon’s physical network.
- By default, each container running in a Swarm service is connected to its local Docker daemon host’s docker_gwbridge network.
- The docker_gwbridge network is created automatically when you initialize or join a swarm. Docker allows users to modify it as shown in this guide.
How I did the curl command tests
I tested the request on the Docker host as follows. I have blurred the URL in the command and output for security purposes:
#curl http://external_server_ip/request_uri_and_arguments
To do a test in one of the containers, I first identified the container and then executed a shell into it in interactive mode so I could run commands in it. Then I installed the curl package and ran the test command:
# docker container ls # docker exec -it 60d77e77c2a1 sh # apt update && apt install curl #curl http://external_server_ip/request_uri_and_arguments
I noticed that the curl request was hanging. After doing some research online, many Docker users who faced a similar issue where pointing to the MTU mismatch issue.
As shown in the following image, my host network interface had an MTU of 1450 and the docker_gwbridge virtual network has a default of 1500. As a result, all the container networks have an MTU of 1500. The solution was to set the same value for the docker_gwbridge network as that of the host network interface.
Changing the MTU Size of docker_gwbridge network
To change the MTU size for the docker_gwbridge network, I had to first remove the stack (fg_apps) in the Swarm, make the host leave the Swarm mode ( it was a single node Swarm ) and stop the Docker service as follows:
#docker stack rm fg_apps #docker swarm leave #docker swarm leave --force
Next, I had to remove the default docker_gwbridge network and create a custom one. So I listed all networks and removed the docker_gwbridge network as follows:
# docker network ls # docker network rm docker_gwbridge # docker network ls
Next, I had to create a new docker_gwbridge network as follows. You can change the subnet to your own but I maintained the default network name:
# docker network create --subnet 172.18.0.1/16 \ --opt com.docker.network.bridge.name=docker_gwbridge \ --opt com.docker.network.bridge.enable_icc=false \ --opt com.docker.network.bridge.enable_ip_masquerade=true \ --opt com.docker.network.driver.mtu=1450 \ docker_gwbridge
After making the above changes and restoring my Docker swarm and stack services, I checked the docker_gwbridge and container networks, the MTU changed to 1450 and the request was failing in the containers started working fine.
Final thoughts
I found out many solutions on the web but they all didn’t fix the problem. It was not until I looked through the Docker networking documentation and attempted to customize the docker_gwbridge network that my issue was fixed. If you have any questions or comments, let me know via the feedback form below.
