04 November 2020
HAProxy Network Error: cannot bind socket
Introduction
An HAProxy cannot bind socket
error message is generated when there is another process listening on the same interface and TCP port combination that HAProxy is configured to use, or when HAProxy attempts to use an IP address that is not assigned to a network interface. Both error conditions derive from the underlying operating system’s network stack.
In the first case, when there is another process that is already using an interface and port that HAProxy is attempting to bind to, the underlying error on Linux is EADDRINUSE
. The issue is that only a single process can be bound to an IP address and port combination at any given time.
In the second case, when HAProxy is attempting to use an IP address that is not assigned to an interface on the system, the underlying error on Linux is EADDRNOTAVAIL
. The issue here is that an IP socket cannot be created using an address that is not available to the operating system.
However, both underlying errors generate the same HAProxy error message, so troubleshooting a cannot bind socket
error requires examining the list of currently used sockets and IP addresses on a Linx system.
To detect a cannot bind socket
error message, you will need to examine systemctl
and journalctl
output to determine the IP address and port combination that are causing the error. Then you can inspect other running processes and network interfaces and decide how to resolve the issue, whether it is by switching servers, changing the IP address or port that HAProxy uses, or any combination of these options.
Troubleshooting with systemctl
Following the troubleshooting steps from the How to Troubleshoot Common HAProxy Errors tutorial at the beginning of this series, the first step when you are troubleshooting an cannot bind socket
error message is to check HAProxy’s status with systemctl
.
The output from systemctl status
will in many cases contain all the diagnostic information that you need to resolve the error. It may include the IP address that HAProxy is using, as well as the port that it is attempting to bind to. The output will also indicate how long HAProxy has been unable to start so that you can determine how long the issue has been affecting HAProxy.
<$>[note]
Note: If you are using Ubuntu or a Debian-derived Linux distribution, systemctl
does not include output from HAProxy with a cannot bind socket
error message that describes the problem. Skip to the the next section of this tutorial, Troubleshooting Using journalctl
Logs to learn how to examine the systemd
logs to find the conflicting IP address or port.
<$>
On CentOS, Fedora and RedHat-derived systems, use this systemctl
command to examine HAProxy’s status:
[label CentOS and Fedora Systems]
sudo systemctl status haproxy.service -l --no-pager
The -l
flag will ensure that systemctl
outputs the entire contents of a line, instead of substituting in ellipses (…
) for long lines. The --no-pager
flag will output the entire log to your screen without invoking a tool like less
that only shows a screen of content at a time.
Since you are troubleshooting a cannot bind socket
error message, you should receive output that is similar to the following:
[secondary_label Output]
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled)
Active: <^>failed<^> (Result: exit-code) since Wed 2020-08-19 14:57:05 UTC; 3s ago
Process: 138738 ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE (code=exited, status=1/FAILURE)
Process: 138736 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q (code=exited, status=0/SUCCESS)
Main PID: 138738 (code=exited, status=1/FAILURE)
Aug 19 14:57:05 92214d8ff5e2 systemd[1]: Starting HAProxy Load Balancer...
Aug 19 14:57:05 92214d8ff5e2 haproxy[138738]: [ALERT] 231/145705 (138738) : Starting frontend main: <^>cannot bind socket [0.0.0.0:80]<^>
. . .
Aug 19 14:57:05 92214d8ff5e2 systemd[1]: <^>Failed to start HAProxy Load Balancer.<^>
This example systemctl
output includes some highlighted lines from the systemd
journal that describes the error. These lines give you all the information about the error that you need to troubleshoot it further. Specifically, the line <^>cannot bind socket [0.0.0.0:80]<^>
describes the socket that HAProxy is trying to use (0.0.0.0:80
), so you can skip the following journalctl
steps and instead proceed to the Troubleshooting with ss
and ps
Utilities section at the end of this tutorial. The other highlighted line indicates the status of the HAProxy process, which in the case of a cannot bind socket
error will show Failed to start HAProxy Load Balancer.
If your systemctl
output does not give specific information about the IP address and port or ports that are causing the error (if you are using Ubuntu or Debian then this applies), then you will need to examine journalctl
output from the systemd
logs. The following section explains how to use journalctl
to troubleshoot a cannot bind socket
error.
Troubleshooting Using journalctl
Logs
If your systemctl
output does not include specifics about a cannot bind socket
error, you should proceed with using the journalctl
command to examine systemd
logs for HAProxy.
On Ubuntu and Debian-derived systems, run the following command:
sudo journalctl -u haproxy.service --since today --no-pager
On CentOS, Fedora, and RedHat-derived systems, use this command to inspect the logs:
sudo journalctl -u haproxy.service --since today --no-pager
The --since today
flag will limit the output of the command to log entries beginning at 00:00:00 of the current day only. Using this option will help restrict the volume of log entries that you need to examine when checking for errors.
If HAProxy is unable to bind to a port that is in use, search through the output for lines that are similar to the following log entries, specifically lines that contain the cannot bind socket
error message as highlighted in this example:
[secondary_label Output]
-- Logs begin at Wed 2020-08-19 19:38:12 UTC, end at Wed 2020-08-19 19:53:53 UTC. --
. . .
Aug 19 19:39:21 92214d8ff5e2 systemd[1]: Starting HAProxy Load Balancer...
Aug 19 19:39:21 92214d8ff5e2 haproxy[135]: [ALERT] 231/193921 (135) : Starting frontend main: <^>cannot bind socket [0.0.0.0:80]<^>
Aug 19 19:39:21 92214d8ff5e2 haproxy[135]: [ALERT] 231/193921 (135) : Starting frontend main: <^>cannot bind socket [:::80]<^>
Aug 19 19:39:21 92214d8ff5e2 systemd[1]: haproxy.service: Main process exited, code=exited, status=1/FAILURE
Aug 19 19:39:21 92214d8ff5e2 systemd[1]: haproxy.service: Failed with result 'exit-code'.
Aug 19 19:39:21 92214d8ff5e2 systemd[1]: Failed to start HAProxy Load Balancer.
. . .
The first highlighted line of output indicates that HAProxy cannot bind to port 80
on all available IPv4 interfaces (denoted by the 0.0.0.0
IP address). Depending on your system’s configuration, the IP addresses may be different and only show individual IPs.
If you are using HAProxy with IPv6, then the output may also include a line like the second one that is highlighted with an IPv6 specific interface and port error, in this case :::80
. The first two ::
characters indicate all available IPv6 interfaces, while the trailing :80
indicates the port.
Even though your own system may have different conflicting interfaces and ports, the errors will be similar to the output shown here. With this output from journalctl
you will be able to diagnose the issue using ss
, ps
, and ip
commands in the following section of this tutorial.
Troubleshooting with ss
and ps
Utilities
To troubleshoot a cannot bind socket
error you need to determine what other process is listening on the IP address and port that HAProxy is attempting to use, or if the IP address is available to HAProxy.
For example, if another server like Nginx is configured to listen on port 8080 on all available IPv4 network interfaces, the full socket would be 0.0.0.0:8080
. If HAProxy is also configured to use 0.0.0.0:8080
then the operating system will throw an EADDRINUSE
error, and HAProxy will show a cannot bind socket
error message, since it cannot claim the socket for itself.
In the previous journalctl
section, something was already bound to all the available IPv4 addresses (denoted by 0.0.0.0:80
). Most modern Linux distributions include a utility called ss
which can be used to gather information about the state of a system’s network sockets.
The following command will determine the name of the process that is already bound to an IPv4 interface on port 80
. Ensure that you substitute the port from the error message if it is different from 80
in the following command:
sudo ss -4 -tlnp | grep <^>80<^>
The flags to the ss
command alter its default output in the following ways:
-4
restrictsss
to only display IPv4-related socket information.-t
restricts the output totcp
sockets only.-l
displays all listening sockets with the-4
and-t
restrictions taken into account.-n
ensures that port numbers are displayed, as opposed to protocol names like ‘httpor
https`. This is important since HAProxy may be attempting to bind to a non-standard port and a service name can be confusing as opposed to the actual port number.-p
outputs information about the process that is bound to a port.| grep 80
limits the output to lines that contain the characters80
so there are fewer lines that you have to examine
<$>[note]
Note: in this IPv4 and the following IPv6 example, if you do not have a line in your output with a matching port, then your cannot bind socket
error may be derived from an EADDRNOTAVAIL
error. Skip to the next section Troubleshooting with the ip
Utility to examine the available IP addresses on your system.
<$>
With all of those flags, you should receive output like the following:
[secondary_label Output]
LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=40,fd=6))
The first three fields are not important when troubleshooting a cannot bind socket
error so they can be ignored. The important fields are the fourth (0.0.0.0:80
), which matches the journalctl
error that you discovered earlier, along with the last users:(("nginx",pid=40,fd=6))
, specifically the pid=40
portion.
If you have a cannot bind socket
error that is related to an IPv6 interface, repeat the ss
invocation, this time using the -6
flag to restrict the interfaces to the IPv6 network stack like this:
sudo ss -6 -tlnp |grep <^>80<^>
The -6
flag limits the ip
command to IPv6 interfaces. If HAProxy is unable to bind to an IPv6 socket, you should have output like the following:
[secondary_label Output]
LISTEN 0 511 [::]:80 [::]:* users:(("nginx",pid=40,fd=7))
Again, substitute the port number in question from your journalctl
output if it is different from the highlighted 80
given here.
In both these cases of IPv4 and IPv6 errors, the ss
output indicates that there is a program with process ID 40 (the pid=40
in the output) that is bound to the 0.0.0.0:80
and [::]:80
interfaces respectively. This process is preventing HAProxy from starting since it already owns the port. To determine the name of the program, use the ps
utility like this, substituting the process ID from your output in place of the highlighted 40
value in this example:
sudo ps -p <^>40<^>
You will receive output that is similar to the following:
[secondary_label Output]
PID TTY TIME CMD
40 ? 00:00:00 <^>nginx<^>
The highlighted nginx
in the output is the name of the process that is listening on the interfaces. Now that you have the name of the program that is preventing HAProxy from starting, you can decide how to resolve the error. You could stop the nginx
process, reconfigure nginx
to listen on a different interface and port, or reconfigure HAProxy to avoid the port collision.
It is important to note that the process may be different from nginx
and the port and IP addresses may not always be 0.0.0.0
or [::]
if you are diagnosing a cannot bind socket
error. Oftentimes, different web servers and proxies will be in use on the same server. Each may be attempting to bind to different IPv4 ports and IPv6 interfaces to handle different web traffic. For example, a server that is configured with HAProxy listening on the IPv4 loopback address (also referred to as localhost
) on port 8080
will show ss
output like this:
[secondary_label Output]
LISTEN 0 2000 127.0.0.1:8080 0.0.0.0:* users:(("haproxy",pid=545,fd=7))
It is important to combine systemctl
output, or journalctl
output that indicates specific IP addresses and ports, with diagnostic data from ss
, and then ps
to narrow down the process that is causing HAProxy to fail to start.
Sometimes when you are troubleshooting a cannot bind socket
error message with ss
and ps
there will not be any output at all, which means that the error may not be caused by a socket conflict. The next section of this tutorial explains how to troubleshoot a cannot bind socket
error using the ip
utility.
Troubleshooting with the ip
Utility
The previous section explained how an EADDRINUSE
operating system error could cause a cannot bind socket
error message. However, if you have examined ss
and ps
output and there is no socket conflict on your system, the issue may be caused by an EADDRNOTAVAIL
operating system error instead. In this case HAProxy may be trying to bind to a socket that is not available to your operating system.
To determine whether a cannot bind socket
error is caused by an EADDRNOTAVAIL
, examine both the IPv4 and IPv6 network interfaces on your system using the ip
command.
sudo ip -4 -c address show
-4
restrictsip
to only display IPv4-related interface information.-c
adds color coding to the output so that it is easier to parse visually.address show
displays the IP address for an interface, with the-4
and-c
flags taken into account.
You should receive output that looks similar to the following on any Linux distribution that includes the ip
tool:
[secondary_label Output]
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
inet <^>127.0.0.1/8<^> scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
inet <^>203.0.113.1/24<^> brd 203.0.113.255 scope global eth0
valid_lft forever preferred_lft forever
inet <^>192.0.2.1/24<^> brd 192.0.2.255 scope global eth0
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
inet <^>198.51.100.1/24<^> brd 198.51.100.255 scope global eth1
valid_lft forever preferred_lft forever
Make a note of your IP addresses that correspond to the highlighted examples in this output. Your IP addresses and network interfaces will be different than the examples shown here. You may have more or fewer interfaces, and each may have more or fewer addresses assigned to them. The important part is to note the IP addresses from ip
.
To examine IPv6 addresses that are assigned to your system, use the ip
command with the -6
flag like this:
sudo ip -6 -c address show
You should receive output like the following:
[secondary_label Output]
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
inet6 <^>::1/128<^> scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
inet6 <^>2604:a880:400:d1::3d3:6001/64<^> scope global
valid_lft forever preferred_lft forever
inet6 fe80::a4ff:aaff:fec9:24f8/64 scope link
valid_lft forever preferred_lft forever
Again note the highlighted values in this example output and look for the corresponding IPv6 addresses in your output.
Once you have a list of addresses that are assigned to your system, you can try to find a matching IP address that corresponds to the cannot bind socket [x.x.x.x:80]
error. If there is no IP address that matches, then HAProxy is likely configured to use an IP address that is not available to your system and the cannot bind socket
error is being caused by the operating system throwing an EADDRNOTAVAIL
error.
To resolve the error you will need to edit your /etc/haproxy/haproxy.cfg
file and change the bind
address or addresses to an IP address that is available to your system based on the output of the ip
command.
For example, if /etc/haproxy/haproxy.cfg
contained a bind
line like the following using 198.51.100.123
as the IP address, but your system has 198.51.100.1
assigned based on the example output above, you will need to edit the bind line.
Following this hypothetical example, this haproxy.cfg
snippet shows the invalid IP address:
[label /etc/haproxy/haproxy.cfg]
. . .
frontend main
bind <^>198.51.100.123<^>:80
A correct bind
line that matches the IP address in the example ip
output would look like this:
[label /etc/haproxy/haproxy.cfg]
. . .
frontend main
bind <^>198.51.100.1<^>:80
Once you have edited /etc/haproxy/haproxy.cfg
with the correct IP address, restart it using the systemctl
command:
sudo systemctl restart haproxy.service
Now examine HAProxy’s status and make sure that the output shows an active (running)
line:
sudo systemctl status haproxy.service
[secondary_label Output]
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
Active: <^>active (running)<^> since Wed 2020-08-19 21:31:46 UTC; 17h ago
Docs: man:haproxy(1)
file:/usr/share/doc/haproxy/configuration.txt.gz
Process: 487 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
. . .
Aug 19 21:31:46 d6cdd0c71489 systemd[1]: <^>Started HAProxy Load Balancer.<^>
If you have resolved the cannot bind socket
error your output should be similar to this example output. The highlighted lines that show HAProxy is active, and that the process was started successfully.
Conclusion
In this tutorial you learned how to troubleshoot an HAProxy cannot bind socket
error message on both IPv4 and IPv6 interfaces. You learned how to use systemctl
to examine the status of the HAProxy server and try to find error messages. You also learned how to use journalctl
to examine the systemd
logs for specific information about a cannot bind socket
error.
With the appropriate error messages from the systemd
logs, you then learned about the ss
utility and how to use it to examine the state of a system’s network sockets. After that you learned how to combine process ID information from ss
with the ps
utility to find the name of the process that is causing HAProxy to be unable to start.
Finally, in the case of a cannot bind socket
error that is related to an unavailable IPv4 or IPv6 address, you learned how to use the ip
utility to examine available network interfaces on your system.