03 March 2021

Understanding Sockets

Introduction

Sockets are a way to enable inter-process communication between programs running on a server, or between programs running on separate servers. Communication between servers relies on network sockets, which use the Internet Protocol (IP) to encapsulate and handle sending and receiving data.

Network sockets on both clients and servers are referred to by their socket address. An address is a unique combination of a transport protocol like the Transmission Control Protocol (TCP) or User Datagram Protocol (UDP), an IP address, and a port number.

In this tutorial you will learn about the following different types of sockets that are used for inter-process communication:

• Stream sockets, which use TCP as their underlying transport protocol
• Datagram sockets, which use UDP as their underlying transport protocol
• Unix Domain Sockets, which use local files to send and receive data instead of network interfaces and IP packets.

In each section of this tutorial you will also learn how to enumerate the respective socket types on a Linux system. You’ll examine each type of socket using a variety of command line tools.

Prerequisites

The examples in this tutorial were validated on an Ubuntu 20.04 server. You can follow this tutorial using most modern Linux distributions on a local computer or remote server, as long as you have the equivalent version of each of the required tools for your distribution installed.

To get started using Ubuntu 20.04, you will need one server that has been configured by following our Initial Server Setup for Ubuntu 20.04 guide.

You will also need a few other packages in order to examine sockets on your system. Ensure that your system’s package cache is up to date using the apt update command:

sudo apt update 

Then install the required packages using this command:

sudo apt install iproute2 netcat-openbsd socat

The iproute2 package contains the ss utility, which is what we’ll use to inspect sockets. We’ll use the netcat-openbsd package to install netcat. Note that netcat is abbreviated to nc when it is invoked on the command line. Finally, we’ll use the socat package to create example sockets.

What is a Stream Socket?

Stream sockets are connection oriented, which means that packets sent to and received from a network socket are delivered by the host operating system in order for processing by an application. Network based stream sockets typically use the Transmission Control Protocol (TCP) to encapsulate and transmit data over a network interface.

TCP is designed to be a reliable network protocol that relies on a stateful connection. Data that is sent by a program using a TCP-based stream socket will be successfully received by a remote system (assuming there are no routing, firewall, or other connectivity issues). TCP packets can arrive on a physical network interface in any order. In the event that packets arrive out of order, the network adapter and host operating system will ensure that they are reassembled in the correct sequence for processing by an application.

A typical use for a TCP-based stream socket would be for a web server like Apache or Nginx handling HTTP requests on port 80, or HTTPS on port 443. For HTTP, a socket address would be similar to 203.0.113.1:80, and for HTTPS it would be something like 203.0.113.1:443.

Creating TCP-based Stream Sockets

In the following example you’ll use the socat (short for <^>SO<^>cket <^>CAT<^>) command to emulate a web server listening for HTTP requests on port 8080 (the alternative HTTP port). Then you’ll examine the socket using the ss, and nc commands.

First, run the following socat commands to create two TCP-based sockets that are listening for connections on port 8080, using IPv4 and IPv6 interfaces:

socat TCP4-LISTEN:8080,fork /dev/null&
socat TCP6-LISTEN:8080,ipv6only=1,fork /dev/null&

• The TCP4-LISTEN:8080 and TCP6-LISTEN:8080 arguments are the protocol type and port number to use. They tell socat to create TCP sockets on port 8080 on all IPv4 and IPv6 interfaces, and to listen to each socket for incoming connections. socat can listen on any available port on a system, so any port from 0 to 65535 is a valid parameter for the socket option.
• The fork option is used to ensure that socat continues to run after it handles a connection, otherwise it would exit automatically.
• The /dev/null path is used in place of a remote socket address. In this case it tells socat to print any incoming input to the /dev/null file, which discards it silently.
• The ipv6only=1 flag is used for the IPv6 socket to tell the operating system that the socket is not configured to send packets to IPv4-mapped IPv6 addresses. Without this flag, socat will bind to both IPv4 and IPv6 addresses.
• The & character instructs the shell to run the command in the background. This flag will ensure that socat continues to run while you invoke other commands to examine the socket.

You will receive output like the following, which indicates the two socat process IDs that are running in the background of your shell session. Your process IDs will be different than the ones highlighted here:

[secondary_label Output]
[1] <^>434223<^>
[2] <^>434224<^>

Now that you have two socat processes listening on TCP port 8080 in the background, you can examine the sockets using the ss and nc utilities.

Examining TCP-Based Stream Sockets

To examine TCP sockets on a modern Linux system using the ss command, run it with the following flags to restrict the output:

• The -4 and -6 flags tell ss to only examine IPv4 or IPv6 sockets respectively. Omitting this option will display both sets of sockets.
• The t flag limits the output to TCP sockets. By default the ss tool will display all types of sockets in use on a Linux system.
• The l flag limits the output to listening sockets. Without this flag, all TCP connections would be displayed, which would include things like SSH, clients that may be connected to a web-server, or connections that your system may have to other servers.
• The n flag ensures that port numbers are displayed instead of service names.

First run the ss -4 -tln command to examine the IPv4 TCP-based sockets that are listening for connections on your system:

ss -4 -tln

You will receive output like the following:

[secondary_label Output]
State      Recv-Q     Send-Q         Local Address:Port           Peer Address:Port     Process     
. . .
LISTEN     0          1                    <^>0.0.0.0:8080<^>                  0.0.0.0:*
. . .

There may be other lines with other ports in your output depending on which services are running on your system. The highlighted 0.0.0.0:8080 portion of the output indicates the IPv4 TCP socket is listening on all available IPv4 interfaces on port 8080. A service that is only listening on a specific IPv4 address will show only that IP in the highlighted field, for example 203.0.113.1:8080.

Now run the same ss command again but with the -6 flag:

ss -6 -tln

You will receive output like the following:

[secondary_label Output]
State    Recv-Q   Send-Q     Local Address:Port     Peer Address:Port  Process  
. . .
LISTEN   0        5                   <^>[::]:8080<^>               [::]:*
. . .

There may be other lines with other ports in your output depending on which services are running on your system. The highlighted [::]:8080 portion of the output indicates the IPv6 TCP socket is listening on all available IPv6 interfaces on port 8080 (as indicated by the :: characters, which are IPv6 notation for an address composed of all zeros). A service that is only listening on a specific IPv6 address will show only that IP in the highlighted field, for example [2604:a880:400:d1::3d3:6001]:8080.

Connecting to TCP-Based Stream Sockets

So far you have learned how to create and enumerate TCP sockets on both IPv4 and IPv6 interfaces. Now that you have two sockets listening for connections, you can experiment with connecting to sockets using the netcat utility.

Using netcat to test TCP connections to local and remote sockets is a very useful troubleshooting technique that can help isolate connectivity and firewall issues between systems.

To connect to the IPv4 socket over the local loopback address using netcat, run the following command:

nc -4 -vz 127.0.0.1 8080

• The -4 flag tells netcat to use IPv4.
• The -v flag is used to print verbose output to your terminal.
• The -z option ensures that netcat only connects to a socket, without sending any data.
• The local loopback 127.0.0.1 IP address is used since your system will have its own unique IP address. If you know the IP for your system you can test using that as well. For example, if your system’s public or private IP address is 203.0.113.1 you could use that in place of the loopback IP.

You will receive output like the following:

[secondary_label Output]
<^>Connection to 127.0.0.1 (127.0.0.1) 8080 port [tcp/http-alt] succeeded!<^>

The highlighted line is the output from netcat. It indicates that netcat connected to the TCP socket listening on the loopback 127.0.0.1 IPv4 address on port 8080. You can ignore the second line, it is from the socat process running in the background in your terminal.

Now you can repeat the same connection test but using IPv6. Run the following netcat command:

nc -6 -vz ::1 8080

You should receive output like the following:

[secondary_label Output]
<^>Connection to ::1 8080 port [tcp/http] succeeded!<^>

The highlighted line is the output from netcat. It indicates that netcat connected to the TCP socket listening on the loopback ::1 IPv6 address on port 8080. Again, you can ignore the second line of output.

To clean up your sockets, you’ll need to run the fg (foreground) command for each socat process that you created. Then you’ll use CTRL+C to close each socat. fg will bring processes to the foreground of your terminal in the reverse order that you ran them, so when you run it, the second socat instance will be the one that you interact with first.

Run fg to bring the second IPv6 socat instance to the foreground of your terminal. Then run CTRL+C to close it.

fg

You will receive output like the following:

[secondary_label Output]
socat TCP6-LISTEN:8080,ipv6only=1,fork /dev/null

Press CTRL+C to stop the process.

Now run fg again to clean up the first IPv4 socket. You should have output like the following:

[secondary_label Output]
socat TCP4-LISTEN:8080,fork /dev/null

Press CTRL+C to stop the process.

You have now created, examined, and connected to IPv4 and IPv6 sockets on your system. These techniques and tools will work on local development systems, or remote production servers, so try experimenting with each tool to become more familiar with how they can be used to test and troubleshoot TCP sockets.

What is a Datagram Socket?

Datagram sockets are connectionless, which means that packets sent and received from a socket are processed individually by applications. Network-based datagram sockets typically use the User Datagram Protocol (UDP) to encapsulate and transmit data.

UDP does not encode sequence information in packet headers, and there is no error correction built into the protocol. Programs that use datagram-based network sockets must build in their own error handling and data ordering logic to ensure successful data transmission.

UDP sockets are commonly used by Domain Name System (DNS) servers. By default, DNS servers use port 53 to send and receive queries for domain names. An example UDP socket address for a DNS server would be similar to 203.0.113.1:53.

<$>[note] Note: Although the protocol is not included in the human-readable version of the socket address, operating systems differentiate socket addresses by including TCP and UDP protocols as part of the address. So a human-readable socket address like 203.0.113.1:53 could be using either protocol. Tools like ss, and the older netstat utility, are used to determine which kind of socket is being used. <$>

The Network Time Protocol (NTP) uses a UDP socket on port 123 to synchronize clocks between computers. An example UDP socket for the NTP protocol would be 203.0.113.1:123.

Creating Datagram Sockets

As in the previous TCP socket example, in this section you’ll use socat again to emulate an NTP server listening for requests on UDP port 123. Then you’ll examine the sockets that you create using the ss and nc commands.

First, run the following socat commands to create two UDP sockets that are listening for connections on port 123, using IPv4 and IPv6 interfaces:

sudo socat UDP4-LISTEN:123,fork /dev/null&
sudo socat UDP6-LISTEN:123,ipv6only=1,fork /dev/null&

You will receive output like the following, which indicates the two socat process IDs that are running in the background of your shell session. Your process IDs will be different than the ones highlighted here:

[secondary_label Output]
[1] <^>465486<^>
[2] <^>465487<^>

• Each command is prefixed with sudo, because ports 0 to 1024 are reserved on most systems. sudo runs a command with administrator permissions, which allows socat to bind to any port in the reserved range.
• The UDP4-LISTEN:123 and UDP6-LISTEN:123 arguments are the protocol type and port to use. They tell socat to create UDP based sockets on port 123 on both IPv4 and IPv6 interfaces, and to listen for incoming data. Again any port in the entire range of 0-65535 is a valid parameter for UDP sockets.
• The fork, ipv6only=1, and /dev/null arguments are used in the same manner as described in the previous TCP example.

Now that you have two socat processes listening on UDP port 123, you can examine the sockets using the ss and nc utilities.

Examining Datagram Sockets

To examine UDP sockets on a modern Linux system using the ss command, run it with the following -4, -6, and uln` flags to restrict the output:

The u flag limits the output to UDP sockets. The other flags are the same as the ones used in the previous TCP example.

First run the ss -4 -uln command to examine the IPv4 UDP sockets that are listening for connections on your system:

ss -4 -uln

You will receive output like the following:

[secondary_label Output]
State      Recv-Q     Send-Q         Local Address:Port           Peer Address:Port     Process     
. . .
UNCONN   0        0                 <^>0.0.0.0:123<^>            0.0.0.0:*
. . .

There may be other lines with other ports in your output depending on which services are running on your system. The highlighted 0.0.0.0:123 portion of the output indicates the IPv4 UDP socket is available on all IPv4 interfaces on port 123. A service that is only available on a specific IPv4 address will show only that IP in the highlighted field, for example 203.0.113.1:123.

Now run the same ss command again but with the -6 flag:

ss -6 -uln

You will receive output like the following:

[secondary_label Output]
State    Recv-Q   Send-Q     Local Address:Port     Peer Address:Port  Process  
. . .
UNCONN   0        0                   <^>[::]:123<^>              [::]:*
. . .

There may be other lines with other ports in your output depending on which services are running on your system. The highlighted [::]:123 portion of the output indicates the IPv6 TCP socket is available on all IPv6 interfaces on port 123 (as indicated by the :: characters). A service that is only available on a specific IPv6 address will show only that IP in the highlighted field, for example [2604:a880:400:d1::3d3:6001]:123.

Testing Datagram Sockets

Now that you are familiar with how to create and enumerate UDP sockets on both IPv4 and IPv6 interfaces, you can experiment with connecting to them. As with TCP sockets, you can experiment with UDP sockets using the netcat utility.

To connect to the example UDP socket on port 123 that you created in the previous section of this tutorial, run the following netcat command:

nc -4 -u -vz 127.0.0.1 123

• The -4 flag tells netcat to use IPv4.
• The -u option instructs netcat to use UDP instead of TCP.
• The -v flag is used to print verbose output to your terminal.
• The -z option ensures that netcat only connects to a socket, without sending any data.
• The local loopback 127.0.0.1 IP address is used since your system will have its own unique IP address. If you know the IP for your system you can test using that as well. For example, if your system’s public or private IP address is 203.0.113.1 you could use that in place of the loopback IP.

You will receive output like the following:

[secondary_label Output]
<^>Connection to 127.0.0.1 123 port [udp/ntp] succeeded!<^>

The output indicates that netcat did not receive an error from the UDP socket listening on the loopback 127.0.0.1 IPv4 address on port 123. This lack of an error response is used to infer that the socket at 127.0.0.1:123 is available. This behaviour is different from TCP sockets, which need to exchange packets to confirm if a socket is available.

<$>[note] Note: If the socket in this example was unavailable, the remote system would return an ICMP type 3 message (Destination Unreachable) with a Code of 3, indicating that the port is unreachable on the remote host.

Inferring that a socket is available based on a lack of error response assumes that there are no firewalls or connectivity issues that are blocking ICMP traffic. Without sending, receiving, and verifying application data over a UDP socket, there is no guarantee that a remote UDP port is open and accepting packets. <$>

Now you can repeat the same connection test but using IPv6. Run the following netcat command:

nc -6 -u -vz ::1 123

You should receive output like the following:

[secondary_label Output]
<^>Connection to ::1 123 port [udp/ntp] succeeded!!<^>

The output indicates that netcat did not receive an error from the UDP socket listening on the loopback ::1 IPv6 address on port 123. Again, this lack of an error response is used to infer that the socket at ::1:123 is available.

To clean up your sockets, you’ll need to run the fg (foreground) command for each socat process that you created. Then you’ll use CTRL+C to close each socat.

Run fg to bring the second IPv6 socat instance to the foreground of your terminal. Then run CTRL+C to close it.

fg

You will receive output like the following:

[secondary_label Output]
sudo socat UDP6-LISTEN:123,ipv6only=1,fork /dev/null

Press CTRL+C to stop the process.

Now run fg again to clean up the first IPv4 socket. You will have output like the following:

[secondary_label Output]
sudo socat UDP4-LISTEN:123,fork /dev/null

Press CTRL+C to stop the process.

You have now created, examined, and tested IPv4 and IPv6 UDP sockets on your system. Try experimenting with each tool to become more familiar with how they can be used to test and troubleshoot UDP sockets.

What is a Unix Domain Socket?

Programs that run on the same server can also communicate with each other using Unix Domain Sockets (UDS). Unix Domain Sockets can be stream-based, or datagram-based. When using domain sockets, data is exchanged between programs directly in the operating system’s kernel via files on the host filesystem. To send or receive data using domain sockets, programs read and write to their shared socket file, bypassing network based sockets and protocols entirely.

Unix Domain Sockets are used widely by database systems that do not need to be connected to a network interface. For example, MySQL on Ubuntu defaults to using a file named /var/run/mysqld/mysql.sock for communication with local clients. Clients read from and write to the socket, as does the MySQL server itself.

PostgreSQL is another database system that uses a socket for local, non-network communication. Typically it defaults to using /run/postgresql/.s.PGSQL.5432 as its socket file.

Creating Unix Domain Sockets

In the previous sections you explored how TCP is used with stream sockets, and how UDP is used with datagram sockets. In this section you’ll use socat to create both stream-based and datagram-based Unix Domain Sockets without using TCP or UDP to encapsulate data to send over networks. Then you’ll examine the sockets that you create using the ss, and nc commands. Finally, you’ll learn about testing Unix Domain Sockets using netcat.

To get started, run the following socat commands to create two socket files:

socat unix-listen:/tmp/stream.sock,fork /dev/null&
socat unix-recvfrom:/tmp/datagram.sock,fork /dev/null&

• The first command instructs socat to create a socket using the unix-listen address type, which will create a stream-based UDS.
• The second command specifies unix-recvfrom as the socket type, which will create a datagram-based UDS
• Both commands specify a filename after the : separator. The filename is the address of the socket itself. For the first stream example it is /tmp/stream.sock and for the second datagram example it is /tmp/datagram.sock. Note that the name of a socket is arbitrary but it helps if it is descriptive when you are troubleshooting.
• The fork, and /dev/null arguments are used in the same manner as described in the Stream and Datagram socket example sections.

Now that you have created your two UDS sockets, you can examine them using the ss and nc utilities.

Examining Unix Domain Sockets

To list all listening Unix Domain Sockets, run the ss -xln command. The x flag ensures that only domain sockets are displayed.

ss -xln

You will receive output like the following:

[secondary_label Output]
Netid State  Recv-Q Send-Q      Local Address:Port   Peer Address:Port  Process
. . .
<^>u_str<^> LISTEN 0      5        <^>/tmp/stream.sock<^> 436470            * 0             
<^>u_dgr<^> UNCONN 0      0      <^>/tmp/datagram.sock<^> 433843            * 0
 . . .

Notice the highlighted <^>u_str<^> portion of the /tmp/stream/sock line. This field indicates that the socket type is a stream-based UDS. The second line shows the type is <^>u_dgr<^> which means the socket type is datagram-based.

Since Unix Domain Sockets are files, the usual Linux user and group permissions and access controls can be used to restrict who can connect to the socket. You can also use filesystem tools like ls, mv, chown and chmod to examine and manipulate UDS files. Tools like SELinux can also be used to label UDS files with different security contexts.

To check if a file is a UDS socket, use the ls, file or stat utilities. However, it is important to note that none of these tools can determine if a UDS is stream or datagram-based. Use the ss tool for the most complete information about a Unix Domain Socket.

To examine a socket on the filesystem, the stat utility shows the most relevant information. Run it on the sockets that you created earlier:

stat /tmp/stream.sock /tmp/datagram.sock

You will receive output like the following:

[secondary_label Output]
  File: <^>/tmp/stream.sock<^>
  Size: 0             Blocks: 1          IO Block: 131072 <^>socket<^>
Device: 48h/72d    Inode: 1742        Links: 1
Access: (0755/<^>s<^>rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-03-01 18:10:25.025755168 +0000
Modify: 2021-03-01 18:10:25.025755168 +0000
Change: 2021-03-01 18:22:42.678231700 +0000
 Birth: -
  File: <^>/tmp/datagram.sock<^>
  Size: 0             Blocks: 1          IO Block: 131072 <^>socket<^>
Device: 48h/72d    Inode: 1743        Links: 1
Access: (0755/<^>s<^>rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-03-01 18:10:25.025755168 +0000
Modify: 2021-03-01 18:10:25.025755168 +0000
Change: 2021-03-01 18:10:25.025755168 +0000
 Birth: -

Notice that for each file, the type is socket (highlighted at the far right of the output) and the access mode has an s character preceding the file’s permissions.

The ls utility will also indicate if a file is a socket. Run ls -l to examine the files:

ls -l /tmp/stream.sock /tmp/datagram.sock

You will receive output like the following. Again note that for sockets, the file mode includes the s character before the file permission fields:

[secondary_label Output]
<^>s<^>rwxr-xr-x 1 root root 0 Mar  1 18:10 /tmp/datagram.sock
<^>s<^>rwxr-xr-x 1 root root 0 Mar  1 18:10 /tmp/stream.sock

Now that you have created Unix Domain Sockets, and learned how to examine them using the ss and various filesystem based tools, the next step is to test the sockets using a tool like netcat.

Testing Unix Domain Sockets

The netcat utility can be used to connect to Unix Domain Sockets, as well as TCP and UDP sockets that you already learned about earlier in this tutorial. To connect to the example sockets that you created, you will need to specify an extra -U flag when running the netcat command. This flag tells netcat to connect to a UDS, as opposed to a TCP or UDP based network socket.

Additionally, if the socket is datagram-based, you will use the -u flag to instruct netcat to use datagrams as we learned in the Datagram Socket section of this tutorial.

Let’s start examining UDS sockets by connecting to the stream-based socket with the following command:

nc -U -z /tmp/stream.sock

The -U tells netcat that it is connecting to a Unix Domain Socket. The -z option ensures that netcat only connects to a socket, without sending any data. The /tmp/stream.sock is the address of the socket on the filesystem.

You will not receive any output from netcat when you run the command. However, if a socket is not available, netcat will output an error message like the following:

[secondary_label Output]
nc: unix connect failed: No such file or directory
nc: /tmp/stream.sock: No such file or directory

So the absence of output from netcat when testing a stream-based UDS socket means that the connection was successful.

Repeat the testing process, this time for the datagram-based UDS:

nc -uU -z /tmp/datagram.sock

The additional -u flag is added to tell netcat that the remote socket is a datagram socket. Again, you will not receive any output if the test is successful.

If there is no socket at the address, you will receive an error like the following:

[secondary_label Output]
nc: unix connect failed: No such file or directory
nc: /tmp/datagram.sock: No such file or directory

To clean up your sockets, you’ll need to run the fg (foreground) command for each socat process that you created. Then you’ll use CTRL+C to close each socat.

Run fg to bring the datagram-based socat instance to the foreground of your terminal:

fg

You will receive output like the following:

[secondary_label Output]
socat unix-recvfrom:/tmp/datagram.sock,fork /dev/null

Run CTRL+C to close it. You will not receive any output.

Now run fg again to clean up the first stream-based UDS socket.

Again you should have output like the following:

[secondary_label Output]
socat unix-listen:/tmp/stream.sock,fork /dev/null

Run CTRL+C to end the process. You will not receive any output.

You have now created, examined, and tested Unix Datagram Sockets sockets on your system. Try experimenting with netcat and socat to become more familiar with how you can send and receive data over a UDS, as well as how you can test and troubleshoot Unix Domain sockets.

Conclusion

In this tutorial you explored how different kinds of sockets are used on a Linux system. You learned about stream-based sockets, which typically use TCP for network communication. You also learned about datagram-based sockets, which use UDP to send data over networks. Finally, you explored how Unix Domain Sockets can be either stream or datagram-based on a local server.

In each section you used the ss utility to gather information about sockets on a Linux system. You learned how the different flags that the ss tool provides can help you limit its output to specific types of sockets when you are examining sockets on a system.

Finally, you used the netcat and socat tools to create and connect to each of the three different types of sockets discussed in this tutorial. The netcat utility is widely used to connect to sockets, but it can also create sockets. Its documentation (man nc) contains many examples of how it can be used in either mode. The socat utility is a more advanced tool that can be used to connect to and many different types of sockets that are not covered in this tutorial. Its documentation (man socat) also contains numerous examples of the different ways that it can be used.

Understanding what sockets are and how they work is a core system administration skill. The tools and techniques that you experimented with in this tutorial will help you become more familiar with sockets, and how to troubleshoot them if your servers and applications are not communicating with each other correctly.