Understanding lsof

As a Liquid Web customer, you’re always looking for ways to get the most out of your server. Sometimes, understanding what’s happening under the hood can be crucial for troubleshooting and ensuring smooth operations. That’s where a powerful command-line tool called lsof comes in!

What is lsof?

lsof stands for “LiSt Open Files.” In simple terms, it helps you see all the files that are currently open by processes running on your server. This might sound basic, but “files” in a Linux environment can include a lot more than just documents! It encompasses:

• Regular files: Your website files, configuration files, logs, etc.
• Directories: The folders on your server.
• Network sockets: Crucial for understanding network connections and services.
• Pipes and devices: Internal communication channels and hardware interfaces.

By listing these open “files,” lsof provides a unique window into your server’s activity.

How lsof can help you

lsof is a versatile tool that can assist with a variety of common server-related questions and troubleshooting scenarios:

Checking port listeners: Is your service online?

Ever wondered if your web server or email service is actually listening for connections on the correct port? lsof can quickly confirm this.

Scenario: You’re trying to connect to your email client, but it’s failing. You suspect the mail server might not be listening on the standard SSL port (465).

Command Example:

lsof -i TCP:465

Example Output:

COMMAND    PID       USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
exim      8471 mailnull    6u  IPv6 40228971      0t0  TCP *:urd (LISTEN)
exim      8471 mailnull    7u  IPv4 40228972      0t0  TCP *:urd (LISTEN)

Note: The COMMAND and USER fields in the output may vary depending on the email service running on your server. The example above is typical for a WHM server.

What the output means:

• If you see output like the example above, it indicates a process (in this case, exim) is actively listening on port 465. This tells you the service is running and ready to accept connections.
• If there’s no output, it means nothing is listening on that port, which could be the reason for your connection issue. In such cases, restarting the relevant service (like Exim for email) often resolves the problem.

This command helps you quickly determine if a service is “listening” for connections, which is often the first step in diagnosing connectivity problems.

Investigating high load or malicious activity

If your server experiences unexpected high load, or you suspect unauthorized activity, lsof can help you pinpoint the processes and users that might be causing the issue.

Scenario: Your server’s load is unusually high, and top command shows a user you don’t recognize consuming a lot of resources.

Command Example:

Let’s say top shows a suspicious user named hackerman.

lsof -u hackerman

What the output means:

This command will list all the files and network connections opened by the hackerman user. This can provide valuable clues, such as:

• COMMAND: What specific programs are they running?
• NAME: What files are they accessing or creating? Are these legitimate?
• TYPE: Are there unusual network connections (IPv4, TCP) that shouldn’t be there?

This information can help you identify suspicious processes, understand what they’re doing, and take appropriate action to secure your server.

More lsof command examples for deeper insight

lsof is a deeply powerful tool with many capabilities beyond basic port and user checks. Here are more examples that can help you troubleshoot and monitor your server:

List processes which opened a specific file

You can list only the processes that have opened a specific file by providing the filename as an argument.

lsof /var/log/syslog

Output:

COMMAND  PID   USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
rsyslogd 721 syslog    1w   REG    8,1     2048 312345 /var/log/syslog

List opened files under a directory

You can list processes that have opened files within a specific directory using the +D option. This option will also look inside subdirectories. If you only want to check the main directory and not its subfolders, use the +d option instead.

lsof +D /var/log/

Output:

COMMAND   PID   USER  FD   TYPE DEVICE SIZE/OFF   NODE NAME
rsyslogd  721 syslog   1w   REG    8,1     2048 312345 /var/log/syslog
rsyslogd  721 syslog   2w   REG    8,1     3100 312346 /var/log/auth.log
httpd     850 apache   3w   REG    8,1     1500 312347 /var/log/apache2/access.log

List opened files based on process names starting with

You can list files opened by processes whose names start with a particular string using the -c option. For example, -c ssh will show files opened by processes like ssh or sshd. You can use multiple -c options in one command.

lsof -c ssh -c init

Output:

COMMAND    PID   USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
init         1       root  txt    REG        8,1   124704  917562 /sbin/init
init         1       root  mem    REG        8,1  1434180 1442625 /lib/i386-linux-gnu/libc-2.13.so
init         1       root  mem    REG        8,1    30684 1442694 /lib/i386-linux-gnu/librt-2.13.so
...
ssh-agent 2105 devuser    1u   CHR        1,3      0t0    5000 /dev/null
ssh-agent 2105 devuser    2u   CHR        1,3      0t0    5000 /dev/null
ssh-agent 2105 devuser    3u  unix 0xdf70e240      0t0   11000 /tmp/ssh-agent-xyz/agent.2105

List processes using a mount point

Sometimes, when you try to unmount a directory, your system might say “Device or Resource Busy.” This means something is still using that directory. You can use lsof to find out which processes are holding it open, so you can stop them and then unmount the directory.

lsof /home

The following command will also work:

lsof +D /home/

Output:

COMMAND    PID       USER   FD   TYPE DEVICE SIZE/OFF       NODE NAME
bash      1234 devuser  cwd    DIR        8,1     4096     123456 /home/devuser
nginx     5678 www-data  cwd    DIR        8,1     4096     123456 /home/devuser/public_html

List all open files by a specific process

You can list all files opened by a specific process using the -p option. This is helpful for getting more detailed information about what a particular program is doing.

lsof -p 1753

Output:

COMMAND  PID       USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
bash    2001 devuser  cwd    DIR    8,1     4096  400000 /home/devuser/my_script.sh
bash    2001 devuser  rtd    DIR    8,1     4096       2 /
bash    2001 devuser  255u   CHR  136,0      0t0       4 /dev/pts/0
...

Kill all processes that belong to a particular user

If you need to stop all processes that a specific user has running and that are holding files open, you can use the -t option. This option will only show you the process IDs (PIDs), which you can then pass to the kill command.

kill -9 `lsof -t -u devuser`

The command above will forcefully stop all processes belonging to devuser that have files open.

Similarly, you can use -t in many ways. For example, to find the process ID of the program that opened /var/log/syslog, you can use:

lsof -t /var/log/syslog

Output:

721

Combine more list options using OR/AND

When you use more than one filter option in lsof, they usually work with an “OR” logic. This means lsof will show you results that match any of the conditions you’ve set. For example:

lsof -u devuser -c init

Output:

COMMAND    PID       USER   FD   TYPE     DEVICE SIZE/OFF       NODE NAME
init         1       root  cwd    DIR        8,1     4096          2 /
init         1       root  mem    REG        8,1  1434180 1442625 /lib/i386-linux-gnu/libc-2.13.so
init         1       root  mem    REG        8,1    30684 1442694 /lib/i386-linux-gnu/librt-2.13.so
...
bash      2001 devuser    2u   CHR      136,2      0t0          6 /dev/pts/2
bash      2001 devuser  255u   CHR      136,2      0t0          6 /dev/pts/2
...

The command above lists processes belonging to devuser OR processes whose names start with init.

However, if you want to find a process that belongs to devuser AND whose name starts with init (meaning both conditions must be true), you can use the -a option.

lsof -u devuser -c init -a

This command will likely not show anything, as there’s typically no process named init running under a regular user like devuser.

Execute lsof in repeat mode

lsof also has a “Repeat mode,” which is useful for continuous monitoring. It will first show open files based on your filters, then pause for a specified number of seconds, and then show the updated list again. You can stop it by pressing Ctrl+C.

You can turn on Repeat mode using -r or +r. If you use +r, the repeating will stop automatically when no open files are found. If you use -r, it will keep repeating the list and delay, even if no files are open, until you manually stop it.

Each cycle of output will be separated by =======. You can also specify the delay time (e.g., -r5 for a 5-second delay).

lsof -u devuser -c init -a -r5

Output:

=======
=======
COMMAND   PID       USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
app_init 3000 devuser  cwd    DIR    8,1     4096  400000 /home/devuser
app_init 3000 devuser  rtd    DIR    8,1     4096       2 /
app_init 3000 devuser  txt    REG    8,1    90000  550000 /usr/bin/python3
app_init 3000 devuser  mem    REG    8,1  1500000 1500000 /lib/x86_64-linux-gnu/libc.so.6
app_init 3000 devuser  mem    REG    8,1   120000 1500001 /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
app_init 3000 devuser    0u   CHR  136,5      0t0       8 /dev/pts/5
app_init 3000 devuser    1u   CHR  136,5      0t0       8 /dev/pts/5
app_init 3000 devuser    2u   CHR  136,5      0t0       8 /dev/pts/5
app_init 3000 devuser   10r   REG    8,1       50  400001 /home/devuser/start_app.sh
=======

In the output above, for the first 5 seconds, there’s no output. After that, a script named “inita.sh” starts, and it lists the output.

List all network connections

You can list all network connections using the -i option.

lsof -i

Output:

COMMAND    PID  USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
systemd    1    root   10u  IPv4   1234      0t0  TCP *:ssh (LISTEN)
nginx     800 www-data  11u  IPv4   5678      0t0  TCP *:http (LISTEN)
mysql     900 mysql    12u  IPv4   9012      0t0  TCP localhost:mysql (LISTEN)

You can also use -i4 or -i6 to list only IPv4 or IPv6 connections, respectively.

List all network files in use by a specific process

You can list all network files being used by a process as follows:

lsof -i -a -p 2345

You can also use the following:

lsof -i -a -c ssh

Output (for lsof -i -a -c ssh):

COMMAND    PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd      2345   root    3u  IPv4  12345      0t0  TCP *:ssh (LISTEN)
sshd      2345   root    4u  IPv6  12346      0t0  TCP *:ssh (LISTEN)
ssh       2346 devuser    5u  IPv4  12347      0t0  TCP 192.168.1.100:54321->192.168.1.1:22 (ESTABLISHED)

The command above will list the network files opened by processes whose names start with ssh.

List all TCP or UDP connections

You can list all TCP or UDP connections by specifying the protocol using -i.

lsof -i tcp; lsof -i udp;

Output (for lsof -i tcp):

COMMAND    PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd      2345   root    3u  IPv4  12345      0t0  TCP *:ssh (LISTEN)
nginx     800 www-data  11u  IPv4   5678      0t0  TCP *:http (LISTEN)
ssh       2346 devuser    5u  IPv4  12347      0t0  TCP 192.168.1.100:54321->192.168.1.1:22 (ESTABLISHED)

Output (for lsof -i udp):

COMMAND    PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
systemd-r 700 systemd-r   10u  IPv4   9876      0t0  UDP localhost:domain
ntpd      450   ntp    12u  IPv4   5432      0t0  UDP *:ntp

List all Network File System (NFS) files

You can list all NFS files by using the -N option. The following lsof command will list all NFS files used by user devuser.

lsof -N -u devuser -a

Output:

COMMAND    PID       USER   FD   TYPE DEVICE SIZE/OFF       NODE NAME
bash      2001 devuser  cwd    DIR    0,3     4096 1000000 /mnt/nfs_share/devuser_files
bash      2001 devuser    4r   REG    0,3     1024 1000001 /mnt/nfs_share/devuser_files/script.sh

Conclusion

Mastering lsof empowers you to understand your server’s processes and open files. It’s key for troubleshooting, security checks, and ensuring smooth operation. Use it for better server control.