How to Troubleshoot SMTP Errors: Part 2

In part one of How to Troubleshoot SMTP Errors, we reviewed email status codes, internal reasons for email issues, and troubleshooting steps to address those issues. In part two of this series, we will examine some of the external reasons for email issues. 

External Reasons for Email Issues

After we have tested the local server and see the local server is working correctly, we can infer that the issue lies with one of the following external problems:

• There exists incorrect DNS information on the remote mail server.
• The remote mail server is malfunctioning or undergoing maintenance.
• The remote mail server may be blocking the sender's IP.
• Port 25 has been blocked (or intermittently blocked) by the ISP or local network.
• The remote mail server has accepted the message, but the delivery is delayed.
• There is a typo in the email address.
• The IP or domain name is on a blacklist.

Who is Affected

Next, we check to see if the issue affects all users by searching the /var/log/exim_mainlog file for the domain in question.

grep domain.com /var/log/exim_mainlog

If we see only one email is affected, ask for login details to test their email address. Otherwise, log into the cPanel account via WHM, create a test email account, and send a message via Webmail. If we can send a message from the test account, check for the email address in the WHM mail queue. If we see it in there, try clicking the "Deliver Now" link to attempt the delivery. WHM will report whether the message was sent or an error occurred.

If sent, it was probably a temporary issue with the remote mail server. Review the exim_mainlog file for the specific reason the error message failed to send.

2021-01-31 16:38:38 [17466] fixed_login authenticator failed for localhost ([51.128.95.01]) [127.0.0.1]:36824 I=[127.0.0.1]:25: 435 Unable to authenticate at present (set_id=test@domain.com): syntax error in Exim::expand_string argument: failed to open /home/user/etc/domain.com/shadow for linear search: Permission denied (euid=47 egid=12) at /etc/exim.pl.local line 673

Often, the error message is provided in the log but can be unclear. In this case, notice the status code error 435 Unable to authenticate at present. In checking our status codes above, code 435 is not listed. If this is the case, a simple google search of the error code provides an answer. This specific error occurs when Exim cannot verify if the username and password provided by the email user are correct. 

Syntax of Error Log Entries

Now that we have identified an error in the logs, we can review how to break down the log entry information. When we send or receive an email, a log entry is generated. Typically that entry will contain a significant amount of information about the message. Part of that information will be the server status code indicating if the message was sent successfully, if it is being held, or rejected. Below are the sent and reply messages indicating their status. 

Using a Linux command like exigrep, we can search through the logs to find our content and identify the status code or error message. 

[root@host /var/log/ ]% exigrep "you@domain.com" exim_mainlog

A typical result will look something like this.

2010-09-13 05:00:13 [1487] 1Ov4tU-0000Nz-Rm H=mailhost.domain.com [208.42.54.2]:51792 I=[67.215.162.175]:25 Warning: "SpamAssassin as theuser detected message as NOT spam (0.0)" 2010-09-13 05:00:13 [1487] 1Ov4tU-0000Nz-Rm <= maillinglist@domain.com H=mailhost.domain.com [208.42.54.2]:51792 I=[67.215.162.175]:25 P=esmtps X=TLSv1:AES256-SHA:256 CV=no S=21778 id=384a86a39e83be0d9b3a94d1feb3119f@domain.com T="Daily Science Maillinglist: Chameleon" from for you@domain.com 2010-09-13 05:00:14 [1534] 1Ov4tU-0000Nz-Rm => user F= P= R=virtual_user T=virtual_userdelivery S=21902 QT=6s DT=0s 2010-09-13 05:00:15 [1534] 1Ov4tU-0000Nz-Rm Completed QT=7s

That is a LOT of information jammed into a minimal log entry. The email was delivered successfully if we see the word Completed on the last line. Using the same method as above, we search for the common error code “421” and see the following error. 

[root@host /var/log/ ]% exigrep "you@domain.com" exim_mainlog

2021-02-07 03:08:49.554 [100522] 1l8Hwo-000KoR-MF Sender identification U=mailnull D=-system- S=mailnull

 
2021-02-07 03:08:49.882 [100524] 1l8Hwo-000KoR-MF H=gmail-smtp-in.l.google.com [142.250.128.27]:25: SMTP error from remote mail server after end of data: 421-4.7.0 [67.227.187.136   15]  

 
Our system has detected that this message is\n421-4.7.0 suspicious due to the nature of the content and/or the links within.\n421-4.7.0 

 
To best protect our users from spam, the message has been blocked.\n421-4.7.0 Please visit\n421 4.7.0  https://support.google.com/mail/answer/188131 for more information. w14si12360049ilu.77 - gsmtp

In this 421 status code error, we see that I tried to send my server an email with a suspicious link in it which was blocked, as it should be. 

How to Verify POP3 is Working

As we demonstrated in part 1 of this series, we can use the telnet command in the terminal to test a connection to the mail server. In this case, instead of connecting to port 25, we attempt a connection to port 110, the port POP3 uses. Below is an example of the results seen when using telnet. 

The commands we will use are as follows.

telnet g33k.fun 110

• User - Enter username.
• Password - Enter a password (password will show in plaintext).
• List - Lists the email messages in the inbox.
• Stat - Shows the number of messages in the inbox and their size in bytes.
• Retr - Lets us select an email message and read it. Below, we see “1 4562”, which is a message in the inbox. We use retr 1 command to display and read the plain text message, including the email headers. I have removed some of the extraneous data to shorten the output quite a bit. 
• Dele - Lets us select and delete an email message. The syntax we use is the same as the Retr command above; “Dele 1” removes message 1.
• Quit - This command drops the connection between the terminal and the POP3 server.

Telnet Verification

Below is the output from the command line when using this method.

root@host:~# telnet g33k.fun 110
Trying 67.227.187.136...
Connected to g33k.fun.
Escape character is '^]'.
+OK Dovecot ready.
user user@g33k.fun
+OK
pass ********************
+OK Logged in.
list
+OK 1 messages:
1 4562
.
RETR 1
+OK 4562 octets
Return-Path: <dsinger@liquidweb.com>
Delivered-To: david@g33k.fun
Received: from host.mylwinfo.com
        by host.mylwinfo.com with LMTP
        id pIbRKPFjJGCPIgEAPdnBzA
        (envelope-from <dsinger@liquidweb.com>)
        for <david@g33k.fun>; Wed, 10 Feb 2021 17:53:37 -0500
Return-path: <dsinger@liquidweb.com>
Envelope-to: david@g33k.fun
Delivery-date: Wed, 10 Feb 2021 17:53:37 -0500
Received: from mail-pg1-f178.google.com ([209.85.215.178]:37155)
        by host.mylwinfo.com with esmtps  (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        (Exim 4.94)
        (envelope-from <dsinger@liquidweb.com>)
        id 1l9yMb-000JKF-9C
        for david@g33k.fun; Wed, 10 Feb 2021 17:53:37 -0500
Wed, 10 Feb 2021 14:52:55 -0800 (PST)
MIME-Version: 1.0
References: <dd4ee71f8870ec287608a2080afe9997@g33k.fun>
In-Reply-To: <dd4ee71f8870ec287608a2080afe9997@g33k.fun>
From: David Singer <dsinger@liquidweb.com>
Date: Wed, 10 Feb 2021 17:52:19 -0500
Message-ID: <CANtJYYuPE0d35G9QkBS-kSZFUT5eW2+mbp1U0AMB+OsCrKnseQ@mail.gmail.com>
Subject: Re: Test
To: david@g33k.fun
Content-Type: multipart/alternative; boundary="00000000000087345705bb0343c9" 

--00000000000087345705bb0343c9
Content-Type: text/plain; charset="UTF-8"

 
This is a reply to the test.

 
Regards,

 
=====================
David Singer
LiquidWeb Marketing Team
Liquid Web Inc <http://www.liquidweb.com>.

 
On Wed, Feb 10, 2021 at 5:45 PM <david@g33k.fun> wrote:

 
> This is a test message.
>

--00000000000087345705bb0343c9
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
--00000000000087345705bb0343c9--
.

DELE 1
+OK Marked to be deleted.

QUIT
+OK Logging out, messages deleted.
Connection closed by foreign host.
root@hosyt:~#

Here, we see the log entries generated for that connection. 

2021-02-10 17:45:45.993 [73521] 1l9yEz-000J7p-Uc <= david@g33k.fun H=(host.mylwinfo.com) [::1]:37548 I=[::1]:587 P=esmtpa L.- A=dovecot_login:user@g33k.fun S=569 M8S=0 RT=0.040s id=dd4ee71f8870ec287608a2080afe9997@g33k.fun T="Test" from <david@g33k.fun> for user@liquidweb.com

2021-02-10 17:45:46.036 [73524] 1l9yEz-000J7p-Uc SMTP connection outbound 1612997146 1l9yEz-000J7p-Uc g33k.fun user@liquidweb.com

2021-02-10 17:45:46.182 [73524] 1l9yEz-000J7p-Uc => user@liquidweb.com F=<user@g33k.fun> P=<user@g33k.fun> R=dkim_lookuphost T=dkim_remote_smtp S=1069 H=bmx03.liquidweb.com [67.227.128.72]:25 I=[67.227.187.136]:35380 L C="250 Ok: queued as 266C827AE5D" QT=0.233s DT=0.120s
root@host [~]#

Using this method, we can test our POP3 server.

Other Typical Email Issues

Email Marked as Spam

Spam is the issue most often seen with email, which includes inbound and outbound spam.

Inbound Spam

If the issue is inbound spam, we typically have to either whitelist the sender’s email address or domain or blacklist it entirely. Whitelisting is typically done within cPanel or the exim.conf file using trusted_users or trusted_groups. Blacklisting is handled using ACL (access control lists) via SpamAssassin or creating the /etc/eximblacklist file and referencing it in the exim.conf file under domainlist exim_blacklist = lsearch;/etc/eximblacklist.

Outbound Spam

If your domain or server IP ends up on a blacklist due to spamming, we must identify the account or domain responsible for the issue. Typically, there are five steps we must take if this issue occurs.

• Stop the source of spam.
• Investigate how the spam began.
• Clean the Exim queue. This command will delete all emails related to the email address or domain.  

grep -R --include='*-H' 'auth_id' /var/spool/exim/input | grep {email_address} | cut -d: -f1 | cut -d/ -f7 | cut -d- -f1-3 | xargs -n 1 exim -Mrm

Once completed, you will want to clear out all the bounceback emails using the following command.

exim -bpr | grep '<>' | awk '{print $3}' | xargs -I % exim -Mrm %

• Prevent future compromises. Update all related passwords on the account. 
• Address any external real-time blacklist (RBL) listings and email reputation. Real-time blacklists are services that publish the IP addresses of servers that send spam or exhibit spam-like characteristics. Once the spam is removed, and exim is cleared, and security issues are corrected on an account, we can then apply for delisting. Delisting involves visiting the service or ISP, which has the domain blacklisted. Using a search tool like Multirbl lookup will identify what services have blacklisted the domain or server. Each of these services must be contacted by the domain or server owner exclusively when removal is requested. The owner will have to answer multiple questions regarding the domain or server to have the delisting enabled.

Warning: It should be noted that each subsequent IP blacklisting will increase the length of time the domain or server is detained. Extreme care should be exercised so there is no evidence of further spam-like behavior. At some point, the IP itself continues to be blacklisted, and a timeframe of at least six months must pass before the IP is removed from the blacklist. 

Email Client Issues

Sometimes, the issue lies with the email client. Outlook profiles become corrupted, the default settings can cause iOS problems, and even ThunderBird has experienced bug problems. Making sure we have the latest versions of software often eliminates concerns. Additionally, if any of the email settings are off, errors will be seen. Ensure you have set up the correct ports, mail server configurations, and authentications. If any of these settings are not accurate, sending and receiving email may not work. 

Server Configurations

Another route to take is to check the email server settings for email, firewall rules, and DNS records (A, MX, and NS records should all resolve correctly).

Next, check the mail queue on the server to verify emails are being sent and received, review the server’s email and error logs, and finally, examine any hosting rules and email policies which can affect email transfers. 

Network Issues

Oftentimes networking problems will be the last thing we evaluate. If a server is offline or experiencing congestion, emails may be bounced back or refused altogether. If a network is unreachable or is misconfigured, the delivery of messages will not work.

We can check on these issues using the traceroute, dig, hostname or other Linux commands. External tools can also be used to provide insight as to why mail is failing. MXtoolbox.com, senderbase.com, dnschecker.org, or verify-email.org can provide in-depth information about why email fails to send. 

Delayed Email

In most cases, delayed email is usually due to a slow or overloaded email server. Other causes include spamming, checking email frequently, or hardware issues.

On servers where a “greylisting” option is available, emails can be delayed. Greylisting is a technique used to defend against spam. A mail transfer agent (or MTA - like Sendmail, Postfix, or Exim) uses greylisting to "temporarily reject" an email if it does not recognize the sender or sends emails too quickly. If the email is legitimate, the sending server will continue to try and deliver the email again after the delay. Once sufficient time has elapsed, the email will be accepted. If looking to diagnose this issue, the logs will show a status code of 429.

Missing Email

One of the most troubling issues for users is when email goes missing. Typically, this occurs when a POP3 account is used to retrieve email. POP3 downloads and then deletes all emails from the server, leaving no specific trace of where they went.

In the past, some clients connected to the server while unknowingly using an email client like Outlook (that was set up as POP3) to retrieve their email. They found out after the fact that all the email was pulled to that client. The only way to address this is to use an IMAP-enabled account to sync the email back up to the server. This involves setting up a second email account using the IMAP settings on the account and then selecting and dragging the emails from one inbox to the other. Because IMAP syncs the email on its account, it is reuploaded to the server. 

If an email is missing or failed to arrive in your inbox after being sent to you, it may have been marked as spam. This can occur for various reasons and can be addressed individually unless the sending domain has been blacklisted. In that case, the sender must follow the Outbound Spam steps listed above. This includes delisting the domain and external validation by the listing agency.

Disk Quota Errors 

If a user has an active email account, the email account can fill and hit a hard limit on space. If this issue occurs, we can usually increase the allocation of space for that user, which addresses the issue.

Note: The account may have thousands of emails that are not specifically relevant or needed. The only way to address this is to remove the older or non-essential emails on the account from the server.

Port 25 is Blocked

Some service providers do not use port 25 for email as this is the default for most of the servers in use today. They can change the receiving port to 26 or another less used port to increase the server’s security level. This helps to prevent spamming from a compromised server. The easiest method to verify this is to use the telnet command noted earlier to check the accessibility via port 26 on the external server.

Conclusion

If none of the above information helps resolve an issue you are experiencing, gather the following data and open a chat, support ticket, or give us a call at 800-580-4985. The details you provide us will be needed to resolve the issue. Below is the info we need to begin troubleshooting every email issue. The more info, the better! 

• The email account(s) affected.
• The type of client used to read/send an email and if you use POP3 or IMAP.
• Include any error messages in full.
• Add any bounceback information received in full.
• When the issue started.
• When does the issue occur? Is it constant, intermittent, or happens at a specific time of day?
• Provide a list of related DNS entries (if possible) using the dig or whois commands.
• Describe what triggers the issue. Ex. When opening an email, it takes five minutes to load, or when trying to send an email, I get the X error message.
• If the issue is regarding a single email, we need the date and time the email was sent, the subject, sender, receiver, full header information, and any related error messages that occurred. 
• Can you connect to and send or receive emails via your web client? 

With this information, we can explain the issues we are seeing and offer a solution to remedy the situation as quickly as possible. 

Need More Help?

We pride ourselves on being The Most Helpful Humans In Hosting™! Our Support Teams are filled with experienced Linux technicians and talented system administrators who have intimate knowledge of multiple web hosting technologies, especially those discussed in this article. Should you have any questions regarding this information, we are available 24 hours a day, 7 days a week, 365 days a year.

If you are a Fully Managed VPS server, Cloud Dedicated, VMWare Private Cloud, Private Parent server, Managed Cloud Servers, or a Dedicated server owner and you are uncomfortable with performing any of the steps outlined, Support can be reached via phone at @800.580.4985, a chat or support ticket to assisting you with this process.