Apache Performance Tuning: Swap Memory

Before we get into the nitty-gritty of Apache tuning, we need to understand what happens when servers go unresponsive due to a poorly optimized configuration. An over-tuned server is one that is configured to allow more simultaneous requests (ServerLimit) than the server’s hardware can manage. Servers set in this manner have a tipping point, once reached, the server will become stuck in a perpetual swapping scenario. Meaning the Kernel is stuck rapidly reading and writing data to and from the system swap file. Swap files have read/write access speeds vastly slower than standard memory space. The swap files’ latency causes a bottleneck on the server as the Kernel attempts to read and write data faster than is physically possible or more commonly known as thrashing. If not caught immediately, thrashing spirals the system out of control leading to a system crash. If trashing is left running for too long, it has the potential of physically harming the hard drive itself by simulating decades of read/write activity over a short period. When optimizing Apache, we must be cautious not to create a thrashing scenario. We can accomplish this by calculating the thrashing point of the server based on several factors.

 

Pre-Flight Check

This article covers all Apache-based servers including but is not limited to, both traditional Dedicated servers and Cloud VPS servers running a variety of Linux distributions. We will include the primary locations where stored Apache configurations on the following Liquid Web system types:

Liquid Web Server Types

Calculating the estimated thrashing point or ServerLimit of a server uses a simple equation:

Caculating the Thrashing Point

  • buff/cache: The total memory used by the Kernel for buffers and cache.
  • Reserved: The amount of memory reserved for non-Apache processes.
  • Available: The difference between buff/cache and Reserved memory.
  • Avg.Apache: The average of all running Apache children during peak operational hours.

 

Important
Calculating the thrashing point/ServerLimit should be done during peak operational hours and periodically reassessed for optimal performance.

The thrashing point value is equal to the number Apache children the server can run; this applies to either threaded or non-thread children. When the number of children running in memory meets the calculated thrashing point, the server will begin to topple. The following example walks through a standard Liquid Web Fully Managed cPanel server to illustrate gathering the necessary details to calculate a system’s estimated thrashing point.

 

Buff/Cache Memory

On modern Linux systems, the buffer/cache can be derived using the /proc/meminfo file by adding the Buffers, Cached and Slab statistics. Using the free command, we can quickly grab this information, as in the example below:

free

Output:

Use the free command to get the buff/cache info

Don’t be fooled by the column labeled “available.” We are solely looking at the memory we can reappropriate, which is the buff/cache column (708436).

 

Reserved Memory

Reserved memory is a portion of memory held for other services aside from Apache. Some of the biggest contenders for additional memory outside of Apache are MySQL, Tomcat, Memcache, Varnish, and Nginx. It is necessary to examine these services configs to determine a valid reserved memory. These configs are outside the purview of this article. However, MySQL is the most commonly encountered service running with Apache. You can find tools online to help analyze and configure MySQL separate from this article.

 

Rule-of-Thumb:
Save 25% of the total buff/cache memory for any extra services ran on the server. Examples:

  • A standard cPanel server runs several services along with Apache and MySQL. A server with these services runs on the heavier side, needing 25% reserved for non-Apache services.
  • A pure Apache web node in a high capacity load balanced configuration does not need to reserve any additional ram for other services.

 

Average Apache Memory

Finding the average size of Apache processes is relatively simple using the ps command to list the RSS (Resident Set Size) of all running httpd processes. Note: some distributions use “apache” instead of “httpd” for process name.

This example uses a short awk script to print out the average instead of listing the sizes.

ps o rss= -C httpd|awk '{n+=$1}END{print n/NR}'

Output:

22200.6

 

This example is easy to average by hand, but larger servers will require more calculation.

ps o rss -C httpd

Output:

RSS
5092
27940
28196
27572

 

Calculate the Thrashing Point

Once collected divided the Available memory by Avg. Apache, rounding down to the nearest whole number. Available memory is the buff/cache memory minus the Reserved memory. Below is a summary of the calculation process in table form.

Thrashing point calculations

Conservative estimates are provided below for various memory configurations. These estimates can be used as a starting configuration but will require additional follow up performance assessments during peak hours to adjust directives by the servers.

General Thrashing Estimates

Remote Desktop Troubleshooting

Remote Desktop Protocol (or RDP) is the most common method of gaining administrative access to a Windows server. RDP is available on all versions of Windows server and a client (called Remote Desktop Connection) is included with all versions of Windows desktop operating systems. Clients are also available for Macintosh operating systems from Microsoft in the iTunes store and for Linux desktops with applications like FreeRDP. Connecting to your server via RDP allows you full control of the server desktop environment, just as if you were sitting in front of the server’s monitor and keyboard. Depending on your permissions and settings, you can copy and delete files, change file permissions or settings, and even print documents from the server.

Pre-Flight Check

Using Remote Desktop Protocol to manage a Windows server generally requires a few basic settings and information about the server.

  • First, the Remote Desktop Service must be running on the server to which you would like to connect (RDP uses port 3389 by default).
  • Second, you need to know the IP address of the server.  
  • Third, you must have a username and password that is allowed to connect to the server remotely (often, this is the primary administrator account, but can also be a secondary account set up specifically for remote access purposes).
  • Finally, the Windows firewall (and any other hardware or software firewalls) needs to be configured to allow Remote Connections from your location.

 

Once you have all of the correct settings enabled, IP address and user account details, you can connect RDP to your server! Just launch the RDP client, enter the IP address of the server and the user credentials, and log in to the server using what looks like the standard Windows desktop environment.

Image of Remote Desktop Connection

As helpful as the Remote Desktop Protocol can be when it comes to managing your Windows server, there are also times when the connection fails, which can be very frustrating as the error message is generally not very helpful (often just the window shown below).         

RDP Connection Error Pop Up

 

The error shown above means that for some reason, your client was unable to make a connection to the Windows server via the Remote Desktop Protocol. When you are experiencing connectivity issues, there are many items that you can check to try to resolve the problem.

 

  1. Ensure you can reach the server via ICMP (or Ping). Most desktop operating systems will allow you to send small bits of information to the computer to verify connectivity and connection speeds. Generally, you just need to open a terminal window (on a Windows desktop, press the Window key, then type cmd and press enter) and enter the following command: ping IP or ping domain.tld. Normally, you’ll receive an output that is similar:Ping Results
  2. This output shows the pings were successful to the destination and took between 50 ms and 150 ms to complete. These pings indicate a successful connection to the server as desired (at least over ICMP). If the output for the command shows a failure to respond, we know there is some network interference.
  3. If the ping test fails (indicated by repeating asterisks), check your internet connectivity to guarantee that you can reach other resources on the internet. If not, you may need to contact your local service provider to restore your internet access.
  4. Reaching other internet sites but not your server indicates your server is refusing connections from your IP address (due to security software or firewall settings). You may need to contact your hosting company to verify there is not an IP address blocked by your server. You can find your current public IP address by going to http://ip.liquidweb.com.
  5. Can you ping your server, but still can’t connect over RDP? It is likely an issue with the RDP service or your firewall. You’ll need to contact your hosting company to get assistance with the service or firewall.

Firewall Issues

Best practices in configuring a firewall is to allow the least amount of access necessary for the various connections to the server. Limiting the connections to a particular service like RDP is called “scoping” the access for that service. If your configured Windows firewall scopes traffic on RDP, it’s possible that a user may not be able to connect due to their IP address not being included in the rule. Access to the server via RDP from one user but another user is not, check the firewall; their IP address may not be included in the allowed list of IPs for Remote Desktop Access.

  1. Log in to the server, click on the Windows icon, and type Windows Firewall into the search bar.Firewall Settings
  2. Click on Windows Firewall with Advanced Security.
  3. Click on Inbound Rules
  4. Scroll down to find a rule labeled RDP (or using port 3389).
  5. Double-click on the rule, then click the Scope tab.Scope Tab
  6. Make sure the user’s current IP address is included in the list of allowed Remote IPs.

If you are unable to connect to the server from your location, contact your hosting company for help in checking the firewall rule for RDP access.

User Connectivity Problems

Can you connect to RDP using the administrator account, but one or more of the other accounts cannot? There may be a problem with the user account permissions.

  1. Make certain the user is a member of the Remote Desktop Users group. Log in to the server with the administrator account, then go to the Local Users and Groups control panel (Open Administrative Tools, then open Computer Management).Local Users and Groups
  2. Navigate to the Remote Desktop Users group and verify that the user is a member of the group. If they are not a member of the group, add them as a member of the group.
    remote desktop users group
  3. Go to the username under the Users tab. Make sure that the user account is not locked out. Accounts can get locked out due to too many attempts to log in with an incorrect password (either by the user or by a brute force attack on the server).
    account lockout screen
  4. Double check the firewall for the IP address of the user and add to the scope of the RDP rule.

No Available Connections/Sessions

By default, Windows server only allows two users to connect via RDP simultaneously. If both sessions are already in use, you will receive an error indicating that no additional users are allowed to connect at this time. Too Many Users Error

To resolve this issue, you will need to wait until one of the other users logs out or you’ll require to purchase additional RDP user licenses from your hosting provider (assuming that you regularly need access for more than two users at a time).

Failed login attempts during a brute force attack can sometimes take up RDP licenses, even though the session isn’t connecting. If you are experiencing unavailable sessions even when no one is logged in to the server, it’s possibly the result of a malicious login. The best remedy for this situation is to scope the firewall rule to prevent access attempts from unauthorized IP addresses.

Data Encryption Errors

If you are using an out of date Remote Desktop Client or are connecting to an older Windows server, you may receive an error that there is a problem with the TLS settings for the connection. Generally, you can resolve this issue by updating your RDP client software on your workstation. It may also be possible to set the client to ignore these errors, but that could leave your workstation and your server vulnerable to malicious attacks.

Sudden Disconnection

If you are using RDP and suddenly lose the connection, the issue is almost always related to your internet connection. Check to make sure that you can stay connected to other services (like running a ping command in the background). If you are not losing internet connectivity, it’s possible that the server is running out of memory or the RDP service may be experiencing an active attacked in a brute force attack. If you’ve confirmed that your internet connection is stable, contact your hosting company to make sure that the server is not the cause of the lost connection.

Slow Connection Issues

If the connection between your location and your server is slow your Remote Desktop Session may not function as smoothly as you would like. However, you may be able to adjust the Desktop Environment settings of the connection before you connect to simplify and speed up the connection.

  1. Open the Remote Desktop Client application (these directions are for the Windows built-in client, but most RDP clients have similar settings available).
  2. Click on the Experience tab to see the various items you can choose to enable or disable to improve your connection speeds. Change the drop-down to select a specific connection speed or select/deselect the various items to optimize performance.Remote Desktop Connection Settings

         

Windows 10 Update Issues

Oddly enough, Microsoft updates often cause problems with RDP connectivity. As recent as April 2018, an update on both the server operating system and the Windows 10 desktop operating system caused connectivity issues for many users. Generally, the best policy is to update both the server and workstation, as connectivity issues most often arise when the two systems are not on the same update cycle. You may be able to resolve a new connectivity issue by removing a recent Windows update (either on the server or the desktop). Many users also reported that disabling the Printer option from the local resources setting resolved the most recent connectivity issue.         

Local Resources

 

While RDP is a great tool for managing your Windows server, connectivity issues can be frustrating. By working through the possible causes of the connection problem, you will generally be able to get reconnected and working again in no time!

Configuring and Troubleshooting WHMCS Crons

Over the years WHMCS has made some changes to where it stores certain directories, specifically directories outside of public_html. The goal of this is to increase overall security by moving sensitive files to a more protected location. While this change does help to improve WHMCS security, it also adds a few steps of complexity.

This article is meant to help simplify this complexity, or at least provide a reference configuration that you can use to troubleshoot cron issues, or gain a better understanding of WHMCS crons in general. I used WHMCS 7.3 for this article, but the general concept and instructions should apply for any 7.0 version of WHMCS.
Continue reading “Configuring and Troubleshooting WHMCS Crons”

How To Modify an Existing Email Account in Thunderbird

Pre-Flight Check

  • These instructions are intended specifically for setting up an email account in Mozilla Thunderbird 38.3.0 on Mac OS X 10.11.1.
  • While the steps should be similar across platforms and operating systems, they may not necessarily apply to older versions of Thunderbird.
  • For help with general email account settings, see How to Set up Any Email Client.

You can edit an email account that already has been configured in Thunderbird, for example should you decide to switch between non-SSL and SSL settings or change the server’s connection port. You change the connection type between standard (non-SSL) and secure (SSL) by changing the hostname and port for the incoming and outgoing servers.

Note: You cannot edit an existing email account to switch its account type from POP3 to IMAP or vice versa. To change the account type, you must add a new account of the desired type (POP3 or IMAP). Adding a new account with a different connection type should not require you to delete the old one in most mail clients.

To avoid data loss, please use caution any time you change an email account’s connection type or delete an email account. Removing an email account from a mail client also will remove all messages associated with it on the device and, specifically in the case of POP accounts that are not configured to retain mail on the server, there may be no way to recover those messages. If you have any doubt or questions, please feel free to contact Heroic Support® for guidance.

Step #1: Edit Incoming Server Settings

  1. To edit the incoming server, select your email address in the left pane and then click on View settings for this account in the main window.
  2. In the account settings window, click on Server Settings to update the Server Name and Port.incomingedit
    • Server Name
      • SSL settings will use the server’s hostname (e.g., host.yourdomainname.com)
      • Standard non-SSL settings will use the domain name (yourdomainname.com or mail.yourdomainname.com).
    • Port
      • SSL settings will use Port 993 for IMAP and Port 995 for POP3.
      • Standard non-SSL settings will use Port 143 for IMAP and Port 110 for POP3.

Step #2: Edit Outgoing Server Settings

  1. To edit the outgoing server settings, click on Outgoing Server (SMTP) in the left pane, select your outgoing server and click the Edit button.editoutgoing
  2. You can edit the server name and port in the popup window.out2
    • Server Name
      • SSL settings will use the server’s hostname (e.g., host.yourdomainname.com)
      • Standard non-SSL settings will use the domain name (yourdomainname.com or mail.yourdomainname.com).
    • Port
      • SSL settings will use Port 465.
      • Standard non-SSL settings will use Port 587 (depending on your server configuration, you may be able to use Port 25 as well).
  3. Click on the OK button to save the outgoing server settings, then click on OK once more to exit the settings menu and begin using your email account with the new settings.

 

How to Install Logwatch on Fedora 21

Logwatch is a Perl-based log management tool for analyzing, summarizing, and reporting on a server’s log files. It is most often used to send a short digest of server’s log activity to a system administrator.

What are log files? Logs are application-generated files useful for tracking down and understanding what has happened in the past.

Pre-Flight Check

  • These instructions are intended specifically for installing the Logwatch on Fedora 21.
  • I’ll be working from a Liquid Web Self Managed Fedora 21 server, and I’ll be logged in as root.

Continue reading “How to Install Logwatch on Fedora 21”

How to Install Logwatch on Fedora 20

Logwatch is a Perl-based log management tool for analyzing, summarizing, and reporting on a server’s log files. It is most often used to send a short digest of server’s log activity to a system administrator.

What are log files? Logs are application-generated files useful for tracking down and understanding what has happened in the past.

Pre-Flight Check
  • These instructions are intended specifically for installing the Logwatch on Fedora 20.
  • I’ll be working from a Liquid Web Self Managed Fedora 20 server, and I’ll be logged in as root.

Continue reading “How to Install Logwatch on Fedora 20”

How to Install Logwatch on Ubuntu 14.04 LTS

Logwatch is a Perl-based log management tool for analyzing, summarizing, and reporting on a server’s log files. It is most often used to send a short digest of server’s log activity to a system administrator.

What are log files? Logs are application-generated files useful for tracking down and understanding what has happened in the past.

Pre-Flight Check
  • These instructions are intended specifically for installing the Logwatch on Ubuntu 12.04 LTS.
  • I’ll be working from a Liquid Web Core Managed Ubuntu 12.04 LTS server, and I’ll be logged in as root.

Continue reading “How to Install Logwatch on Ubuntu 14.04 LTS”

How to Install Logwatch on Ubuntu 12.04 LTS

Logwatch is a Perl-based log management tool for analyzing, summarizing, and reporting on a server’s log files. It is most often used to send a short digest of server’s log activity to a system administrator.

What are log files? Logs are application-generated files useful for tracking down and understanding what has happened in the past.

Pre-Flight Check
  • These instructions are intended specifically for installing the Logwatch on Ubuntu 12.04 LTS.
  • I’ll be working from a Liquid Web Core Managed Ubuntu 12.04 LTS server, and I’ll be logged in as root.

Continue reading “How to Install Logwatch on Ubuntu 12.04 LTS”

How to Install Logwatch on CentOS 7

Logwatch is a Perl-based log management tool for analyzing, summarizing, and reporting on a server’s log files. It is most often used to send a short digest of server’s log activity to a system administrator.

What are log files? Logs are application-generated files useful for tracking down and understanding what has happened in the past.

Pre-Flight Check
  • These instructions are intended specifically for installing the Logwatch on CentOS 7.
  • I’ll be working from a Liquid Web Core Managed CentOS 7 server, and I’ll be logged in as root.

Continue reading “How to Install Logwatch on CentOS 7”

How to Install Logwatch on CentOS 6

Logwatch is a Perl-based log management tool for analyzing, summarizing, and reporting on a server’s log files. It is most often used to send a short digest of server’s log activity to a system administrator.

What are log files? Logs are application-generated files useful for tracking down and understanding what has happened in the past.

Pre-Flight Check
  • These instructions are intended specifically for installing the Logwatch on CentOS 6.
  • I’ll be working from a Liquid Web Core Managed CentOS 6.5 server, and I’ll be logged in as root.

Continue reading “How to Install Logwatch on CentOS 6”