Apache Performance Tuning: Swap Memory

Before we get into the nitty-gritty of Apache tuning, we need to understand what happens when servers go unresponsive due to a poorly optimized configuration. An over-tuned server is one that is configured to allow more simultaneous requests (ServerLimit) than the server’s hardware can manage. Servers set in this manner have a tipping point, once reached, the server will become stuck in a perpetual swapping scenario. Meaning the Kernel is stuck rapidly reading and writing data to and from the system swap file. Swap files have read/write access speeds vastly slower than standard memory space. The swap files’ latency causes a bottleneck on the server as the Kernel attempts to read and write data faster than is physically possible or more commonly known as thrashing. If not caught immediately, thrashing spirals the system out of control leading to a system crash. If trashing is left running for too long, it has the potential of physically harming the hard drive itself by simulating decades of read/write activity over a short period. When optimizing Apache, we must be cautious not to create a thrashing scenario. We can accomplish this by calculating the thrashing point of the server based on several factors.

 

Pre-Flight Check

This article covers all Apache-based servers including but is not limited to, both traditional Dedicated servers and Cloud VPS servers running a variety of Linux distributions. We will include the primary locations where stored Apache configurations on the following Liquid Web system types:

Liquid Web Server Types

Calculating the estimated thrashing point or ServerLimit of a server uses a simple equation:

Caculating the Thrashing Point

  • buff/cache: The total memory used by the Kernel for buffers and cache.
  • Reserved: The amount of memory reserved for non-Apache processes.
  • Available: The difference between buff/cache and Reserved memory.
  • Avg.Apache: The average of all running Apache children during peak operational hours.

 

Important
Calculating the thrashing point/ServerLimit should be done during peak operational hours and periodically reassessed for optimal performance.

The thrashing point value is equal to the number Apache children the server can run; this applies to either threaded or non-thread children. When the number of children running in memory meets the calculated thrashing point, the server will begin to topple. The following example walks through a standard Liquid Web Fully Managed cPanel server to illustrate gathering the necessary details to calculate a system’s estimated thrashing point.

 

Buff/Cache Memory

On modern Linux systems, the buffer/cache can be derived using the /proc/meminfo file by adding the Buffers, Cached and Slab statistics. Using the free command, we can quickly grab this information, as in the example below:

free

Output:

Use the free command to get the buff/cache info

Don’t be fooled by the column labeled “available.” We are solely looking at the memory we can reappropriate, which is the buff/cache column (708436).

 

Reserved Memory

Reserved memory is a portion of memory held for other services aside from Apache. Some of the biggest contenders for additional memory outside of Apache are MySQL, Tomcat, Memcache, Varnish, and Nginx. It is necessary to examine these services configs to determine a valid reserved memory. These configs are outside the purview of this article. However, MySQL is the most commonly encountered service running with Apache. You can find tools online to help analyze and configure MySQL separate from this article.

 

Rule-of-Thumb:
Save 25% of the total buff/cache memory for any extra services ran on the server. Examples:

  • A standard cPanel server runs several services along with Apache and MySQL. A server with these services runs on the heavier side, needing 25% reserved for non-Apache services.
  • A pure Apache web node in a high capacity load balanced configuration does not need to reserve any additional ram for other services.

 

Average Apache Memory

Finding the average size of Apache processes is relatively simple using the ps command to list the RSS (Resident Set Size) of all running httpd processes. Note: some distributions use “apache” instead of “httpd” for process name.

This example uses a short awk script to print out the average instead of listing the sizes.

ps o rss= -C httpd|awk '{n+=$1}END{print n/NR}'

Output:

22200.6

 

This example is easy to average by hand, but larger servers will require more calculation.

ps o rss -C httpd

Output:

RSS
5092
27940
28196
27572

 

Calculate the Thrashing Point

Once collected divided the Available memory by Avg. Apache, rounding down to the nearest whole number. Available memory is the buff/cache memory minus the Reserved memory. Below is a summary of the calculation process in table form.

Thrashing point calculations

Conservative estimates are provided below for various memory configurations. These estimates can be used as a starting configuration but will require additional follow up performance assessments during peak hours to adjust directives by the servers.

General Thrashing Estimates

Install Multiple PHP Versions on Ubuntu 16.04

As a default, Ubuntu 16.04 LTS servers assign the PHP 7.0 version. Though PHP 5.6 is coming to the end of life in December of 2018, some applications may not be compatible with PHP 7.0. For this tutorial, we instruct on how to switch between PHP 7.0 and PHP 5.6 for Apache and the overall default PHP version for Ubuntu.

Step 1: Update Apt-Get

As always, we update and upgrade our package manager before beginning an installation. If you are currently running PHP 7.X, after updating apt-get, continue to step 2 to downgrade to PHP 5.6.

apt-get update && apt-get upgrade

Step 2: Install PHP 5.6
Install the PHP5.6 repository with these two commands.

apt-get install -y software-properties-common
add-apt-repository ppa:ondrej/php
apt-get update
apt-get install -y php5.6

Step 3: Switch PHP 7.0 to PHP 5.6
Switch from PHP 7.0 to PHP 5.6 while restarting Apache to recognize the change:

a2dismod php7.0 ; a2enmod php5.6 ; service apache2 restart

Note
Optionally you can switch back to PHP 7.0 with the following command: a2dismod php5.6 ; a2enmod php7.0 ; service apache2 restart

Verify that PHP 5.6 is running on Apache by putting up a PHP info page. To do so, use the code below in a file named as infopage.php and upload it to the /var/www/html directory.

<? phpinfo(); ?>

By visiting http://xxx.xxx.xxx.xxx/infopage.php (replacing the x’s with your server’s IP address), you’ll see a PHP info banner similar to this one, confirming the PHP Version for Apache:

Example of PHP Info page

Continue onto the section PHP Version for Ubuntu to edit the PHP binary from the command line.

Step 4: Edit PHP Binary

Maintenance of symbolic links or the /etc/alternatives path through the update-alternatives command.

update-alternatives --config php

Output:
There are 2 choices for the alternative php (providing /usr/bin/php).
Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/bin/php7.0 70 auto mode
1 /usr/bin/php5.6 56 manual mode
2 /usr/bin/php7.0 70 manual mode
Press to keep the current choice[*], or type selection number:

Select php5.6 version to be set as default, in this case, its the number one option.

You can now verify that PHP 5.6 is the default by running:
php -v

Output:
PHP 5.6.37-1+ubuntu16.04.1+deb.sury.org+1 (cli)
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies

Install Nginx on Ubuntu 16.04

Nginx is an open source Linux web server that accelerates content while utilizing low resources. Known for its performance and stability Nginx has many other uses such as load balancing, reverse proxy, mail proxy and HTTP cache. With all these qualities it makes a definite competitor for Apache. To install Nginx follow our straightforward tutorial.

Pre-Flight Check

  • Logged into an as root and are working on an Ubuntu 16.04 LTS server powered by Liquid Web! If using a different user with admin privileges use sudo before each command.

Step 1: Update Apt-Get

As always, we update and upgrade our package manager.

apt-get update && apt-get upgrade

Step 2: Install Nginx

One simple command to install Nginx is all that is needed:

apt-get -y install nginx

Step 3: Verify Nginx Installation

When correctly installed Nginx’s default file will appear in /var/www/html as index.nginx-debian.html . If you see the Apache default page rename index.html file. Much like Apache, by default, the port for Nginx is port 80, which means that if you already have your A record set for your server’s hostname you can visit the IP to verify the installation of Nginx. Run the following command to get the IP of your server if you don’t have it at hand.

ip addr show eth0 | grep inet | awk '{ print $2; }' | sed 's/\/.*$//'

Take the IP given by the previous command and visit via HTTP. (http://xxx.xxx.xxx.xxx) You will be greeted with a similar screen, verifying the installation of Nginx!

Nginx Default Page

Note
Nginx, by default, does not execute PHP scripts and must be configured to do so.

If you already have Apache established to port 80, you may find the Apache default page when visiting your host IP, but you can change this port to make way for Nginx to take over port 80. Change Apache’s port by visiting the Apache port configuration file:

vim /etc/apache2/ports.conf

Change “Listen 80” to any other open port number, for our example we will use port 8090.

Listen 8090

Restart Apache for the changes to be recognized:

service apache2 restart

All things Apache can now be seen using your IP in the replacement of the x’s. For example http://xxx.xxx.xxx.xxx:8090

Whitelisting in ModSecurity

Broken down into two parts our article’s first section hits on “how to whitelist IPs or URIs,” for people who are somewhat familiar with ModSecurity but want to know further about the process. Our second section examines why we configure ModSecurity and how to prevent the security of the server from getting in the way of our work. If you have a Fully Managed Liquid Web server reach out to our Heroic Support team for assistance with whitelisting!

How to Whitelist IPs or URIs

“ModSecurity is a toolkit for real-time web application monitoring, logging, and access control.” (modsecurity.org).  In simple terms, this means that ModSec, also called mod_security or ModSecurity, is a web application firewall that can actively look for attacks to the system and stop malicious activity. However, sometimes these rules trigger when legitimate work is taking place, blocking your IP and stopping you or your developer’s until you can remove the IP block. The way around for being blocked is known as whitelisting, which essentially allows for a specific IP to access the server.   There are a few ways to whitelist a request in ModSec, either by IP or by URI (URIs are specific pages on the website). 

Getting Started

  1. Find your IP or ask your developer for theirs. (You can find this by going to ip.liquidweb.com)If you or your developer have a static IP (one that will not change), one way you can whitelist the ModSec rules is by IP.
  2. Find the ModSec error in the Apache error logs with the following command (Be sure to modify the command with your IP in place of “IP here.”):
    grep ModSec /usr/local/apache/logs/error_log | grep “IP here”.
  3. The output of this command will give you a list of hits for ModSecurity from you or your developer’s IP, which you can see below. While this looks intimidating, you will only want to pay attention to 3 bits of information highlighted.  Please note, the output will not show these colors when you are viewing the files.
Note
Blue = client, the IP which tripped the rule
Red
= ID number of tripped rule within ModSec
Green = URI, the location where the error started from

[Fri May 25 23:07:04.178701 2018] [:error] [pid 78007:tid 139708457686784] [client 61.14.210.4:30095] [client 61.14.210.4] ModSecurity: Access denied with code 406 (phase 2). Pattern match "Mozilla/(4|5)\\\\.0$" at REQUEST_HEADERS:User-Agent. [file "/etc/apache2/conf.d/modsec2.liquidweb.conf"] [line "109"] [id "20000221"] [hostname "67.227.209.163"] [uri "/db/index.php"] [unique_id "WwjPWChxvG1CO4kz-D55eQAAACU"]

 

Whitelist By IP:

1. Once you have the correct ModSec error, you will need to edit the ModSec configuration. If you are using Easy Apache 4 you will find the configuration file with this path:
/etc/apache2/conf.d/modsec2/whitelist.conf

2. Open the file with your favorite text editor, such as vim, nano, or file manager like so:

vim /etc/apache2/conf.d/whitelist.conf

3.  The blue text above will be the IP address that you are whitelisting from the original error. You must keep the backslashes (\) and up-carrot (^) in order for the IP to be read correctly. Thus it will always look something like:

“^192\.168\.896\.321”

For for the id, noted in red, you will change the number after the colon, which will be the Apache error log like we saw above. This will look similar to:

Id:2000221

Add the following code with the colored sections edited to match your intended IP.

SecRule REMOTE_ADDR "^64\.14\.210\.4"
"phase:1,nolog,allow,ctl:ruleEngine=off,id:20000221"

 

Whitelist By URI:

If your IP is dynamic (changing) and you keep getting blocked in the firewall, it is best to whitelist it via URI, the yellow item in the ModSec error.

1. Begin by opening the Easy Apache 4 configuration file:

vim /etc/apache2/conf.d/whitelist.conf

2. Add the following text to the configuration. Remember to pay attention to the highlighted parts.  Change the yellow “/db/index.php” to match your URI and the red id to match the id of your error (Do not use the colon in this one).

<LocationMatch "/db/index.php">
SecRuleRemoveById 20000221
</LocationMatch>

3. The final step for whitelisting, before you finalize the process, is to ensure you have correctly set up the whitelist. For Easy Apache 4 you will run the command:
apachectl -t

As long as the command returns “Syntax Ok” you are safe to make the whitelist active by restarting Apache. Otherwise, review the whitelists to make sure the syntax matches up correctly with the above directions.

4. Lastly, restart Apache with the following command.

/scripts/restartsrv_httpd

You have successfully whitelisted yourself in ModSec!

 

Using ModSec

Cyber Security is a hydra; once one threat is cut off, two more grow back. While this is not a new analogy, it’s important to understand as we battle threats to our network, computers, and servers. With all the complexities that come with security, I want to talk about adequately configuring ModSec to deter threats while still allowing you to work on your websites. Often, when it comes to server security, too much protection can hinder effectiveness.

For example, say you have the following set up on your server:

  • You do not allow root SSH login to the server
  • utilize dual-factor authentication for any SSH logins
  • use an SSH key for the sudo user and require other security safeguards

While this type of configuration is secure, it takes longer to log into your system to make a quick edit to your settings, a double-edged sword; how can you keep the server safe while not tying your own hands?  A great example of how this plays out is using ModSec.

ModSec can block your IP if it falsely flags your work. While this module improves system security, you’ll need to be aware of properly implementing and “scoping” the technology. Scoping in this sense means to manage risks, the focus of what is important for security while still allowing work on the server with minimal interference. To tune out legitimate requests to your server, such as when you are editing your website’s code via a plugin, ModSec has the options to whitelist rules or IPs and keep your work on track.

Whitelisting an IP from the rules that ModSec follows is a great option so long as the IP never changes (i.e. a static IP, see article here to learn more https://support.google.com/fiber/answer/3547208?hl=en) and is limited to only people you trust. This method prevents ModSec from viewing your requests as malicious and blocking your IP. This practice has the drawback that if someone (say an unhappy employee) has access to your network, they now have a way around ModSec to attack your server.

With non-static (dynamic) IPs the problems of whitelisting an IP are readily apparent. With the continual change of a dynamic IP, it creates the potential of exploiting your server, as someone could use an old IP to access the server. Whitelisting specific rules comes to save the day! When you whitelist by rules, you can edit with granularity and limit the rules to particular domains and URIs, protecting the rest of the server from attacks related to that same rule!

Example of ModSecurity

ModSec reads a series of rules and applies them to incoming requests being made to the web server. An example of what a block looks like is:

[Sat Jun 30 02:21:56.013837 2018] [:error] [pid 79577:tid 139862413879040] [client 120.27.217.223:24397] [client 120.27.217.223] ModSecurity: Access denied with code 406 (phase 2). Pattern match "Mozilla/(4|5)\\\\.0$" at REQUEST_HEADERS:User-Agent. [file "/etc/apache2/conf.d/modsec2.liquidweb.conf"] [line "109"] [id "2000064"] [hostname "67.227.192.139"] [uri "/mysql/index.php"] [unique_id "WzchhAjuZ6wPAzo9AwW1WwAAAE8"]

This error shows Apache stopped a potential attack on a file at /mysql/index.php. This is an error similar to what appears when the code is being written or edited within programs like Drupal or WordPress.

Evaluating ModSecurity

If you are persistently being blocked in your firewall while working on your code, ModSec is the likely culprit. The ModSec errors can be found in the Apache error log (in cPanel the path is /usr/local/apache/logs/error_log). The phrase “ModSec” can be quickly isolated from the log (via the command ‘grep “ModSec” /usr/local/apache/logs/error_log’). By comparing you or your developer(s) IP to the log, you’ll be able to identify stopped requests that are legitimate. Verify these are valid requests by double-checking that someone in your organization made them. Once you have done so, you can move forward in setting up a whitelist for the error, per the steps above.

Again, we want to scope to allow the least amount of wiggle room for an attack and ensure we can keep working. If you are unable to have a trusted static IP, you’ll need to use the whitelist URI  method, providing the specific page as an exemption. Once completed, remove both whitelisted items from the configuration file, in case of a genuine attack.

On a parting note, I encourage you to explore ModSec and learn more of the ins and outs of the software. Exploring different methods of whitelisting can be a lot of for to learn and most importantly helps to tighten server security. As always, our Fully Supported Customers can contact our Helpful Human Support team for assistance. Check out articles on security in our Knowledge Base, like this one on Maldet! It’s another excellent way to learn about your server and develop an understanding of server security.

What is mod_deflate?

How mod_deflate works

When a visitor accesses a website, a request is made to the web server for a specific kind of data. An example might be a home page of a site. Next, the web server locates that data and delivers it to the client who is requesting that data – basically back to the web browser.

In this example, the speed at which the home page loads can depend on a variety of factors. One of them could be how long it takes to find and deliver the data for that page. This is just one example.

Some of that data – such as javascript files, css files, and php files – can actually be compressed into smaller sizes before they are delivered back to the visiting client or browser at the smaller size. The visitor can now have a more optimized browsing experience.

This is where mod_deflate comes in.

Continue reading “What is mod_deflate?”

How to Install and Configure phpMyAdmin on Ubuntu 15.04

phpMyAdmin is an open source tool used for the administration of MySQL. In addition to offering the capability to perform administration tasks such as creating, editing, or deleting databases, and managing users and permissions, phpMyAdmin provides a graphical user interface to do all of these tasks and more.

Pre-Flight Check

  • These instructions are intended specifically for installing phpMyAdmin on Ubuntu 15.04.
  • I’ll be working from a Liquid Web Self Managed Ubuntu 15.04 server, and I’ll be logged in as root.
  • A LAMP, Linux, Apache, MySQL and PHP, must be installed on your server.

Continue reading “How to Install and Configure phpMyAdmin on Ubuntu 15.04”

How to Install and Configure phpMyAdmin on Fedora 22

phpMyAdmin is an open source tool used for the administration of MySQL. In addition to offering the capability to perform administration tasks such as creating, editing, or deleting databases, and managing users and permissions, phpMyAdmin provides a graphical user interface to do all of these tasks and more.

Pre-Flight Check

  • These instructions are intended specifically for installing phpMyAdmin on Fedora 22.
  • I’ll be working from a Liquid Web Self Managed Fedora 22 server, and I’ll be logged in as root.
  • A LAMP stack, meaning Linux, Apache, MySQL and PHP, must be installed on your server.

Continue reading “How to Install and Configure phpMyAdmin on Fedora 22”

How to Install Apache on Fedora 21

The Apache web server is one of the most popular and powerful web servers in the world, due to its ease of administration and flexibility. In this tutorial we will install Apache on a server that doesn’t have a web server or database server already installed.

Pre-Flight Check
  • These instructions are intended for installing Apache on a single Fedora 21 node.
  • I’ll be working from a Liquid Web Self Managed Fedora 21 server, and I’ll be logged in as non-root user. If you need more information then visit our tutorial on How to Add a User and Grant Root Privileges on Fedora 21.

Continue reading “How to Install Apache on Fedora 21”

How to Install Apache on Fedora 20

The Apache web server is one of the most popular and powerful web servers in the world, due to its ease of administration and flexibility. In this tutorial we will install Apache on a server that doesn’t have a web server or database server already installed.

Pre-Flight Check
  • These instructions are intended for installing Apache on a single Fedora 20 node.
  • I’ll be working from a Liquid Web Self Managed Fedora 20 server, and I’ll be logged in as non-root user. If you need more information then visit our tutorial on How to Add a User and Grant Root Privileges on Fedora 20.

Continue reading “How to Install Apache on Fedora 20”

List Which Apache 2 Modules are Enabled on Fedora 21

The Apache web server is one of the most popular and powerful web servers in the world due to its ease of administration and flexibility. This flexibility comes Apache’s modular design, and allows for such features as: URL rewriting for SSL encryption natively, and Outlook Anywhere passthrough support in reverse proxy setups. Modularity allows Administrators to modify Apache to meet their needs; adding modules that are needed and removing ones that are not.

Pre-Flight Check
  • These instructions are intended specifically for viewing which Apache modules are enabled on Fedora 21.
  • I’ll be working from a Liquid Web Self Managed Fedora 21 server with Apache 2 installed, and I’ll be logged in as root.

Continue reading “List Which Apache 2 Modules are Enabled on Fedora 21”