Transfer an SSL to Ubuntu 16.04 or CentOS 7

SSL certificates have become a de facto part of every website. If you don’t yet have an SSL on your site to encrypt data, you should. Rather than showing an extra layer of security on sites protected by SSL, modern browsers instead now display a warning when a website does not have an SSL, essentially requiring sites to maintain their positive image.

When moving from one server to another, what needs to happen to your SSL to maintain your secure status? We’ll cover the basics for transferring traditional and Let’s Encrypt SSLs to Ubuntu 16.04 and CentOS 7.

Note:
This article will address SSLs in Apache specifically, but the same concepts apply to any service that supports SSL encryption.

Can SSLs be transferred between servers?

Absolutely! An SSL consists just of a handful of text files and a bit of configuration for the secure service. Additionally, there is a misconception that certificates are tied permanently to their IP address or physical server, but this is not true. In reality, they are linked only to the domain name(s) listed the certificate and are very migratable. Technically, an SSL can be used on multiple servers or services to protect the same domain name. We will leverage this concept to help us move our SSLs.

Prerequisites

You will need to have a copy of your SSL Certificate and SSL Private Key, as well as any ‘Chain Certificates.’ Often these will be two separate files, called domain.com.crt for the certificate, and domain.com.key for the private key, as well as an additional signing_authority.crt file for the chain certificate. Another popular naming convention for these files is cert.pem and key.pem, but sometimes they have no file suffix at all (.pem or .crt).

How can you tell where to get these files?  We’ll get into how to determine the locations for these configurations first.

Step 1: Collecting SSL Files

Debian SSL File Location

In distributions of Linux, including Ubuntu, you can check the loaded virtual hosts on Apache using:

apache2ctl -S

 

CentOS SSL File Location

For RHEL based distributions, including CentOS, you can use:

httpd -S

 

Both of these commands will list the running configuration settings, including all loaded virtual hosts and their configuration file locations. We’re looking for the domain name of the SSL we want to copy, and the HTTPS port 443. On my Ubuntu test machine, for example, this line is the output for the domain in question:

port 443 namevhost domain.com (/etc/apache2/sites-enabled/domain.com.conf:1)

Since we are copying the SSL for the domain.com domain, we’ll check this configuration file, and look for the virtual host block that references port 443.

vim /etc/apache2/sites-enabled/domain.com.conf

Searching through the file, we find:

<VirtualHost *:443>
ServerName domain.com
ServerAlias www.domain.com
DocumentRoot /data/www/domain.com
SSLCertificateFile /etc/ssl/domain.com/cert.pem
SSLCertificateKeyFile /etc/ssl/domain.com/privkey.pem
SSLCertificateChainFile /etc/ssl/domain.com/chain.pem
</VirtualHost>

This block of text describes all three files that we need to bring to the new server: the certificate file (cert.pem), the private key (privkey.pem) and the intermediate certificates (chain.pem).

 

Step 2: Copying SSL Files

From the file paths we discovered in the last step, we know that all of the SSL files for this site exist in the /etc/ssl/domain.com directory. So, we can copy this whole folder to the new server using the rsync tool. Assuming that the old server has an IP of 123.45.67.89, hop over to your new server, and run the command below. Trailing slashes (/) are very important for this command!

mkdir -p /etc/ssl
rsync -avz root@123.45.67.89:/etc/ssl/domain.com /etc/ssl/

This method preserves the ownership and permissions of these files, so we should not have to do anything else in that regard.  Our next section, step 3, is split into two sections.  Pick step 3A if you’re new server is Ubuntu 16.04 or step 3B if you new server is CentOS 7.

 

Step 3A: Setting up Apache on the new server (Ubuntu 16.04)

A non-SSL virtual host set up is necessary for your domain on the new server. If you have not yet done so, you will need to install and activate mod_ssl to allow Apache to parse SSL communication on the target machine:

apt-get install mod_ssl
a2enmod ssl

If you get a message that mod_ssl is not available for installation, its possibly installed through a different package. Check to see if that is the case:

dpkg -S mod_ssl.so

If this returns a positive result, you ‘ll run:

a2enmod ssl to complete the activation. After enabling mod_ssl, restart Apache:

systemctl restart apache2

Lastly, there is one configuration file to change, which will allow serving named virtual hosts. The activation of the SSL module lets Apache listen on port 443, but we need to be able to host multiple sites per IP. Open up the /etc/apache2/ports.conf file:

vim /etc/apache2/ports.conf

Ensure that the contents mimic this output:

<IfModule ssl_module>
NameVirtualHost *:80
Listen 80
NameVirtualHost *:443
Listen 443
</IfModule>

Specifically, we are ensuring that the NameVirtualHost lines are present. Once this is true, we can test and reload the configuration to bring it into effect:

apachectl configtest
systemctl reload apache2

Note:
Even though enabled sites have configurations listed in /etc/apache2/sites-enabled/, the actual configuration files should live in /etc/apache2/sites-available/, so that the apache control scripts can turn sites on and off. The sites-enabled folder consist exclusively of symlinks to configurations in sites-available.

Next, we must add a configuration for the website to call the SSL files we just copied over. If you are using the same document root on the new server as on the old, you can copy and paste the original virtual host block into a new configuration file in /etc/apache2/sites-available/. Be sure to update referenced IPs to match the new server.

If there are differences in the running modules or other customizations performed on the new machine, make a copy of the existing non-SSL virtual host block on the target server into a separate file. Also, under /etc/apache2/sites-available/, update the port number, and add the three lines for the SSL files. For our example, I copied and pasted the exact contents of the old file into /etc/apache2/sites-available/domain.com_ssl.conf.

After having done so, we run:

a2ensite domain.com_ssl

This links the configuration into the /etc/apache2/sites-enabled/ folder. This site name will change depending on what you named the configuration file (just cut off the .conf). Next, we test this new configuration:

apachectl configtest

If everything comes back OK, we can reload the configuration files for the apache2 service:

systemctl reload apache2

 

Step 3B: Setting up Apache on the new server (CentOS 7)

Each command line operation for CentOS is very different, but the overall procedure is the same, and the configuration files will have similar content. First, confirm that Apache is configured to support SSL traffic:

yum install mod_ssl

This installation procedure should automatically set up a ssl.conf file for you in /etc/httpd/conf.d/ for listening on port 443 and restart Apache, but it may not enable named virtual hosts. Make sure you add the following line to your httpd.conf file as well if it is not already present. Adding these lines allows you to serve multiple sites per IP address using virtual host blocks.

<IfModule mod_ssl.c>
NameVirtualHost *:443
</IfModule>

Next, we must add a configuration for the website to call the SSL files we just copied over. If you are using the same document root on the new server as on the old, copy and paste the original virtual host block into a new configuration file in /etc/httpd/conf.d/, making sure that the IP addresses are updated to match those on the target server.

If there are differences in the running modules or other customizations performed on the new machine, you can make a copy of the existing non-SSL virtual host block on the target server into a separate file. Also, in /etc/httpd/conf.d/, update the port number, and add the three lines for the SSL files.

Once created, test the configuration syntax of the file:

httpd -tThe output will either say ‘Syntax OK’ or tell you what file and line is problematic. If everything is okay, you can reload the configuration for Apache:

systemctl reload httpd

 

What if I use Let’s Encrypt?

Let’s Encrypt SSLs are just as transferable as any other SSL, but manually creating an SSL virtual host config file sometimes causes conflict with future installations of Let’s Encrypt. So, there is a particular procedure to be followed.

Transferring Let’s Encrypt Installations

First, Let’s Encrypt should be installed on the new server. For Ubuntu 16.04, that looks like this:

add-apt-repository ppa:certbot/certbot
apt-get update
apt-get install python-certbot-apache

 

For CentOS 7, use these commands:

yum install epel-release
yum install python-certbot-apache

Once installed on either distro, set a cron to renew the certificates automatically. You can set these up by running

crontab -e as the root user. The cron on my Ubuntu machine looks like this:

45 2 * * 6 /usr/local/letsencrypt/certbot-auto renew && systemctl reload apache2

On my CentOS server, it looks like this:

45 2 * * 6 /usr/local/letsencrypt/certbot-auto renew && systemctl reload httpd

Next, we can bring over the entire /etc/letsencrypt directory, which houses the configuration files. This directory holds the certificates and the keys themselves. On the new server, we run:

rsync -avz 123.45.67.89:/etc/letsencrypt /etc/

Now that the files are in place, we can have Let’s Encrypt reinstall all of the certificates we synced over:

/usr/local/letsencrypt/certbot-auto install --apache

Running the command presents a numbered list of the certificates in the /etc/letsencrypt folder. Select the number for the site you’d like to set up a configuration for, and it will create your configuration file; Once done, you only need to reload Apache:

systemctl reload apache2 #ubuntu
systemctl reload httpd #centos

 

Step 4: Confirming operation

The new server can now be tested using hosts file modification, detailed in this article. It’s truly is the best way to test the functionality of a migration target, as both browser and server believe that this is real traffic on the real domain name.

Once you set up your hosts file and flush your DNS cache, you should simply be able to load up the website in your browser using https, and see your website secured with the original certificate on the target machine! No worries though; live traffic is still routed to the original server with the same SSL, so your site visitors will not notice your testing.

After you finish testing your SSL installations and have confirmed that your websites are working well on the new server, DNS for the migrated sites can be updated to make the new server live.

If you are using Let’s Encrypt, it is important to keep your certificate expiration dates in mind. For instance, if your certificates are slated to expire in 7 days, you will want to update DNS promptly after the transfer, so that the new server can take over certificate renewals. If you miss this window and the certificate renews on the source server, you can simply re-run the rsync command where we collected /etc/letsencrypt:

rsync -avz 123.45.67.89:/etc/letsencrypt /etc/

Need help ordering a signed SSL or ordering a new server for your migration target? Chat with our solutions team!

 

Useful Command Line for Linux Admins

The command line terminal, or shell on your Linux server, is a potent tool for deciphering activity on the server, performing operations, or making system changes. But with several thousand executable binaries installed by default, what tools are useful, and how should you use them safely?

We recommend coming to terms with the terminal over SSH. Learn how to connect to your server over SSH, and get started with a few basic shell commands. This article will expand on those basic commands and show you even more useful and practical tools.

Warning:
Warning: These commands can cause a great deal of harm to your server if misused. Computers do precisely what you tell them to do. If you command your server to delete all files, it will remove every single file without question, and feasibly crash because it deleted itself. Please take precautions when working on your server, and ensure you have good local and remote backups available.

In the basic shell commands tutorial, you learned about basic navigation and file manipulation commands like ls, rm, mv, and cd. Below are a few essential commands for learning about your Linux system. (Display a user manual for each command by using man before each command, like so: man ps)

pipe

The pipe command (which is the | between two or more commands) is possibly the most useful tool in the shell language. This command allows the output of one command to be fed into the input of another command directly, without temporary files. The pipe command useful if you are dealing with a huge command output that you would like to format further, or to be processed by some other program without using a temporary file.

The basic tutorial showed the commands w and grep. Let’s connect them using pipe to format the output. Using the w command allows us to view users logged into the server while passing the output for the grep command to filter by the ‘root’ user type:

# w
08:56:43 up 27 days, 22:17, 2 users, load average: 0.00, 0.00, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root pts/0 10.1.1.206 08:52 0.00s 0.06s 0.00s w
jeff pts/1 191.168.1.6 09:02 1:59 0.07s 0.06s -bash

# w | grep root
root pts/0 10.1.1.206 08:52 0.00s 0.06s 0.00s w

The format of the last command is much more digestible and becomes much more important with the output from commands like ps.

ps

The ps command shows a ‘process snapshot’ of all currently running programs on the server. It is particularly useful in conjunction with the grep command to pare down its verbose results down to a certain keyword. For instance, let’s see if the Apache process ‘httpd,’ is running:

# ps faux | grep httpd
root 27242 0.0 0.0 286888 700 ? Ss Aug29 1:40 /usr/sbin/httpd -k start
nobody 77761 0.0 0.0 286888 528 ? S Sep17 0:03 \_ /usr/sbin/httpd -k start
nobody 77783 0.0 1.6 1403008 14416 ? Sl Sep17 0:03 \_ /usr/sbin/httpd -k start
...

We can see that there are several ‘httpd’ processes running here. The one owned by ‘root’ is the core one (the ‘forest’ nodes, \_, help identify child processes, too). If we did not see any httpd processes, it could safely assume, Apache is not running, and we should restart it to serve websites request again.

The common flags used for ps are ‘faux’, which displays processes for all users in a user-oriented format, run from any source (terminal or not, which is signified by the x), paired with a process tree (forest). The ‘aux’ command ensures the view of every single process on the server, while the ‘f’ in aux helps to determine which processes are parents and which are children.

top

Like the ps command, the top command helps to determine which processes are running on a server, but top has an advantage in its ability to display in real-time while filtering by several different factors. Simply, it dynamically shows the ‘top’ resource utilizers and is executed by running:

# top

Once inside of top, you will see a lot of process threads moving around. The ones at the top, by default, will show you processes that are using the most CPU at the moment. Holding shift to type ‘M’ will change the sort to processes that are using the most MEMory. Hold shift and press ‘P’ to change the sort back to CPU. When you want to quit, you can simply press ‘q’.

Since top writes information live, its output cannot be parsed by grep and thus seldom used in conjunction with a pipe. Top is most useful for discovering what is causing a server to run out of memory, or what is causing a load. For instance, on a server with high load, if the first command is using 100% CPU and its name is php-fpm, then we can assume that an inefficient PHP script is causing the load. In this case, php-fpm should be restarted (this is achieved on cPanel with /scripts/restartsrv_apache_php_fpm).

netstat

netstat is another tool to show what service is running on a server, but in particular, it shows processes that are listening for traffic on any particular network port. It can also display other interface statistics. Here is how you would display all publicly listening processes:

# netstat -tunlp

The command flags ‘-tunlp’ will show program names listening for UDP or TCP traffic, with numeric addresses. This can be further scoped down by grep to see, for instance, what program is listening on port 80:

# netstat -tunlp | grep :80
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 27242/httpd
tcp 0 0 :::80 :::* LISTEN 27242/httpd

There are four listeners listed, two each for all IPv4 (0.0.0.0) and all IPv6 (::) addresses on the local machine. There are two unique PID numbers (1863 and 1993), indicating that there are two, actively running memcached processes. The active ports for each PID, respectively, are 11211 and 11213. I can use this information to guarantee correct connects against my configurations and to provide the correct ports.

ip

The ip command shows network devices, their routes, and a means of manipulating their interfaces. LiquidWeb IP addresses are statically assigned, so you will not need to make any changes to the IPs on your server, but you can use the ip command to read the information on the interfaces:

# ip a

This command is short for ‘ip address show’, and shows you the active interfaces on the server:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:00:00:00 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.3/24 brd 192.168.0.255 scope global eth0
inet 192.168.0.1/24 brd 192.168.0.255 scope global eth0:cp1
inet 192.168.0.2/24 brd 192.168.0.255 scope global secondary eth0:cp2
inet6 fe80::5054:ff:face:b00c/64 scope link
valid_lft forever preferred_lft forever

In our case, there are two interfaces numbered 1 and 2: lo (the localhost loopback interface), and eth0. eth0 has three IP addresses assigned to it, on eth0, eth0:cp1, and eth0:cp2, which are 192.168.0.1, 2, and 3. We can also see that my MAC address for eth0 is 52:54:00:00:00:00, which can be helpful for troubleshooting connections to other devices like firewalls and switches. This interface also supports IPv6, and our IP is fe80::5054:ff:face:b00c.

lsof

lsof stands for ‘list open files,’ and it does just that; lists the files that are in use by the system. Listing open files is very helpful in determining what script is especially complex, or for finding a file that is in a state of writing.

Let’s use PHP as an example. We want to figure the location or path for the PHP default error logs, but Apache’s configuration is a large group of nested folders. The ps command only tells us if PHP is running, not which file is being written. lsof will show me this handily:

# lsof -c php | grep error
php-fpm 13366 root mem REG 252,3 16656 264846 /lib64/libgpg-error.so.0.5.0
php-fpm 13366 root 2w REG 252,3 185393 3139602 /opt/cpanel/ea-php70/root/usr/var/log/php-fpm/error.log
php-fpm 13366 root 5w REG 252,3 185393 3139602 /opt/cpanel/ea-php70/root/usr/var/log/php-fpm/error.log
php-fpm 13395 root mem REG 252,3 16656 264846 /lib64/libgpg-error.so.0.5.0
php-fpm 13395 root 2w REG 252,3 14842 2623528 /opt/cpanel/ea-php56/root/usr/var/log/php-fpm/error.log
php-fpm 13395 root 7w REG 252,3 14842 2623528 /opt/cpanel/ea-php56/root/usr/var/log/php-fpm/error.log

The ‘-c’ flag will only list processes that match a certain command name, in my case, ‘php’. I pipe this output into grep to search for the files that match the name ‘error’, and I see that there are two open error logs: /opt/cpanel/ea-php56/root/usr/var/log/php-fpm/error.log and /opt/cpanel/ea-php70/root/usr/var/log/php-fpm/error.log. Check each of these files (with tail or cat) to see recently logged errors.

If using the rsync command for the transfer of large folder(s), in this case, /backup, we can search for open rsync processes inside:

# lsof -c rsync | grep /backup
rsync 48479 root cwd DIR 252,3 4096 4578561 /backup
rsync 48479 root 3r REG 252,3 5899771606 4578764 /backup/2018-09-12/accounts/jeff.tar.gz
rsync 48480 root cwd DIR 252,3 4096 4578562 /backup/temp
rsync 48481 root cwd DIR 252,3 4096 4578562 /backup/temp
rsync 48481 root 1u REG 252,3 150994944 4578600 /backup/temp/2018-09-12/accounts/.jeff.tar.gz.yG6Rl2

The process has two regular files open in the /backup directory: /backup/2018-09-12/accounts/jeff.tar.gz and /backup/temp/2018-09-12/accounts/.jeff.tar.gz.yG6Rl2. Even with quiet output on rsync, we can see that it is currently working on copying the jeff.tar.gz file.

df

df is a swift command that displays how much space used on the mounted partitions of a system. It only reads data from the partition tables, so it is slightly less accurate if you are actively moving files around, but it beats enumerating and adding up every file.

# df -h

This ‘-h’ flag gets human readable output in nice round numbers (it can be omitted to print output in KB):

Filesystem Size Used Avail Use% Mounted on
/dev/vda3 72G 49G 20G 72% /
tmpfs 419M 0 419M 0% /dev/shm
/dev/vda1 190M 59M 122M 33% /boot
/usr/tmpDSK 3.1G 256M 2.7G 9% /tmp

Some of the information we see is the primary partition mounted on / is 72% used space with 20GB being free. Since we’re not planning on adding any more sites our server right, this is not a problem. But, some of the information we don’t see also is telling. There is no separate /backup partition mounted on my server, so my cPanel backups are filling up the primary partition. If I want to retain more backups, I should consider adding another physical or networked disk to store them.

df can also show inode (file and folder) count of mounted filesystems from the same partition table information:

# df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/vda3 4.6M 496K 4.1M 11% /
tmpfs 105K 2 105K 1% /dev/shm
/dev/vda1 50K 44 50K 1% /boot
/usr/tmpDSK 201K 654 201K 1% /tmp

Our main partition has 496,000 inodes used, and just over 4 million inodes free, which is plenty for general use. If we stored a lot of small files, like emails, my inode count could be much higher for the same disk usage in bytes. If you run out of inodes on your partition, it won’t be able to record the location of any more files or folders, even if you have free disk space, the system will function like your disk is full.

du

Like the df command, du will tell you disk usage, but it works by recursively counting folders and files that you specify. This command can take a long time on large folders, or those with a lot of inodes.

# du -hs /home/temp/
2.4M /home/temp/

My flags ‘-hs’ give human-readable output, and only displays the summary of the enumeration, rather than each nested folder. One of the other useful flags is –max-depth, which can define how deep you would like to list folder summaries. This flag is like increasing the depth of the -s flag (-s is basically –max-depth=0,  root — level 1, and one sub-directory — level 2):

# du -hs public_html/
5.5G public_html/

# du -h public_html/ --max-depth=0
5.5G public_html/

# du -h public_html/ --max-depth=1
8.0K public_html/_vti_txt
8.0K public_html/_vti_cnf
257M public_html/storage
64K public_html/cgi-bin
8.0K public_html/_vti_log
5.0G public_html/images
64K public_html/scripts
8.0K public_html/.well-known
8.0K public_html/_private
5.0M public_html/forum
56K public_html/_vti_pvt
24K public_html/_vti_bin
360K public_html/configs
5.5G public_html/

These commands help to find out if any specific folders inside of public_html are significantly larger than others. We add this to a pipe along with grep to get only folders that are 1GB or larger:

# du -h public_html/ --max-depth=1 | grep G
5.0G public_html/images
5.5G public_html/

Clearly, we have some pictures to delete or compress if we need more disk space.

free

The free command shows the instant reading of free memory on your system. Also displayed by top, but when only needing total memory information, the free is a lot faster.

# free -m
total used free shared buffers cached
Mem: 837 750 86 5 66 201
-/+ buffers/cache: 482 354
Swap: 1999 409 1590

Our free command with the megabytes flag displays output in MB. Without it, it would default to -k (kilobytes), but you can also pass -g for gigabytes (though the output is rounded and thus less accurate).

In our output, the total RAM on the system is 837MB, or about 1GB. Of this, 750MB is ‘used,’ but 66MB is in buffers and 201M is cached data, so subtracting those, the total ‘free’ RAM on the server is around 354MB. Because the calculations are made in KB and rounded for output, the numbers won’t always accurately add up (750 plus 86 is not 837).

The final line shows swap usage, which you want to avoid using. My output says that there is 409MB used in the on-disk swap space, but since there is free RAM at the moment, my swap usage was in the past, and the system stopped using swap space.

If there was an amount in the ‘used’ column for swap, and there was 0 free RAM after calculating the buffers and cache, then the system will be very sluggish. We call this ‘being in swap.’ The reading/writing to swap is very slow compared to RAM, and you should avoid going into swap space by tuning your programs to use memory appropriately. If you run out of RAM and swap space, then your server will be out of memory (OOM), and will immediately freeze.

Advanced Commands

Useful Pipelines

Now that we have a few advanced commands under our belt let’s learn more about how we can use pipe to our advantage in making useful command strings or scripts, aka ‘one-liners’ or ‘pipelines.’ These use several formatting commands, such as sed, awk, sort, uniq, or column, which fall outside of the scope of this article for description (you can learn more about them using the man command).

Disk Usage Formatting

This command will use du and awk, an output manipulation tool, to nicely format and sort the output of a du command in the current working directory by size. First, change directory (cd) to your intended folder for analysis, and run:

# du -sk ./* | sort -nr | awk 'BEGIN{ pref[1]="K"; pref[2]="M"; pref[3]="G";} { total = total + $1; x = $1; y = 1; while( x > 1024 ) { x = (x + 1023)/1024; y++; } printf("%g%s\t%s\n",int(x*10)/10,pref[y],$2); } END { y = 1; while( total > 1024 ) { total = (total + 1023)/1024; y++; } printf("Total: %g%s\n",int(total*10)/10,pref[y]); }'

The above command will add dynamic suffixes to the on-disk sizes, so you can see output in GB, MB, and KB, instead of just one of those powers. The top listed folder will be your largest in that directory.

Check Connection Count

This string of commands checks active connections to the server using netstat, pares the output down to HTTP and HTTPS connections using grep, formats and sorts the output using a series of other commands. This example shows how many times each IP address listed has connected to the server.

# netstat -tn 2>/dev/null | grep -E '(:80|:443)' | awk '{print $5}' | cut -f1 -d: | sort | uniq -c | sort -rn
2 46.229.168.149
1 46.229.168.147
1 46.229.168.145
1 46.229.168.144
1 46.229.168.142
1 46.229.168.141
1 46.229.168.138
...

In our case, a subnet that was hitting us with requests to scrape data, caused a lot of load on the server. Add this IP range the firewall, in the deny list, to stop the attack for now.

You can also get a quick summary of just the total number of connections with this command:

# netstat -tan | grep -E ‘(:80|:443)’ | wc -l
8

Format error_log Output

Here is some advanced usage for grep output. The sed command is like awk, where it edits the output stream as it is printed. In this case, we want to look at logged modsec errors, but I want to add some whitespace between the errors so it’s easier to read:

# grep -i modsec /usr/local/apache/logs/error_log | tail -n100 | sed -e “s/$/\\n/”

This command will give me the output of the last 100 logged modsec, ModSec, or ModSecurity line (since the ‘-i’ flag for grep will ignore case sensitivity) and replace the end of each line with a newline.

Memory Usage By Account

Add up all of the percentages of memory usage by user for a running program as defined by ps and give you a sorted output.

# tmpvar=””; for each in `ps aux | grep -v COMMAND | awk '{print $1}' | sort -u`; do tmpvar="$tmpvar\n`ps aux | egrep ^$each | awk 'BEGIN{total=0};{total += $4};END{print total, $1}'`"; done; echo -e $tmpvar | grep -v ^$ | sort -rn | head; unset tmpvar

Usage Count In /var/tmp

When searching for a file count per user, if you encounter a number as the file owner, you can conclude that the user has been removed, and should the file should be deleted.

# find /var/tmp/ ! -user root ! -user mysql ! -user nobody ! -group root ! -group mysql | xargs ls -lh | awk '{print $3, $5, $9}' | sort | awk '{print $1}' | uniq -c | sort -rh

Top Processes By Memory Usage

This command outputs the processes using the highest memory, sorting the 4th column of ps and displaying the top 10 commands with head.

# ps aux | sort -rnk 4 | head

Whether you are brushing up on your Linux Admin interview or just want to get more familiar these commands are sure to be useful to your repertoire.

Free Website Migration Service

How To Request Free Website Migrations from Liquid Web

The Migration team at Liquid Web is dedicated to providing you with an efficient and as uneventful a migration as possible. Whether you are migrating from a current Liquid Web server (internal migration) or from another host (external migration) into Liquid Web, it is important that we work together to ensure an effective transfer of information.

If you would like to skip the overview and go straight to the request form, you can do that clicking on this link for Requesting A Migration. At the bottom of this article we guide you through making a migration request. To request a Windows server migration, please open a Support Ticket with the Windows Team indicating that you are requesting a migration.

Otherwise, check out some helpful terms to know before your migration begins. Still have questions about what to expect? We have a handy guide called What to Expect During a Site Migration.

Before Your Migration Begins

Before you start your migration, there are a few terms that we use that you will need to be familiar with:

  • Initial Sync – This is the first of three stages of a migration. In this stage, access levels are determined and tested, version matching occurs, and the initial seed of data for your websites being migrated is brought to the new server.
  • Hosts File – The hosts file is a computer file used by an operating system to map hostnames to IP addresses. This file is a plain text file and stored on your computer or workstation, it is the first stop when your browser looks up a domain name via DNS. You can edit the hosts file to re-route requests for a particular domain name to a different IP Address. This is the preferred method of testing your site on your new server. This allows you to view the site as if it were live on the new server at Liquid Web and verify that all pages are working as intended prior to going live with a DNS update.
  • Final Sync – The final sync is the last transfer of data in the migration process. This is completed after you confirm that all testing has come back without any major issues to fix before the site goes live. The final sync typically updates files, email, and databases that have been changed since the initial migration of data. This is done with the source server no longer serving requests and is most effective when a DNS update is performed in tandem.
  • DNS Update – A DNS update is part of the final sync of your migration which makes the target server (the new server at Liquid Web) live. The DNS update can be performed in conjunction with a final sync, or on its own if a final sync is not possible.
Note:
Your site is not live on the new server until the DNS update has occurred. It is normal to see some errors during testing and most will resolve once the DNS has been updated.
  • Authoritative Nameservers – a specific nameserver which holds the authoritative DNS records for a specific domain. Authoritative nameservers are defined for a specific domain at that domain’s registrar. A change to DNS at the authoritative nameserver will make its way around the internet through propagation. This is why it can sometimes take a little while for your DNS changes to take effect.
  • Nameserver – A nameserver is computer hardware or software that implements a network service for providing responses to DNS queries. Nameservers serve several types of information for a certain domain name, including A Records, MX Records, and CNAME records.
  • Nameserver Glue – Nameserver glue is a record which associates a named nameserver with an IP address on the internet, much like a A Record associates a domain name with an IP address. This record is stored at the domain registrar. During migrations, if nameserver authority is moving from one machine to another, the glue records at the registrar will need to be changed after the final sync and DNS update.

Requesting a Migration

Migrations begin by filling out our form through your Liquid Web account.
1. Once you log into your account, click on the Migration Center link at the top of the page.

2. You will be directed to the Migration Center home page. Click on Create a Migration Request to begin filling out the form.

Note:
If you want more information about migrations to Liquid Web, we have linked our Help Center Migration articles in the Migrations tab. Here you can read our articles What to Expect During a Site Migration and Testing Best Practices for Migrations.

3. Once the Request Migration form opens, you will be given the option to name your migration and add your source account. Click on the Add a Source Account button to add information about the hosting account you are migrating from.

4. Click on the Add a Source button to add server access information for the source account. This is where you would add SSH or panel login details for the server or cPanel account you are migrating.

5. You can add more than one source by clicking the link Add a Source Account at the bottom of the section.

6. Next, you will select your destination. You can choose the server from the drop-down menu.  If you do not currently have a server at Liquid Web, click on the Create a New Server link to be directed to a page for you to purchase and create a server.

7. Provide us with information on what software you’d like to have updated with the migration, or if you don’t need or want updates, you can leave this as it is.

8. We will need information from you on the domains on the incoming server and DNS settings. You can choose what domains you want tested and how you want the DNS handled.

9. The final step before submitting your request is to review the information you’ve provided. When you are ready, click the Submit button and a ticket will be sent to our Heroic Migration team.

We will contact you to schedule the migration and stay in contact with you through the entire process. Once the migration is complete, the last step is to test your site.

Please see our article Editing Your DNS Hosts File for information on how to securely test your site. If at any time you have questions or concerns, please feel free to contact our Heroic Support team by chat, phone or support ticket.

 

Restarting Services From the Command Line

One of the more common management tasks performed on web servers is restarting services (such as your web server daemon, mail daemon, FTP server, or DNS service).

You might need to restart the service(s) if:

  • The service may be crashing or stalled
  • The load the process is causing on the server might be too high
  • A restart might be needed to produce a configuration change to go into effect.

This article is geared towards operating systems using the systemd service.

  • CentOS 6
  • CentOS 7
  • most newer Ubuntu or Debian servers.

Note
For this article, general program names are bolded and italicised, binary commands or operands are highlighted in blue.

There are many ways to do anything on a computer, particularly in Linux, and performing service restarts is no exception. Here are three ways to achieve the same action, restarting Apache for typical Liquidweb Fully Managed servers:

  • With the first method, we use the service command, also known as systemd, to issue commands to the desired service. For Apache, this looks like service httpd restart will issue a restart command to Apache. You can check to see if you are able to restart services this way by running which service. CentOS 7 uses this method by default, CentOS 6 has it available, and it is also available on other major operating systems.
  • An older method is to call an init script on the server, known as the upstart method. An upstart command looks like /etc/init.d/httpd restart, which will issue a restart command to Apache also. You can check to see if you can use the init scripts by listing the scripts inside the init.d directory with ls /etc/init.d/. CentOS 5 and 6 used this type of service control by default, as do other aging OSes. Since this is an older method, we won’t discuss it in this article.
  • Finally, on cPanel machines, there are service restart scripts provided by cPanel that begin with /scripts/restartsrv_. For instance, the module to restart Apache is /scripts/restartsrv_httpd. This script performs a clean restart of Apache. If your web server has cPanel installed, these scripts are available for use and will always use the proper restart method for your operating system. This method is particularly helpful when the service name can vary based on the engine in use, such as proftpd or pure-ftpd for the FTP client; both use /scripts/restartsrv_ftpd and cPanel selects the appropriate service automatically. You can list the available scripts by running ls /scripts/restartsrv_*.

 

Choosing a Script

As demonstrated above, all commands will result in the same action. Therefore, the script you use comes down to a mix of personal preference and OS support. This article will focus on the first method, systemd, and the third method, for cPanel servers.

The typical formulation for using systemd with the service binary looks like this:

Service Command

The command followed by the name of the service you want to affect, and then an option called the ‘usage flag’, which tells service what to do.

 

Usage Flags

Every service typically at the least can stop, start, restart, as well as status usage flags. However, some services have other usage flags that can be used to give more information about how the service performs.

The configtest usage flag for Apache is an excellent example of this. You can test the written configuration for errors without having to load it into Apache first. You can then verify whether the changes made to the Apache configuration might cause Apache not to start, and correct them, without affecting your sites.

Note
cPanel has a background daemon called chkservd, which a daemon that checks to see if specific services are running. If you stop a service that is monitored, like Apache, and chkservd finds out, it will assume the service died and will attempt to restart it. Keep this in mind if you stop any services from the command line, and if needed, turn off chkservd from inside of WHM before you start. Make sure to turn it back on when done.

Below are some examples of typical services that you might need to restart on your server, as well as all of their usage flags for systemd.

Apache

If you are making configuration changes to Apache from the command line, such as creating new include files or optimizations, restarting the service is necessary for the new configuration to be operational. Additionally, if Apache (or httpd) is causing undue load on the server, restarting it usually kills its child processes and starts over with new ones, alleviating memory usage.

# service httpd
usage: httpd {start|stop|restart|fullstatus|status|graceful|configtest|help}

The usage flags may not always make sense to you when printed on the command line. Here is a basic outline of what they mean for Apache. These same general descriptions can apply to other services that use the same usage flags (though some may be Apache-specific).

 

startstart httpd
startsslstart httpd with SSL enabled
stop

stop httpd

restart

restart httpd if running by sending a SIGHUP or start if not running

fullstatusdump a full status screen; requires lynx and mod_status enabled
statusdump a short status screen; requires lynx and mod_status enabled
gracefuldo a graceful restart by sending a SIGUSR1 or start if not running
configtestdo a configuration syntax test
help list the commands available

 

You would use any of the optional usage flags as an extra argument after the service name as described above, like so:

# service httpd status
httpd (pid  74669) is running…

The cPanel restart script is essentially the same as service httpd restart:
# /scripts/restartsrv_httpd
Waiting for “httpd” to restart gracefully …

 

Mail

On cPanel servers the default mail service is exim.  Configuration or include changes to exim will require a restart to take effect.

# service exim
Usage: exim {start|stop|restart|status}

 

On Plesk servers, the default mail service is qmail instead.

# service qmail
Usage: qmail {start|stop|status|reload|condrestart|restart}

 

SSH

Any change to the SSH configuration file (such as changing the SSH port) requires a restart to the server daemon to take effect. Restarting sshd does not normally break SSH connections, but if you do change the SSH port, make sure you have opened the right ports in your firewall and are ready to make a new connection if needed!

# service sshd
Usage: sshd {start|stop|restart|reload|condrestart|status}

 

FTP

On cPanel servers the default FTP program is pure-ftp.  Any configuration changes to pure-ftp will require a restart to take effect. The cPanel restart script is convenient in this instance, as we covered earlier, even if the FTP program changes through cPanel, the restartsrv_ftpd script will restart the right service.

# service pure-ftpd
Usage: pure-config.pl {start|stop|restart|condrestart|status}

# /scripts/restartsrv_ftpd

 

Mysql

Mysql is mostly often restarted for configuration changes. Some newer machines will use mariadb instead of mysql or mysqld, though MariaDB normally responds to service requests made through systemd for MySQL as well. Again, the cPanel restart script is handy for cPanel servers, because usage either mariadb or mysql, /scripts/restartsrv_mysql will restart the database engine.

# service mysql
Usage: mysql (start|stop|restart|reload)

# /scripts/restartsrv_mysql

 

Firewall

On managed Liquidweb servers, the firewall of choice is CSF. The CSF firewall interacts with the iptables service.  Any configuration changes made to CSF the service must need a restart for the changes to take effect. However, you should not use the systemd or init scripts for this. CSF has a special restart procedure:

# csf -ra

 

If you try to use the service command, CSF will warn you not to:

# service csf restart
This script should ONLY be used by the init process. To restart csf use the CLI command 'csf -r'

 

Cron

Within Linux, the Cron service controls the scheduled tasks that run on the server. Stopping the Cron service will result in schedule task to be skipped. Unlike other services, the cron service does not need a restart if there is an additional cron added; it is generally only restarted when there is a change to the system clock or timezone, or when crons are not running due to various reasons.

# service crond
Usage: crond {start|stop|status|reload|restart|condrestart}

 

DNS

The named service is the default nameserver daemon for cPanel servers. If you run a different daemon, like MyDNS or NSD, use these names instead of ‘bind’ below.

# service bind
Usage: named {start|stop|status|restart|try-restart|reload|force-reload}

# /scripts/restartsrv_bind