How to Use Ansible

Ansible symbolAnsible is an easy to use automation software that can update a server, configure tasks, manage daily server functions and deploys jobs as needed on a schedule of your choosing. It is usually administered from a single location or control server and uses SSH to connect to the remote servers. Because it employs SSH to connect, it is very secure and, there is no software to install on the servers being managed. It can be run from your desktop, laptop or other platforms to assist with automating the tedious tasks which every server owner faces.

Once it is configured, Ansible performs tasks based on an ordered list of assignments in what is called a Playbook. The Playbook outlines what tasks need to be run on the remote server and in what order. Once this is configured, Ansible acts like a bash for-loop command that allows a section of code to be repeated over and over again. The difference between using a bash command and Ansible is that Ansible is idempotent. The term Idempotent sounds a little scary, but it merely means that you can make the same type of request over and over again and unless something has changed, you will get the same result.

Pre-flight: Server Requirements

Source Server Requirements

Ansible requires the installation of Python 2.7 or Python 3.5 on the source server. The source server is where you will be running the tasks in the playbook for the remote servers. The remote servers receive commands defined in the playbook.  A playbook is a file which defines the scripts that will be run on the remote servers.

Note:
Unfortunately, Windows is unsupported as a source server. Certain Ansible plugins and/or modules will have other needs or requirements. Usually, these plugins or modules are installed on the same server of the Ansible installation.

Let’s start by installing Python on the source server.
root@test:~# apt-get install python

 

Target Server Requirements

The only requirement from the target server is an open SSH port. Access can also be granted for scp (secure copy) and/or SFTP connections if configured in the /etc/ansible/ansible.cfg file.

Install Ansible On Ubuntu 16.04

To install Ansible on a source Ubuntu server, let’s follow these steps:

Note:
The PPA for Ansible is here: https://launchpad.net/~ansible/+archive/ubuntu/ansible if you would like to review the versions available for your variant of Ubuntu.

root@test:~# apt-get update
root@test:~# apt-get install software-properties-common
root@test:~# apt-add-repository ppa:ansible/ansible
root@test:~# apt-get update
root@test:~# apt-get install ansible
(install text)After this operation, 42.0 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Answer “Y” to the prompt. The install will complete and take you back to the command prompt. Now, let’s check the version of Ansible installed.

Check Ansible Version

root@test:~# ansible --version
ansible 2.7.0
 config file = /etc/ansible/ansible.cfg
 configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
 ansible python module location = /usr/lib/python2.7/dist-packages/ansible
 executable location = /usr/bin/ansible
 python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]

As an alternative, you can also install Ansible on your CentOS 7 server.
Ansible also can be installed on RedHat, Debian, MacOS, and any of the BSD flavors!

Install Ansible on CentOS 7

In order to install Ansible on a source CentOS 7 server, follow these steps.
First, we need to make sure that the CentOS 7 EPEL repository is added:

[root@test ~]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)

[root@test ~]# yum install epel-release
Loaded plugins: fastestmirror, priorities, universal-hooks
Loading mirror speeds from cached hostfile

...
Resolving Dependencies
--> Running transaction check
---> Package epel-release.noarch 0:7-11 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
==========================================================================================================
Package Arch Version Repository Size
==========================================================================================================
Installing:
epel-release noarch 7-11 system-extras 15 k
Transaction Summary
==========================================================================================================
Install 1 Package
Total download size: 15 k
Installed size: 24 k
Is this ok [y/d/N]: y
Downloading packages:
epel-release-7-11.noarch.rpm | 15 kB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : epel-release-7-11.noarch 1/1
Verifying : epel-release-7-11.noarch 1/1
Installed:
epel-release.noarch 0:7-11
Complete!

Select “y”. The EPEL repo will then be added. Once the repository is enabled, we can install Ansible with yum:

root@test:~# yum install ansible
Loaded plugins: fastestmirror, priorities, universal-hooks
Loading mirror speeds from cached hostfile
epel/x86_64/metalink                                                               | 18 kB 00:00:00
* EA4: 208.100.0.204
* cpanel-addons-production-feed: 208.100.0.204
* epel: mirrors.liquidweb.com
epel                                                                               | 3.2 kB 00:00:00
(1/3): epel/x86_64/group_gz                                                        | 88 kB 00:00:00
(2/3): epel/x86_64/updateinfo
(3/3): epel/x86_64/primary                                                         | 3.6 MB 00:00:00
epel                                                                                          12756/12756
Resolving Dependencies
… (dependencies check)
Dependencies Resolved
==========================================================================================================
Package                        Arch Version            Repository Size
==========================================================================================================
Installing:
ansible                        noarch 2.4.2.0-2.el7            system-extras 7.6 M
Installing for dependencies:
21 k
Transaction Summary
==========================================================================================================
Install  1 Package (+17 Dependent packages)
Total download size: 12 M
Installed size: 58 M
Is this ok [y/d/N]:


Select “y” to start the Ansible install:

Is this ok [y/d/N]: y
… Downloading 18 packages:
----------------------------------------------------------------------------------------------------------
Total                                                                      30 MB/s | 12 MB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
… (installing 18 python related software)
...
Installed:
ansible.noarch 0:2.4.2.0-2.el7
Dependency Installed:
... (dependencies verified)
Complete!

Check Ansible Version on CentOS 7

Now, let’s verify the version installed:

root@test ~]# ansible --version
ansible 2.4.2.0
config file = /etc/ansible/ansible.cfg
configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.5 (default, Jul 13 2018, 13:06:57) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

 

Setting Up Ansible Connections

Initially, we will be adding server names or IP’s to the /etc/ansible/hosts file to identify which “ungrouped” servers and “groups” of servers we are going to be connecting to. We mention ungrouped and grouped in this specific order because this is the way the Ansible hosts file is usually arranged.

We can use any name we like for the hosts file itself but typically it is just called hosts. Ansible also states that the hosts file can also be identified as an inventory file and, you can have multiple inventory files.

Let’s start by opening the hosts file with vim and inserting some entries into the file.

root@test:~vim /etc/ansible/hosts
Here is what the default hosts file will look like:

# This is the default ansible 'hosts' file.
#
# It should live in /etc/ansible/hosts
#
# - Comments begin with the '#' character
# - Blank lines are ignored
# - Groups of hosts are delimited by [header] elements
# - You can enter hostnames or ip addresses
# - A hostname/ip can be a member of multiple groups
# Example 1: Ungrouped hosts, specify before any group headers.
#green.example.com
#blue.example.com
#192.168.100.1
#192.168.100.10
# Example 2: A collection of hosts belonging to the 'webservers' group
#[webservers] #alpha.example.org
#beta.example.org
#192.168.1.100
#192.168.1.110
# If you have multiple hosts following a pattern you can specify
# them like this:
#www[001:006].example.com
# Example 3: A collection of database servers in the 'dbservers' group
#[dbservers] #
#db01.intranet.mydomain.net
#db02.intranet.mydomain.net
#10.25.1.56
#10.25.1.57
# Here's another example of host ranges, this time there are no
# leading 0s:
#db-[99:101]-node.example.com


As you can see, there are individual servers “#
green.example.com”, and groups like #[webservers] which have multiple servers under the group and, another section with multiple servers listed like #db-[99:101]-node.example.com which identifies all of the individual servers from 99-101; eg.

  • db-99-node.example.com
  • db-100-node.example.com
  • db-101-node.example.com

So, let’s quickly add another server to the #[webservers] group:

#[webserver1]
#alpha.example.org
#beta.example.org
#192.168.1.100
#192.168.1.110gamma.example.com

Now, simply save the file using :wq

Note:
Make sure you uncomment any ‘#’ entries you place in the file otherwise, the entry is excluded!

 

SSH Keys

You can set up public SSH keys from the control server to log in to those remote servers noted in the hosts file. In this case, you simply want to make sure your local SSH keys are located in the /root/.ssh/authorized_keys file on the remote systems. (Depending on your setup, you may wish to use Ansible’s –private-key option to specify a .pem file instead)

 

Verify Ansible Connections

The ansible inventory file (/etc/ansible/hosts) contains the server names you will have control over and can run tasks against. To verify Ansible’s connectivity, run:

ansible remote -m ping

 

Ansible Playbooks

Ansible playbooks (also called inventory files) define the tasks ran on the remote servers. You can have one playbook or multiple playbooks to accomplish different tasks on different servers.  To easily apply a task to all of the servers in a pool, use the ‘group’ name to apply the task for that group (using the example above, you would use the [webserver1] in the command.

 

Create a Playbook

Step 1: In order to create a playbook, let’s create a new file in the /etc/ansible/playbooks/ folder:

mkdir -p /etc/ansible/playbooks/ && touch /etc/ansible/playbooks/playbook.yml && vim /etc/ansible/playbooks/playbook.yml

 

Step 2: Let’s add a server and file entry into the playbook filer:

(Click the insert key to open VIM’s edit access on the file)

- hosts: gamma.example.com
 tasks:
     - name: Create file
       file:
           path: /tmp/test.txt
           state: touch


Once we have added the entry, let’s save the file using

:wq

Step 3: Now, to set up 0644 permissions on that file, we can reopen it and add another line defining the permission set:

- hosts: gamma.example.com
 tasks:
     - name: Create file
       file:
           path: /tmp/test.txt
           state: touch            mode: "u=rw,g=r,o=r"

Again, let’s save the file using

:wq

Step 4: Next, let’s create a folder and then place a text file in it using Ansible. We will add another section defining the element needed:

- hosts: gamma.example.com
 Tasks:       - name: Create folder
       path: /home/tmp/
           state: directory
           mode: 0755
     - name: Create file
       file:
           path: /home/tmp/test.txt
           state: touch            mode: "u=rw,g=r,o=r"

Once we have added this entry, save the file using

:wq

 

Running a Playbook

To start a playbook, simply run:

ansible-playbook /etc/ansible/playbooks/playbook.yml

or if you have multiple playbooks in a folder, can run a specific playbook using the -i <path> option from the command line:

ansible-playbook -i /etc/ansible/playbooks/playbook1.yml
In addition to .yaml files, Ansible can use .json files as well to control the playbook. It is also very easy to convert 
bash or shell scripts into playbooks as well!

Schedule a Playbook Using Cron

As an additional option, you can schedule a playbook to run at a specific time using your servers cron command! To accomplish this, log in to your server as root and run the following command:

crontab -e
This command opens a temporary cron file in your system’s 
default text editor and then simply add a line like so:

0 4 * * * /usr/bin/ansible-playbook /etc/ansible/playbooks/playbook.yml
this will run the
/etc/ansible/playbooks/playbook.yml file at 0400 a.m. using the ansible-playbook command.

 

Troubleshooting A Playbook

Sometimes, a set of commands in the playbook file may fail. If it does, you have several options to address this. Generally, playbooks will simply stop completing the commands in the playbook. If this occurs, you can define a follow-up task in the playbook to overlook the error by adding another section like so:

- name: ignore this error
 command: /bin/false
 ignore_errors: yes


Unreachable Hosts

this command will only work when the task is run but returns a “failed” value.

If Ansible fails to connect to a server, it will set the host as being ‘UNREACHABLE’. This effectively removes the server temporarily from the list of active hosts for that task. To correct this, we can use an entry to reactivate them and have all current hosts previously indicated as being unreachable cleared, so subsequent tasks can use the playbook again.

meta: clear_host_errors


Handlers and Failure

A handler is simply a specially named task that runs when told to by another task. Handlers are executed at the end of the playbook by default as opposed to other tasks, which are executed immediately when defined within the playbook. This behavior can be modified by using the

--force-handlerscommand-line option, or by including

force_handlers: Truein a playbook, or addingforce_handlers = Truein the ansible.cfg file.


If you want to force a handler to run in the middle of two separate tasks instead of at the end of the playbook, you will need to add this entry between the two tasks:

- meta: flush_handlers

When handlers are “forced” like this, they will run when notified even if a task fails on that host.

Note:
Certain errors can still prevent the handler from running, such as a host becoming unreachable.
Handlers will only be visible in the output if they have actually been executed. Also, handlers are only fired when there are changes made by a task. For example, a task may update a specific configuration file and then notify a handler to restart a service. If a task in the same playbook fails later on, the service will not be restarted despite the previous configuration change.

Overall, Ansible is an indispensable tool for managing and administrating a single server or an entire group of geographically diverse servers.

 

Kubernetes Tutorial

What is Kubernetes?

The name Kubernetes has its origins from the original Greek term for helmsman or pilot. Kubernetes, or ‘k8s’ (pronounced “Kate’s”) as it’s sometimes referred to, is an open-source software tool that was originally created by Google and is now being maintained by the Cloud Native Computing Foundation. Kubernetes is used for arranging and coordinating containers that an application needs to run into easy to handle groups.

In order to manage your Kubernetes cluster effectively, we recommend using kubectl as the command-line tool of choice. Basically, kubectl communicates with the master node (or server) which in turn submits those commands to the worker nodes to manage the cluster.

The Kubernetes cluster consists of two basic types of resources;

  • Master server – a master server organizes the cluster
  • Node server – Nodes are the workers that contain and run the applications

Each node contains a Kubelet, which is the agent for managing the node and communicating with the master. You can use kubectl to deploy, explore, review and remove Kubernetes objects (like nodes, images or containers).

Let’s next look at setting up kubectl.

The Master communicates with containers through the worker node.

Note:
This tutorial assumes you have a Kubernetes cluster already setup and running.

In order to setup kubectl, we will need the following:

Prerequisites

  1. A working internet connection
  2. The cURL or wget utilities installed
  3. Basic knowledge of the Linux command line

Installing kubectl

On an Ubuntu 16.04 LTS server, here are the commands to use if logged in as root to install kubectl:

apt-get update && sudo apt-get install -y apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
touch /etc/apt/sources.list.d/kubernetes.list
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | tee -a /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-get install -y kubectl

Kubectl Commands for Basic Kubernetes Functions

Now that we have kubectl up and running, let’s review a few of the basic commands available. Here are the five most basic commands we will be reviewing along with their fundamental definition:

  • kubectl create – the create command constructs a resource from a configuration file or stdin (or standard input). A resource is defined as “something that can be “requested by”, “allocated to”, “or consumed by” a pod or container.”
  • kubectl get – the get command displays a table of the most relevant information about one or multiple relevant resources.
  • kubectl run –  the run command will kickoff one or more instances of a container in the cluster.
  • kubectl expose – the expose command will start to load balance inbound traffic across your running instances. This command can also create a High Availability proxy for the client to access the running containers from outside the cluster.
  • kubectl delete – the delete command removes defined resources by
    • filenames
    • stdin
    • resources and names
    • resources and label selector

Kubectl App Management

  • kubectl edit – Alters the characteristics of a resource on a server using the default editor.
  • kubectl apply – Applies a change to a resource from a file or stdin.
  • kubectl label – Adds or updates a specific attribute to specifically identify an object

Working with Apps Using Kubectl

  • kubectl exec – Runs a command on a container in a pod
  • kubectl logs – Prints a container log
  • kubectl describe – Displays the status or state of the resources.

Kubectl Cluster Management

  • kubectl cluster-info – Displays information about the master and services in the cluster.
  • kubectl drain – removes pods in preparation for maintenance
  • kubectl certificate – Approves a CSR or certificate signing request

Kubectl Settings and Usage

  • kubectl api-resources (eg. Pods and Services) – Lists all of the supported resources and their shortnames, API grouping, if namespaced, and Kind
  • kubectl config –  Changes or alters kubeconfig files
  • kubectl version – Displays the Kubernetes version

 

These are just some of the many basic command examples that are available for use in setting up and maintaining your Kubernetes environment.

 

How To Install Docker on Ubuntu 16.04

Adding Docker to an Ubuntu server.

Docker is an open-source software tool designed to automate and ease the process of creating, packaging, and deploying applications using an environment called a container. The use of Linux containers to deploy applications is called containerization. A Container allows us to package an application with all of the parts needed to run an application (code, system tools, logs, libraries, configuration settings and other dependencies) and sends it out as a single standalone package deployable via Ubuntu (in this case 16.04 LTS). Docker can be installed on other platforms as well. Currently, the Docker software is maintained by the Docker community and Docker Inc. Check out the official documentation to find more specifics on Docker. Docker Terms and Concepts

Docker is made up of several components:

  • Docker for Linux: Software which runs Docker containers on the Ubuntu Linux OS.
  • Docker Engine: Used for building Docker images and creating Docker containers.
  • Docker Registry: Used to store various Docker images.
  • Docker Compose: Used to define applications using multiple Docker containers.

 

Some of the other essential terms and concepts you will come into contact with are:

  • Containerization: Containerization is a lightweight alternative to full machine virtualization (like VMWare) that involves encapsulating an application within a container with its own operating environment.

Docker also uses images and containers. The two ideas are closely related, but very distinct.

  • Docker Image: A Docker Image is the basic unit for deploying a Docker container. A Docker image is essentially a static snapshot of a container, incorporating all of the objects needed to run a container.  
  • Docker Container: A Docker Container encapsulates a Docker image and when live and running, is considered a container. Each container runs isolated in the host machine.
  • Docker Registry: The Docker Registry is a stateless, highly scalable server-side application that stores and distributes Docker images. This registry holds Docker images, along with their versions and, it can provide both public and private storage location. There is a public Docker registry called Docker Hub which provides a free-to-use, hosted Registry, plus additional features like organization accounts, automated builds, and more. Users interact with a registry by using Docker push or pull commands. Example:

docker pull registry-1.docker.io/distribution/registry:2.1.

  • Docker Engine: The Docker Engine is a layer which exists between containers and the Linux kernel and runs the containers. It is also known as the Docker daemon. Any Docker container can run on any server that has the Docker-daemon enabled, regardless of the underlying operating system.
  • Docker Compose: Docker Compose is a tool that defines, manages and controls multi-container Docker applications. With Compose, a single configuration file is used to set up all of your application’s services. Then, using a single command, you can create and start all the services from that file.
  • Dockerfiles: Dockerfiles are merely text documents (.yaml files) that contains all of the configuration information and commands needed to assemble a container image. With a Dockerfile, the Docker daemon can automatically build the container image.

    Example: The following basic Dockerfile sets up an SSHd service in a container that you can use to connect to and inspect other containers volumes, or to get quick access to a test container.

FROM ubuntu:16.04
RUN apt-get update && apt-get install -y openssh-server
RUN mkdir /var/run/sshd
RUN echo 'root:screencast' | chpasswd
RUN sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin
yes/' /etc/ssh/sshd_config
# SSH login fix. Otherwise user is kicked off after login
RUN sed 's@session\s*required\s*pam_loginuid.so@session optional
pam_loginuid.so@g' -i /etc/pam.d/sshd
ENV NOTVISIBLE "in users profile"
RUN echo "export VISIBLE=now" >> /etc/profile
EXPOSE 22
CMD ["/usr/sbin/sshd", "-D"]

Docker Versions

There are three versions of Docker available, each with its own unique use:

  • Docker CE is the simple, classic Docker Engine.
  • Docker EE is Docker CE with certification on some systems and support by Docker Inc.
  • Docker CS (Commercially Supported) is kind of the old bundle version of Docker EE for versions <= 1.13.

We will be installing Docker CE.

 

Docker logo

Step 1 — Checking Prerequisites

To begin, start with the following server environment: 

  1. 64-bit Ubuntu 16.04 server
  2. Logged in as the root user
Important:
Docker on Ubuntu requires a 64-bit architecture for installation and, the Linux Kernel version must be 3.10 or above.

Before installing Docker, we need to set up the repository which contains the latest version of the software (Docker is unavailable in the standard Ubuntu 16.04 repository). Adding the repository allows us to easily update the software later as well.

Step 2 — Installing Docker

The next step is to remove any default Docker packages from the existing system before installing Docker on a Linux VPS. Execute the following commands to start this process:

root@test:~# apt-get remove docker docker-engine docker.io lxc-docker
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package 'docker-engine' is not installed, so not removed
Package 'docker' is not installed, so not removed
Package 'docker.io' is not installed, so not removed
E: Unable to locate package lxc-docker

Note:
In certain instances, a specific variant of the linux kernel is slimmed down by removing less common modules (or drivers). If this is the case, the “linux-image-extra” package contains all of the “extra” kernel modules which were left out. Use this command to re-add them: root@test:~# sudo apt-get install linux-image-extra-$(uname -r) linux-image-extra-virtual

Step 3 — Add required packages

Now, we need to install some required packages on your system. Run the commands below to accomplish this:

root@test:~# apt-get install curl apt-transport-https ca-certificates software-properties-common

Note:
If you get the error: “E: Unable to locate package curl”, Use the commands “curl -V” to see if curl is already installed; if so, move on to step 4.

Step 4 — Verify, Add and Update Repositories

Add the Docker GPG key to your system:

root@test:~# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
OK

Next, update the APT sources to add the source:

root@test:~# add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable" | tee /etc/apt/sources.list.d/docker.list

Run the update again so the Docker packages are recognized:

root@test:~# apt-get update
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [107 kB]
Hit:2 http://us.archive.ubuntu.com/ubuntu xenial InRelease                              
Get:3 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]             
Get:4 http://us.archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]                 
Fetched 323 kB in 0s (827 kB/s)                             
Reading package lists... Done
E: The method driver /usr/lib/apt/methods/https could not be found.
N: Is the package apt-transport-https installed?
E: Failed to fetch https://download.docker.com/linux/ubuntu/dists/xenial/InRelease  
E: Some index files failed to download. They have been ignored, or old ones used instead.

Note:
If you get the error seen above: “N: Is the package apt-transport-https installed?”, Use the following command to correct this. root@test:~# sudo apt-get install apt-transport-https

Let’s rerun the update:

root@test:~# apt-get update
Hit:1 http://us.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [107 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]        
Get:4 http://us.archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]                 
Hit:5 https://download.docker.com/linux/ubuntu xenial InRelease
Fetched 323 kB in 0s (656 kB/s)
Reading package lists... Done

Success! Now, verify we are installing Docker from the correct repo instead of the default Ubuntu 16.04 repo:

root@test:~# apt-cache policy docker-ce
docker-ce:
 Installed: (none)
 Candidate: 18.06.0~ce~3-0~ubuntu
 Version table:
    18.06.0~ce~3-0~ubuntu 500
       500 https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages

Step 5 — Install Docker

Finally, let’s start the Docker install:

root@test:~# apt-get install -y docker-ce
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
 aufs-tools cgroupfs-mount libltdl7 pigz
Suggested packages:
 mountall
The following NEW packages will be installed:
 aufs-tools cgroupfs-mount docker-ce libltdl7 pigz
0 upgraded, 5 newly installed, 0 to remove and 0 not upgraded.
Need to get 40.3 MB of archives.
After this operation, 198 MB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 pigz amd64 2.3.1-2 [61.1 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 aufs-tools amd64 1:3.2+20130722-1.1ubuntu1 [92.9 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 cgroupfs-mount all 1.2 [4,970 B]
Get:4 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 libltdl7 amd64 2.4.6-0.1 [38.3 kB]
Get:5 https://download.docker.com/linux/ubuntu xenial/stable amd64 docker-ce amd64 18.06.0~ce~3-0~ubuntu [40.1 MB]
Fetched 40.3 MB in 1s (38.4 MB/s)    
...
...

Docker should now be installed, the daemon started, and the process enabled to start on boot. Let’s check to see if it’s running:

root@test:~# systemctl status docker
* docker.service - Docker Application Container Engine
  Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
  Active: active (running) since Wed 2018-08-08 13:51:22 EDT; 2min 13s ago
    Docs: https://docs.docker.com
Main PID: 6519 (dockerd)
  CGroup: /system.slice/docker.service
          |-6519 /usr/bin/dockerd -H fd://
          `-6529 docker-containerd --config /var/run/docker/containerd/containerd.toml

Aug 08 13:51:22 test.docker.com dockerd[6519]: time="2018-08-08T13:51:22.192600502-04:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Aug 08 13:51:22 test.docker.com dockerd[6519]: time="2018-08-08T13:51:22.192630873-04:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42020f6a0, CONNECTING" module=grpc
Aug 08 13:51:22 test.docker.com dockerd[6519]: time="2018-08-08T13:51:22.192854891-04:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42020f6a0, READY" module=grpc
Aug 08 13:51:22 test.docker.com dockerd[6519]: time="2018-08-08T13:51:22.192867421-04:00" level=info msg="Loading containers: start."
Aug 08 13:51:22 test.docker.com dockerd[6519]: time="2018-08-08T13:51:22.340349000-04:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Aug 08 13:51:22 test.docker.com dockerd[6519]: time="2018-08-08T13:51:22.397715134-04:00" level=info msg="Loading containers: done."
Aug 08 13:51:22 test.docker.com dockerd[6519]: time="2018-08-08T13:51:22.424005987-04:00" level=info msg="Docker daemon" commit=0ffa825 graphdriver(s)=overlay2 version=18.06.0-ce
Aug 08 13:51:22 test.docker.com dockerd[6519]: time="2018-08-08T13:51:22.424168214-04:00" level=info msg="Daemon has completed initialization"
Aug 08 13:51:22 test.docker.com dockerd[6519]: time="2018-08-08T13:51:22.448805942-04:00" level=info msg="API listen on /var/run/docker.sock"
Aug 08 13:51:22 test.docker.com systemd[1]: Started Docker Application Container Engine.
~
~
~
(press q to quit)

Excellent! Good to go!

If Docker is not started automatically after the installation, run the following commands:

root@test:~# systemctl start docker.service
root@test:~# systemctl enable docker.service

Step 6 — Test Docker

Let’s check the new Docker build by downloading the hello-world test image.
To start testing, issue the following command:

 


root@test:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
9db2ca6ccae0: Pull complete
Digest: sha256:4b8ff392a12ed9ea17784bd3c9a8b1fa3299cac44aca35a85c90c5e3c7afacdc
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
   (amd64)
3. The Docker daemon created a new container from that image which runs the
   executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
   to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/

For more examples and ideas, visit:
https://docs.docker.com/engine/userguide/

Step 7 — The ‘Docker’ Command

With Docker installed and working, now is the time to become familiar with the command line utility. The ‘Docker’ command consists of using Docker with a chain of options followed by arguments. The syntax takes this form:

root@test:~# docker
Usage: docker [OPTIONS] COMMAND
A self-sufficient runtime for containers
Run 'docker COMMAND --help' for more information on a command.


To view all of the available Options and Management Commands, simply type:

docker

To view the switches available for a specific command, type:

docker docker-subcommand --help

Lastly, To view system-wide information about Docker, use:

docker info

Docker is a dynamic, robust and responsive tool that makes it very simple to run applications within a containerized environment. It is portable, less resource-intensive, and more reliant on the host operating system which allows for multiple uses. Overall, Docker is a ‘must have’ system and should be included in your toolkit for automation, deployment, and scaling of your applications!

Our Support Teams are filled with talented admins with an intimate knowledge of multiple web hosting technologies, especially those discussed in this article. If you are uncomfortable walking through the steps outlined here, we are a phone call, chat or ticket away from assisting you with this process. If you’re running one of our fully Managed Cloud VPS Servers, we can provide more information on directly implementing the software described in this article.

 

VPS Server Space/Disk Quota

The term “server space” refers to the amount of disk space that is available on your server’s hard disk drive. This space varies according to server type, hosting plan and possibly by additional services that are set up and available on your Liquid Web account.

Some of the largest hard disk drives on the market now can hold up to 100TB of data. To better visualize this, 100 terabytes of data is approximately equivalent to:

  • 42,000,000,000 trillion single-spaced typewritten pages
  • 8,000,000 phone books
  • 160,000 regular compact discs
  • 20,000 DVDs
  • 200 average-sized hard disks (500GB)
  • 80 human brains (the capacity of a human being’s functional memory is estimated to be 1.25 terabytes by futurist Raymond Kurzweil in The Singularity Is Near)

Your disk space can hold many types of data including file types like HTML, TXT, CSS, PHP, PDF, JPG, MP3, MP4, compressed (tar.gz) backups, SQL databases and more. These files are in specific folders which are defined by the applications configuration files or locations you determine.

How do I locate the folders containing a particular set of data?

The location of a file depends primarily on the type of file. On a Linux server, your typical cPanel account is set up under the /home/username folder, and your cPanel account username specifies the username folder. This folder is sometimes called the top-level or root folder of your cPanel account. This root folder is not publicly accessible on the web but, contains folders which are accessible via a web browser. The root folder holds other cPanel specific system folders that use a variety of functions.

As you can see, when uploading files to your account, you’ll likely want them to be in public_html to be accessible on the web. Uploading an image.jpg file to the public_html folder makes it available at domain.com/image.jpg. Additionally, if you create a folder inside of the public_html directory and add the same image there, it would be accessible at yourdomain.com/foldername/image.jpg.

To see the location of a file, you have several options;

  1. Log into your cPanel and open the File Manager under the Files section
    cPanel >> Home >> Files >> File Manager’ here you can view all of the files and folders in your account’s root directory.cPanel File Manager
  2. When a cPanel account is initially set up, it also creates the main FTP user. You can use the servers FTP functionality to access folders from a remote location to view the file listings. Several software titles like Filezilla, Cyberduck, and WinSCP are available for this type of connection.
  3. Lastly, you can connect to the server via SSH and get access to folders/files on the server.

How do I see how much space I’m using?

Disk Usage Graph

Let’s start by reviewing a few command line examples; mainly the “du” and “df” commands.

Note
The ‘du’ command sums up the total space of files that exist on the filesystem, while the ‘df’ command shows blocks available in the file system.

The ‘df’ command (abbreviation for disk free) simply lists the space used per partition:

df Command Output

The ‘du’ command (abbreviation for disk usage) reports the sizes of directories including all of their contents and the sizes of individual files:

du Command Output

Note
There are times when the ‘du’ and ‘df’ commands show different usage amounts. Previously removed files can cause this discrepancy from a running process holding open that file. Open processes cause the ‘df’ command to report that space as still being used. The solution to this is to restart the service to close any open process.

You can also use cPanel to determine the amount of space used and where its located. If you log in to cPanel, you would need to go to cPanel >> Home >> Files >> Disk Usage to get graphical of your disk usage.

cPanel Disk Usage

Lastly, to view your server’s disk usage in your Manage account server resource graph

  1. Log into your Manage portal.
  2. Navigate to the Servers section and then click on the Plus sign (+) next to the server of focus.
  3. Click on the Dashboard button and click the link next to the Disk Usage text as seen below

graphical statistics

This view provides a graphical representation of your disk space and the used amount.

How do I prevent disk space overages?

Disk space overages can result in lost emails, backups or even websites or the server going down! Just like your car, your server requires regular server maintenance. Attention to server maintenance reduces lost data. One way to prevent disk space overages is to use cPanel’s built-in tools.

cPanel possess the ability to send “Disk Quota Warning”  emails that denote when your server is using too much space. They contain specific locations to check, and the space used. The settings for these emails notifications are in WHM (Web Hosting Manager) under the Home »Server Configuration »Tweak Settings .

Email Notifications AreaOther areas of server maintenance to check on regularly include:

  • Pruning backups
  • Logs are rotating correctly (including Domlogs, Apache2, MySql, and Chkservd)
  • Regularly archiving email
  • Using the /home directory for large user accounts

What are the dangers of being too close to Disk Quota?

When a server gets close to or is at its max disk space capacity, strange errors and problems can manifest themselves in many ways including:

  • Services (like MySQL or Apache) can error out or stop
  • Websites can become very sluggish
  • The servers overall responsiveness can become slower
  • The server may exhibit a high load
  • You may see degraded disk performance
  • The server may display an increase in I/O wait
  • The server may demonstrate an increase in CPU usage
  • The file system can go into “read-only” mode
  • The server can run out of inodes
  • Files can become corrupted
  • Decreased swap space may occur causing issues

So what do I do if I’m running out of space?

As Benjamin Franklin stated, “By failing to prepare, you are preparing to fail.” In light of this knowledge, taking steps in advance to prevent these issues is always the best course of action. Directly monitoring your server disk space on a weekly or monthly basis prevents most space issues from turning into actual problems.

If you have already reached the point where immediate action needs to be taken to bring a server back in line with normal space expectations, you have several options. Using the “du” and “df” commands are your primary weapons in tracking down used server space.

The primary steps needed are:

    1. Log into your server
    2. Run a df -h command to locate which partitions are using the most spacedf Command Output
    3. Change directories into the affected folders using the most space.
    4. Run the following command:

du-sk Command Output(This is an advanced du command that sorts the contents of a directory by size. Use this to drill down into a folder to see used space.)

  1. Move files (to a backup drive or folder) or, remove the files that are no longer needed using the ‘rm’ command.
  2. Repeat steps 2 through 6 as needed until reaching desired space level.

Final Thoughts

Over time, any operating system can become overcrowded with addition and removal of programs or accounts. Actively monitoring your servers disk space is the most effective method to prevent server space issues. If you do run into issues, using the du, df command line tools or, using the graphical interface in your account allows you to view files as needed. As always, if you have further thoughts or questions about this topic, please contact our Linux Support department for more information.