Java is a programming language used to build Android apps and real life web applications like Gmail and Google Docs. This ubiquitous language can be installed onto an Ubuntu server and its what we’ll be teaching in this tutorial today. Continue reading “Install Oracle Java 8 on Ubuntu 18.04”
WP-CLI is a very handy set of commands. You can run anything that you would run in wp-admin on a WordPress site but from the command line. Useful commands which WP-CLI employs to keep WordPress core updated plugins including the default themes which come with WordPress.
Cron is a service for Linux servers that automatically executes scheduled commands. A cron job can be a series of shell commands, scripts, or other programs. Cron tasks or jobs can perform a variety of functions and once ran can send out an e-mail message to inform you of its completion or errors. If you receive an error, there are many ways to troubleshoot the cron task. Use this article for troubleshooting assistance or a tutorial on the basics of cron jobs. If you would like to learn more about creating a cron job check out our Knowledge Base tutorials on the subject. Continue reading “Troubleshooting: Cron Jobs”
The command line terminal, or shell on your Linux server, is a potent tool for deciphering activity on the server, performing operations, or making system changes. But with several thousand executable binaries installed by default, what tools are useful, and how should you use them safely? Continue reading “Useful Command Line for Linux Admins”
What if you have dozens of SQL databases and manually backing up/restoring each database is too time-consuming for your project? No problem! We can script out a method that will export and import all databases at once without needing manual intervention. For help with transferring SQL Logins and Stored Procedures & Views take a look at our MSSQL Migration with SSMS article.
1. Open SSMS (Microsoft SQL Server Management Studio) on the source server, log in to the SQL instance and open a New Query window. Run the following query:
SELECT name FROM master.sys.databases
This command will output a list of all MSSQL databases on your server. To copy this list out, click anywhere in the results and use the keyboard shortcut CTRL+A (Command + A for Mac users) to select all databases. After highlighting all the databases right click and select copy.
2. Open Notepad, paste in your results and delete all databases (in the newly copied notepad text) you do NOT wish to migrate, as well as deleting the following entries:
These entries are the system’s databases, and copying them is not necessary. Make sure to delete everything except explicitly the databases you need to migrate. You should now have a list of all required databases separated by a line. i.e.
3. Save this result on the computer as C:\databases.txt.
4. Create a new Notepad window, copy/paste the following into the document and save it as C:\db-backup.bat
for /F "tokens=*" %%a in (databases.txt) do ( sqlcmd.exe -Slocalhost -Q"BACKUP DATABASE %%a TO DISK ='%systemdrive%\dbbackups\%%a.bak' WITH STATS" )
5. Now that you’ve saved the file as C:\db-backup.bat, navigate to the Start menu and type cmd and right click on Command Prompt to select Run as Administrator.Type the following command:
And hit enter. Afterward, type db-backup.bat and hit enter once again.
At this point, your databases have begun exporting and you will see the percentage progress of each databases export (pictured below).
Take note of any failed databases, as you can re-run the batch file when it’s done, using only the databases that may have failed. If the databases are failing to back up, take note of the error message displayed in the command prompt, address the error by modifying the existing C:\databases.txt file to include only the failed databases and re-run db-backup.bat until all databases are successfully exported.
By now you have the folder C:\dbbackups\ that contains .bak files for each database you want to migrate. You will need to copy the folder and your C:\databases.txt file to the destination server. There are numerous ways to move your data to the destination server; you can use USB, Robocopy or FTP. The folder on the C drive of the destination server should be called C:\dbbackups . It’s important to accurately name the file as our script will be looking for the .bak files here. Be sure that the destination server has your C:\databases.txt file as well, as our script will be looking for the database names here.
1. Open a Notepad and copy/paste the following into the document and save it as C:\db-restore.bat
for /F "tokens=*" %%a in (C:\databases.txt) do (
sqlcmd.exe -E -Slocalhost -Q"RESTORE DATABASE %%a FROM DISK='%systemdrive%\dbbackups\%%a.bak' WITH RECOVERY"
2. Save the file as C:\db-restore.bat
3. Navigate to the Start menu and type cmd.
4. Right click on Command Prompt and select Run as Administrator. Type the following command:
and hit Enter. Now type db-restore.bat and hit Enter.
Your databases have now begun importing. You will see the percentage of each databases restoration and the message “RESTORE DATABASE successfully processed” for each database that has been successfully processed.
Take note of any failed databases, as you can re-run the batch file when it’s done, using only the databases that have failed. If the databases are failing to back up, take note of the error message displayed in the command prompt, address the error (you can change the batch file as necessary), modify C:\databases.txt to include only the failed databases and re-run db-restore.bat until all databases are successfully exported.
Congratulations, you have now backed up and restored all of your databases to the new server. If you have any login issues while testing the SQL connections on the destination server, refer to the Migrating Microsoft SQL Logins (anchor link) section of this article and follow the steps therein. To migrate views or stored procedures please refer to the Migrating Views and Stored Procedures section. Every SQL server will have it’s own configurations and obstacles to face but we hope this article has given you a strong foundation for your Microsoft SQL Server Migration.
Since 1996, XFCE Desktop gives users the ability to have a graphical user interface (GUI) environment, visually turning your Linux server into an environment more like your desktop computer. With its no-frills look, XFCE does not weigh heavy on the server’s hardware and is faster than GNOME and KDE to boot. Once completed with this small tutorial, you’ll be able to share and connect to the XFCE GUI by continuing to the next tutorial on How To Install VNC.
- These instructions are intended for installing Xfce Desktop Environment on an Ubuntu 16.04 LTS server.
- Logged in as a root user, but for non-root users precede all commands with the word sudo.
Step 1: Update apt-get
With best practices in mind, we will update before proceeding to install XFCE 4
Step 2: Install XFCE4 Desktop Environment
With one command we can install Xfce itself and some useful utilities that come with Xfce:
apt-get install -y xfce4 xfce4-goodies
Run each of these commands so that apt-get can utilize them during the purge of Xfce.
apt-get -f install
Purge Xfce from your Ubuntu server:
apt-get purge xfce4
As mentioned in our opening paragraph the next step is to configure VNC (virtual network computing) Installing VNC is necessary to open the recently installed Xfce interface. It’s optional but advisable to set up an SSH tunnel that connects to VNC for a secure connection. Check out our Knowledge Base on the subject of VNC to find your choice of articles.
User management includes removing users who no longer need access, removing their username and any associate root privileges are necessary for securing your server. Deleting a user’s access to your Linux server is a typical operation which can easily be performed using a few commands.
- We are logged in as root on an Ubuntu 16.04 VPS powered by Liquid Web!
Step 1: Remove the User
Insert the username you want to delete by placing it after the userdel command. In our example, I’ll be deleting our user, Tom.
Simultaneous you can delete the user and the files owned by this user with the -r flag. Be careful these files are not needed to run any application within your server.
userdel -r tom
If the above code produces the message below, don’t be alarmed, it is not an error, but rather /home/tom existed but /var/mail/tom did not.
userdel: tom mail spool (/var/mail/tom) not found
Step 2: Remove Root Privileges
By removing Tom’s username from our Linux system we are halfway complete, but we still need to remove their root privileges.
Navigate to the following section:
## Allow root to run any commands anywhere
root ALL=(ALL:ALL) ALL
tom ALL=(ALL:ALL) ALL
## User privilege specification
root ALL=(ALL:ALL) ALL
tom ALL=(ALL:ALL) ALL
With either result, remove access for your user by deleting the corresponding entry:
tom ALL=(ALL:ALL) ALL
Save and exit this file by typing :wq and press the enter key.
To add a user, see our frequently used article, How to Add a User and Grant Root Privileges on Ubuntu 16.04. Are you using a different Ubuntu version? We’ve got you covered, check out our Knowledge Base to find your version.
Broken down into two parts our article’s first section hits on “how to whitelist IPs or URIs,” for people who are somewhat familiar with ModSecurity but want to know further about the process. Our second section examines why we configure ModSecurity and how to prevent the security of the server from getting in the way of our work. If you have a Fully Managed Liquid Web server reach out to our Heroic Support team for assistance with whitelisting!
How to Whitelist IPs or URIs
“ModSecurity is a toolkit for real-time web application monitoring, logging, and access control.” (modsecurity.org). In simple terms, this means that ModSec, also called mod_security or ModSecurity, is a web application firewall that can actively look for attacks to the system and stop malicious activity. However, sometimes these rules trigger when legitimate work is taking place, blocking your IP and stopping you or your developer’s until you can remove the IP block. The way around for being blocked is known as whitelisting, which essentially allows for a specific IP to access the server. There are a few ways to whitelist a request in ModSec, either by IP or by URI (URIs are specific pages on the website).
- Find your IP or ask your developer for theirs. (You can find this by going to ip.liquidweb.com)If you or your developer have a static IP (one that will not change), one way you can whitelist the ModSec rules is by IP.
- Find the ModSec error in the Apache error logs with the following command (Be sure to modify the command with your IP in place of “IP here.”):
grep ModSec /usr/local/apache/logs/error_log | grep “IP here”.
- The output of this command will give you a list of hits for ModSecurity from you or your developer’s IP, which you can see below. While this looks intimidating, you will only want to pay attention to 3 bits of information highlighted. Please note, the output will not show these colors when you are viewing the files.
[Fri May 25 23:07:04.178701 2018] [:error] [pid 78007:tid 139708457686784] [client 220.127.116.11:30095] [client 18.104.22.168] ModSecurity: Access denied with code 406 (phase 2). Pattern match "Mozilla/(4|5)\\\\.0$" at REQUEST_HEADERS:User-Agent. [file "/etc/apache2/conf.d/modsec2.liquidweb.conf"] [line "109"] [id "20000221"] [hostname "22.214.171.124"] [uri "/db/index.php"] [unique_id "WwjPWChxvG1CO4kz-D55eQAAACU"]
1. Once you have the correct ModSec error, you will need to edit the ModSec configuration. If you are using Easy Apache 4 you will find the configuration file with this path:
2. Open the file with your favorite text editor, such as vim, nano, or file manager like so:
3. The blue text above will be the IP address that you are whitelisting from the original error. You must keep the backslashes (\) and up-carrot (^) in order for the IP to be read correctly. Thus it will always look something like:
For for the id, noted in red, you will change the number after the colon, which will be the Apache error log like we saw above. This will look similar to:
Add the following code with the colored sections edited to match your intended IP.
SecRule REMOTE_ADDR "^64\.14\.210\.4" "phase:1,nolog,allow,ctl:ruleEngine=off,id:20000221"
If your IP is dynamic (changing) and you keep getting blocked in the firewall, it is best to whitelist it via URI, the yellow item in the ModSec error.
1. Begin by opening the Easy Apache 4 configuration file:
2. Add the following text to the configuration. Remember to pay attention to the highlighted parts. Change the yellow “/db/index.php” to match your URI and the red id to match the id of your error (Do not use the colon in this one).
3. The final step for whitelisting, before you finalize the process, is to ensure you have correctly set up the whitelist. For Easy Apache 4 you will run the command:
As long as the command returns “Syntax Ok” you are safe to make the whitelist active by restarting Apache. Otherwise, review the whitelists to make sure the syntax matches up correctly with the above directions.
4. Lastly, restart Apache with the following command.
You have successfully whitelisted yourself in ModSec!
Cyber Security is a hydra; once one threat is cut off, two more grow back. While this is not a new analogy, it’s important to understand as we battle threats to our network, computers, and servers. With all the complexities that come with security, I want to talk about adequately configuring ModSec to deter threats while still allowing you to work on your websites. Often, when it comes to server security, too much protection can hinder effectiveness.
For example, say you have the following set up on your server:
- You do not allow root SSH login to the server
- utilize dual-factor authentication for any SSH logins
- use an SSH key for the sudo user and require other security safeguards
While this type of configuration is secure, it takes longer to log into your system to make a quick edit to your settings, a double-edged sword; how can you keep the server safe while not tying your own hands? A great example of how this plays out is using ModSec.
ModSec can block your IP if it falsely flags your work. While this module improves system security, you’ll need to be aware of properly implementing and “scoping” the technology. Scoping in this sense means to manage risks, the focus of what is important for security while still allowing work on the server with minimal interference. To tune out legitimate requests to your server, such as when you are editing your website’s code via a plugin, ModSec has the options to whitelist rules or IPs and keep your work on track.
Whitelisting an IP from the rules that ModSec follows is a great option so long as the IP never changes (i.e. a static IP, see article here to learn more https://support.google.com/fiber/answer/3547208?hl=en) and is limited to only people you trust. This method prevents ModSec from viewing your requests as malicious and blocking your IP. This practice has the drawback that if someone (say an unhappy employee) has access to your network, they now have a way around ModSec to attack your server.
With non-static (dynamic) IPs the problems of whitelisting an IP are readily apparent. With the continual change of a dynamic IP, it creates the potential of exploiting your server, as someone could use an old IP to access the server. Whitelisting specific rules comes to save the day! When you whitelist by rules, you can edit with granularity and limit the rules to particular domains and URIs, protecting the rest of the server from attacks related to that same rule!
Example of ModSecurity
ModSec reads a series of rules and applies them to incoming requests being made to the web server. An example of what a block looks like is:
[Sat Jun 30 02:21:56.013837 2018] [:error] [pid 79577:tid 139862413879040] [client 126.96.36.199:24397] [client 188.8.131.52] ModSecurity: Access denied with code 406 (phase 2). Pattern match "Mozilla/(4|5)\\\\.0$" at REQUEST_HEADERS:User-Agent. [file "/etc/apache2/conf.d/modsec2.liquidweb.conf"] [line "109"] [id "2000064"] [hostname "184.108.40.206"] [uri "/mysql/index.php"] [unique_id "WzchhAjuZ6wPAzo9AwW1WwAAAE8"]
This error shows Apache stopped a potential attack on a file at /mysql/index.php. This is an error similar to what appears when the code is being written or edited within programs like Drupal or WordPress.
If you are persistently being blocked in your firewall while working on your code, ModSec is the likely culprit. The ModSec errors can be found in the Apache error log (in cPanel the path is /usr/local/apache/logs/error_log). The phrase “ModSec” can be quickly isolated from the log (via the command ‘grep “ModSec” /usr/local/apache/logs/error_log’). By comparing you or your developer(s) IP to the log, you’ll be able to identify stopped requests that are legitimate. Verify these are valid requests by double-checking that someone in your organization made them. Once you have done so, you can move forward in setting up a whitelist for the error, per the steps above.
Again, we want to scope to allow the least amount of wiggle room for an attack and ensure we can keep working. If you are unable to have a trusted static IP, you’ll need to use the whitelist URI method, providing the specific page as an exemption. Once completed, remove both whitelisted items from the configuration file, in case of a genuine attack.
On a parting note, I encourage you to explore ModSec and learn more of the ins and outs of the software. Exploring different methods of whitelisting can be a lot of for to learn and most importantly helps to tighten server security. As always, our Fully Supported Customers can contact our Helpful Human Support team for assistance. Check out articles on security in our Knowledge Base, like this one on Maldet! It’s another excellent way to learn about your server and develop an understanding of server security.
Ubuntu 16.04 LTS provides you the ability to add a user for anyone who plans on accessing your server. Creating a user is a basic setup but an important and critical one for your server security. In this tutorial, we will create a user and grant administrative access, known as root, to your trusted user.
- Open a terminal and log in as root.
- Work on a Linux Ubuntu 16.04 server
Step 1: Add The User
Create a username for your new user, in my example my new user is Tom:
You’ll then be prompted to enter a password for this user. We recommend using a strong password because malicious bots are programmed to guess simple passwords. If you need a secure password, this third party password generator can assist with creating one.
~# adduser tom
Adding user `tom' ...
Adding new group `tom' (1002) ...
Adding new user `tom' (1002) with group `tom' ...
Creating home directory `/home/tom' ...
Copying files from `/etc/skel' ...
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Prompts will appear to enter in information on your new user. Entering this information is not required and can be skipped by pressing enter in each field.
Enter the new value or press ENTER for the default
Full Name :
Room Number :
Work Phone :
Home Phone :
Lastly, the system will ask you to review the information for accuracy. Enter Y to continue to our next step.
Is the information correct? [Y/n]
Step 2: Grant Root Privileges
Assigning a user root access is to grant a user the highest power. My user, Tom, can then make changes to the system as a whole, so it’s critical to allow this access only to users who need it. Afterward, Tom will be able to use sudo before commands that are usually designed to be used by the root user.
usermod -aG sudo tom
Step 3: Verify New User
As root, you can switch to your new user with the su – command and then test to see if your new user has root privileges.
su - tom
If the user has properly been granted root access the command below will show tom in the list.
grep '^sudo' /etc/group
A major factor in database performance is the storage engine used by the database, and more specifically, its tables. Different storage engines provide better performance in one situation over another. For general use, there are two contenders to be considered. These are MyISAM, which is the default MySQL storage engine, or InnoDB, which is an alternative engine built-in to MySQL intended for high-performance databases. Before we can understand the difference between the two storage engines, we need to understand the term “locking.”
To protect the integrity of the data stored within databases, MySQL employs locking. Locking, simply put, means protecting data from being accessed. When a lock is applied, the data cannot be modified except by the query that initiated the lock. Locking is a necessary component to ensure the accuracy of the stored information. Each storage engine has a different method of locking used. Depending on your data and query practices, one engine can outperform another. In this series, we will look at the two most common types of locking employed by our two storage engines.
Table locking: The technique of locking an entire table when one or more cells within the table need to be updated or deleted. Table locking is the default method employed by the default storage engine, MyISAM.
|Example: MyISAM Table Locking||Column A||Column B||Column C|
|Query 1 UPDATE||Row 1||Writing||data||data|
|Query 2 SELECT (Wait)||Row 2||data||data||data|
|Query 3 UPDATE (Wait)||Row 3||data||data||data|
|Query 4 SELECT (Wait)||Row 4||data||data||data|
|Query 5 SELECT (Wait)||Row 5||data||data||data|
|The example illustrates how a single write operation locks the entire table causing other queries to wait for the UPDATE query finish.|
Row-level locking: The act of locking an effective range of rows in a table while one or more cells within the range are modified or deleted. Row-level locking is the method used by the InnoDB storage engine and is intended for high-performance databases.
|Example: InnoDB Row-Level Locking||Column A||Column A||Column A|
|Query 1 UPDATE||Row 1||Writing||data||data|
|Query 2 SELECT||Row 2||Reading||data||data|
|Query 3 UPDATE||Row 3||data||Writing||data|
|Query 4 SELECT||Row 4||Reading||Reading||Reading|
|Query 5 SELECT||Row 5||Reading||data||Reading|
|The example shows how using row-level locking allows for multiple queries to run on individual rows by locking only the rows being updated instead of the entire table.|
By comparing the two storage engines, we get to the crux of the argument between using InnoDB over MyISAM. An application or website that has a frequently used table works exceptionally well using the InnoDB storage engine by resolving table-locking bottlenecks. However, the question of using one over the other is a subjective as neither of them is perfect in all situations. There are strengths and limitations to both storage engines. Intimate knowledge of the database structure and query practices is critical for selecting the best storage engine for your tables.
MyISAM will out-perform InnoDB on large tables that require vastly more read activity versus write activity. MyISAM’s readabilities outshine InnoDB because locking the entire table is quicker than figuring out which rows are locked in the table. The more information in the table, the more time it takes InnoDB to figure out which ones are not accessible. If your application relies on huge tables that do not change data frequently, then MyISAM will out-perform InnoDB. Conversely, InnoDB outperforms MyISAM when data within the table changes frequently. Table changes write data more than reading data per second. In these situations, InnoDB can keep up with large amounts of requests easier than locking the entire table for each one.
Should I use InnoDB with WordPress, Magento or Joomla Sites?
The short answer here is yes, in most cases. Liquid Web’s Most Helpful Humans in Hosting Support Teams have encountered several table-locking bottlenecks when clients are using some of the standard web applications of today. Most users of popular third-party applications like WordPress, Magento, and Joomla have limited knowledge of the underlying database components or code involved to make an informed decision on storage engines. Most table-locking bottlenecks from these content management systems (CMS) are generally resolved by changing all the tables for the site over to InnoDB instead of the default MyISAM. If you are hosting many of these types of CMS on your server, it would be beneficial to change the default storage engine in MySQL to use InnoDB for all new tables so that any new table installations start off with InnoDB.
Set your default storage engine to InnoDB by adding default_storage_engine=InnoDB to the [mysqld] section of the system config file located at: /etc/my.cnf . Restarting the MySQL service is necessary for the server to detect changes to the file.
~ $ cat /etc/my.cnf
Unfortunately, MySQL does not inherently have an option to convert tables, leaving each table to be changed individually. Liquid Web’s support team has put together an easy to follow maintenance plan for this process. The script, which you can run on the necessary server via shell access (SSH) will convert all tables between storage engines.
Step 1: Prep
Plan to start at a time of day where downtime would have minimal consequences. This process itself does not require any downtime, however, downtime may be necessary to recover from unforeseen circumstances.
Step 2: Backup All Databases To A File
The command below creates a single file backup of all databases named all-databases-backup.sqld and can be deleted once the conversion has succeeded and there are no apparent problems.
mysqldump --all-databases > all-databases-backup.sql
Step 3: Record Existing Table Engines To A File
Run the following script to record the existing table engines to a file named table-engine-backup.sql. You can then “import” or “run” this file later to convert back to their original engines if necessary.
mysql -Bse 'SELECT CONCAT("ALTER TABLE ",table_schema,".",table_name," ENGINE=",Engine,";") FROM information_schema.tables WHERE table_schema NOT IN("mysql","information_schema","performance_schema");' | tee table-engine-backup.sql
If you need to revert the table engines back for any reason, run:
mysql < table-engine-backup.sql
Step 4a: Convert MyISAM Tables To InnoDB
The below command will proceed even if a table fails and lets you know which tables failed to convert. The output is saved to the file named convert-to-innodb.log for later review.
mysql -Bse 'SELECT CONCAT("ALTER TABLE ",table_schema,".",table_name," ENGINE=InnoDB;") FROM information_schema.tables WHERE table_schema NOT IN ("mysql","information_schema","performance_schema") AND Engine = "MyISAM";' | while read -r i; do echo $i; mysql -e "$i"; done | tee convert-to-innodb.log
Step 4b: Convert All InnoDB Tables To MyISAM
This command will proceed even if a table fails and lets you know which tables failed to convert. The output is also saved to the file named convert-to-myisam.log for later review.
mysql -Bse 'SELECT CONCAT("ALTER TABLE ",table_schema,".",table_name," ENGINE=MyISAM;") FROM information_schema.tables WHERE table_schema NOT IN ("mysql","information_schema","performance_schema") AND Engine = "InnoDB";' | while read -r i; do echo $i; mysql -e "$i"; done | tee convert-to-myisam.log
The following commands illustrate how converting a single table is accomplished.
Backup A Single Table To A File
mysqldump database_name table_name > backup-table_name.sql
Convert A Single Table To InnoDB
mysql -Bse ‘ALTER TABLE database_name.table_name ENGINE=InnoDB;’
Convert A Single Table To MyISAM:
mysql -Bse ‘ALTER TABLE database_name.table_name ENGINE=MyISAM;’
Check out our other articles in this series, MySQL Performance: Identifying Long Queries, to pinpoint slow queries within your database. Stay tuned for our next article where we will cover caching and optimization.