How To Employ Basic Elasticsearch Commands

Reading Time: 5 minutes

What is Elasticsearch?

Elasticsearch is a distributed, full-text, open-source search engine. It provides multi-tenant capabilities in analyzing aggregate data types from sources like Logstash or Kibana. This application stores and indexes information, which can then be queried for specific data. It returns useful details about a particular program, log analysis, application performance data, or other information. 

Installation Order

To install the Elastic Stack, deploy these applications in the following order.

  1. Elasticsearch (install instructions)
  2. Kibana (install)
  3. Logstash (install)
  4. Beats (install instructions)
  5. APM Server (install instructions)
  6. Elasticsearch Hadoop (install instructions)

Installation 

In order to install Elasticsearch, see our kb article for more in depth instructions. Here are the basic installation steps.

  1. Download and unpack the Elasticsearch official distribution.
  2. Next, run bin/elasticsearch on Linux or macOS. Run bin\elasticsearch.bat on Windows.
  3. Then, curl -X GET http://localhost:9200/.
  4. Start more servers

What is Indexing?

Indexing is simply the process of adding data into Elasticsearch. Elasticsearch stores and retrieves this data in Apache Lucene indexes. We will not be discussing Lucene specifically in this article because we need to delve deeper into that application to truly understand the role Elasticsearch plays. This article is for primarily for new users to employ basic GET and PUT requests in Lucene.

Requests

Put

We send PUT requests when we know, or want to specify an ID of the data type. We can use POST if we want Elasticsearch to generate an ID for that item on its own. An Example of a simple POST command would look like this.

curl -XPOST 'localhost:9200/logs/test_app' -H 'Content-Type: application/json' -d'
{
"timestamp": "2020-08-20 09:10:11",
"message": "Test user is logged in",
"user_id": 2,
"admin": false
}
'

And the output looks like this:

{"_index":"logs","_type":"test_app","_id":"e8rHCnQBSXUbYazxinrq","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1}

Output

We can see in the output provided above that id was generated by Elasticsearch., We can also note the “version” and “created” fields as well. This implies that the test_app file was created using our POST command which did not exist before. Let’s review how we can index something using a PUT request.

curl -X PUT 'localhost:9200/app/users/4' -H 'Content-Type: application/json' -d '
{
  "id": 2,
  "username": "Dean",
  "last_login": "2020-08-20 09:10:11"
}
'

Using this command, we get this output.

{"_index":"app","_type":"users","_id":"4","_version":2,"result":"updated","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1}

With this entry, we indexed the user Dean under /app/users/ directory. 

All the data that was entered is sent as a JSON object. If we ask how this works without any data structure, the answer is succinct. Elasticsearch usually works as a NoSQL database, thus structure is not needed. Next, we will check how we can perform queries using Elasticsearch.

What is Querying?

Querying is simply using a crafted query search to locate specific information within a given location. Using a query, we can separate questions in elasticsearch into two main categories. 

  • Leaf Queries – Leaf queries are looking for specific values in certain fields. We can run Leaf Query independently and some are termed match, term, and range queries.  
  • Compound Queries – Compound queries are combinations of leaf queries and compound queries. Compound queries combine multiple queries into a single one we use to get what we’re seeking more easily. 

Query Types

A complete classification of Elasticsearch queries looks like this:

  • Leaf Queries
    • Full text
    • Term queries
    • Geo queries
    • Span Queries
    • Joining Queries
    • Specialized queries
  • Compound Queries
    • Bool
    • Boosting
    • Dis max
    • Constant score
    • Function score

We do not have to know all these query types to run some basic commands with elasticsearch as many of these are advanced and require more than basic knowledge of the program. We list them here to get a better overall picture of elasticsearch’s capabilities. Let’s review some basic query commands that can be used with Elasticsearch. 

Note:
To query something with the Elasticsearch, we must index it first!

Examples

Since we indexed our app using the previous commands, let’s try to query it now. A simple command would look like this.

curl -XGET 'localhost:9200/app/users/4?pretty'

And this would be the output:

{
  "_index" : "app",
  "_type" : "users",
  "_id" : "4",
  "_version" : 1,
  "_seq_no" : 0,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "id" : 2,
    "username" : "Dean",
    "last_login" : "2020-08-20 09:10:11"
  }
}

Using this command, we queried all the users under “app”. It would return basic information such as username, id, time of last login and so on. Every field with the underscore in front of it is the meta field of our query. Under _source, we can see the original document that was indexed. The simplest way to query Elasticsearch is to use a “URI Search”. We provide a single search term or word and Elasticsearch will comb through every field of every document in our cluster. For example, we can use this query:

curl -XGET 'localhost:9200/_search?q=dean'

And Elasticsearch will find every field that contains the word “Dean” in it. On our end, the output query will look like this.

{"took":60,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":1,"relation":"eq"},"max_score":0.2876821,"hits":[{"_index":"app","_type":"users","_id":"4","_score":0.2876821,"_source":
{
  "id": 2,
  "username": "Dean",
  "last_login": "2020-08-20 09:10:11"
}
}]}}

Output

As we can see, the output produces both search terms, and the result of the search. Let’s do a quick breakdown of this information.

took = number of seconds that query took to complete

timed_out = pretty straightforward; false means that query didn’t time out, true means that it did time out

shards = number of documents that Elasticsearch went through and how many of them were searched successfully, how many were skipped, and how many failed

hits = how many fields actually met the query criteria, along with the meta information

In the example above, we searched for specific fields within all of our indexed information noted in Elasticsearch. We can narrow this query down further by providing the path to a limited subset of documents. This way, we can cut down on the query time as well. So, instead of: 

curl -XGET 'localhost:9200/_search?q=dean'

We will use:

curl -XGET 'localhost:9200/app/users/_search?q=dean'

The output would be the same, but in this case “took” (search timeframe) was only 3 seconds as opposed to the 60 seconds that we observed in the previous query. This is handy when we know the specific directory where some information is stored, but are not sure which document holds the information that we need. 

Now that we know the very basics of indexing and querying with Elasticsearch, let’s move on to the deletion of indexed documents.

What is Deleting?

In the same way we used previous commands, we can use the Delete function to remove data stored in our cluster. The difference is, we use DELETION via an HTTP request. 

curl -XDELETE 'localhost:9200/app/users/4?pretty'

Our output will look like this:

[root@host ~]# curl -XDELETE 'localhost:9200/app/users/4?pretty'
{
  "_index" : "app",
  "_type" : "users",
  "_id" : "4",
  "_version" : 1,
  "result" : "not_found",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 2,
  "_primary_term" : 1
}

In order to delete a single index, following command can be used.

curl -XDELETE 'localhost:9200/logs?pretty'

If we want to delete an entire document that has been indexed, we can use this command.

curl -XDELETE 'localhost:9200/path/to/document'

The response that we get in both cases will look like this.

{
 "acknowledged" : true
}

Conclusion

In this article, we covered some of the basic Elasticsearch commands, but barely scratched the surface of all its capabilities. Multiple query variations can be employed to search through our information, but we would need multiple articles to cover all of them in depth. We hope that this article provides some of the basics of Elasticsearch as it is only written as an introduction to this powerful and versatile search engine.

Join Us!

Contact us today at 1.800.580.4985 to speak to a knowledgeable Solutions Provider who can get you the info you need on any one of our product lines, to assist you in making an informed decision right away.

Too busy to talk? Click HERE to open a quick chat with us to find out more. Would you like the information in an email you can review at your leisure? Email us today to get solid advice on which product in our line up would best suit your needs.

We look forward to hearing from you!

How to Commit and Push Changes from Ubuntu 18.04 to GitHub

Reading Time: 2 minutes

In our previous tutorial, we showed you how to create and sync a GitHub repository to your Ubuntu VPS server.  Using the last tutorial as a springboard, we’ll continue on and show you how to commit and push a script from your Ubuntu server to your GitHub account.

Continue reading “How to Commit and Push Changes from Ubuntu 18.04 to GitHub”

How to Solve the Upgrade Ubuntu Install Updates Error

Reading Time: < 1 minute

When updating from Ubuntu 16.04 to 18.04 we used the

do-release-upgrade command as one of the steps to get the newer versions.  While using that method the update Ubuntu’s core we came across a known bug that would not allow us to continue with the upgrade. Instead, it would give the following error message:

Checking for a new Ubuntu release
Please install all available updates for your release before upgrading.

Continue reading “How to Solve the Upgrade Ubuntu Install Updates Error”

How to Install Logstash

Reading Time: 4 minutes

What is Logstash?

logstash logo

Logstash is a free, open source, server-side data collection and processing engine which uses dynamic pipelining capabilities. It takes in information from multiple data sources, reconstructs it, and then sends it to a destination of our choice. Logstash also cleans and modifies the data for use in an advanced outbound analytics and visualization use cases. It is the L in the ELK stack (Elasticsearch, Logstash and Kibana) and is typically responsible for sending data to Elasticsearch.

Continue reading “How to Install Logstash”

How to Install and Configure Elasticsearch

Reading Time: 8 minutes

What is Elasticsearch?

elasticsearch logo

Elasticsearch is a distributed, open-source, full-text search engine which provides multi-tenant capabilities for analyzing multiple data types. It stores and indexes data shipped from sources like Logstash or Kibana. Elasticsearch can then be queried for specific data to return useful information about a particular application, log analysis, application performance data, or other information. 

Continue reading “How to Install and Configure Elasticsearch”

How to Install Pyenv-virtualenv on Ubuntu 18.04

Reading Time: 3 minutes
python-logo

Pyenv is an outstanding tool for managing multiple Python installations. Pyenv-virtualenv is a pyenv plugin that facilitates the creation and management of Python virtual environments with pyenv. This is a compelling proposition, making it possible to manage multiple Python versions with pyenv and provide the means to control the Python environment in a more granular manner.

Continue reading “How to Install Pyenv-virtualenv on Ubuntu 18.04”

How to Install and Configure Gradle

Reading Time: 3 minutes

What is Gradle?

Gradle is a high-performance open-source build automation tool that makes it possible to build just about any type of software. It is highly configurable and extensible via a well-developed plugin ecosystem. Because build automation is an integral part of the consistent delivery of software projects, Gradle integrates seamlessly with multiple IDE’s, including Android Studio. This tutorial covers the manual installation of Gradle on Linux, MacOS, and Windows.

Continue reading “How to Install and Configure Gradle”

How to Install Pyenv on Ubuntu 18.04

Reading Time: 3 minutes

What is Pyenv?

Pyenv is a fantastic tool for installing and managing multiple Python versions. It enables a developer to quickly gain access to newer versions of Python and keeps the system clean and free of unnecessary package bloat. It also offers the ability to quickly switch from one version of Python to another, as well as specify the version of Python a given project uses and can automatically switch to that version. This tutorial covers how to install pyenv on Ubuntu 18.04.

Continue reading “How to Install Pyenv on Ubuntu 18.04”

How to Install and Configure Apache Maven on Ubuntu 18.04

Reading Time: 3 minutes

What is Apache Maven?

Apache Maven is one of the most popular build management tools for Java projects. It employs a Project Object Model or POM utilizing an XML file to define the structure of a project, including its dependencies, documentation, and build parameters. Instead of having to manage those things individually, Maven consolidates that functionality allowing for a more streamlined and efficient project management process. In this tutorial, we are going to cover how to install Maven on Ubuntu 18.04.

Continue reading “How to Install and Configure Apache Maven on Ubuntu 18.04”

How to Install INXI on Ubuntu 18.04

Reading Time: 6 minutes

INXI is one of the best tools that offer a straightforward and comprehensive method for obtaining a wealth of information about a server with a single command.

There are a myriad of individual tools and commands that can be utilized to glean this information from a Linux system. Understanding the specific hardware that underlies a Linux server is an integral part of understanding that server’s capabilities. In this tutorial, we will cover the installation of INXI on an Ubuntu 18.04 server. It will also include some basic command-line usage of the INXI tool.

Continue reading “How to Install INXI on Ubuntu 18.04”