Grails ElasticSearch

Learn how to use ElasticSearch within a Grails application

Authors: Puneet Behl, Sergio del Amo

Grails Version: 3.3.2

1 Training

Grails Training - Developed and delivered by the folks who created and actively maintain the Grails framework!.

2 Getting Started

Elasticsearch is a highly scalable open-source full-text search and analytics engine. It allows you to store, search, and analyze big volumes of data quickly and in near real time. It is generally used as the underlying engine/technology that powers applications that have complex search features and requirements.

The Grails ElasticSearch plugin intends to implement a simple integration of Grails with the Open Source Search Engine ElasticSearch, which is based on Lucene and provide distributed capabilities.

In this guide, you will learn how to use Grails Elasticsearch plugin to create index in Elasticsearch for faster search.

2.1 Features

Maps domain classes to their corresponding index in Elasticsearch
Provides an ElasticSearch service for cross-domain searching
Injects domain class methods for specific domain searching, indexing and unindexing
Automatically mirrors any changes made through GORM to the index
Allow to use the Groovy Content Builder DSL for search queries
Support for term highlighting

2.2 What will you need

To complete this guide, you will need the following:

Some time on your hands
Elasticsearch v5.4.1 installed on your machine with ES_HOME configured properly
A decent text editor or IDE
JDK 1.7 or greater installed with JAVA_HOME configured appropriately

2.3 How to complete the guide

To get started do the following:

Download and unzip the source

Clone the Git repository:
git clone https://github.com/grails-guides/grails-elasticsearch.git

The Grails guides repositories contain two folders:

initial Initial project. Often a simple Grails app with some additional code to give you a head-start.
complete A completed example. It is the result of working through the steps presented by the guide and applying those changes to the initial folder.

To complete the guide, go to the initial folder

cd into grails-guides/grails-elasticsearch/initial

and follow the instructions in the next sections.

You can go right to the completed example if you cd into grails-guides/grails-elasticsearch/complete

3 Writing the Guide

3.1 Install Elasticsearch via Docker

If you do not already have Docker installed, you’ll need to install it.

Depending on your environment you may need to increase Docker’s available memory to 4GB or more.

Make sure Docker machine is running by executing following command:

docker-machine status default

If default Docker machine is not running then start the machine by executing following bash command:

docker-machine start default

After executing the above command you should see something on the terminal as follows:

Starting "default"...
(default) Check network to re-create if needed...
(default) Waiting for an IP...
Machine "default" was started.
Waiting for SSH to be available...
Detecting the provisioner...
Started machines may have new IP addresses. You may need to re-run the `docker-machine env` command.

Now we have started Docker machine successfully. So let’s note down the IP address of default Docker machine, which we will use in future to connect our application to the Elasticsearch. Run the following command to see IP address of Docker machine:

docker-machine ls
NAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER        ERRORS
default   -        virtualbox   Running   tcp://192.168.99.100:2376           v17.12.0-ce

Start the Elasticsearch Docker container

eval $(docker-machine env default)

docker run \
    -p 9200:9200 \
    -p 9300:9300 \
    -e "discovery.type=single-node" \
    -e ES_JAVA_OPTS="-Xms512m -Xmx512m" \
    -e "http.cors.enabled=true" \
    -e "http.cors.allow-origin=/https?:\/\/192.168.99.100(:[0-9]+)?/" \
    -e "http.cors.allow-headers=Authorization,X-Requested-With,Content-Length,Content-Type" \
    -e "http.cors.allow-credentials=true" \
    -e "cluster.name=elasticsearch" \
    -e "xpack.security.enabled=false" \
    docker.elastic.co/elasticsearch/elasticsearch:5.4.3

3.2 Configure ElasticSearch plugin

To install the plugin add following properties to build.gradle:

build.gradle

def elasticsearchVersion = '5.4.1' (1)
ext['elasticsearch.version'] = elasticsearchVersion

1	Define Elasticsearch version.

compile "org.grails.plugins:elasticsearch:2.4.0"

In order to connect Application to Elasticsearch, define Elasticsearch plugin configurations in application.yml as follows:

grails-app/conf/application.yml

elasticSearch:
    datastoreImpl: hibernateDatastore (1)
    client:
        mode: transport (2)
        hosts:
          - {host: 192.168.99.100, port: 9300} (3)
    cluster.name: elasticsearch (4)
    disableAutoIndex: false (5)
    bulkIndexOnStartup: true (6)

1	The Hibernate datastore implementation should be watched for storage events.
2	The plugin creates a transport client that will connect to a remote ElasticSearch instance without joining the cluster. The `TransportClient` connects remotely to an Elasticsearch cluster using the transport module and communicates with them in round robin fashion on each action.
3	Host address for `TransportClient` to connect to.
4	Name of the Elasticsearch cluster.
5	Determines if the plugin should reflect any database save/update/delete automatically on the ES instance. Default to false.
6	Application should launch a bulk index operation upon startup.

3.3 Elasticsearch logs

Add at the end of logback.groovy the next statement to see ElasticSearch related logs

grails-app/conf/logback.groovy

logger("grails.plugins.elasticsearch", DEBUG, ['STDOUT'], false)

3.4 Create a Domain Class

You can create a domain class

grails-app/domain/demo/Book.groovy

package demo

class Book {
    String title
    String author
    String about
    String href
    static mapping = {
        about type: 'text'
    }
}

3.5 Mark Domain Searchable

In order to create the index for this domain in Elasticsearch, define a static property called searchable in the previoulsy defined Book domain class

grails-app/domain/demo/Book.groovy

    static searchable = {
        title boost: 2.0  (1)
        about boost: 1.0  (2)
        except = ['href'] (3)
    }

1	'boost' is commonly used to fine-tune the the relevance `_score` for each document by giving more weight to documents. In this case, if the search query matches with the `title` of the document then the relevance `_score` of the document will be double when compared to other results.
2	If the search query matches with the `about` field then the relevant `_score` will be `1.0` times the others results.
3	The `href` property is included in the list of properties that are not `searchable`.

The previous example will create the following Elasticsearch mappings:

{
  "mappings": {
    "book": {
      "properties": {
        "author": {
          "include_in_all": true,
          "term_vector": "with_positions_offsets",
          "type": "text"
        },
        "about": {
          "include_in_all": true,
          "term_vector": "with_positions_offsets",
          "type": "text"
        },
        "title": {
          "include_in_all": true,
          "term_vector": "with_positions_offsets",
          "boost": 2,
          "type": "text"
        }
      }
    }
  }
}

3.6 GORM Data Service

Create a GORM Data Service to simplify Book CRUD operations.

grails-app/services/demo/BookDataService.groovy

package demo

import grails.gorm.services.Service
import groovy.transform.CompileStatic

@CompileStatic
@Service(Book)
interface BookDataService {
    Book save(String title, String author, String about, String href)
    Number count()
}

3.7 Save Initial Data

Create some books by modifying the BootStrap.groovy as follows:

grails-app/init/demo/BootStrap.groovy

package demo

import grails.util.BuildSettings
import grails.util.Environment
import groovy.transform.CompileStatic

@CompileStatic
class BootStrap {

    public final static List< Map<String, String> > GRAILS_BOOKS = [
            [
                    title : 'Grails 3 - Step by Step',
                    author: 'Cristian Olaru',
                    href: 'https://grailsthreebook.com/',
                    about : 'Learn how a complete greenfield application can be implemented quickly and efficiently with Grails 3 using profiles and plugins. Use the sample application that accompanies the book as an example.'
            ],
            [
                    title : 'Practical Grails 3',
                    author: ' Eric Helgeson',
                    href  : 'https://www.grails3book.com/',
                    about : 'Learn the fundamental concepts behind building Grails applications with the first book dedicated to Grails 3. Real, up-to-date code examples are provided, so you can easily follow along.'
            ],
            [
                    title : 'Falando de Grails',
                    author: 'Henrique Lobo Weissmann',
                    href  : 'http://www.casadocodigo.com.br/products/livro-grails',
                    about : 'This is the best reference on Grails 2.5 and 3.0 written in Portuguese. It&#39;s a great guide to the framework, dealing with details that many users tend to ignore.'
            ],
            [
                    title : 'Grails Goodness Notebook',
                    author: 'Hubert A. Klein Ikkink',
                    href  : 'https://leanpub.com/grails-goodness-notebook',
                    about : 'Experience the Grails framework through code snippets. Discover (hidden) Grails features through code examples and short articles. The articles and code will get you started quickly and provide deeper insight into Grails.'
            ],
            [
                    title : 'The Definitive Guide to Grails 2',
                    author: 'Jeff Scott Brown and Graeme Rocher',
                    href  : 'http://www.apress.com/9781430243779',
                    about : 'As the title states, this is the definitive reference on the Grails framework, authored by core members of the development team.'
            ],
            [
                    title : 'Grails in Action',
                    author: 'Glen Smith and Peter Ledbrook',
                    href  : 'http://www.manning.com/gsmith2/',
                    about : 'The second edition of Grails in Action is a comprehensive introduction to Grails 2 focused on helping you become super-productive fast.'
            ],
            [
                    title : 'Grails 2: A Quick-Start Guide',
                    author: 'Dave Klein and Ben Klein',
                    href  : 'http://www.amazon.com/gp/product/1937785777?tag=misa09-20',
                    about : 'This revised and updated edition shows you how to use Grails by iteratively building a unique, working application.'
            ],
            [
                    title : 'Programming Grails',
                    author: 'Burt Beckwith',
                    href  : 'http://shop.oreilly.com/product/0636920024750.do',
                    about : 'Dig deeper into Grails architecture and discover how this application framework works its magic.'
            ]
    ] as List< Map<String, String> >

    public final static List< Map<String, String> > GROOVY_BOOKS = [
            [
                    title: 'Making Java Groovy',
                    author: 'Ken Kousen',
                    href: 'http://www.manning.com/kousen/',
                    about: 'Make Java development easier by adding Groovy. Each chapter focuses on a task Java developers do, like building, testing, or working with databases or restful web services, and shows ways Groovy can make those tasks easier.'],
            [
                    title: 'Groovy in Action, 2nd Edition',
                    author: 'Dierk König, Guillaume Laforge, Paul King, Cédric Champeau, Hamlet D\'Arcy, Erik Pragt, and Jon Skeet',
                    href: 'http://www.manning.com/koenig2/',
                    about: 'This is the undisputed, definitive reference on the Groovy language, authored by core members of the development team.'],
            [
                    title: 'Groovy for Domain-Specific Languages',
                    author: 'Fergal Dearle',
                    href: 'http://www.packtpub.com/groovy-for-domain-specific-languages-dsl/book',
                    about: 'Learn how Groovy can help Java developers easily build domain-specific languages into their applications.'
            ],
            [
                    title: 'Groovy 2 Cookbook',
                    author: 'Andrey Adamovitch, Luciano Fiandeso',
                    href: 'http://www.packtpub.com/groovy-2-cookbook/book',
                    about: 'This book contains more than 90 recipes that use the powerful features of Groovy 2 to develop solutions to everyday programming challenges.'
            ],
            [
                    title: 'Programming Groovy 2',
                    author: 'Venkat Subramaniam',
                    href: 'http://pragprog.com/book/vslg2/programming-groovy-2',
                    about: 'This book helps experienced Java developers learn to use Groovy 2, from the basics of the language to its latest advances.'
            ],
    ] as List< Map<String, String> >

    BookDataService bookDataService

    def init = { servletContext ->
        for (Map<String, String> bookInfo : (GRAILS_BOOKS + GROOVY_BOOKS)) {
            bookDataService.save(bookInfo.title, bookInfo.author, bookInfo.about, bookInfo.href)
        }
    }
}

3.8 Create Service

Create BookSearchService.groovy which queries Elasticsearch with the help of the ElasticSearch Grails Plugin.

grails-app/services/demo/BookSearchService.groovy

package demo

import grails.plugins.elasticsearch.ElasticSearchService
import groovy.transform.CompileStatic

@CompileStatic
class BookSearchService {
    ElasticSearchService elasticSearchService

    Map search(String query) {
        elasticSearchService.search(query, [indices: Book, types: Book, score: true]) as Map (1)
    }
}

1	Search through an index for the specified search query.

The service returns a Map containing a total entry, representing the total number of hits found, searchResults entry, containing the hits and a scores entry, containing the hits scores.

Create a unit test to verify that the elasticSearchService is invoked once with the supplied query method argument.

src/test/groovy/demo/BookSearchServiceSpec.groovy

package demo

import grails.plugins.elasticsearch.ElasticSearchService
import grails.testing.services.ServiceUnitTest
import spock.lang.Specification

class BookSearchServiceSpec extends Specification implements ServiceUnitTest<BookSearchService> {

    def "search uses indices and types `Book` and score `true`"() {

        service.elasticSearchService = Mock(ElasticSearchService)

        when:
        service.search('Grails')

        then:
        1 * service.elasticSearchService.search('Grails', [indices: Book, types: Book, score: true])
    }
}

3.9 Create Controller

Implement a search action in BookController.groovy which will query Elasticsearch.

grails-app/controllers/demo/BookController.groovy

package demo

import grails.plugins.elasticsearch.ElasticSearchService
import grails.validation.ValidationException
import groovy.transform.CompileStatic

import static org.springframework.http.HttpStatus.*

@CompileStatic
class BookController {

    BookSearchService bookSearchService

    static responseFormats = ['json']
    static allowedMethods = [
            search: "GET"
    ]

    def search(String q) {
        if ( !q ) {
            render status: NOT_FOUND (1)
            return
        }
        respond bookSearchService.search(q)
    }
}

1	If no query string is passed within the request then simply return `Not Found(404)`.

Create a test which verifies the controller only responds to HTTP GET requests.

src/test/groovy/demo/BookControllerSpec.groovy

package demo

import grails.testing.web.controllers.ControllerUnitTest
import spock.lang.Specification
import spock.lang.Unroll
import static javax.servlet.http.HttpServletResponse.SC_METHOD_NOT_ALLOWED
import static javax.servlet.http.HttpServletResponse.SC_NOT_FOUND

class BookControllerSpec extends Specification implements ControllerUnitTest<BookController> {

    @Unroll
    def "test BookController.search does not accept #method requests"(String method) {
        when:
        request.method = method
        controller.search()

        then:
        response.status == SC_METHOD_NOT_ALLOWED

        where:
        method << ['PATCH', 'DELETE', 'POST', 'PUT']
    }

    def "test BookController.search accepts GET requests"() {
        when:
        request.method = 'GET'
        controller.search()

        then:
        response.status == SC_NOT_FOUND
    }
}

3.10 JSON View

Now create a view named search under /views/book/search.gson which will be used to display the resulting JSON.

grails-app/views/book/search.gson

model {
    Long total
    List searchResults
    Map scores
}

json {
    totalCount total
    results g.render(searchResults)
    scores g.render(scores)
}

3.11 Modify UrlMapping

Modify UrlMappings.groovy by adding the following to the end of the mappings block:

grails-app/controllers/demo/UrlMappings.groovy

        get "/book/search/$q?"(controller: 'book', action: "search")

Test the mapping

src/test/groovy/demo/UrlMappingsSpec.groovy

package demo

import grails.testing.web.UrlMappingsUnitTest
import spock.lang.Specification

class UrlMappingsSpec extends Specification implements UrlMappingsUnitTest<UrlMappings> {

    void setup() {
        mockController(BookController)
    }

    void "book/search endpoint mapping includes query term in path"() {
        expect:
        verifyForwardUrlMapping("/book/search/grails", controller: 'book', action: 'search') {
            q = 'grails'
        }
    }
}

3.12 Functional Test

Functional tests require the app to be running, and are designed to exercise the application almost as a user would, by making HTTP requests against it. These tend to be the most complex tests to write.

The testing framework used by Grails is Spock. Spock provides an expressive syntax for writing test cases, based on the Groovy language, and so is a great fit for Grails. It includes a JUnit runner, which means IDE support is effectively universal (any IDE that can run JUnit tests can run Spock tests).

Following is a functional test for the search action of BookController

src/integration-test/groovy/demo/RestSpec.groovy

package demo

import grails.plugins.rest.client.RequestCustomizer
import grails.plugins.rest.client.RestBuilder
import grails.plugins.rest.client.RestResponse
import spock.lang.Specification

class RestSpec extends Specification {

    RestBuilder rest = new RestBuilder()

    RestResponse get(String path, @DelegatesTo(RequestCustomizer) Closure customizer = null) {
        rest.get("http://localhost:${serverPort}${path}", customizer)
    }
}

src/integration-test/groovy/demo/BookControllerFunctionalSpec.groovy

package demo

import grails.plugins.rest.client.RestResponse
import grails.testing.mixin.integration.Integration
import spock.lang.IgnoreIf

@IgnoreIf( { System.getenv('TRAVIS') as boolean } )
@Integration
class BookControllerFunctionalSpec extends RestSpec {

    BookDataService bookDataService

    void "Test the search action correctly searches"() {
        expect:
        bookDataService.count()

        when:
        RestResponse rsp = get("/book/search/Beckwith")

        then: "The list is returned with only one instance"
        rsp.json.totalCount == 1
        rsp.json.results.first().author == 'Burt Beckwith'
    }
}

You will need to have an ElasticSearch instance running for this test to pass.

4 Summary

During this guide, we learned how to configure Elasticsearch Grails plugin. The plugin focus on exposing Grails domain classes for the moment. It highly takes the existing Searchable Plugin as reference for its syntax and behaviour.

Please, refer to the Grails Elasticsearch plugin’s documentation to learn more.

5 Help with Grails

Object Computing, Inc. (OCI) sponsored the creation of this Guide. A variety of consulting and support services are available.

OCI is Home to Grails

Meet the Team