Documentation is a Living System with Asciidoctor

In Agile methodology is very important to iterate over and over again walking small steps so all team (which I also include the customers), can receive quick feedback of how the project is going on. 

Aligned with this requirement, is where Continuous Integration/Continuous Delivery has sense. Every new commit, implies among a lot of other stages compile, execute unit/integration tests, execute code quality or execute acceptance tests. All these kind of tests generate some kind of documentation (usually an HTML file) as a report. To improve the visibility of the project to all team members, it is a good practice to publish these reports to a webserver so all members can access to it and inspect the result of that compilation and the progress of the project.

But with the advent of lightweight markup languages like Markdown or AsciiDoc, the documentation of the projects is (or should be) treated as source code too, so it should be delivered as any other part of the system like for example test reports, so that every member of the team should see exactly the progress of the documentation at the time of packaging. 

In this post we are going to see how to acquire this we are going to use Asciidoctor-Maven-Plugin and Maven-Assembly-Plugin, to create a war file containing your documentation which at the end could be uploaded to a server using for example Cargo-Maven-Plugin or in your Jenkins pipeline.

The first thing to do is create a project with source code, test code and of course documentation. 

Because it is a good idea to treat documentation as code (it would require another post to talk about it), we can put documentation inside src/main/resources, and create a document for each module inside the package of the module. So for example if in src/main/java/com/mycompany/users we have all code developed about user management, you will put documentation about user management in src/main/resources/com/mycompany/users, or if you prefer src/main/docs/com/mycompany/users, but as you can see the documentation should be very close to developer, in fact in same project workspace. 

You can see a full example in https://github.com/lordofthejars/foobank.

And now it is time to render documentation each time the build is done and package it into a war file.

To render the documentation we are going to use Asciidoctor-Maven-Plugin and Assembly-Maven-Plugin.

Let's see how to configure Asciidoctor-Maven-Plugin:

The plugin renders all AsciiDoc files that are at src/main/resources folder and its subfolders. They are rendered in HMTL format, and by default are rendered at target/generated-docs.

Now the we have all our documentation files rendered as HTML, it is time to package them inside a WAR file so we can deploy them into a web server using Assembly-Maven-Plugin.

The name of the output file is foobank because we are using finalBuild tag, and finally Assembly plugin requires a file called assembly.xml to configure which files are included in package and the output directory.

Note that we are adding a suffix to the output file by using id tag, so the final name will be foobank-doc.war. And finally we setup that all files inside target/generated-docs will be packaged.

And now you can deploy this WAR file in your documentation server, so every team on your project will be able to read the latest content generated at the same time of compiling, packaging or deploying your application. Your documentation is treated as code.

You can see a full example in https://github.com/lordofthejars/foobank.

We keep learning,
Alex.

Desolation comes upon the sky, Now I see fire, Inside the mountain, I see fire (I See Fire - Ed Sheeran)

Music: http://www.youtube.com/watch?v=DzD12qo1knM

Asciidoctor Meets JBehave.

User story is one or more sentences in natural language or business language which captures the requirements of what end user wants to be implemented in the product we are developing.

User Stories typically have the form:

"In order to <receive benefit> as a <role>, I want <goal/desire>"

User stories are also documentation for our project, first of all because they express what the product should do, and second one because they will be used by our clients/stakeholders, to validate that we are building the right thing.

JBehave is a framework for Behaviour-Driven Development (BDD). BDD is an evolution of test-driven development (TDD) and acceptance-test driven design, and is intended to make these practices more accessible and intuitive to newcomers and experts alike. The core of JBehave from the point of view of end users are user story files, where we define stories that our product should meet to validate that we are aligned with what our client wants. A user story example in JBehave is:

Note that first sentence is a description, then a narrative section which contains sentences of the form in order to, as a, I want. And finally the definition of scenarios that must meet this story to be considered completed.

So as you can see, story files are project documentation, and as project documentation would be awesome if we could add them inside our documents as documentation and not as simple plain text. And here is where Asciidoctor comes to action. Asciidoctor is a Ruby, Java and Javascript processor for converting AsciiDoc source files and strings into HTML 5, DocBook 4.5 and other formats. Your project may contain different kind of documents written in AsciiDoc like Software Design Specifications (SDS), manuals, or for example a test plan. And it is in test plan where including user stories, could be a really good improvement. And for this reason jbehave-asciidoctorj-extension has been developed.

Let's see an example. Suppose that we are writing a document where we want to test some user stories manually (apart of automatically of course) due to the criticality of the process:

As you can see we have created a simple AsciiDoc document with some information. But the interesting part is include::jbehave::userStories:src/**/*.story[initialLevel=1, manual=true]. Note that we are including all jbehave userstories that are placed on src folder. This extension supports any valid glob expression. And at the end we can set two attributes, the first one is the initial level of the section where user stories will be rendered (remember that the first one is 0), and finally if the steps will be executed manually or not, which has a deep impact on final rendered document as we will see soon.

Our project has two story files:

As you can see the second story has a dependency to the first story, which means that this second story is only valid if first story has passed successfully. Apart from this fact, we can set examples as tables as well.

So let's see an screenshot of previous document rendered:

I think it is important to note two blocks. The first one is the NOTE block. See that GivenStories are rendered as Admonition, and with a link pointing to another story; this is pretty important because clicking there it will move the page where that (precondition) story was defined, so you can also inspect what was that story about. The other interesting part is the Result part. Because we have set user stories as manual, the extension will render three checkboxes so tester can mark if the scenario passes, fails or also if he wants to add some comment.

So now there is no excuse to not adding user stories in your documents in a really fashion way.

You can see the code of this extension at: https://github.com/lordofthejars/asciidoctorj-extensions/tree/master/jbehave-asciidoctorj-extensions

We keep learning,

Alex.

Maman dit Annotate"travailler c'est bien", Bien mieux qu'être mal accompagné, Pas vrai ? (Papaoutai - Stromae)

Music: http://www.youtube.com/watch?v=oiKj0Z_Xnjc

Are Configuration Files Documentation? ... Yes!!! TomEE + Asciidoctor Example

Normally in our projects we generate a considerable amount of documentation. Some documentation is directly a document written as is, for example Software Requirements Specifications (SRS) or Software Design Specifications (SDS), but there are other pieces of our projects that although they are not a document directly, they are consulted by developers/testers/managers, customers ... and treat it as is, for example user stories written by developers and/or QA in Cucumber or JBehave format or  javadoc. And now I ask to myself, are configuration files documentation too? Are people interested in having them in a document and treat them as documentation? And the answer is YES!!!. In this post we are going to see how to transform TomEE's configuration file into a document using an Asciidoctor extension I have developed specially for this occasion.

In TomEE, resources and containers can be configured inside a file called tomee.xml among other places. But what really makes interesting is that a subset of this file can be put inside the project, so each web application is responsible of configuring their own resources, for example DataSources, JMS Queues, ....  And having the ability of putting specific configuration of the server inside the application also opens the doors of creating more than one configuration file per application, and load one or another depending on the environment we are going to deploy the application. For example:

• In developer environment the DataSource could point to an embedded database.
• In QA environment the DataSource could point to a database installed on localhost.
• In Stage environment the same DataSource points to an external IP1.
• In Production environment the same DataSource points to an external IP2.

So as you can see, it would be interesting to have all these information as readable documentation, so everyone involved in the project could review it and for example know exactly which servers are being used in each environment as datasources, or configuration parameters like pool size of stateless manager.

And this is exactly what tomee-asciidoctor-extension does. It allows us to embed Apache TomEE configuration files into an AsciiDoc documentation.

Let's see an example of Apache TomEE configuration file:

This is a typical configuration file where for example you can see some DataSources defined, custom resource or definition of TransactionManager.

Probably your projects will have a document called for example infrastructure (or something similar) where all information related with the software that is being used in the project and  environments used will be described.

A simple example of these kind of documents can be:

Note that besides describing the software that is going to be used for solving our problem, in this case an Apache TomEE, we are using the tomee-asciidoctor-extension to add the Apache TomEE's container and resources configuration file, so when someone wants to inspect the documentation will also see exactly the configuration values that had Apache TomEE at time of generating the document. If you look carefully you only need to add next line to transform a Apache TomEE configuration file into documentation:

include::tomee:/home/alex/git/asciidoctorj-extensions/tomee-asciidoctorj-extensions/src/test/resources/resources.xml[initialLevel=2]

In this case we are including a file called resources.xml. Note that in order to extension detects an Apache TomEE configuration file, we need to use the special keyword tomee: followed by the location of the file. Also note that we are using an attribute called initialLevel. Because we are embedding one document into another one, we need a way to set at which level the document will be rendered, and this is what this attribute is designed for.

Previous document rendered with default style sheet looks like:

See that thanks of Asciidoctor and its extension system, we are able to convert configuration files to documentation.

You can see the code of this extension at: https://github.com/lordofthejars/asciidoctorj-extensions/tree/master/tomee-asciidoctorj-extensions

We keep learning,
Alex.

Siente y baila y goza, Que la vida es una sola, Voy a reír, voy a bailar, Vive, sigue, Siempre pa'lante (Vivir Mi Mida - Marc Anthony)

Music: http://www.youtube.com/watch?v=YXnjy5YlDwk

Is there an Asciidoctor on board ?

On October 21, 2013 Dan Allen and I were interviewed in JBoss Asylum by Emmanuel Bernard and Max Rydahl Andersen about Asciidoctor.

It was an honor to share this talk with all of them, and you can listen the whole podcast at http://jbosscommunityasylum.libsyn.com/podcast-32-is-there-an-asciidoctor-on-board and listen why AsciiDoc and Asciidoctor rocks.

Tomcat y Java EE con TomEE {y mucho más}

Aquí podéis ver la presentación que hice sobre Apache TomEE en los JavaDays 2013 en la GuateJug (La comunidad Java en Guatemala).

Tomcat y Java EE con TomEE {y mucho más} from Alex Soto

También podéis visualizar el video en el canal de YouTube de la comunidad GuateJug.

Dar las gracias a David por pensar en mi como su sustituto, y ofrecerme la oportunidad de presentar Apache TomEE a la comunidad Java de Guatemala. Un placer, y un abrazo muy fuerte a la comunidad GuateJug, por tratarme tan bién.

Resistance is futile mocks will be assimilated

My slides of my recently talk about testing at Codemotion Madrid. The summary of the talk was:

Everybody is unit testing, everybody is mocking. Everybody mocks database access, software dependencies, external systems, container services, ... to test what? You end up testing a bit of business logic and writing a lot of test code, when in production this code will run within a container under different circumstances. In this talk we are going to see how some tools like Arquillian, Byteman, NoSQLUnit and other programs, may help us writing real tests in a Java EE environment.

Resistance is futile mocks will be assimilated from Alex Soto

I would like to say thank you very much to Laura and all the team to arrange such a great conferences. Hope to see you in next Codemotion.

NoSQLUnit 0.7.7 Released

NoSQLUnit is a JUnit extension to make writing unit and integration tests of systems that use NoSQL backend easier. Visit official page for more information.

In 0.7.7 release, next changes has been added:

• Allow for array types in GraphML (https://github.com/lordofthejars/nosql-unit/issues/82)
• Enhancing manual index support in GraphML (https://github.com/lordofthejars/nosql-unit/pull/81)
• Bug Issue #79 (https://github.com/lordofthejars/nosql-unit/issues/79)
• Updated Cassandra-Unit to 1.1.2.1 and Cassandra to 1.2.0 (https://github.com/lordofthejars/nosql-unit/pull/76)

I would like to say thank you to javahelp, ochrons, Jdourd, JordiAranda and anton-kostyliev for their help.

We keep learning,
Alex.

I belong with you, You belong with me, You're my sweetheart (Ho Hey - The Lumineers)

Music: http://www.youtube.com/watch?v=zvCBSSwgtg4

AsciiDoc Editor with MarkItUp

In this post I am going to show you how to use MarkItUp jquery library to create a markup editor, concretely an editor for AsciiDoc format.  Apart from MarkItUp library we are going to use asciidoctor.js library for rendering purposes.

MarkItUp! is a JavaScript plugin built on the jQuery library. It allows you to turn any textarea into a markup editor. And in next screencast I am going to show you how to use it.

We keep learning,
Alex.

Your love is blind, blind as a bat, The way that you're leading me home like that, Your love is blind, blind as bat (Blind as a Bat - Meatloaf)

Code: https://github.com/lordofthejars/asciidoctor-markitup
Music: http://www.youtube.com/watch?v=EpzeSSeOvw0

Searchable documents? Yes You Can. Another reason to choose AsciiDoc

Elasticsearch is a flexible and powerful open source, distributed real-time search and analytics engine for the cloud based on Apache Lucene which provides full text search capabilities. It is document oriented and schema free.

Asciidoctor is a pure Ruby processor for converting AsciiDoc source files and strings into HTML 5, DocBook 4.5 and other formats. Apart of Asciidoctor Ruby part, there is an Asciidoctor-java-integration project which let us call Asciidoctor functions from Java without noticing that Ruby code is being executed.

In this post we are going to see how we can use Elasticsearch over AsciiDoc documents to make them searchable by their header information or by their content.

Let's add required dependencies:

Lambdaj library is used to convert AsciiDoc files to a json documents.

Now we can start an Elasticsearch instance which in our case it is going to be an embedded instance.

Next step is parse AsciiDoc document header, read its content and convert them into a json document.

An example of json document stored in Elasticsearch can be:

And for converting an AsciiDoc File to a json document we are going to use XContentBuilder class which is provided by Elasticsearch Java API to create json documents programmatically.

Basically we are building the json document by calling startObject methods to start a new object, field method to add new fields, and startArray to start an array. Then this builder will be used to render the equivalent object in json format. Notice that we are using readDocumentHeader method from Asciidoctor class which returns header attributes from AsciiDoc file without reading and rendering the whole document. And finally content field is set with all document content.

And now we are ready to start indexing documents. Note that populateData method receives as parameter  a Client object. This object is from Elasticsearch Java API and represents a connection to Elasticsearch database.

It is important to note that the first part of the algorithm is converting all our AsciiDoc files (in our case two) to XContentBuilder instances by using previous converter class and the method convert of Lambdaj project.

If you want you can take a look to both documents used in this example in https://github.com/asciidoctor/asciidoctor.github.com/blob/develop/news/asciidoctor-java-integration-0-1-3-released.adoc and https://github.com/asciidoctor/asciidoctor.github.com/blob/develop/news/asciidoctor-maven-plugin-0-1-2-released.adoc.

Next part is inserting documents inside one index. This is done by using prepareIndex method, which requires an index name (docs), an index type (asciidoctor), and the id of the document being inserted. Then we call setSource method which transforms the XContentBuilder object to json, and finally by calling execute().actionGet(), data is sent to database.

The final step is only required because we are using an embedded instance of Elasticsearch (in production this part should not be required), which refresh the indexes by calling refresh method.

After that point we can start querying Elasticsearch for retrieving information from our AsciiDoc documents.

Let's start with very simple example, which returns all documents inserted:

Next we are going to search for all documents that has been written by Alex Soto which in our case is one.

Note that I am searching for field author the string Alex Soto, which returns only one. The other document is written by Jason. But it is interesting to say that if you search for Alexander Soto, the same document will be returned; Elasticsearch is smart enough to know that Alex and Alexander are very similar names so it returns the document too.

More queries, how about finding documents written by someone who is called Alex, but not Soto.

And of course no results are returned in this case. See that in this case we are using a field query instead of a term query, and we use +, and - symbols to exclude and include words.

Also you can find all documents which contains the word released on title.

And finally let's find all documents that talks about 0.1.2 release, in this case only one document talks about it, the other one talks about 0.1.3.

Now we only have to send the query to Elasticsearch database, which is done by using prepareSearch method.


Note that in this case we are printing the AsciiDoc content through console, but you could use asciidoctor.render(String content, Options options) method to render the content into required format.

So in this post we have seen how to index documents using Elasticsearch, how to get some important information from AsciiDoc files using Asciidoctor-java-integration project, and finally how to execute some queries to inserted documents. Of course there are more kind of queries in Elasticsearch, but the intend of this post wasn't to explore all possibilities of Elasticsearch.

Also as corollary, note how important it is using AsciiDoc format for writing your documents. Without much effort you can build a search engine for your documentation. On the other side, imagine all code that would be required to implement the same using any proprietary binary format like Microsoft Word. So we have shown another reason to use AsciiDoc instead of other formats.

We keep learning,

Alex.

I work hard (he works hard) every day of my life, I work till I ache in my bones, At the end (at the end of the day), I take home my hard earned pay all on my own (Somebody To Love - Queen)

Music: http://www.youtube.com/watch?v=2pMM4iwC-ag

Download code

Implementing Timeouts with FutureTask

Sometimes when you are writing some code that needs communication with external systems, not necessarily by using a socket, for example it can be using a serial port, we may need a way to implement a timeout algorithm, so if after some specified time, the request does not return a result, a timeout error should be thrown so user can act in consequence, and not wait indefinitely for a result that maybe will never come.

If the API being used contains a read method with timeout, then the problem is easily fixed by catching the exception, but if not, then you must implement yourself the timeout logic. To do that we can use FutureTask class.

Basically what we are doing is creating a FutureTask object, and creating a Callable interface which executes the call method which requires a timeout. Then we create a thread to execute it. And finally we call get method, which waits until the result is calculated, or if the timeout exceeds (in this case 5 seconds), TimeoutException is thrown.

So we have seen a really easy way to implement a timeout feature for our application.

We keep learning,

Alex

When the garden flowers baby are dead yes, And your mind [, your mind] is [so] full of RED, Don't you want somebody to love, Don't you need somebody to love (Somebody to love - Jefferson Airplane)

Music: http://www.youtube.com/watch?v=5Jj3wZVc7nw

Git Bisect in Action

Today it is time for a screencast where I show an amazing git command, called git bisect. Git bisect command command uses git to find in which commit a bug was introduced. After that, I am sure you will wonder where it's been you whole career. Thanks Jason for showing me the way.

You can watch it in HD to see clearly each command.

We keep learning,
Alex.

Siempre quise ir a L.A. Dejar un día esta ciudad. Cruzar el mar en tu compañía.  (Cadillac Solitario - Loquillo)

Music: http://www.youtube.com/watch?v=kg0Jlpgzel4

NoSQLUnit 0.7.6 Released

NoSQLUnit is a JUnit extension to make writing unit and integration tests of systems that use NoSQL backend easier. Visit official page for more information.

In 0.7.6 release, next changes has been implemented:

• FongoDB version has been updated to 1.0.7. Issue #73 (https://github.com/lordofthejars/nosql-unit/issues/73)
• Update logging dependencies. Issues #71 and #69. Thanks sslavic.
• Update pom configuration. Issue #70. (https://github.com/lordofthejars/nosql-unit/pull/70). Thanks sslavic.
• Now you can add options (like options, geospatial stuff, ...)  in MongoDB indexes. Issue #67. (https://github.com/lordofthejars/nosql-unit/issues/67). See an example for creating unique indexes here.
• Improved the way for testing Redis replication. Issue #62 (https://github.com/lordofthejars/nosql-unit/issues/62). See an example of how to define replication scenario here.
• All Redis tests has been passed using Redis 2.6.x. Redis 2.4.x is deprecated and will be removed  its support at then end of 2013.  

We keep learning,
Alex.

De dia vivire pensando en tu sonrisa, De noche las estrellas me acompañaran, Seras como un luz que alumbra en mi destino, Me voy pero te juro que mañana volvere (Un Beso Y Una Flor - Niño Bravo)

Music: http://www.youtube.com/watch?v=Q3UqusEWGeo

Enjoy the magic of asciidoctor in java with asciidoctor-java-integration

The asciidoctor-java-integration is the official means of using Asciidoctor to render all your AsciiDoc documentation using Java instead of Ruby.

Installation

Since asciidoctor-java-integration is a standard jar file, the only thing you should do is add library into classpath.

Dependency declaration in Maven

... <dependencies> <dependency> <groupId>org.asciidoctor</groupId> <artifactId>asciidoctor-java-integration</artifactId> <version>${asciidoctor.version}</version> <1> ... </dependency> </dependencies> ...

1. As this library tracks the version of asciidoctor, you can use which every version of asciidoctor you prefer.

Usage

The core interface of asciidoctor-java-integration is Asciidoctor interface. It provides two methods for rendering asciidoc content, render and renderFile. Both of them returns a string with rendered content.

Also a factory method is provided to create an instance of Asciidoctor interface.

Creation of Asciidoctor interface

import static org.asciidoctor.Asciidoctor.Factory.create; import org.asciidoctor.Asciidoctor; ... Asciidoctor asciidoctor = create(); ...

And then we can call render methods depending on our requirements.

Rendering a String

... String rendered = asciidoctor.render("*This* is it.", Collections.EMPTY_MAP); System.out.println(rendered); ...

But also you can render the content of a file.

Rendering a File

... String rendered = asciidoctor.renderFile("target/test-classes/rendersample.asciidoc", Collections.EMPTY_MAP); System.out.println(rendered); ...

Options

Asciidoctor supports different kind of options, like in_place which renders the output inside a file, template_dir used to provide a directory of Tilt-compatible templates to be used instead of the default built-in templates, or for example attributes option where we can set key-value pairs of attributes that will be used within asciidoc document.

The second parameter of render methods are a java.util.Map where we can set all these options.

Example of using in_place Option and backend Attribute

Map<String, Object> attributes = new HashMap<String, Object>(); attributes.put("backend", "docbook"); Map<String, Object> options = new HashMap<String, Object>(); options.put("in_place", true); options.put("attributes", attributes); String render = asciidoctor.renderFile("target/test-classes/rendersample.asciidoc", options);

See that in previous example we have created a Map, where we have put the options and attributes (creating a Map too) required to render input as docbook and generate an output file.

But asciidoctor-java-integration also provides two builder classes to create these maps in a more readable form.

AttributesBuilder is provided for creating a map with required attributes set, and OptionsBuilder can be used for options. Previous example but using these classes looks like:

Example setting attributes and options

import static org.asciidoctor.AttributesBuilder.attributes; import static org.asciidoctor.OptionsBuilder.options; ... Map<String, Object> attributes = attributes().backend("docbook").asMap(); Map<String, Object> options = options().inPlace(true).attributes(attributes).asMap(); String render = asciidoctor.renderFile("target/test-classes/rendersample.asciidoc", options); ...

Utilities

A utility class for searching all asciidoc files present in a root folder and all its subfolders is given. In fact it finds all files that end up with .asc, .asciidoc, .ad or .adoc. This class is AsciidocDirectoryWalker.

Example of finding all asciidoc

DirectoryWalker directoryWalker = new AsciidocDirectoryWalker("target/test-classes/src"); List<File> asciidocFiles = directoryWalker.scan();

We keep learning,
Alex.

Cold, cold heart, Hard done by you, Some things look better, baby, Just passing through (Sacrifice - Elton John)

http://www.youtube.com/watch?v=NrLkTZrPZA4

Installing TomEE from Puppet

Apache TomEE is an all-Apache stack aimed at Java EE 6 Web Profile certification where Tomcat is top dog. It is the conjunction of Tomcat + Java EE. 

Puppet is a tool designed to manage the configuration of our systems declaratively. We only have to describe system resources and their state. This description is stored in the core-files of Puppet, which are called Puppet manifests.

In current post we are going to see how to define TomEE as Puppet resource, so it can be installed automatically in all computers that are managed by Puppet. I am using Ubuntu 12.04, but of course this could be adapted to your system.

Because TomEE is written in Java, we obviously need to ensure that a JDK is present in our system, for our example we are going to use OpenJDK 1.6. The way to install a package to our system is using Package resource, which uses underlying package manager to find, download and install it.

Let's create an init.pp manifest file and the first thing we are going to do is create an exec task which updates the package manager with a list of last packages available.

Then we can define a class that is in charge of installing the OpenJDK. A class in Puppet can be understood as a collection of resources that will be seen by Puppet as a unit.

It is pretty intuitive I guess, first of all we are ensuring that openjdk-6-jdk package is already present in system. If it is not installed, then Exec["update-package-list"] resource is executed, and finally the package manager installs OpenJDK into the system. 

After executing this part we can run java -version or javac -version without any problem, OpenJDK is there.

Next step is installing TomEE. Probably TomEE package is not in your distribution package repository in software package format (for example in case of Debian .deb). For this reason we need a different approach to the one followed by OpenJDK. We are going to download tar.gz file from TomEE site and uncompress it to an installation directory.

As with OpenJDK we are creating a class called tomee. First we are creating the directory where TomEE will be installed (tar command requires that destination directory should be already created). 

Then we are downloading TomEE from Apache site using wget command, to finally uncompressing it to already created directory.

Note the use of ->. Puppet does not guarantee the order of execution of defined resources, so you cannot infer "a priori" if TomEE will be downloaded first and then installed or vice versa. To define an order we are using -> operator which sets a priority between resources.

Now we have got Apache TomEE installed on computer, but obviously is not started and stopped automatically, you must execute /opt/tomee-1.5.1/bin/startup.sh to have TomEE available. Let's change this by using service resource. As its name suggests it registers as service an installed server. So inside tomee class define next service resource:

Keep an eye in two points of previous declaration, the first one is the provider/start/stop declarations, provider is set to init, which means we want to use the standard init-style service management, but for example you could use launchd (case of Mac OS X), upstart, windows (for Windows machines), ... see [http://docs.puppetlabs.com/references/latest/type.html#service] for more information. And because TomEE is not implemented as a Linux service by default (basically because has not been installed from a native package like .deb), we need to specify which command should be executed to start and stop TomEE. 

The second point is that service resource requires to have TomEE unpacked and OpenJDK installed, for this reason require attribute contains two declarations.

See full script here.

Final notes about example:

• In Puppet, creates attribute inside exec task, is used to know if the resource should be executed or not (case that the file exists, exec will not be executed). In our case we are downloading TomEE in tmp directory. Most OS removes periodically this directory so it is a bad location for downloading it, but for this tutorial it works perfectly because I could re-execute the script every time as a new execution.
• For simplifications we have add all content inside a single file, in your enterprise I suggest creating a TomEE module so you could share across all your projects.
• TomEE version should be set as a variable/parameter/hiera so same class can be reused when new version of TomEE is released. 

We keep learning,

Alex.

Ven al desfile de sirenas, de cuerpos bien cebados, sube al séptimo cielo (Ellos Las Prefieren Gordas - Orquesta Mondragon)

Music:  http://www.youtube.com/watch?v=_Z_LTvQcw38

Testing Spring Data Neo4j Applications with NoSQLUnit

Spring Data Neo4j

Spring Data Neo4j is the project within Spring Data project which provides an extension to the Spring programming model for writing applications that uses Neo4j as graph database.

To write tests using NoSQLUnit for Spring Data Neo4j applications, you do need nothing special apart from considering that Spring Data Neo4j uses a special property called type in graph nodes and relationships which stores the fully qualified classname of that entity.

Apart from type property at node/relationship level, we also need to create one index for nodes and one index for relationships. In case of nodes, types index name is required, meanwhile rel_types is required for relationships. In both cases we must set key value to className and value to full qualified classname.

Type mapping IndexingNodeTypeRepresentationStrategy and IndexingRelationshipTypeRepresentationStrategy are used as default type mapping implementation, but you can also use SubReferenceNodeTypeRepresentationStrategy which stores entity types in a tree in the graph representing the type and interface hierarchy, or you can customize even more by implementing NodeTypeRepresentationStrategy interface.

Hands on Work

Application

Starfleet has asked us to develop an application for storing all starfleet members, with their relationship with other starfleet members, and the ship where they serve.

The best way to implement this requirement is using Neo4j database as backend system. Moreover Spring Data Neo4j is used at persistence layer.

This application is modelized into two Java classes, one for members and another one for starships. Note that for this example there are no properties in relationships, so only nodes are modelized.

Member class

@NodeEntity public class Member { private static final String COMMANDS = "COMMANDS"; @GraphId Long nodeId; private String name; private Starship assignedStarship; public Member() { super(); } public Member(String name) { this.name = name; } @Fetch @RelatedTo(type=COMMANDS, direction=Direction.OUTGOING) private Set<Member> commands; public void command(Member member) { this.commands.add(member); } public Set<Member> commands() { return this.commands; } public Starship getAssignedStarship() { return assignedStarship; } public String getName() { return name; } public void assignedIn(Starship starship) { this.assignedStarship = starship; } //Equals and Hash methods }

Starship class

@NodeEntity public class Starship { private static final String ASSIGNED = "assignedStarship"; @GraphId Long nodeId; private String starship; public Starship() { super(); } public Starship(String starship) { this.starship = starship; } @RelatedTo(type = ASSIGNED, direction=Direction.INCOMING) private Set<Member> crew; public String getStarship() { return starship; } public void setStarship(String starship) { this.starship = starship; } //Equals and Hash methods }

Apart from model classes, we also need two repositories for implementing CRUD operations, and spring context xml file. Spring Data Neo4j uses Spring Data Commons infrastructure allowing us to create interface based compositions of repositories, providing default implementations for certain operations.

MemberRepository class

public interface MemberRepository extends GraphRepository<Member>, RelationshipOperationsRepository<Member> { Member findByName(String name); }

See that apart from operations provided by GrapRepository interface like save, findAll, findById, … we are defining one query method too called findByName. Spring Data Neo4j repositories (and most of Spring Data projects) provide a mechanism to define queries using the known Ruby on Rails approach for defining finder queries.

StarshipRepository class

public interface StarshipRepository extends GraphRepository<Starship>, RelationshipOperationsRepository<Starship> { }

application-context file

<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context" xmlns:neo4j="http://www.springframework.org/schema/data/neo4j" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.1.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.1.xsd http://www.springframework.org/schema/data/neo4j http://www.springframework.org/schema/data/neo4j/spring-neo4j.xsd"> <context:component-scan base-package="com.lordofthejars.nosqlunit.springdata.neo4j"/> <context:annotation-config/> <neo4j:repositories base-package="com.lordofthejars.nosqlunit.springdata.repository"/> </beans>

Testing

Unit Testing

As it has been told previously, for writing datasets for Spring Data Neo4j, we don’t have to do anything special beyond creating node and relationship properties correctly and defining the required indexes. Let’s see the dataset used to test the findByName method by seeding Neo4j database.

star-trek-TNG-dataset.xml file

<?xml version="1.0" ?> <graphml xmlns="http://graphml.graphdrawing.org/xmlns" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://graphml.graphdrawing.org/xmlns http://graphml.graphdrawing.org/xmlns/1.0/graphml.xsd"> <key id="name" for="node" attr.name="name" attr.type="string"></key> <key id="__type__" for="node" attr.name="__type__" attr.type="string"></key> <key id="starship" for="node" attr.name="starship" attr.type="string"></key> <graph id="G" edgedefault="directed"> <node id="3"> <data key="__type__">com.lordofthejars.nosqlunit.springdata.neo4j.Member</data> <data key="name">Jean-Luc Picard</data> <index name="__types__" key="className">com.lordofthejars.nosqlunit.springdata.neo4j.Member</index> </node> <node id="1"> <data key="__type__">com.lordofthejars.nosqlunit.springdata.neo4j.Member</data> <data key="name">William Riker</data> <index name="__types__" key="className">com.lordofthejars.nosqlunit.springdata.neo4j.Member</index> </node> <node id="4"> <data key="__type__">com.lordofthejars.nosqlunit.springdata.neo4j.Starship</data> <data key="starship">NCC-1701-E</data> <index name="__types__" key="className">com.lordofthejars.nosqlunit.springdata.neo4j.Starship</index> </node> <edge id="11" source="3" target="4" label="assignedStarship"></edge> <edge id="12" source="1" target="4" label="assignedStarship"></edge> <edge id="13" source="3" target="1" label="COMMANDS"></edge> </graph> </graphml>

See that each node has at least one type property with full qualified classname and an index with name types, key className and full qualified classname as value.

Next step is configuring application context for unit tests.

application-context-embedded-neo4j.xml

<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context" xmlns:neo4j="http://www.springframework.org/schema/data/neo4j" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.1.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.1.xsd http://www.springframework.org/schema/data/neo4j http://www.springframework.org/schema/data/neo4j/spring-neo4j.xsd"> <import resource="classpath:com/lordofthejars/nosqlunit/springdata/neo4j/application-context.xml"/> <neo4j:config storeDirectory="target/config-test"/> </beans>

Notice that we are using Neo4j namespace for instantiating an embedded Neo4j database.

And now we can write the JUnit test case:

WhenInformationAboutAMemberIsRequired

@RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration("application-context-embedded-neo4j.xml") public class WhenInformationAboutAMemberIsRequired { @Autowired private MemberRepository memberRepository; @Autowired private StarshipRepository starshipRepository; @Autowired private ApplicationContext applicationContext; @Rule public Neo4jRule neo4jRule = newNeo4jRule() .defaultSpringGraphDatabaseServiceNeo4j(); @Test @UsingDataSet(locations = "star-trek-TNG-dataset.xml", loadStrategy = LoadStrategyEnum.CLEAN_INSERT) public void information_about_starship_where_serves_and_members_under_his_service_should_be_retrieved() { Member jeanLuc = memberRepository.findByName("Jean-Luc Picard"); assertThat(jeanLuc, is(createMember("Jean-Luc Picard"))); assertThat(jeanLuc.commands(), containsInAnyOrder(createMember("William Riker"))); Starship starship = starshipRepository.findOne(jeanLuc.getAssignedStarship().nodeId); assertThat(starship, is(createStarship("NCC-1701-E"))); } private Object createStarship(String starship) { return new Starship(starship); } private static Member createMember(String memberName) { return new Member(memberName); } }

There are some important points in the previous class to take under consideration:

1. Recall that we need to use Spring ApplicationContext object to retrieve embedded Neo4j instance defined into Spring application context.
2. Since lifecycle of database is managed by Spring Data container, there is no need to define any NoSQLUnit lifecycle manager.

Integration Test

Unit tests are usually run against embedded in-memory instances, but in production environment you may require access to external Neo4j servers by using Rest connection, or in case of Spring Data Neo4j instantiating SpringRestGraphDatabase class. You need to write tests to validate that your application still works when you integrate your code with a remote server, and this tests are typically known as integration tests.

To write integration tests for our application is as easy as defining an application context with SpringRestGraphDatabase and allow NoSQLUnit to control the lifecycle of Neo4j database.

.application-context-managed-neo4j.xml

<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context" xmlns:neo4j="http://www.springframework.org/schema/data/neo4j" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.1.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.1.xsd http://www.springframework.org/schema/data/neo4j http://www.springframework.org/schema/data/neo4j/spring-neo4j.xsd"> <import resource="classpath:com/lordofthejars/nosqlunit/springdata/neo4j/application-context.xml"/> <bean id="graphDatabaseService" class="org.springframework.data.neo4j.rest.SpringRestGraphDatabase"> <constructor-arg index="0" value="http://localhost:7474/db/data"></constructor-arg> </bean> <neo4j:config graphDatabaseService="graphDatabaseService"/> </beans>

Note how instead of registering an embedded instance, we are configuring SpringRestGraphDatabase class to connect to localhost server. And let’s implement an integration test which verifies that all starships can be retrieved from Neo4j server.

WhenInformationAboutAMemberIsRequired

@RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration("application-context-managed-neo4j.xml") public class WhenInformationAboutStarshipsAreRequired { @ClassRule public static ManagedNeoServer managedNeoServer = newManagedNeo4jServerRule() .neo4jPath( "/Users/alexsotobueno/Applications/neo4j-community-1.7.2") .build(); @Autowired private StarshipRepository starshipRepository; @Autowired private ApplicationContext applicationContext; @Rule public Neo4jRule neo4jRule = newNeo4jRule() .defaultSpringGraphDatabaseServiceNeo4j(); @Test @UsingDataSet(locations = "star-trek-TNG-dataset.xml", loadStrategy = LoadStrategyEnum.CLEAN_INSERT) public void information_about_starship_where_serves_and_members_under_his_service_should_be_retrieved() { EndResult<Starship> allStarship = starshipRepository.findAll(); assertThat(allStarship, containsInAnyOrder(createStarship("NCC-1701-E"))); } private Object createStarship(String starship) { return new Starship(starship); } }

Because defaultSpringGraphDatabaseServiceNeo4j method returns a GraphDatabaseService instance defined into application context, in our case it will return the defined SpringRestGraphDatabase instance.

Conclusions

There is not much difference between writing tests for none Spring Data Neo4j applications than the ones they use it. Only keep in mind to define correctly the type property and create required indexes.

Also see that from the point of view of NoSQLUnit there is no difference between writing unit or integration tests, apart of lifecycle management of the database engine.

Download Code

We keep learning,
Alex.

Gonna rise up, Burning black holes in dark memories, Gonna rise up, Turning mistakes into gold (Rise - Eddie Vedder)

Music: http://www.youtube.com/watch?v=uuVf4xiRsJQ