Apache Camel Tutorial – Introduction to EIP, Routes, Components, Testing, and other Concepts

Data exchanges between companies increase a lot. The number of applications, which must be integrated increases, too. The interfaces use different technologies, protocols and data formats. Nevertheless, the integration of these applications shall be modeled in a standardized way, realized efficiently and supported by automatic tests. Such a standard exists with the Enterprise Integration Patterns (EIP) [1], which have become the industry standard for describing, documenting and implementing integration problems. Apache Camel [2] implements the EIPs and offers a standardized, internal domain-specific language (DSL) [3] to integrate applications. This article gives an introduction to Apache Camel including several code examples.

 

Enterprise Integration Patterns

EIPs can be used to split integration problems into smaller pieces and model them using standardized graphics. Everybody can understand these models easily. Besides, there is no need to reinvent the wheel every time for each integration problem.

Using EIPs, Apache Camel closes a gap between modeling and implementation. There is almost a one-to-one relation between EIP models and the DSL of Apache Camel. This article explains the relation of EIPs and Apache Camel using an online shop example.

 

Use Case: Handling Orders in an Online Shop

The main concepts of Apache Camel are introduced by implementing a small use case. Starting your own project should be really easy after reading this article. The easiest way to get started is using a Maven archetype [4]. This way, you can rebuild the following example within minutes. Of course, you can also download the whole example at once[5].

Figure 1 shows the example from EIP perspective. The task is to process orders of an online shop. Orders arrive in csv format. At first, the orders have to be transformed to the internal format. Order items of each order must be split because the shop only sells dvds and cds. Other order items are forwarded to a partner.

Figure 1: EIP Perspective of the Integration Problem

 

This example shows the advantages of EIPs:  The integration problem is split into several small, perseverative subproblems. These subproblems are easy to understand and solved the same way each time. After describing the use case, we will now look at the basic concepts of Apache Camel.

 

Basic Concepts

Apache Camel runs on the Java Virtual Machine (JVM). Most components are realized in Java. Though, this is no requirement for new components. For instance, the camel-scala component is written in Scala. The Spring framework is used in some parts, e.g. for transaction support. However, Spring dependencies were reduced to a minimum in release 2.9 [6]. The core of Apache Camel is very small and just contains commonly used components (i.e. connectors to several technologies and APIs) such as Log, File, Mock or Timer.

Further components can be added easily due to the modular structure of Apache Camel., Maven is recommended for dependency management, because most technologies require additional libraries. Though, libraries can also be downloaded manually and added to the classpath, of course.

The core functionality of Apache Camel is its routing engine. It allocates messages based on the related routes. A route contains flow and integration logic. It is implemented using EIPs and a specific DSL. Each message contains a body, several headers and optional attachments. The messages are sent from a provider to a consumer. In between, the messages may be processed, e.g. filtered or transformed. Figure 1 shows how the messages can change within a route.

Messages between a provider and a consumer are managed by a message exchange container, which contains an unique message id, exception information, incoming and outgoing messages (i.e. request and response), and the used message exchange pattern (MEP). „In Only“ MEP is used for one-way messages such as JMS whereas „In Out“ MEP executes request-response communication such as a client side HTTP based request and its response from the server side.

After shortly explaining the basic concepts of Apache Camel, the following sections will give more details and code examples. Let’s begin with the architecture of Apache Camel.

 

Architecture

Figure 2 shows the architecture of Apache Camel. A CamelContext provides the runtime system. Inside, processors handle things in between endpoints like routing or transformation. Endpoints connect several technologies to be integrated. Apache Camel offers different DSLs to realize the integration problems.

Figure 2: Architecture of Apache Camel

 

CamelContext

The CamelContext is the runtime system of Apache Camel and connects its different concepts such as routes, components or endpoints. The following code snipped shows a Java main method, which starts the CamelContext and stops it after 30 seconds. Usually, the CamelContext is started when loading the application and stopped at shutdown.

 

public class CamelStarter {

public static void main(String[] args) throws Exception {

CamelContext context = new DefaultCamelContext();

context.addRoutes(new IntegrationRoute());

context.start();

Thread.sleep(30000);

context.stop();

}

}

 

The runtime system can be included anywhere in the JVM environment, including web container (e.g. Tomcat), JEE application server (e.g. IBM WebSphere AS), OSGi container, or even in the cloud.

 

Domain Specific Languages

DSLs facilitate the realization of complex projects by using a higher abstraction level. Apache Camel offers several different DSLs. Java, Groovy and Scala use object-oriented concepts and offer a specific method for most EIPs. On the other side, Spring XML DSL is based on the Spring framework and uses XML configuration. Besides, OSGi blueprint XML is available for OSGi integration.

Java DSL has best IDE support. Groovy and Scala DSL are similar to Java DSL, in addition they offer typical features of modern JVM languages such as concise code or closures. Contrary to these programming languages, Spring XML DSL requires a lot of XML. Besides, it offers very powerful Spring-based dependency injection mechanism and nice abstractions to simplify configurations (such as JDBC or JMS connections). The choice is purely a matter of taste in most use cases. Even a combination is possible. Many developer use Spring XML for configuration whilst routes are realized in Java, Groovy or Scala.

Routes

Routes are a crucial part of Apache Camel. The flow and logic of an integration is specified here. The following example shows a route using Java DSL:

public class IntegrationRoute extends RouteBuilder {

@Override

public void configure() throws Exception {

from(“file:target/inbox”)

                      .process(new LoggingProcessor())

                      .bean(new TransformationBean(),

                                  “makeUpperCase”)                                                  

.to(“file:target/outbox/dvd”);

}

}

 

The DSL is easy to use. Everybody should be able to understand the above example without even knowing Apache Camel. The route realizes a part of the described use case. Orders are put in a file directory from an external source. The orders are processed and finally moved to the target directory.

Routes have to extend the „RouteBuilder“ class and override the „configure“ method. The route itself begins with a „from“ endpoint and finishes at one or more „to“ endpoints. In between, all necessary process logic is implemented. Any number of routes can be implemented within one „configure“ method.

 

The following snippet shows the same route realized via Spring XML DSL:

 

<beans … >

 

<bean class=”mwea.TransformationBean” id=”transformationBean”/>

<bean class=”mwea.LoggingProcessor” id=”loggingProcessor”/>

 

<camelContext xmlns=”http://camel.apache.org/schema/spring”>

<package>mwea</package>

 

<route>

        <from uri=”file:target/inbox”/>

                    <process ref=”loggingProcessor”/>

                    <bean ref=”transformationBean”/>

        <to uri=”file:target/outbox”/>

    </route>

 

</camelContext>

 

</beans>

 

Besides routes, another important concept of Apache Camel is its components. They offer integration points for almost every technology.

 

Components

In the meantime, over 100 components are available. Besides widespread technologies such as HTTP, FTP, JMS or JDBC, many more technologies are supported, including cloud services from Amazon, Google, GoGrid, and others. New components are added in each release. Often, also the community builds new custom components because it is very easy.

The most amazing feature of Apache Camel is its uniformity. All components use the same syntax and concepts. Every integration and even its automatic unit tests look the same. Thus, complexity is reduced a lot. Consider changing the above example: If orders should be sent to a JMS queue instead of a file directory, just change the „to“ endpoint from „file:target/outbox“ to „jms:queue:orders“. That’s it! (JMS must be configured once within the application before, of course)

While components offer the interface to technologies, Processors and Beans can be used to add custom integration logic to a route.

 

Processors and Beans

Besides using EIPs, you have to add individual integration logic, often. This is very easy and again uses the same concepts always: Processors or Beans. Both were used in the route example above.

Processor is a simple Java interface with one single method: „process“. Inside this method, you can do whatever you need to solve your integration problem, e.g. transform the incoming message, call other services, and so on.

 

public class LoggingProcessor implements Processor {

@Override

public void process(Exchange exchange) throws Exception {

System.out.println(“Received Order: ” +

exchange.getIn().getBody(String.class));

}

}

 

The „exchange“ parameter contains the Messsage Exchange with the incoming message, the outgoing message, and other information. Due to implementing the Processor interface, you have got a dependency to the Camel API. This might be a problem sometimes. Maybe you already have got existing integration code which cannot be changed (i.e. you cannot implement the Processor interface)? In this case, you can use Beans, also called POJOs (Plain Old Java Object). You get the incoming message (which is the parameter of the method) and return an outgoing message, as shown in the following snipped:

 

public class TransformationBean {

public String makeUpperCase(String body) {

String transformedBody = body.toUpperCase();

return transformedBody;

}

}

 

The above bean receives a String, transforms it, and finally sends it to the next endpoint. Look at the route above again. The incoming message is a File. You may wonder why this works? Apache Camel offers another powerful feature: More than 150 automatic type converters are included from scratch, e.g. FileToString, CollectionToObject[] or URLtoInputStream. By the way: Further type converters can be created and added to the CamelContext easily [7].

If a Bean only contains one single method, it even can be omitted in the route. The above call therefore could also be .bean(new TransformationBean()) instead of .bean(new TransformationBean(), “makeUpperCase”).

 

Adding some more Enterprise Integration Patterns

The above route transforms incoming orders using the Translator EIP before processing them. Besides this transformation, some more work is required to realize the whole use case. Therefore, some more EIPs are used in the following example:

 

public class IntegrationRoute extends RouteBuilder {

@Override

public void configure() throws Exception {

from(“file:target/inbox”)

.process(new LoggingProcessor())

.bean(new TransformationBean())

.unmarshal().csv()

.split(body().tokenize(“,”))

.choice()

.when(body().contains(“DVD”))

.to(“file:target/outbox/dvd”)

.when(body().contains(“CD”))

.to(“activemq:CD_Orders”)

.otherwise()

.to(“mock:others”);

}

}

 

Each csv file illustrates one single order containing one or more order items. The camel-csv component is used to convert the csv message. Afterwards, the Splitter EIP separates each order item of the message body. In this case, the default separator (a comma) is used. Though, complex regular expressions or scripting languages such as XPath, XQuery or SQL can also be used as splitter.

Each order item has to be sent to a specific processing unit (remember: there are dvd orders, cd orders, and other orders which are sent to a partner). The content-based router EIP solves this problem without any individual coding efforts. Dvd orders are processed via a file directory whilst cd orders are sent to a JMS queue.

ActiveMQ is used as JMS implementation in this example. To add ActiveMQ support to a Camel application, you only have to add the related maven dependency for the camel-activemq component or add the JARs to the classpath manually. That’s it. Some other components need a little bit more configuration, once. For instance, if you want to use WebSphere MQ or another JMS implementation instead of ActiveMQ, you have to configure the JMS provider.

All other order items besides dvds and cds are sent to a partner. Unfortunately, this interface is not available, yet. The Mock component is used instead to simulate this interface momentarily.

The above example shows impressively how different interfaces (in this case File, JMS, and Mock) can be used within one route. You always apply the same syntax and concepts despite very different technologies.

 

Automatic Unit and Integration Tests

Automatic tests are crucial. Nevertheless, it usually is neglected in integration projects. The reason is too much efforts and very high complexity due to several different technologies.

Apache Camel solves this problem: It offers test support via JUnit extensions. The test class must extend CamelTestSupport to use Camel’s powerful testing capabilities. Besides additional assertions, mocks are supported implicitly. No other mock framework such as EasyMock or Mockito is required. You can even simulate sending messages to a route or receiving messages from it via a producer respectively consumer template. All routes can be tested automatically using this test kit. It is noteworthy to mention that the syntax and concepts are the same for every technology, again.

The following code snipped shows a unit test for our example route:

 

public class IntegrationTest extends CamelTestSupport {

@Before

public void setup() throws Exception {

super.setUp();

context.addRoutes(new IntegrationRoute());

}

 

@Test

public void testIntegrationRoute() throws Exception {

 

// Body of test message containing several order items

String bodyOfMessage = “Harry Potter / dvd, Metallica / cd, Claus Ibsen –

Camel in Action / book “;

// Initialize the mock and set expected results

MockEndpoint mock = context.getEndpoint(“mock:others”,

MockEndpoint.class);

mock.expectedMessageCount(1);

mock.setResultWaitTime(1000);

 

// Only the book order item is sent to the mock

// (because it is not a cd or dvd)

String bookBody = “Claus Ibsen – Camel in Action / book”.toUpperCase();

mock.expectedBodiesReceived(bookBody);

// ProducerTemplate sends a message (i.e. a File) to the inbox directory

template.sendBodyAndHeader(“file://target/inbox”, bodyOfMessage, Exchange.FILE_NAME, “order.csv”);

Thread.sleep(3000);

// Was the file moved to the outbox directory?

File target = new File(“target/outbox/dvd/order.csv”);

assertTrue(“File not moved!”, target.exists());

// Was the file transformed correctly (i.e. to uppercase)?

String content = context.getTypeConverter().convertTo(String.class, target);

String dvdbody = “Harry Potter / dvd”.toUpperCase();

assertEquals(dvdbody, content);

// Was the book order (i.e. „Camel in action“ which is not a cd or dvd) sent to the mock?

mock.assertIsSatisfied();

 

}

}

 

The setup method creates an instance of CamelContext (and does some additional stuff). Afterwards, the route is added such that it can be tested. The test itself creates a mock and sets its expectations. Then, the producer template sends a message to the „from“ endpoint of the route. Finally, some assertions validate the results. The test can be run the same way as each other JUnit test: directly within the IDE or inside a build script. Even agile Test-driven Development (TDD) is possible. At first, the Camel test has to be written, before implementing the corresponding route.

If you want to learn more about Apache Camel, the first address should be the book „Camel in Action“ [8], which describes all basics and many advanced features in detail including working code examples for each chapter. After whetting your appetite, let’s now discuss when to use Apache Camel…

 

Alternatives for Systems Integration

Figure 3 shows three alternatives for integrating applications:

 

• Own custom Solution: Implement an individual solution that works for your problem without separating problems into little pieces. This works and is probably the fastest alternative for small use cases. You have to code all by yourself.

 

• Integration Framework: Use a framework, which helps to integrate applications in a standardized way using several integration patterns. It reduces efforts a lot. Every developer will easily understand what you did. You do not have to reinvent the wheel each time.

 

• Enterprise Service Bus (ESB): Use an ESB to integrate your applications. Under the hood, the ESB often also uses an integration framework. But there is much more functionality, such as business process management, a registry or business activity monitoring. You can usually configure routing and such stuff within a graphical user interface (you have to decide at your own if that reduces complexity and efforts). Usually, an ESB is a complex product. The learning curve is much higher than using a lightweight integration framework. Though, therefore you get a very powerful tool, which should fulfill all your requirements in large integration projects.

Figure 3: Alternatives for Systems Integration

 

If you decide to use an integration framework, you still have three good alternatives in the JVM environment: Spring Integration [9], Mule [10], and Apache Camel. They are all lightweight, easy to use and implement the EIPs.  Therefore, they offer a standardized way to integrate applications and can be used even in very complex integration projects. A more detailed comparison of these three integration frameworks can be found at [11].

My personal favorite is Apache Camel due to its awesome Java, Groovy and Scala DSLs, combined with many supported technologies. Spring Integration and Mule only offer XML configuration. I would only use Mule if I need some of its awesome unique connectors to proprietary products (such as SAP, Tibco Rendevous, Oracle Siebel CRM, Paypal or IBM’s CICS Transaction Gateway). I would only use Spring Integration in an existing Spring project and if I only need to integrate widespread technologies such as FTP, HTTP or JMS. In all other cases, I would use Apache Camel.

Nevertheless: No matter which of these lightweight integration frameworks you choose, you will have much fun realizing complex integration projects easily with low efforts. Remember: Often, a fat ESB has too much functionality, and therefore too much, unnecessary complexity and efforts. Use the right tool for the right job!

 

Apache Camel is ready for Enterprise Integration Projects

Apache Camel already celebrated its fourth birthday in July 2011 [12] and represents a very mature and stable open source project. It supports all requirements to be used in enterprise projects, such as error handing, transactions, scalability, and monitoring. Commercial support is also available.

The most important gains is its available DSLs, many components for almost every thinkable technology, and the fact, that the same syntax and concepts can be used always – even for automatic tests – no matter which technologies have to be integrated. Therefore, Apache Camel should always be evaluated as lightweight alternative to heavyweight ESBs. Get started by downloading the example of this article. If you need any help or further information, there is a great community and a well-written book available.

 

About the author

Kai Wähner works as an IT-Consultant at MaibornWolff et al in Munich, Germany. His main area of expertise lies within the fields of Java EE, SOA and Cloud Computing.

He is speaker at international IT conferences such as Jazoon or Confess, writes articles for professional journals, and shares his experiences with new technologies on his blog (www.kai-waehner.de/blog). Contact: kontakt@kai-waehner.de or Twitter: @KaiWaehner.

 

Sources:

• [1] „Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions“, ISBN: 0321200683, Gregor Hohpe, Bobby Woolf
• [2] Apache Camel http://camel.apache.org
• [3] Internal DSL http://martinfowler.com/bliki/DomainSpecificLanguage.html
• [4] Camel Archetypes http://camel.apache.org/camel-maven-archetypes.html
• [5] Example Code for this Article at github https://github.com/megachucky/camel-infoq
• [6] Reduced Dependency on Spring JARs http://davsclaus.blogspot.com/2011/08/apache-camel-29-reduced-dependency-on.html
• [7] Camel Type Converter http://camel.apache.org/type-converter.html
• [8] “Camel in Action”, ISBN: 1935182366, Claus Ibsen, Jonathan Anstey, Hadrian Zbarcea
• [9] Spring Integration www.springsource.org/spring-integration
• [10] Mule ESB http://www.mulesoft.org
• [11] Comparison of Apache Camel, Mule ESB and Spring Integration https://www.kai-waehner.de/blog/2012/01/10/spoilt-for-choice-which-integration-framework-to-use-spring-integration-mule-esb-or-apache-camel
• [12] Fourth Birthday of Apache Camel http://camel.apache.org/2011/07/07/happy-birthday-camel.html