When to use Apache Camel?

  • 6 minute read
Apache Camel is one of my favorite open source frameworks in the JVM / Java environment. It enables easy integration of different applications which use several protocols and technologies. This article shows when to use Apache Camel and when to use other alternatives.
Total
8
Shares
8
0
0
Share
8 people shared the story
8
0
0
0

Apache Camel is one of my favorite open source frameworks in the JVM / Java environment. It enables easy integration of different applications which use several protocols and technologies. This article shows when to use Apache Camel and when to use other alternatives.

The Problem: Enterprise Application Integration (EAI)

Enterprise application integration is necessary in almost every company due to new products and applications. Integrating these applications creates several problems. New paradigms come up every decade, for example client / server communication, Service-oriented Architecture (SOA) or Cloud Computing.

Besides, different interfaces, protocols and technologies emerge. Instead of storing data in files in the past (many years ago), SQL databases are used often today. Sometimes, even NoSQL databases are required in some usecases. Synchronous remote procedure calls or asynchronous messaging is used to communicate via several technologies such as RMI, SOAP Web Services, REST or JMS. A lot of software silos exists. Nevertheless, all applications and products of these decades have to communicate with each other to work together perfectly.

Enterprise Integration Patterns (EIP)

Of course, you could reinvent the wheel for each problem, write some spaghetti code and let the applications work together. Unfortunately, your management will not like the long-term perspective of this solution.

Enterprise Integration Patterns (www.eaipatterns.com) help to fragment problems and use standardized ways to integrate applications. Using these, you always use the same concepts to transform and route messages. Thus, it is a good idea to forget about reinventing the wheel each time you have a problem.

Alternatives for integrating Systems

Three alternatives exist for integrating applications. EIPs can be used in each solution.

Solution 1: Own custom Solution

Implement a individual solution that works for your problem without separating problems into little pieces. This works and is probably the fastest alternative for small use cases. You have to code all by yourself. Maintenance will probably be high if team members change.

Solution 2: Integration Framework

Use a framework which helps to integrate applications in a standardized way using several integration patterns. It reduces efforts a lot. Every developer will easily understand what you did (if he knows the used framework).

Solution 3: Enterprise Service Bus (ESB)

Use an enterprise service bus to integrate your applications. Under the hood, the ESB also uses an integration framework. But there is much more functionality, such as business process management, a registry or business activity monitoring. You can usually configure routing and such stuff within a graphical user interface – you have to decide at your own if that reduces complexity and efforts. Usually, an ESB is a complex product. The learning curve is much higher. But therefore you get a very powerful tool which should offer all your needs.

What is Apache Camel?

Apache Camel is a lightweight integration framework which implements all EIPs. Thus, you can easily integrate different applications using the required patterns. You can use Java, Spring XML, Scala or Groovy. Almost every technology you can imagine is available, for example HTTP, FTP, JMS, EJB, JPA, RMI, JMS, JMX, LDAP, Netty, and many, many more (of course most ESBs also offer support for them). Besides, own custom components can be created very easily.

You can deploy Apache Camel as standalone application, in a web container (e.g. Tomcat or Jetty), in a JEE application Server (e.g. JBoss AS or WebSphere AS), in an OSGi environment or in combination with a Spring container.

If you need more information about Apache Camel, please go to its web site as starting point: http://camel.apache.org. This article is no technical introduction J

When to use Apache Camel?

Apache Camel is awesome if you want to integrate several applications with different protocols and technologies. Why? There is one feature (besides supporting so many technologies and besides supporting different programming languages) which I really appreciate a lot: Every integration uses the same concepts! No matter which protocol you use. No matter which technology you use. No matter which domain specific language (DSL)  you use – it can be Java, Scala, Groovy or Spring XML. You do it the same way. Always! There is a producer, there is a consumer, there are endpoints, there are EIPs, there are custom processors  / beans (e.g. for custom transformation) and there are parameters (e.g. for credentials).

Here is one example which contains all of these concepts using the Java DSL:

from(„activeMQ:orderQueue“)..transaction().log(„processing order“).to(mock:“notYetExistingInterface“)

Now let’s look at another example using the Scala DSL:

„file:incomingOrders?noop=true“ process(new TransformationProcessor) to „jdbc:orderDatastore“

If you are a developer, you should be able to recognize what these routes do, don’t you?

Two other very important features are the support for error-handling (e.g. using a dead letter queue) and automatic testing. You can test EVERYTHING very easily using a Camel-extension of JUnit! And again, you always use the same concepts, no matter which technology you have to support.

Apache Camel is mature and production ready. It offers scalability, transaction support, concurrency and monitoring. Commercial support is available by FuseSource: http://fusesource.com/products/enterprise-camel

When NOT to use Apache Camel?

Well, yes, there exist some use cases where I would not use Apache Camel. I have illustrated this in the following graphic (remember the three alternatives I mentioned above: own custom integration, integration framework, enterprise service bus).

When to use Apache Camel?

If you have to integrate just one or two technologies, e.g. reading a file or sending a JMS message, it is probably much easier and faster to use some well known libraries such as Apache Commons IO or Spring JmsTemplate. But please do always use these helper classes, pure File or JMS integration with try-catch-error is soooo ugly!

Although FuseSource offers commercial support, I would not use Apache Camel for very large integration projects. An ESB is the right tool for this job in most cases. It offers many additional features such as BPM or BAM. Of course, you could also use several single frameworks or products and „create“ your own ESB, but this is a waste of time and money (in my opinion).

Several production-ready ESBs are already available. Usually, open source solutions are more lightweight than commercial products such as WebSphere Message Broker (you probably need a day or two just to install the evaluation version of this product)! Well-known open source ESBs are Apache ServiceMix, Mule ESB and WSO2 ESB. By the way: Did you know that some ESB base on the Apache Camel framework (e.g. Apache Service Mix and the Talend ESB). Thus, if you like Apache Camel, you could also use Apache ServiceMix or the commercial Fuse ESB which is based on ServiceMix.

Conclusion

Apache Camel is an awesome framework to integrate applications with different technologies. The best thing is that you always use the same concepts. Besides, support for many many technologies, good error handling and easy automatic testing make it ready for integration projects.

Because the number of applications and technologies in each company will increase further, Apache Camel has a great future. Today we have application silos, in ten years we will probably have cloud silos which are deployed in Goggle App Engine, CloudFoundry, Amazon EC3, or any other cloud service. So I hope that Apache Camel will not oversleep to be ready for the cloud era, too (e.g. by offering components to connect to cloud frameworks easily). But that’s future… At the moment you really should try this framework out, if you have to integrate applications in the JVM / Java environment.

By the way: I know that I praise Camel in this article, but I am neither a Camel committer nor working for FuseSource. I just really like this framework.

Best regards,

Kai Wähner (Twitter: @KaiWaehner)

Total
8
Shares
Share 8
Tweet 0
Pin it 0

Dont‘ miss my next post. Subscribe!

We don’t spam! Read our privacy policy for more info.
If you have issues with the registration, please try a private browser tab / incognito mode. If it doesn't help, write me: kontakt@kai-waehner.de

Related Tags

builds cloud-native event streaming infrastructures for real-time data processing and analytics

37 comments

    1. Hey Jeff,

      I do not have experience in a project with Spring Integration. My personal opinion is that Camel has an easier, fluent API (especially the Java / Scala DSL). You can integrate several protocols without coding a lot. Spring Integration could also do the job, but I do not like the Spring programming model – I needs much more (XML-) configuration.

      Thus, if you already use a Spring project which uses several other spring projects, then I would at least evaluate Spring Integration, too. In all other cases, I would choose Camel as integration framework.

      DZone has a great (neutral) comparison of Spring Integration and Camel (unfortunately not using the Java DSL for Camel): http://java.dzone.com/articles/spring-integration-and-apache

      Best regards,
      Kai Wähner (Twitter: @KaiWaehner)

  2. Hi,
              Good post.
              >> Well-known open source ESBs are Apache ServiceMix, Mule ESB and WSO2 ESB
              When you mention Apache Servicemix ESB are refering to Servicemix 3.x (JBI) or Servicemix 4.x (OSGi)?
              Please let me know your comment.

    With Best Regards,
    Diwakar

  3. Hey Diwakar,
    I (and most other guys) think that the JBI standard is dead already. It always was too complex. Thus, I refer to the current version of ServixMix: 4.x.
    I would not consider any JBI product for a new project. Mule, WSO2 and ServiceMix are all great ESBs because they are up-to-date, lightweight and easy to use.
     
    Best regards,
    Kai

  4. Hi Kai,

    Thank you for your thoughtful treatment of the topic of when to use Camel.  However, I wish to draw your attention to the correct use of the term DSL (Domain-specific Language).  Java, Scala and Groovy are general-purpose languages, the opposite of a DSL.  Examples of DSLs are: HTML, SQL,  Mathematic and YACC.  For reference, please note the Wikipedia article at http://en.wikipedia.org/wiki/Domain-specific_language.

    David Foote 

  5. Kai

    “You can integrate several protocols without coding a lot.” – When you say without coding a lot do you mean XML, Java, Scala? I am always skeptical when I read such statements. There has got to be a place where you provide a machine instruction, right otherwise how does a runtime knows what to do?

  6. @Oleg:
    What I want to say is: No matter which API you need, you just have to write the route using the same syntaxt and concepts, no matter if you use the Java, XML or Scala DSL.
    Example (Java DSL):
    from(“file:input)”.to(“jms:outputQueue”)
    Now, again the statement: “You can integrate several protocols without coding a lot” => i.e.: You do not have to use the File API, you do not have to use the JMS API, you do not have to code a lot. The Camel components do this for you, their source code contains calls to the File API or JMS API. No matter which other Camel component you use, it’s the same…
    Of course, you still have to write some code: The routes, and you have to do the configuration once (e.g. configuring JMS ressources). Though, it is much less boilerplate code.
    I hope you agree with this explanation 🙂
     

  7. Yes I do agree and wanted to say that the same applies for Spring Integration where pretty sophisticated message flows could be constructed without writing a single line of Java code and only with few configuration elements configured via XML and soon Scala. Let me know and I’ll point you to some samples.
    Also, on the related note its nice how you said “. . .you just have to write the route using. . .” FWIW this is one of the fundamental differences between Camel approach and SI. EIP does not define ‘routes’, instead core EIP patterns is what you use to construct ‘message flows’ which themselves could be used in isolation, extended or included as part of some other flows etc., which means IMHO the idea behind the construct that begins with ‘from’ and ends with ‘to’ contradicts the main idea behind Messaging and EIP in general and that is “physical and logical decoupling of producers and consumers”.
    I can go on, but don’t want to hijack the main idea behind the post 😉
    May be in the future you car write something that compares SI and Camal and than we can duke it out 😉
    Cheers
     

  9. Nice post. For the cloud era, I think mule is moving to the correct direction. Mule ESB already provides connectors for salesforce and twitter, etc. Also Mule iON claims to be a iPaaS, though I have not tried this. If the DSL is not of priority, the connection ability is critical to integration architects. Camel has to hurry up.

  10. Hey Hongchao,
    I agree about the components, Mule has several awesome connectors to proprietary products. I have discussed this topic in another blog post: “Comparison of Apache Camel, Mule ESB and Spring Integration”.
    Though, Camel also has connectors for many cloud services such as AWS, GAE, Twitter, and many others (using the jclouds component) already.
    Besides, I think Fuse Fabric is going into the same direction as Mule iON… So I think, has not to hurry up 🙂
    Best regards,
    Kai Wähner (Twitter: @KaiWaehner)
     

  11. In this article and other articles of yours, you refer to MuleESB in both Integration Framework context and ESB context. Do you consider MuleESB as more of an Integration Framework or an ESB, because MuleESB itself, based on my understanding, does not have support for BPM/BAM/Registry/Repository which you say are typical features of an ESB.

  13. Nice article – however don’t agree with the concept that ESBs can do BPM. Common mistake – BPM is for “Process orchestration” mainly built on BPEL and more for manual+auto tasks. ESB do orchestration however on “Business Service” level. BPMS can use ESBs for delegating service tasks or ESBs may invoke a BPMS process. To summarise ESBs can do in-memory stateless micro-flows i.e. collection of some services – while BPMS does long runnning process orchestration

  15. Hi – This ia very short and nice article. I was wondering if there is any list of features to compare for different products like Camel, Active Matrix (TIBCO) or might be WCF?

  17. Can you please help me to decide which ETL tool should I use for my application over spring framework to integrate with HL7 (health domain) format. for now we are considering FTP for HL7 transform. I am currently looking into Apache Camel and spring integration. Is there be any other good option (open source) for this purpose.

  20. Apache Camel does NOT require to be connected to the internet.

    Of course, if you use Maven, then it tries to download packages from internet. But this is an optional feature. You can also just distribute JARs which you need.

    Kai

  21. Nice article.  What factors in integration complexity would make you move up to ESB from Camel? 
     
    In other words; I like the graph.  Its easy to understand.  What are the factors for deciding between one or the other?  Number of Servers? Number of Routes?  Number of different Protocols?  etc.
     
    Thanks,
     
    Andrew

  22. Andrew,
    it is NOT about number of routes, servers, etc.
    Use an ESB if you need good tooling (graphical designer, code generator, monitoring) and commercial support. If you want to write source code by yourself and debug etc. just with an IDE such as Eclipse, then Camel without an ESB is a good choice.
    These are the main differentiators between an integration framework and an ESB whereby an ESB might be on top of of the framework, e.g. Talend ESB and JBoss Fuse both generate Camel code under the hood.
    Kai

  23. Very helpful article, thanks! Now, few years after writing this, has something dramatically changed? New better frameworks emerged? Other frameworks are more suitable for the cloud era?

    We want to build an e-commerce integration hub (like wombat.co) for CEE market, because we have a custom e-commerce solution (written in PHP) and we are constantly connecting it to various local ERP system (point-to-point integration – it’s difficult to reuse between projects). Building from scratch seemed like too much of reinventing-the-wheel, on the other hand ESB seemed like an overkill. Integration framework like Camel seems like a nice sweet spot in-between. What do you think Kai? I would appreciate your insight.

  25. Thank you for this article, I am hoping that you can give some guidance on this for me please? I am creating a component to accept, validate and save orders. Normally I would not consider Camel as this is not an integration problem, more an endpoint in of itself. However, I need to support REST, SOAP and MQ endpoints so I was considering using Camel to simplify building support for these endpoints and routing all inbound messages to central validation and persistance beans. At the enterprise level you can use Camel to route messages between applications / datastores but you “can” also use it to route messages between methods in different java classes within a single application – just wondering if you think it makes sense at this level? Thank you for you time.

  26. Peter,

    this sounds very interesting. Yes, I think using an integration solution often makes sense for implementing this kind of “business application”. If you need to build services (SOAP, REST, etc) and integrate technologies such as MQ, then you are on the right way.

    Therefore, we often use an integration framework or ESB not just as “Integration Platform”, but also as “Service Delivery Platform”…

  27. Dear folks, this is really a great articule, hope you continue sharing your time and knowledge. well, here some questions/doubts,
    1- when you say that ESB is recommended for large integration applications, what do you mean with “large”? can you share an example?.
    2- if just less effort for integration development is needed when ESB is chosen, and more development effort (even coding the solution) is needed implementing Camel, is correct my understanding?
    3- ESB as Service Delivery Platform, is this an advantage versus Camel ?
    Thanks!

  28. Good questions:

    1) There is no hard definition for that. Think about if a Java framework fits your needs regarding implementation efforts / monitoring and if you need commercial support. For example, a mission-critical system, which has to be up 24/7 might be a good choice for an ESB.

    2) An ESB (no matter if open source or commercial) offers zero-coding tooling (yes, you still can and have to write some custom code somtimes nevertheless) which improves the speed of development and maintenance significantly for most developers.

    3) Not really. You can use an integration framework like Camel in the same way like an ESB to build services (e.g. REST or SOAP).

  29. That was an informative article.
    I had a use-case , I need to perform an ETL on data in mongodb and data in sql ,load the transformed data in sql.Whats more suitable to start with ? apache camel,mule ESB or Talend ESB.

  30. If you wanna do ETL (i.e. batch processing), then Talend is the best tool. It allows “visual development” which helps a lot for ETL instead of coding with a framework like Camel.
    However, you need to use Talend DI (Data Integration), not Talend ESB. Mule ESB is like Talend ESB focused on “real time integration”, i.e. Web Services etc. You can also do ETL, but the tooling for that is not as sophisticated for this use case.

  31. Hi Kai,
    Very nice article. I am just a beginner in SOA and EIP. Could you please help me to understand how SOA is different from EIP? Somewhere I read that SOA can be replaced EIP because of its advance features, EIP can be obsolete.

    Thanks,
    Santu

  32. Hi Santu,

    well, I recommend to do more research and read more articles. SOA is a very generic principle or architecture style with many best practices around it. EIP is “just” some design patterns which can be applied when realizing a SOA. Thus, EIP is just a small part of a SOA.

  33. Hi Kai,

    I’m building a bunch of API’s in nodejs and I have to make sure it supports multiple message formats and protocols. I couldn’t find any integration framework in node packages. Will I be able to use Camel in my node project in some way? Do you have any other suggestions to achieve this?

    Thanks,
    Jai

  34. Hi Jai, I think node-red is a good open source integration framework based on node.js. This might fit better (but it also depends on the use case). Node-red is IoT focused while Camel is a general, sophisticated (and very powerful) integration framework. But you can also integrate with Apache Camel or another Streaming platform (e.g. Apache Kafka + Kafka Connect) via standard interfaces like HTTP.

  35. Good Post !

    I am working on an asset management project of a well established bank.
    We are planning to use Apache camel as Integration layer and mainly concentrating on :
    1.File Transfer to/From Amazon S3.
    2. Performing ETL operations on data files of various platforms- Deals with business transformation , combining multiple file content
    3. Publishing the transformed files to target source.

    Will camel be the right choice here ? Considering huge data flow, I am not confident on the choice of camel over ETL tools.

  36. @Rashmi.

    For huge data, the better option is Apache Kafka and its integration API Kafka Connect. Kafka is built for high volume and large scale. It became more or less the de facto standard for such a use case.

Leave a Reply
You May Also Like
Read More
  • 20.4K views
  • 3 minute read

Apache Kafka vs. Middleware (MQ, ETL, ESB) – Slides + Video

This post shares a slide deck and video recording of the differences between an event-driven streaming platform like Apache Kafka and middleware like Message Queues (MQ), Extract-Transform-Load (ETL) and Enterprise Service Bus (ESB).
Read More
0
0
0