This post started out as an email response to a question Steve Rittler asked me about integrating ColdFusion with an Enterprise Service Bus (ESB). Since it's a fairly interesting topic, I figured it might benefit the community so I asked him if he minded if I posted my response as a series of blog entries. Before I speak to the specifics of connecting ColdFusion to an ESB, I think it might be useful to talk about what an ESB is, how it came about, and what sorts of problems it helps solve.
To give you some background, I've been working for the past two years now as the architect for a project to transform the way our company does integration. We're going from point-to-point integration using mainly shared databases, flat files, journals, and FTP to a service oriented architecture that uses XML messaging and an Enterprise Service Bus as the foundation. To understand how we arrived where we are today, it's useful to understand the evolution of integration patterns. As I said, my company had been doing the majority of its integration point-to-point. Each system was integrated with every other system directly – through database calls, sending flat files over FTP, etc. A typical point-to-point architecture looks something like this:
As you can see, it looks kind of messy – and it is for a large enterprise like mine. If you are a small company, though, chances are this is how your integration looks as well, and that might be fine given your situation. Just like the procedural vs. OOP debate or the framework debate, the integration architecture debate is discussed in similar fashion. That said, the main drawback to the point-to-point architecture is that it's an n2 problem. That means for each new system you add to your architecture that needs to be integrated, you end up with a total of n2 integration points. If you only have a handful of systems, you may be able to support this, but it can become unmanageable fast. It just doesn't scale well in most cases. Point-to-point integration is popular because it's so easy to do and can usually be implemented fairly quickly. "I'll just write data to this staging file" or "I'll just grab data from this table over here" probably sounds familiar.
This brings us to the next evolution in integration architectures – hub and spoke integration. Hub and spoke integration attempted to address the scalability and management problems of point-to-point architectures by introducing the idea of a central hub or broker. You'll often hear this referred to as EAI or Enterprise Application Integration as well. The idea is that all systems in an organization would connect to and send their data through a central hub. This way, connectivity went from n2 connections to n:
Integration is usually performed via adapters for each specific application. These adapters can be as generic as file drops and database queries or API based adapters for connecting to packaged software such as SAP.
In order to integrate a new system in this architecture, all you needed to do is connect it to the central hub and it is then capable of sending data to or receiving data from any other system on the hub. Hub and spoke architectures make for more loosely coupled systems.
While the hub and spoke architecture was great at reducing dependencies as well as the number of connectivity points, it introduced several new problems as well. First, hub and spoke architectures tend to be proprietary in nature. They also introduce a single point of failure that can be difficult to mitigate. Additionally, they have historically have had scalability problems because of an inability to support large transaction volumes.
Because nothing in IT is ever the be-all and end-all solution, integration architecture continued to evolve. Seeking to take the best parts of EAI (centralized management, loose coupling, n lines of connectivity), a new integration pattern known as the Enterprise Service Bus or ESB emerged. Initially, the ESB was only an integration pattern – there were no commercial applications you could buy and implement. Of course, that has all changed. Today, you can find ESB product offerings from pretty much every major integration software vendor out there as well as a growing list of open source alternatives.
There are a number of features and characteristics that make an ESB different from traditional EAI/hub and spoke integration. It's difficult to get everyone to agree 100% on a universal definition of an ESB, but for my purposes, I've used this high level definition from David Chappell's book Enterprise Service Bus: Theory in Practice (O'Reilly):
- A distributed services architecture, which includes a lightweight container model for hosting integration components as remote services
- An enterprise messaging backbone that provides reliable delivery of messages between applications and services
- XML Data transformation
- Service orchestration and intelligent routing of messages based on their content
- A flexible security framework
- A management infrastructure that lets you configure, deploy, monitor, and manage your remote services
The implementation of the ESB looks something like this:
One of the most important aspects here and one that people often don't fully get at first is the distributed nature of the ESB. In hub and spoke EAI, all data has to be brought to the central hub in order to be processed. In an ESB, you actually deploy ESB end-points where your applications are located. If your data center is all in one building, it's as simple as pushing the processing out to other servers located in the same room. In a distributed enterprise, the remote processing means you can physically locate processing components where the systems you need to integrate reside. This makes the ESB much, much more scalable than hub and spoke architectures.
Another thing to note is that the distributed services architecture is implemented using lightweight containers. This means that you don't need to have an application server installed everywhere you want to deploy the ESB containers. That can easily lower your overall integration costs, especially if you use commercial application servers in your environment. All that's required to deploy the remote ESB containers is (usually) a JVM.
A third difference between hub and spoke and the ESB is that the ESB is built on a SOA foundation. Not only does the ESB act as the underlying backbone for SOA, but it is itself SOA based – all of its parts are loosely coupled and can generally be deployed separately as necessary. One benefit of this is that you can deploy only what functionality you need where and when you need it. If you have an intensive transformation process that occurs in one of the applications you are integrating with the bus, you can deploy the necessary horsepower there to deal with the transformation, without having to have the same configuration everywhere.
Applications can connect to ESB endpoints using a variety of mechanisms including but not limited to Web Services, JMS, JCA, HTTP, email, FTP, JDBC, adapters, and many more. How you connect will depend on what you are trying to connect, your overall architecture, and other factors. The bottom line is that there's no one way to do ESB connectivity, although your choice can impact ease of implementation, performance, scalability, etc.
I hope you've enjoyed this general overview of integration architectures. If you have questions or comments, I'd love to hear them. Stay tuned for the next post in this series in which I answer the question "how to get ColdFusion on the Bus".
