As more and more businesses move to an API-led integration strategy to connect their cloud, web and mobile technologies, we are often being asked:
- Should we be doing Event-Driven Architecture (EDA) or should we be doing APIs?
- How can APIs and Events work together?
In this blog I’m going to discuss how Event-Driven Architecture can fit into an API integration strategy and how it will benefit your business.
The Distinction Between APIs and Event-Driven Architecture
According to Gartner about 50% of managed APIs will support Event-Driven IT by 2020. These Event-Driven APIs are part of the technology mix that will help a business achieve a competitive advantage with real-time integration.
MuleSoft defines an API as ‘a software intermediary that allows two applications to talk to each other. In other words, an API is the messenger that delivers your request to the provider that you’re requesting it from and then delivers the response back to you.’
So, APIs are message-driven, where messaging is the packaging of an information exchange between a sender and a receiver into a data record structure that can be transmitted through a channel. An event message is a type of messaging where senders publish a message about a state change that occurred in the past, without knowing the receivers of the information.
Wikipedia defines Event-Driven Architecture as a software architecture pattern promoting the production, detection, consumption of, and reaction to events. An event can be defined as "a significant change in state". For example, when a consumer purchases a car, the car's state changes from "for sale" to "sold".
From a formal perspective, what is produced, published, propagated, detected or consumed is a message (typically asynchronous) called the event notification, not the event (details) itself which is the state change that triggered the message emission. Events do not travel, they just occur, but the term ‘event’ is often used to mean the notification message itself, which may lead to some confusion.
Conceptually APIs and Event-Driven Architecture (EDA) are both messaging based, but APIs aim to agree the structure of the messages exchanged while EDA aims to agree a structure that notifies any subscriber of a particular change of state. So APIs have a wide scope and EDA has a narrower scope. There is no reason why the scope of EDA cannot be part of the wider API catalog (defined message structure) scope, but it is a special subset that goes beyond defined message structures and also recommends a message exchange sequencing approach, much like how a BPM orchestrates APIs together to fulfil a business process but the business process itself can also be exposed as an API e.g. a command message triggering the business process.
There are a few scenarios driving the need for Event-Driven Architecture, e.g.:
- Integrating with modern SaaS applications - they heavily depend on EDA approaches to ensure high availability and scalability.
- Internet of Things - the sheer volume of events streamed from devices that needs to be accessible in real time, at scale, makes synchronous request-response APIs alone unfeasible.
How APIs and Event-Driven Architecture Can Work Together
Event-Driven APIs supplement, not replace, RESTful API request-and-response technologies. Event APIs can be a useful subset in the API catalog defining what events are published that could be of use and how to subscribe to these events. Event Streaming is an even narrower subset of that catalog where there is a high volume of events that are published and might make use of different protocols, tools and analytics or decision processes as subscribers, these may (or may not) be well-defined APIs.
When an API only offers request/response support, consumers are required to poll to see if there are any changes to important data. With API based Event notifications, consumers can receive a well-defined trigger (notification of the event) when data changes and execute the desired automation flow, without needing to constantly poll an API. Publish-Subscribe (e.g. Salesforce Platform Event Bus) and Webhooks (e.g. GitHub webhooks + events) are some examples of (API) Event Driven integration.
To cater for Event-Driven APIs, your integration capability will need to consider the following:
- Publish-Subscribe topic infrastructure
- Subscription management ('Subscribe here' / WebHook APIs, persistence of subscription management information)
- Event sourcing or message stores (event persistence) for various purposes
If you ‘externalize’ events, you should consider how your consumers will need to consume them, perhaps with a combination of WebHooks, event streaming, or the less efficient long polling for some circumstances. This is where APIs come in - APIs provide the low friction consumption of your services by providing visibility, documentation & guides, catalog, versioning, security and management.
API events expand the kinds of conversations that our APIs need to have as we use them to solve day-to-day problems - it is an additional catalog of APIs to subscribe to events explaining not only how to subscribe but also the structure of the event payload a subscriber will receive. A great example of this is GitHub's API catalog that includes a request-response synchronous API catalog, a webhook and an event catalog (https://developer.github.com/v3/):
GitHub request-response APIs:
- GitHub Database API: https://developer.github.com/v3/git/
GitHub WebHook & Event APIs:
- GitHub 'Activity' API provides access to notifications, subscriptions, and timelines: https://developer.github.com/v3/activity/
- According to GitHub, Webhooks allow you to build or set up integrations which subscribe to certain events on GitHub.com. When one of those events is triggered, we'll send a HTTP POST payload to the webhook's configured URL (https://developer.github.com/webhooks/#events)
In summary, Event APIs can be a useful subset in the API catalog to define what events are published that could be of use, how to publish those events and how to subscribe to these events. Adding an event-driven domain to your API catalog will add another tool in the toolbox covering the situations where synchronous request-response only APIs may not be optimal.