There is no question that integration platforms make it easier to solve integration problems. But what do you do when the lines start to blur between integration, application development and business process requirements? In recent years we have seen capabilities such as workflow and persistence creeping into “integration” solution requirements and it often isn’t possible to find a single technology that can cater for all of these needs. In these scenarios you can use an integration platform with plugins or you could look to an ecosystem to fulfil your end-to-end requirements.
In this blog I will explore using the AWS (Amazon Web Services) ecosystem for enterprise integration solutions: the benefits, my insights from a recent engagement and some areas you need to consider ahead of implementation.
Benefits of an Ecosystem for Integration
Specialised integration solutions such as iPaaS (Integration Platform as a Services) platforms are best suited to the integration patterns they specialise in, such as API Management or Orchestration. For example, the MuleSoft Anypoint Platform is the market-leading solution for API integration and it is very quick and easy to implement services such as ‘lookup customer account details’ exposed as APIs on the Mule platform. Delivering the same functionality via an ecosystem would be more technically complex.
However, when you have non-traditional integration requirements such as File Transfer or Database Migration, that’s where an ecosystem starts to show its value. iPaaS platforms don’t typically provide SFTP servers for File Transfer; you would need to provide your own via a plug-in if there is a requirement for it. Whereas in an ecosystem environment such as AWS, a serverless SFTP server is part of their product stack that you can simply ‘turn on’ if you have File Transfer requirements.
An ecosystem also provides a lot of flexibility for businesses that have varied capacity requirements over a year; there is no upfront cost and you only pay for what you use. For businesses that have a very large file to transfer at the end of financial year or annual large data events such as student enrolment, procuring an iPaaS solution can become quite expensive when you have to pay for your maximum usage requirements, even if it’s only needed once or twice a year.
AWS Integration Capabilities
Whilst AWS is not synonymous with integration, they have one of the most comprehensive technology ecosystems in the world and over the last few years their ability to service enterprise integration patterns has strengthened considerably.
The table below details the products that AWS highlights as being part of their Application Integration product suite.
This is an excellent starting point. However to build a solution for inter-application integration there are a few other capabilities you will need such as AWS Lambda for transformation, data mappings and decisions and AWS Glue for data analytics.
It should be noted that API Management is one area where AWS is lower in maturity than iPaaS platforms. For example, there is no AWS service to catalog APIs with proper domains and metadata to make large numbers of APIs searchable.
Insights from an AWS Integration Solution
Having used the AWS ecosystem for the integration of SaaS platforms with other enterprise systems, I want to share some of my observations.
- Persistence: AWS provides good options for a wide variety of persistence needs e.g. NoSQL, object storage, event storage, etc.
- Orchestration & Choreography: AWS provides some rudimentary orchestration capabilities through step functions but it is more of a technical tool rather than an easy drag ‘n drop flow editor.
- Analytics & Search: AWS provides good options for analytics and searching whereas most iPaaS platforms focus on API-based visibility only.
- Security: Typically an iPaaS wouldn’t supply you with an identity repository but it is readily available in the AWS ecosystem.
- Serverless: Try to use a serverless (AWS SAM) architecture as far as possible for your integration solution as this will lower overall integration costs.
- Open Source: Using at least two lambda open source options such as Python and Node will make your solution more robust by giving you options when you encounter technical challenges e.g. debatching a specific format, adapters, library of packages to solve problems.
Considerations for Optimal AWS Integration Solutions
If you are intending to use the AWS ecosystem for your integration solution here are a few things you should consider.
API Management: API management needs to be carefully thought through. How will API consumers be able to review a catalog of APIs or services? For an API-first integration approach you would typically need to supplement the AWS ecosystem with a good API Management capability.
Continuous Integration/ Continuous Delivery: DevOps setup to enable CI/CD is often seen as an easy add-on to integration solutions but in reality there is a bit more to it. AWS provides best-in-class tools for CI/CD enablement but this is a very technical capability that needs to be maintained by a technical resource.
Skill Set: AWS is much more technical than an iPaaS so you will need to have people and / or partners with the appropriate skill set to deliver, manage, and support the solution. For example, if customisation is required you will need to do open source coding in node or python.
Cost Optimisation: Some AWS components can still be expensive if used in the incorrect manner. For example, spinning up an EC2 to run a Spring container to do integration when serverless would suffice for a much lower cost.
Managed Service: If you have a small IT team but want to leverage AWS for integration you could consider outsourcing your integration. Integration as a managed service allows your business to access leading edge technology without the upfront investment of having to upskill your internal people.
In summary, as with any technology there is no one right way to deliver integration solutions. Your strategic objectives, solution requirements, internal skill set, partner network and investment appetite will all be determining factors when you are weighing up how to best deliver the required integration capabilities for your business needs.
