How to get a Mulesoft application to page large volumes of data without resetting
Recently I’d been developing a Mulesoft application that uses another as a source API. The response from the source API looks something like this:
{
"nextLink": "https://server:port/external-api/endpoint?page=1",
"nextOffset": 1000,
"morePages": true,
"totalRows": 12345,
"totalPages": 13,
"results":
[
{...}
]
}
There are a few things that could potentially be used for paging the response data. As a .NET engineer, I would have approached it by using a While or Until loop on ‘morePages: true’, used the nextLink URI and appended the response payload to a typed list.
Unfortunately Anypoint Studio doesn’t have While or Until components. Also, a Mulesoft consultant pointed out that the nextLink and morePages values were intended for another integration - I probably could use them, but a better option would be to use a ‘For Each’ component on the totalPages and nextOffset values, because the source API accepted nextOffset as a query parameter. This turned out to be much easier.
I added a ‘Parallel For Each’ container to the application (disable concurrency for this!), with the idea of using totalPages as the collection. I also added a target variable called ‘records’, which would be appended with the payload from the HTTP Request during each iteration of the ‘For Each’ loop. I also needed to flatten the JSON to make it usable:
%dw 2.0
output application/json
---
flatten(flatten(vars.records.payload).results)
Within the ‘Parallel For Each’ container, I added another variable for nextOffset, with an intial value of ‘(payload*1000)’. The offset happens to be the number of times a payload was received, multiplied by 1,000.
After the nextOffset variable, I added another HTTP Request component, which is identical to the initial HTTP Request, but sends ‘vars.nextOffset’ as a query parameter.
This worked fine locally, and on a staging environment with a relatively small amount of data.
Part II
After I’d deployed the application to production, it ran into a couple of problems around the large volumes of data and the resource limitations of CloudHub. The source API occasionally closed the connection, and sometimes reset itself after maxxing out the memory. This would cause the application to process incomplete data.
There were a couple of things that needed to be done: First, make the HTTP Requests more resilient by increasing the timeouts and number of retries. Secondly, to avoid data corruption, design the application to terminate when it can’t get a page of data.
HTTP Request Component
The first thing to do is configure the HTTP Request component to increase the number of reconnection attempts to 5, and add a longer delay between each attempt (60,000ms). Also important is the HTTP request configuration in the Global Element Properties:
- Reconnection strategy: Standard
- Max connections: -1
- Connection idle timeout: 400,000
Until Successful Container
We can place the HTTP Request component inside an ‘Until Successful’ container, just to further ensure the request is retried until the correct response is received from the source API. For this, I’d set the ‘Max Retries’ to 5, and set the delay between retries to 60,000ms.
Try/Catch Container
Lastly, I put the ‘Until Successful’ container inside a ‘Try/Catch’ container. This will call an exception handler I’d added specifically for paging errors. If the max number of retries fail in both the HTTP Request and the ‘Until Successful’ container, the exception handler will call another flow that logs the error and terminates the application.
