Why am I seeing this error after batching the contacts?
Hey Team, I’m seeing this error even after batching the contacts. Could you please help me understand why?
Hey Team, I’m seeing this error even after batching the contacts. Could you please help me understand why?
Hi Navya, Hi! The flagged step isn't broken. It's just where the job hit Workato's standard job timeout of 90 minutes (5400s) of active execution time. Yours ran 5403.6s. Limits: docs.workato.com/en/recipes/limits Your WHILE loop pulls one page at a time and keeps growing contacts_list, so runtime scales with page count and API response time. A good fix is to break the work into child recipe functions: 1. In the parent, work out total_pages and split it into ranges (e.g. 25 to 50 pages each). 2. Call a recipe function for each range. It runs the HTTP pagination for just its pages, so it gets its own job and time budget. A couple of things to watch: - Synchronous calls make the parent wait on every child, so the parent's own 90 minutes still applies. Calling the functions asynchronously avoids that, but then results need to go to a shared place (data table, file, or Salesforce) instead of coming back as one list. - Keep the data passed between recipes modest. For large payloads, store them externally and pass a reference.
Adding to the above 👆: calling a child recipe function per page works well too. It gives you good control in the job report:
How many records were fetched in that page
How many succeeded
How many failed
And if an error happens( due to any reason), you only reprocess the job for that page, not the whole run.
I believe that even if we split the total number of pages, the process may still fail depending on the API call executed at Step 28. Therefore, I would recommend considering the following points to make the solution more robust and reliable. This works as a quick stopgap. Its weakness is that a fixed page count is fragile: if the API slows down, 20 pages can take as long as 25 did. A better version of the same idea:
Use a time guard. Record the start time and stop the loop when elapsed time passes about 75 minutes (not a page count).
Save a checkpoint. Store the current_page in a lookup table, and continue from there in the next job or the next scheduled run.
Incase current_page is fixed then I would suggest go with time guard approach, we can skip 2nd step
Process each page right away. Send each page’s contacts to destination and clear the list, instead of collecting everything in contacts_list.
Thankyou Shiva, dheeraj k., Vinay K.! Will update the recipe accordingly and let you know the progress
Hi Navya K., just to understand this better:
For the job that failed, how many pages did it process in that time frame?
What is the typical volume per page? Do you have control on the volume, like by putting limit clause?
How long does your API take to post that volume?
Navya K. Here’s the step-by-step approach I suggest we take to fix this cleanly and prevent future timeouts:
Process Data "In-Flight": First, stop accumulating everything into contacts_list across pages. Clear or process the list on each iteration so memory overhead doesn't grow exponentially and slow down late iterations.
Move to Async Child Recipes: Decouple the loop by having the parent recipe trigger a Recipe Function asynchronously per page (or page range). This gives each page its own fresh 90-minute job budget, makes job logs much cleaner, and means a failed page only requires retrying that single page rather than re-running a multi-hour job.
Add a Time Guard & Checkpoint: Track job_start_time and exit the loop gracefully around the 75-minute mark. Store the last successful current_page in a Lookup/Data Table so the next scheduled run picks up seamlessly right where it left off.