I believe that even if we split the total number of pages, the process may still fail depending on the API call executed at Step 28. Therefore, I would recommend considering the following points to make the solution more robust and reliable.
This works as a quick stopgap. Its weakness is that a fixed page count is fragile: if the API slows down, 20 pages can take as long as 25 did. A better version of the same idea:
- 1.
Use a time guard. Record the start time and stop the loop when elapsed time passes about 75 minutes (not a page count).
- 2.
Save a checkpoint. Store the current_page in a lookup table, and continue from there in the next job or the next scheduled run.
- 3.
Incase current_page is fixed then I would suggest go with time guard approach, we can skip 2nd step
- 4.
Process each page right away. Send each page’s contacts to destination and clear the list, instead of collecting everything in contacts_list.
