I have a question regarding the retrieval of survey responses when working with the LimeSurvey API.
It's possible to fetch responses in batches (for example, 1000 at a time), but it seems that the API always starts from the earliest (oldest) responses. This means that, every time you want to pull in new responses (from a long-term survey), you must iterate through all existing ones until you finally reach the most recent entries.
However, many users would like to periodically sync/retrieve only the most recent responses, for instance to add them to an external database on a regular basis. This issue comes up frequently in different ways. As it stands now, it looks like you need to process all previous responses again and again, just to get to the newest ones.
My question: Is there a way, using the LimeSurvey API, to fetch responses 'backwards' starting from the latest entries, rather than always beginning at the start? Then it would be possible to stop the retrieval once a previously processed response is encountered.
Another option would be retrieving only new responses since a last known point, for instance a response ID.
The current method requires downloading and processing entire response sets repeatedly, followed by 'manually' filtering out already processed responses. This leads to significant computational overhead and potential data redundancy.
Has anyone found an elegant solution to this issue?
Is there a API feature I might have overlooked?
