In order to facilitate the generation of download lists, add a function to the store along those lines: repeated calls to get_download_urls() until there are no more URLs to download.
while True:
bufferlist = self.get_download_urls()
if bufferlist or self.done:
break
sleep(sleep_time)
return bufferlist