feat: Add comprehensive batch processing API with download capabilities #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Owner
stretchcloud
commented
Jun 7, 2025
- Add batch scraping endpoint (/api/batch/scrape) for processing multiple URLs efficiently
- Implement controlled concurrency (1-10 URLs) with Redis-based job management
- Add comprehensive download options:
- Individual file downloads by job ID with format selection (markdown/html/text/json)
- ZIP archive downloads with organized file structure and batch summary
- Consolidated JSON exports with metadata and processing statistics
- Include webhook notifications for async completion events
- Add exponential backoff retry mechanism with configurable max retries
- Implement proper input validation and comprehensive error handling
- Add batch monitoring endpoints for real-time progress tracking
- Include batch cancellation and cleanup functionality
- Update TypeScript interfaces for batch operations and job management
- Add archiver and express-validator dependencies for ZIP creation and validation
- Integrate batch routes into main Express router
- Update README with complete batch processing documentation and examples
- Add batch scraping endpoint (/api/batch/scrape) for processing multiple URLs efficiently - Implement controlled concurrency (1-10 URLs) with Redis-based job management - Add comprehensive download options: * Individual file downloads by job ID with format selection (markdown/html/text/json) * ZIP archive downloads with organized file structure and batch summary * Consolidated JSON exports with metadata and processing statistics - Include webhook notifications for async completion events - Add exponential backoff retry mechanism with configurable max retries - Implement proper input validation and comprehensive error handling - Add batch monitoring endpoints for real-time progress tracking - Include batch cancellation and cleanup functionality - Update TypeScript interfaces for batch operations and job management - Add archiver and express-validator dependencies for ZIP creation and validation - Integrate batch routes into main Express router - Update README with complete batch processing documentation and examples
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.