E-commerce Automation System (Afterbuy)
A high-performance ecosystem for parsing, processing, and managing data, which reduced manual operation time from 1 hour to 5 minutes and handles >100,000 requests per day.
Click on the image to enlarge.
Technical Deep Dive:
Architectural Decisions
- Layered Architecture: Strict separation into `Routers`, `Services`, and `Repositories` for logic isolation and easier testing.
- Dependency Injection: Active use of `FastAPI.Depends` for managing dependencies and DB sessions, ensuring clean code.
- Lazy DB Initialization: A custom `DatabaseHelper` for `SQLAlchemy` that creates the `engine` only on the first call, which is critical for compatibility with `Celery`.
- Reverse-Engineering: Instead of fragile HTML parsing, HTTP request analysis and Afterbuy's internal API were used.
Scalability and Performance
- Asynchronous Pipeline: `FastAPI` + `AIOHTTP` + `asyncpg` provide a fully non-blocking I/O, allowing hundreds of concurrent requests to be processed.
- Background Processing: All long tasks (>1 sec) are offloaded to `Celery`, ensuring an instant API response.
- Aggressive Caching: `Redis` is used for caching API responses, product data, and sessions, reducing the load on the DB.
- Rate Limiting: `slowapi` protects the service from excessive client load.