Scraping Load Testing - Test #5

Summary

The load test was conducted with a higher timeout configuration to address previous timeout issues. The test involved 9000 requests with a timeout set to 30 seconds. The system handled the load well, with only 4 HTTP 502 responses (0.04%). The average response time was 5661.8 ms, with a peak response time of 18924 ms. Further analysis is recommended to optimize response times.

Table of Contents

• Scraping Load Testing - Test #5
  • Summary
  • Table of Contents
  • Test environment
    • Machines
  • Load Test Configuration
    • Configuration
    • Results
    • Metrics
  • Conclusions and Next Steps
    • Conclusions
    • Next Steps

Test environment

Machines

Load Test Configuration

Configuration

  http:
    timeout: 30

Results

Date: 15:59:50(-0300)

Metrics

Conclusions and Next Steps

Conclusions

1. Performance: The system handled 9000 requests with a mean response time of 5661.8 ms. There were only 4 HTTP 502 responses which represent a 0.04% failure rate.
2. Response Times: The peak response time was 18924 ms, indicating that while the system handled the load, there is room for optimization.

Next Steps

2. Testing Scraping Strategies: Conduct further testing on the Playwright instance to ensure it can handle increased load and identify any potential bottlenecks.
3. Load Testing Other Functionalities: Evaluate the performance of other critical routes, such as the crawl route, through additional load tests to ensure comprehensive system reliability.
4. Optimize Response Times: Investigate and implement strategies to reduce the peak response time from 18924 ms. This could involve optimizing database queries, improving server configurations, or enhancing caching mechanisms.
5. Error Handling Improvements: Analyze the causes of the 4 HTTP 502 responses and implement robust error handling and recovery mechanisms to minimize such occurrences in future tests.
6. Scalability Assessment: Assess the system's scalability by gradually increasing the load beyond 9000 requests to determine its breaking point and plan for necessary infrastructure upgrades.

By following these steps, we can further enhance the system's performance and reliability under varying load conditions.