perf(scraping): parallel HTTP fan-out, sequential DB persist (HIGH)
The Pull*UseCase implementations issued one HTTP request at a time despite
Scraping:MaxConcurrentRequests=4. With 30–80 live events and ~1s per
fetch, a 5–10s live cadence target was unreachable; cycles overflowed
the configured interval.
* New Marathon.Application.Configuration.ScrapingThrottle bound from the
shared Scraping:* section. Exposes only MaxConcurrentRequests so the
Application layer doesn't pull in the Infrastructure-side ScrapingOptions.
* PullLiveOddsUseCase + PullUpcomingEventsUseCase split into two phases:
- Phase 1 — Parallel.ForEachAsync over the event list with
MaxDegreeOfParallelism = throttle.MaxConcurrentRequests. The scraper's
Polly rate limiter still throttles to RequestsPerSecond underneath
this fan-out, so spikes are smoothed before they hit the bookmaker.
- Phase 2 — sequential foreach over the (Event, Snapshot) tuples
captured in Phase 1, doing event upsert + snapshot insert. EF Core
DbContext is not thread-safe so all DB writes stay on a single thread.
* InfrastructureModule binds ScrapingThrottle alongside AnomalyOptions.
* Failed snapshot scrapes in Phase 1 mean the event row is also NOT
persisted in Phase 2 — previously we'd persist the row even when the
snapshot scrape failed, leaving an orphan event with no odds. Updated
the regression test accordingly.
* Test fixture exposes TestFixtures.Throttle(maxConcurrentRequests=1) for
deterministic sequential test runs.
* One existing NSubstitute setup that chained Arg.Is<>() across two
configurations was rewritten to use a single Arg.Any<>() with inline
branching — chained matchers were leaking and returning wrong results.
This commit is contained in:
@@ -1,5 +1,6 @@
|
||||
using FluentAssertions;
|
||||
using Marathon.Application.Abstractions;
|
||||
using Marathon.Application.Configuration;
|
||||
using Marathon.Application.UseCases;
|
||||
using Marathon.Domain.Entities;
|
||||
using Marathon.Domain.Enums;
|
||||
@@ -47,11 +48,19 @@ public sealed class LiveOddsPollerTests
|
||||
sp.GetRequiredService<IOddsScraper>(),
|
||||
sp.GetRequiredService<IEventRepository>(),
|
||||
sp.GetRequiredService<ISnapshotRepository>(),
|
||||
StaticThrottle(),
|
||||
NullLogger<PullLiveOddsUseCase>.Instance));
|
||||
|
||||
return services.BuildServiceProvider();
|
||||
}
|
||||
|
||||
private static IOptionsMonitor<ScrapingThrottle> StaticThrottle()
|
||||
{
|
||||
var monitor = Substitute.For<IOptionsMonitor<ScrapingThrottle>>();
|
||||
monitor.CurrentValue.Returns(new ScrapingThrottle { MaxConcurrentRequests = 1 });
|
||||
return monitor;
|
||||
}
|
||||
|
||||
private static IOptionsMonitor<WorkerOptions> BuildOptions(
|
||||
bool enabled = true,
|
||||
int intervalSeconds = 0)
|
||||
|
||||
Reference in New Issue
Block a user