maraphon-app/spike/SCRAPE_FINDINGS.md

# Phase 0 Spike — Scraping Findings for marathonbet.by

**Date:** 2026-05-05
**Probe environment:** Windows 10, Poland-routed IP (countryCode `PL` reported by site,
`isBelarus: true` flag set in `initData`, `jurisdiction: BELARUS`).
**Tooling used:** `curl` with browser User-Agent, ~10 sequential requests with
≥1-second pacing.

---

## TL;DR — Decision Matrix

| Question | Answer |
|---|---|
| Is anonymous scraping feasible? | **YES — confirmed.** Site returns full server-rendered HTML for `/su/`, `/su/live`, sport listings, and event detail pages with HTTP 200 to a plain GET with browser User-Agent. |
| Cloudflare / JS challenge? | **No.** `Server: nginx`, no `cf-ray`, no challenge cookies. Only standard JSESSIONID + analytics cookies. No reCAPTCHA on listing pages. |
| Geo-block from probe environment? | **No.** Probe was made from a non-Belarus IP; site served full HTML. The site treats us as `region:"PL"` but still serves Russian-language `/su` content. |
| Recommended scraping technology | **HttpClient + AngleSharp.** All the data needed (event list, full odds, breadcrumb taxonomy, period markets) is present in the raw SSR HTML. Playwright is not required for read-only scraping. |
| Recommended polling cadence | Pre-match: **30 seconds** (default in `appsettings`). Live: 3-second native cadence is too aggressive — recommend **5–10 seconds** for our analyzer (anomaly detection doesn't need sub-second resolution). |
| WebSocket / API alternative? | STOMP-over-WebSocket exists at `/su/websocket/endpoint` for authenticated clients. Anonymous clients should stick to plain HTML scraping. The JSONP endpoint at `/su/liveupdate/popular/` only returns refresh-page signals, not full odds. |

---

## 1. Probe Outcomes

### 1.1 Pre-match landing — `https://www.marathonbet.by/su`

```
HTTP/1.1 200 OK
Server: nginx
Content-Type: text/html;charset=UTF-8
Set-Cookie: visitedNavBarItems=HOME; HttpOnly; SameSite=None; Secure
Set-Cookie: lastSitePart=SPORT; ...
Set-Cookie: puid=rBWP3Wn5...; expires=2037; domain=.marathonbet.by
Strict-Transport-Security: max-age=31536000
Cache-Status: MISS
Cache-Control: no-store, no-cache, must-revalidate
```

- **Render type:** Server-Side Rendered (SSR). Body is ~590 KB of HTML containing
  the full event grid for live + popular pre-match events. There IS a `<div id="app">`
  wrapper but the content inside is fully populated server-side; the JS layer enhances
  rather than hydrates from empty.
- **Rich data attributes embedded:**
  - `data-event-eventId="<bookmakerEventCode>"` — bookmaker's stable numeric event ID
  - `data-event-treeId="<treeId>"` — tree position ID (used in URLs)
  - `data-event-name="..."` — event display name
  - `data-event-path="<sport>/<league-path>/<teams> - <treeId>"` — URL fragment to
    construct event detail link
  - `data-live="true|false"` — live vs pre-match flag
  - `data-sport-treeId="<sportId>"` — sport identifier (matches customer's "Sport_Code")
  - `data-coeff-uuid` + `data-sel='{...}'` JSON — selection metadata (ewc, cid, prt, epr)
  - `data-selection-key="<eventId>@<MarketType>[N].<Outcome>"` — canonical bet identifier
- **Embedded `initData` JSON blob** (line 6 of every page) exposes runtime config:
  - `serverTime: "2026,05,05,00,43,28"` (Moscow TZ)
  - `liveUpdatePath: "/su/liveupdate/popular/"`
  - `liveUpdateTransport: "JSONP"`
  - `update_interval: 3000` (ms — live update polling cadence used by the site itself)
  - `stomp.url: "/su/websocket/endpoint"` (authenticated stream)
  - `region`, `isBelarus`, `jurisdiction`, `currencyCode` — geo/legal flags
  - `treeIds` — for the event detail page, holds the focal treeId

### 1.2 Live landing — `https://www.marathonbet.by/su/live`

- HTTP 200, ~250 KB body — same `nginx` server, same SSR pattern.
- Same `data-event-*` attributes as pre-match. Live events show `data-live="true"`,
  with extra `score-state` and `time` markers (e.g., `2:1 (1:1)`, `83:30`).
- The site polls `/su/liveupdate/popular/?treeIds=...` every 3 s but the response
  is just a refresh signal (`{"modified":[{"type":"refreshPage"}],"updated":...}`)
  — **the site relies on full HTML re-fetch for live updates**, which is good for us
  (no separate JSON contract to track).

### 1.3 Sport-specific listing — `/su/popular/Basketball` / `/su/betting/Basketball+-+6`

- HTTP 200, ~470 KB.
- Lists all current basketball categories (NBA Playoffs etc.) with full odds.
- URL by name (`Basketball`) and URL by sport tree ID (`Basketball+-+6`) both work.
- Date display: events on the same day show **time only** (`03:00`); events on
  later days show **`DD <month-ru> HH:MM`** (e.g., `06 мая 02:00`). The "today"
  anchor is implicit — must be derived from `initData.serverTime`.

### 1.4 Event detail — `/su/betting/<event-path>`

- HTTP 200, ~500 KB to ~1.6 MB depending on market count.
- URL pattern: `/su/betting/<Sport>/<League+Path>/<Sub+Stage>/<Team1+vs+Team2+-+<treeId>>`.
- Exposes ~140–250 unique market types per event. Each market is a `<div>` containing
  a labeled `<table>` of selections with `data-selection-key`, prices, and handicap/total
  values in `<span class="middle-simple">`.
- **Schema.org breadcrumb** at the bottom of the page provides clean taxonomy:
  Sport → Country/Group → League → Stage → Event. Each level has its own treeId visible
  in `href="/su/betting/<path>+-+<treeId>"`.
- Sample (Football, Arsenal vs Atletico Madrid, treeId 28089645, eventId 26456117):
  - Sport = `Football+-+11`, Country group = `Clubs.+International+-+4409575`,
    League = `UEFA+Champions+League+-+21255`, Stage = `Play-Offs / Semi+Final / 2nd+Leg`.
  - Match-level markets: `Match_Result.{1,draw,3}`, `To_Win_Match_With_Handicap{N}.{HB_H,HB_A}`,
    `Total_Goals{N}.{Under_X,Over_X}`.

### 1.5 Results / archive — **NOT publicly available**

- `https://www.marathonbet.by/su/results` → **HTTP 404**.
- `https://www.marathonbet.by/su/results/` → **HTTP 404**.
- `https://www.marathonbet.by/su/results.htm` → **HTTP 404**.
- No `/results`, `/archive`, or `/history` link anywhere in the public landing-page HTML.
- The `eventJsonInfo` `<td>` on each event has a `matchIsComplete` boolean and a
  `resultDescription` (e.g., `"2:1 (1:1)"`), so **final scores can be captured by
  re-scraping the event detail page after match end** — but only while the event is
  still hosted (likely a few hours / days post-match). After cleanup, results are gone.
- **Implication for Phase 8 (Results loader):** results must be harvested by
  continuing to poll the event detail page until `matchIsComplete=true`, then storing
  the final score. There is no historical archive endpoint to back-fill from. We
  should also evaluate scraping a third-party results aggregator
  (flashscore, livescore, sofascore) as a fallback — that's a Phase 8 design decision.

---

## 2. Anti-bot Posture

| Signal | Observation |
|---|---|
| Cloudflare | Absent. `Server: nginx`, no `cf-*` headers. |
| reCAPTCHA / hCAPTCHA | Not on public listing or event pages (only on `/captchaData.htm` for login). |
| User-Agent filtering | A browser UA returns 200. We did not test with `curl/8.x` or empty UA — recommend always sending a real UA. |
| Cookie requirement | None for read-only access. The site sets `puid`, `JSESSIONID`, `lastSitePart`, etc., but we observed full HTML on the very first request without prior cookies. |
| IP rate-limit | 5 sequential requests at ~1s pacing all returned 200 in <1 s. No throttling observed within our budget (10 total requests). The customer should test heavier loads from their environment. |
| Geo-block | Probe environment is geo-routed as Poland; site still serves `/su` Russian content. Customer (Belarus) should see same or better access. |
| Fingerprinting | Standard analytics (GTM, dataLayer); no JS-fingerprint cookies or canvas hashing detected in the entry-page payload. |

**Mitigations to bake into the scraper anyway** (defense-in-depth):

- **Rotate User-Agents** from a small pool of recent Chrome/Firefox/Edge versions
  (configurable via `Scraping:UserAgents[]`).
- **Polite pacing:** default `Scraping:RateLimit:RequestsPerSecond = 1`,
  `MaxConcurrentRequests = 4`. Per-host token-bucket rate limiter using Polly v8 +
  `Microsoft.Extensions.Http.Resilience`.
- **Honor `Cache-Control: no-store`** — do NOT cache responses; that's the site's intent.
- **Handle 403 / 429 / 503** with exponential backoff and circuit breaker; alert the user
  when circuit opens for >5 minutes.
- **Cookie jar per scraper instance** — accept set-cookies and replay them. This avoids
  a session-creation latency on every request.
- **Belarus-specific:** if customer's environment ever sees a `/forbidden` redirect,
  we fall back to the `afterForbiddenRedirectUrl` documented in `initData`.

---

## 3. URL Templates Phase 3 Will Use

| Purpose | Template | Notes |
|---|---|---|
| Pre-match top page | `https://www.marathonbet.by/su/` | Mixed live + popular pre-match. Use only for landing/health-check. |
| Live top page | `https://www.marathonbet.by/su/live` | Mixed sports. Use for live-event discovery. |
| Live popular | `https://www.marathonbet.by/su/live/popular` | Same data as `/su/live`. |
| All-events index | `https://www.marathonbet.by/su/all-events/` | Long full list; use for discovery seed. |
| Sport listing (by ID) | `https://www.marathonbet.by/su/betting/{Sport}+-+{sportId}` | e.g., `/su/betting/Basketball+-+6`. **Preferred** because sport-id stable. |
| Sport listing (by name) | `https://www.marathonbet.by/su/popular/{Sport}` | e.g., `/su/popular/Basketball`. Convenient for humans. |
| Category / league listing | `https://www.marathonbet.by/su/betting/{Sport}/{League+Path}+-+{categoryTreeId}` | From breadcrumbs / `category-label-link`. |
| Event detail | `https://www.marathonbet.by/su/betting/{event-path}` | `event-path` from `data-event-path`, ends in `-+{treeId}`. |
| Live update signal | `https://www.marathonbet.by/su/liveupdate/popular/?treeIds={csv}` | Returns `{"modified":[...],"updated":<ts>}`. Use only as "hey something changed" hint; full odds still come from event-detail re-fetch. |
| Server time sync | `https://www.marathonbet.by/su/stateless/synctime` | Use to anchor "today" date interpretation. |

URL paths use `+` for spaces, `%2C` for `,`, etc. — standard `Uri.EscapeDataString`.

---

## 4. Sport ID Inventory (observed)

From the pre-match landing page (`data-sport-treeId` attributes + `category-label`
breadcrumb hrefs):

| Sport ID | Russian name | English path |
|---|---|---|
| **6** | Баскетбол | `Basketball` |
| **11** | Футбол | `Football` |
| **537** | (TBD — verify on populated day) | — |
| **2398** | (TBD) | — |
| **22723** | Теннис | `Tennis` |
| **26418** | Футбол (alt? duplicate live) | `Football` |
| **43658** | Хоккей | `Hockey` |
| **45356** | Баскетбол (live tree) | `Basketball` |
| **139722** | Гандбол | `Handball` |
| **414329** | Настольный теннис | `Table+Tennis` |
| **1372932** | Киберспорт | `Esports` |
| **3083982** | Лотереи | `Lotteries` |
| **11308234** | Шорт хоккей | `Short+Hockey` |
| **23054364** | Кибербаскетбол | `eBasketball` |
| **23054392** | Киберфутбол | `eFootball` |

**Important observation:** the site has **two parallel tree IDs per sport** — one
"canonical" (e.g., `6` for Basketball) used on event-detail breadcrumb, and a
"category" tree ID (e.g., `45356`) used inside the live grouping. Phase 1 domain
needs to recognize the canonical ID as `SportCode` and ignore the category tree ID.

The customer-spec field `Sport_Code = 6` for Basketball matches the canonical ID
in `data-sport-treeId="6"` and in the breadcrumb URL `/su/betting/Basketball+-+6`.

---

## 5. Bet Selection Naming Convention

Format: `{eventId}@{MarketName}{LineIndex?}.{Outcome}`

Where:

- `eventId` = bookmaker's `data-event-eventId` (numeric, ~26-million range, stable).
- `MarketName` = `Match_Result`, `To_Win_Match_With_Handicap`, `Total_Points`,
  `1st_Half_Result`, `To_Win_1st_Half_With_Handicap`, `1st_Set_Total_Games`, etc.
- `LineIndex?` = optional integer suffix when a market has multiple lines/spreads
  (e.g., `Total_Points10`, `Total_Points11` are different total thresholds for the
  same event). Empty / `0` is the "main" line.
- `Outcome` codes:
  - `1`, `draw`, `3` — for 3-way result markets
  - `HB_H`, `HB_A` — handicap home/away
  - `Under_<X>`, `Over_<X>` — total under/over (X is the threshold, embedded in name)
  - `HD`, `AD` — half-time/full-time draw combinations
  - `yes` / `no` — for yes/no markets

The handicap value (`+1.0`, `-2.5`) and total threshold (`213.5`) are NOT in the
selection key as parseable numbers — they live in the `<span class="middle-simple">`
display element OR they are embedded in the outcome name (e.g., `Under_213.5`).

---

## 6. Period Scope per Sport (observed)

| Sport | Period scopes available | Spec field prefix |
|---|---|---|
| Football (11) | 1st Half, 2nd Half | `Bet_Period-1_*`, `Bet_Period-2_*` |
| Basketball (6) | 1st/2nd Half, 1st/2nd/3rd/4th Quarter | Customer must clarify whether Period-N maps to halves or quarters. **Recommend halves** as default (Period-1, Period-2) with an `appsettings` toggle for quarter-mode. |
| Tennis (22723) | 1st Set, 2nd Set, ... (variable count) | `Bet_Period-1_*` = 1st Set, etc. **No Draw outcome.** |
| Hockey (43658) | 1st/2nd/3rd Period | `Bet_Period-1_*`, `Bet_Period-2_*`, `Bet_Period-3_*` (not yet sampled — revalidate in Phase 3). |

The internal market-name token is sport-dependent:
- `1st_Half_Result`, `To_Win_1st_Half_With_Handicap`
- `1st_Quarter_Result`, `To_Win_1st_Quarter_With_Handicap`
- `1st_Set_Result`, `To_Win_1st_Set_With_Handicap`

**Phase 3 should encapsulate this** in a sport-aware mapping table
(`PeriodScopeMapper`) keyed on `SportCode`, returning the set of expected period
markets and their token names.

---

## 7. Open Questions / Risks

1. **Results storage cleanup:** how long does marathonbet keep finished events on
   the event detail URL? Must be empirically tested over Phase 8. Recommend retaining
   our own snapshot with `matchIsComplete=true` permanently in SQLite as soon as
   we observe it, so we never depend on the site for historical data.
2. **Sport ID duplication** (e.g., `26418` and `11` both = Football):
   verify with customer that we should use the canonical breadcrumb ID. The
   "category" trees may exist for live grouping or alphabetization purposes.
3. **Localization:** site labels are Russian on `/su/`. There appears to be `/en/`
   path support (untested). Customer wants RU + EN — Phase 5 must verify EN locale
   page parses identically.
4. **Period total markets in basketball:** sampled NBA event did NOT explicitly
   expose "Total points 1st quarter" as a clean market in the public HTML — only
   `AllInningsGoalsOver` (combined). Customer's spec implies `Bet_Period-N_Total_*`
   is universal — Phase 3 must gracefully degrade and emit `null` rates for fields
   the site doesn't surface for that sport+league.
5. **Belarus geo-restriction risk:** we tested from non-BY. If customer's BY IP
   gets a different page (KYC overlay, deposit prompt, etc.), the parser must be
   robust to unexpected wrapping. Defensive parsing only — never assume strict
   structure.
6. **`isLogged: false` overlay risk:** initData reports we are anonymous. Some
   markets may be hidden behind login (we did not detect any in samples, but the
   parser should treat missing markets as `null`, not throw).

---

## 8. Recommended Phase 3 Architecture

```
IOddsScraper (Application)
   │
   └── MarathonBetScraper : IOddsScraper (Infrastructure)
        ├── HttpClient (resilient via Polly v8)
        │     ├── User-Agent rotator
        │     ├── Token-bucket rate limiter (config: RequestsPerSecond)
        │     ├── Retry policy (3x exponential backoff, jitter)
        │     └── Circuit breaker (open after N consecutive 5xx)
        │
        ├── EventDiscoveryParser  ← parses /su/, /su/live, /su/popular/{sport}
        │     produces List<EventListItem>
        │
        ├── EventDetailParser     ← parses /su/betting/<path>
        │     produces FullOddsSnapshot with all markets
        │
        ├── BreadcrumbParser      ← extracts Sport / Country / League / Stage taxonomy
        │
        └── BetMarketMapper       ← AngleSharp QuerySelector → spec field name
              (sport-aware; uses PeriodScopeMapper)
```

**Use AngleSharp for parsing** — it handles malformed HTML well, has a CSS-selector
API, and is the established `.NET` choice. JSON islands inside attributes (`data-sel`,
`data-json`) decode cleanly with `System.Text.Json`.

**No Playwright required** for the scraper. Keep Playwright as a documented
fallback in `appsettings` (`Scraping:UsePlaywright = false`) so we can flip it on
later if the site adds JS challenges. This adds <100 LOC of optional code, costs
nothing if unused.

---

## 9. Customer Validation Plan

If our environment ever stops working (geo-block, IP ban, etc.) the customer in
Belarus can:

1. Open https://www.marathonbet.by/su in a browser, verify it renders.
2. View page source (Ctrl+U), search for `data-event-eventId` — confirm same
   structure as our captured `spike/captures/pre-match-landing.html`.
3. Save the HTML and email it to dev — the parser is environment-agnostic and
   should handle their captured HTML byte-for-byte.

This decouples scraper development from probe environment and makes Phase 3
testable offline.

---

## 10. Captured Samples (gitignored, local only)

| File | Purpose |
|---|---|
| `spike/captures/pre-match-landing.html` | `/su/` snapshot, 587 KB, full grid |
| `spike/captures/live-landing.html` | `/su/live` snapshot, 250 KB |
| `spike/captures/basketball-listing.html` | `/su/popular/Basketball`, 471 KB |
| `spike/captures/event-basketball-28405506.html` | NBA Knicks vs 76ers full event, 505 KB |
| `spike/captures/event-football-28089645.html` | UCL Arsenal vs Atletico full event, 1.58 MB |
| `spike/captures/event-tennis-28430484.html` | ATP Rome qualif full event, 244 KB |
| `spike/captures/liveupdate-popular.json` | Live-update API sample response |
| `spike/captures/results-page.html` | `/su/results` response (~20 KB) — captured to evidence the missing public archive endpoint (Phase 8 deviation). |

These artifacts are **not committed** but should be kept locally to back parser unit
tests in Phase 3.

> **Caveats on captures:**
>
> - `live-landing.html` was captured at a moment when no live events were
>   in-progress for popular sports. As a result, the `.score-state` element
>   referenced in `SCHEMA_DRAFT.md` §1 is NOT present in this particular capture.
>   Phase 3 should re-verify the score selector against a live event during
>   parser implementation (the selector itself is well-known across bookmaker
>   sites and not in doubt).
> - Hockey events were not sampled directly. Period-result selection key tokens
>   for hockey (`1st_Period_Result0.RN_H` etc.) are extrapolated from the
>   football/basketball/tennis pattern and marked TBD in `SCHEMA_DRAFT.md`. Phase 3
>   must verify against a real hockey event before relying on those tokens.