whoscored-event-data v2.0.0

Overview

A rewrite of the original notebook-based scraper (v1.0.0) into a proper package. The event feed is decoded as real JSON, match pages are fetched over HTTP with rate limiting and caching, and a browser is only used when the site throws a bot challenge.

• One request per match — the whole event feed is embedded in a single page.
• Polite by default — 7 s delay + jitter between requests, on-disk caching.
• Auto browser fallback — a 403 challenge is retried through a real browser automatically.
• Backwards compatible — the v1 notebook functions still work via main.py.

Install

pip install -r requirements.txt   # or: pip install -e .
# extras: pip install -r requirements-visuals.txt   (plots)

Dependencies: pandas, numpy, requests, beautifulsoup4, lxml, tqdm. selenium is optional and only needed for Cloudflare-protected listing pages.

Quick start

from whoscored import WhoscoredClient

with WhoscoredClient() as client:
    match = client.get_match(1650630)   # id or full match URL

print(f"{match.home.name} {match.score} {match.away.name}")
# -> Barcelona 0 : 0 Rayo Vallecano

events = match.events          # DataFrame, one row per event (258 cols)
events = match.add_epv()       # + Expected Possession Value column
events.to_csv("events.csv", index=False)

WhoscoredClient

Main entry point. Use it as a context manager.

Constructor options

Match · Fixture · Team

League & fixture discovery

Listing pages are Cloudflare-protected and need a browser. A snapshot of every league is bundled, so list_leagues() works with no network.

with WhoscoredClient(backend="browser") as client:
    fixtures = client.list_fixtures("spain-laliga", "2022/2023")
    lfc = client.team_fixtures("Barcelona", fixtures)

EPV analysis

Expected Possession Value (a model by Laurie Shaw) is bundled with the SDK.

Utilities

Proxies

from whoscored import WhoscoredClient, ProxyRotator

# Static
with WhoscoredClient(proxy="host:port") as client:
    client.get_match(1650630)

# Rotating pool, validated against a neutral endpoint
pool = ProxyRotator(proxies=["p1:8080", "p2:3128"], validate=True)
with WhoscoredClient(proxy_pool=pool) as client:
    client.get_matches(urls)

With a pool, a blocked or dead proxy is skipped automatically; the request only fails once every proxy has been tried. Proxies are validated against gstatic.com, never against Whoscored.

CLI

whoscored --version
whoscored leagues --out leagues.json
whoscored match 1650630 --out data/ --epv
whoscored fixtures spain-laliga 2023/2024 --out fixtures.json
whoscored team "Real Madrid" --fixtures fixtures.json
whoscored scrape --fixtures fixtures.json --out scraped/ --delay 8

python -m whoscored is equivalent to whoscored.

Errors

Politeness & avoiding bans

• Rate limiting on by default (7 s + jitter); tune with request_delay=/jitter= or --delay.
• Set cache_dir so re-runs never touch the site.
• Realistic browser headers are sent by default.
• Challenged pages go through a real headed browser instead of hammering HTTP.

Browser setup

• Ubuntu/Debian snap Firefox: /usr/bin/firefox is a shell wrapper, not a real binary, so geckodriver fails with binary is not a Firefox executable. The SDK auto-detects the real binary (override via binary_location= or FIREFOX_BIN).
• Driver shutdown noise (the PermissionError/Error terminating service process tracebacks and the might not be compatible warning) is silenced automatically. Set SE_DEBUG=1 to restore diagnostics.
• Driver path: found via PATH, or set SE_GECKODRIVER/SE_CHROMEDRIVER.

Examples

Fetch many matches, add EPV, save

from whoscored import WhoscoredClient, save_dataframe

urls = ["https://www.whoscored.com/Matches/1650630/Live/...",
        "https://www.whoscored.com/Matches/1650634/Live/..."]
with WhoscoredClient(cache_dir=".whoscored_cache") as client:
    matches = client.get_matches(urls)

for match in matches:
    save_dataframe(match.add_epv(), f"data/events_{match.match_id}.csv")

A whole season for one team

with WhoscoredClient(backend="browser") as client:
    fixtures = client.list_fixtures("spain-laliga", "2022/2023")
    barca = client.team_fixtures("Barcelona", fixtures)

with WhoscoredClient() as client:
    matches = client.get_matches([f.url for f in barca])

Testing

The suite runs entirely offline against a captured match page — no network, no ban risk.

python test.py        # offline suite (47 tests)
python test.py --live  # + one real match fetch

v1 notebook API

The original notebook functions still work via main.py (with a DeprecationWarning): getLeagueUrls, getMatchUrls, getTeamUrls, getMatchesData, getMatchData, getFixtureData, createEventsDF, createMatchesDF, translateDate, getSortedData, load_EPV_grid, get_EPV_at_location, to_metric_coordinates_from_whoscored, addEpvToDataFrame, main_url.

Repo layout

whoscored/            the SDK package
  client.py           WhoscoredClient (main entry point)
  parser.py           match-centre JSON extraction
  dataframe.py        event / match DataFrame builders
  epv.py              EPV helpers
  models.py           Match / Fixture / League / Team
  transports.py       HTTP + Selenium browser backends
  discovery.py        league / fixture listing
  proxies.py          rotating proxy pool
  cache.py            on-disk cache
  utils.py            RateLimiter, retry, save/load helpers
  cli.py              the `whoscored` command
main.py               v1 compatibility shims
tests/                offline test suite + captured fixture
docs/index.html       this documentation site