Validation
How the engine is checked: against reference calculations, against tournament results, and next against measurements on the lake.
In plain words
Three kinds of checks, each answering a different question:
- Golden tests: does the app compute what the reference scripts compute? Every formula is ported from Python scripts, and the ports must reproduce their results.
- The tournament backtest: does the plan point to the water where tournaments were actually won, using only what was known before each event?
- Measurements: do modeled waves, over-water wind and lake temperature match what buoys and gauges measure? This is the next step.
Golden tests
Run now, in this browser, on the same engine code the plan uses, against the reference values in data/golden/engine_golden.json:
| Check | Largest difference | Tolerance | Result |
|---|---|---|---|
| Wave growth, 45 cases: SPM height | 0.0005 ft | 0.05 ft | pass |
| Wave growth: SPM period | 0.0005 s | 0.05 s | pass |
| Wave growth: Young-Verhagen height | 0.0005 ft | 0.05 ft | pass |
| Water temperature projection, 12 days | 0.0005 F | 0.01 F | pass |
| Largest wave ratio, sqrt(ln 1000 / 2) | 1.8585 shown as 1.86 | rounding | pass |
The full test suite (npm test, before every change) adds more: the area wave table (320 rows) and the route waves from the Metropark (240 rows) of the trip analysis to 0.1 ft; fetch for 10 areas in 8 directions to 0.1 km; the Oct 7-11 regression and climatology; the daily front rule's 15 front days on Selfridge data; sunrise and sunset; season phases and depth bands; area scoring, lure scoring and the day schedule on Lake St. Clair scenarios; and that the figures in the wave disclaimer follow from the wave model.
Source: SPEC 13.1; packages/engine/test.
Tournament backtest
For each in-lake tournament with a known winning area, the engine builds the plan for day 1 from the weather that actually happened (hourly observations at Selfridge, gaps filled from Detroit Metro, land wind times 1.2 for over-water wind), the GLSEA lake temperature, and the generic profile of SPEC 13.3: deep-V, 2.0 ft wave limit, both licences, all lures, the Metropark launch. Only tournaments before each event count as evidence (no peeking). The gate passes when the winner's area is in the plan's top 3 and the winner's main lure family is in the top 3 lures, each for at least 8 events.
Latest run, 2026-10-02: PASS. Winner's area in the top 3: 9 of 11 (needed 8). Main lure family in the top 3: 8 of 10 events that reported their baits (needed 8).
- For comparison, a plan that simply picks the 3 areas with the most earlier wins hits 6 of 11.
- With the app's crossing rule on, as for a user: 9 of 11. The gate scores with it off, because tournament anglers crossed open water in winds the rule does not allow.
| Event, day 1 | Lake | Phase, front | Winner's area | Plan's top 3 (score) | Hit | Winner's lure | Top 3 lures there | Hit |
|---|---|---|---|---|---|---|---|---|
| Bassmaster Elite 2015, 2015-08-27 | 69.3 F | Summer, stable | CA1 | US1 68.9, US5 65.1, CA4 64.1 | no | drop shot | Drop shot, Tube, Paddle-tail swimbait | yes |
| Bassmaster Elite 2017, 2017-08-24 | 71.9 F | Summer, second day after a front | CA2 | US1 66.5, US5 62.7, CA4 61.1 | no | tube | Drop shot, Tube, Paddle-tail swimbait | yes |
| FLW Tour 2018, 2018-06-28 | 72.5 F | Summer, day after a front | CA1 | US1 70.2, CA1 72.5, US5 66.8 | yes | tube | Drop shot, Tube, Paddle-tail swimbait | yes |
| BFL Michigan Division 2018, 2018-07-14 | 75.3 F | Summer, stable | CA1 | CA1 79.8, US1 70.2, US5 67.5 | yes | not reported | Tube, Drop shot, Paddle-tail swimbait | n/a |
| BFL Michigan Division 2019, 2019-08-03 | 74.7 F | Summer, front passing | CA4 | CA1 86.0, US1 70.2, CA4 72.0 | yes | tube | Drop shot, Tube, Paddle-tail swimbait | yes |
| Bassmaster AOY Championship 2019, 2019-09-29 | 65.6 F | Early fall, second day after a front | CA2 | US1 80.2, US5 64.8, CA2 63.7 | yes | crankbait | Tube, Drop shot, Suspending jerkbait 110-115 mm | no |
| Bassmaster Elite 2020, 2020-08-20 | 74.4 F | Summer, before a front | US1 | US1 75.2, US3 63.9, US5 63.6 | yes | drop shot | Drop shot, Paddle-tail swimbait, Medium-diving crankbait | yes |
| MLF Bass Pro Tour Stage Six 2023, 2023-06-29 | 68.6 F | Post-spawn, stable | CA1 | US2 74.3, CA1 71.0, CA2 66.3 | yes | drop shot | Drop shot, Tube, Paddle-tail swimbait | yes |
| Bassmaster Elite 2023, 2023-07-27 | 75.2 F | Summer, stable | CA4 | CA1 86.0, CA4 77.3, US1 74.0 | yes | drop shot | Drop shot, Tube, Paddle-tail swimbait | yes |
| Bassmaster Open 2024, 2024-07-11 | 73.6 F | Summer, day after a front | US1 or CA4 | CA4 80.9, CA1 80.5, US1 74.0 | yes | drop shot | Drop shot, Tube, Paddle-tail swimbait | yes |
| Bassmaster Elite 2025, 2025-08-07 | 75.5 F | Summer, stable | US1 | US1 81.5, CA4 81.4, CA1 78.7 | yes | finesse | Drop shot, Tube, Paddle-tail swimbait | no |
Limits of this test: the observed weather stands in for the forecast, so it tests the scoring, not forecast error. The record is mostly June-August events; only one event falls in early fall. Scoring was tuned in one recorded round and then frozen (2026-10-01), because further tuning on the same 11 events would make the gate meaningless; the next check is on new data: the 2026 events and anglers' logbooks.
Source: SPEC 13.3; full report: reports/backtest-20261002.md (npm run backtest, scripts/backtest/run.ts).
Checks against measurements
Planned: task 1.8 Fit the land-to-water wind factor from buoy 45147 against Selfridge (Wind over water); refit the water temperature coefficients by month on full-year data; compare modeled wave heights with the buoy's measured waves and the NWS wave forecast. The wave disclaimer then gets the measured error in place of today's estimate (Waves).
Plan confidence
Each plan carries a confidence label for how far its data can be trusted, not the chance of a catch (SPEC 5.11):
- High: the date is 3 days ahead or less, the lake temperature is inside the tournament record's 65.7-76.3 F, and the top area has 2 or more supporting records.
- Low: 8 or more days ahead (past the 7-day forecast, climatology only), or the lake more than 5 F outside that range, or a key data feed out of date.
- Medium: everything else.
Source: Engine: planConfidence in packages/engine/src/confidence.ts.
Known limits
- Area positions are approximate centers or outlines from published descriptions, not waypoints.
- The water temperature model was fitted on October days only and runs warm in strong wind (Water temperature page).
- The land-to-water wind factor of 1.2 is a placeholder until task 1.8 measures it.
- The wave models have not yet been checked against measured waves on this lake; gusts, wind shifts, wakes, reflection from walls and current against wind are not modeled.
- Season-phase limits, front thresholds, lure attributes and scoring weights are starting values; the backtest and logbooks check them.
- The NWS forecast has no pressure, so forecast days use the daily front rule.
- The clarity model is not yet calibrated, and river plumes are a constant until river gauges are added.
- The tournament record is mostly June-August; one event falls in early fall.