Deep Reinforcement Learning for Weather-Aware Maritime Route Optimisation:
A Hexagonal Grid Approach

International shipping accounts for approximately 3% of global greenhouse gas emissions. The IMO's 2023 revised strategy targets net-zero emissions by 2050, with an interim 20–30% reduction by 2030 relative to 2008 levels. Voyage optimisation — choosing the safest, most fuel-efficient route through dynamic, weather-varying ocean conditions — is one of the most tractable near-term levers available to fleet operators today.

Classical methods (A*, Dijkstra, Tabu Search) are fast and interpretable, but static: they plan once and cannot adapt to evolving weather mid-voyage. Reinforcement learning agents, by contrast, learn a policy that can be continuously updated online as new forecasts arrive.

Latinopoulos et al. (JMSE, 2025) is the state of the art: DDQN and DDPG agents on a uniform 0.25° lat/lon grid, achieving 8.7–12.6% fuel savings over a shortest-path baseline on a real Mediterranean route. It is a strong result — and it was this paper that motivated this project. But it has two structural limitations:

• The uniform 28 km grid cannot navigate straits narrower than one cell. Canals are handled with teleportation shortcuts, not actual navigation.
• The rectangular grid introduces diagonal bias (NE step = 41% longer than N step), forcing the Q-network to learn a geometry correction.

Before training any neural network, a full suite of heuristic pathfinding algorithms was implemented on the same routes, mirroring the baseline comparison in Latinopoulos et al.

• Neighbour ordering: H3's grid_ring() returns neighbours in unspecified order. Fix: sorted by bearing (clockwise from North), hex-ID as tiebreaker. Cached per cell. Action semantics stable across entire training run.
• Global RNG state: NumPy and Python's random module seeded at grid construction (seed=42) and at each episode reset. Environment transitions fully reproducible for a given seed.
• BFS potential function: Precomputed once after canal integration, cached as a static dict. Every episode uses the identical distance-to-goal mapping — the reward signal does not drift.

Combined effect: given the same seed, two training runs produce identical environment transitions. Variance in results reflects only policy learning, not environment noise.

The adaptive boundary is computed automatically: any res-5 water cell adjacent to a land cell is replaced by its 7 res-6 children, each independently classified by polygon land-fraction (threshold ≥ 0.20 → excluded). This is a polygon-level check, not a centroid check. Result: 10,937 water/coastal cells across the Mediterranean.

The 6.3 km wide, 6 km long Corinth Canal connects the Aegean to the Ionian. Rather than teleporting the agent, we model it as a navigable res-6 hex corridor extracted from OpenStreetMap geometry:

• Sample canal centreline at ~80 m resolution
• Snap to res-6 hexes → compact corridor of water cells
• Add ring-1 buffer (neighbouring hexes) for stability
• Bridge corridor endpoints to open-sea res-5 cells with explicit bidirectional edges

The agent learns to enter, traverse, and exit the canal as part of its policy — no special-case logic in the reward or step function. Other straits handled automatically by res-6 refinement: Bonifacio (~12 km), Gibraltar (~14 km), Messina (~3 km, borderline).

Action space: 7 discrete actions — 6 H3 ring-1 neighbours sorted clockwise from North + STAY. Binary action mask m ∈ {0,1}⁷ prevents stepping onto land; computed once and cached per hex.

Failure on Piraeus: reward rose from −773 → +315/episode over 1,500 episodes while success rate stayed at exactly 0%. Agent farmed the progress signal through oscillation. The non-Markovian step-efficiency term violated the MDP assumption entirely.

The reward function has 13 interacting parameters. Manual grid search is infeasible. Optuna's TPE sampler (Tree-structured Parzen Estimator) was run in parallel on NVIDIA L4 GPUs via Modal cloud compute (~$50 / 50 trials). Search space: progress_scale [8–30], goal_reward [20–100], land_penalty, ping-pong/loop3 coefficients, revisit_multiplier, time_penalty, gamma [0.93–0.999], clip bounds.

Fitness: success_rate − 0.2×clipping_rate − 0.05×orbit_rate. H3 grid built once per worker; only env.reset() between trials.

Cold-start problem: random-walk exploration covers ≈√500 ≈ 22 hops in a 500-step episode; the goal is 107 hops away — effectively invisible.

BFS-seeded replay buffer pre-populates with optimal trajectories, bootstrapping Q-values before exploration begins.

• Bug 1 — goal_threshold = 0: Required landing on 1 exact hex out of 10,937. P(cold-start success in 500 steps) ≈ 1.6×10⁻⁵. Fix: threshold = 2 hops (19 hexes within ~56 km count as success).
• Bug 2 — Canal exit asymmetry: res-6 canal hexes could not exit to res-5 open sea. BFS debug confirmed a hard wall at exactly 54 hops — 661 episodes reached that boundary; zero crossed it in 1,500 episodes. Fix: symmetric bidirectional cross-resolution edges using max(res5_edge, res6_edge) threshold.

Data: ECMWF ERA5 (Copernicus CDS) — 10m wind (U,V), significant wave height, mean wave direction & period, SST, MSLP. Resolution: 0.25°×0.25°, 3-hourly. Coverage: 34°N–46°N, −6°E–30°E. Processing: WeatherProcessor — load GRIB, spatial interpolation onto H3 nodes, temporal interpolation for ship ETA at each hex, cache as NetCDF.

Integration plan: (1) Add weather features to 8D state vector; (2) Wave-heading penalty; (3) Validate weather-aware vs weather-agnostic agent; (4) Online fine-tuning every 6 hours (Latinopoulos: ~15 min GPU update — viable for operational shipping).

Current limitation (shared with Latinopoulos): fuel modelled with fixed analytical formula. Real ships burn non-linearly as a function of sea state, heading, loading, and hull fouling. Proposed: MLP or LSTM trained on AIS voyage logs + ERA5 (no proprietary data required). Inputs: trim, speed, currents, wave height/direction/period, SST. Output: L/hour. Integration: replace r_fuel = fixed × distance with r_fuel = FOC_model(state, action, weather) × Δt — closing the loop so the agent directly optimises real predicted fuel burn.

• H3 hexagons eliminate diagonal bias — Q-values directly encode navigational value, not a geometry correction artefact. All 6 moves are equidistant.
• Adaptive resolution unlocks coastal navigation — 3.7 km res-6 cells make the Strait of Bonifacio (12 km) and Corinth Canal (6 km) naturally navigable without teleportation.
• Reward hacking is real and detectable — potential-based shaping (Ng et al. 1999) with a BFS-derived potential eliminates this class of failure by construction.
• Penalty scaling is not optional — the same values that work on 371 km produce an 818% burden on 1,647 km; λ = min(0.4, BFS_hops/100) makes reward portable across route lengths.
• Short and medium routes are solved — Port Torres 100%, Cagliari 99%. Piraeus bugs identified, fixed, re-run in progress.
• The remaining work is integration, not invention — weather pipeline built, canal model corrected, reward validated, Optuna infrastructure proven.