Climate change is likely to affect economies not only through warming, but also via an increase in prolonged extreme events like heat waves. However, the impacts of heat waves on economic output are not well captured by standard empirical approaches that ignore when hot days occur. Using a global dataset spanning 1979–2016, we show agricultural losses from past heat waves are up to an order of magnitude larger than suggested by standard approaches. Combining these estimates with a suite of climate models implies that by the end of the century, climate damages in agriculture may be 5–10 times larger than is predicted by a focus on mean temperature shifts alone. These findings have important implications for targeting and evaluating climate adaptation efforts.