A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

The paper proposes PatchTST, a Transformer-based model for long-term multivariate time series forecasting and self-supervised representation learning. Its core method combines two components: segmenting each time series into subseries-level patches that serve as input tokens, and a channel-independence scheme in which every channel holds a single univariate series and shares the same embedding and Transformer weights across all series. Patching keeps local semantic information in the embeddings, reduces the computation and memory of attention maps quadratically for a given look-back window, and allows the model to attend to a longer history.

The authors report that PatchTST significantly improves long-term forecasting accuracy relative to state-of-the-art Transformer-based models. Applied to self-supervised pre-training, it achieves excellent fine-tuning performance that surpasses supervised training on large datasets, and masked pre-trained representations transferred from one dataset to others also reach state-of-the-art forecasting accuracy. This mattered because it showed a simple, efficient Transformer design that advances both accuracy and scalable representation learning for time series.