Distribution shift occurs when the test distribution differs from the training distribution, and it can considerably degrade performance of machine learning models deployed in the real world. Temporal shifts - distribution shifts arising from the passage of time – often occur gradually and have the additional structure of timestamp metadata. By leveraging timestamp metadata, models can potentially learn from trends in past distribution shifts and extrapolate into the future. While recent works have studied distribution shifts, temporal shifts remain underexplored. To address this gap, we curate Wild-Time, a benchmark of 5 datasets that reflect temporal distribution shifts arising in a variety of real-world applications, including patient prognosis and news classification. We use two evaluation strategies: evaluation with a fixed time split (Eval-Fix) and evaluation with a data stream (Eval-Stream).

We briefly introduce the dataset below. Please refer to our paper for more detailed dataset descriptions

• Yearbook: Social norms, fashion styles, and population demographics change over time. This is captured in the Yearbook dataset, which consists of 37,921 frontal-facing American high school yearbook photos. We exclude portraits from 1905 − 1929 due to the limited number of examples in these years, resulting in 33,431 examples from 1930−2013. Each photo is a 32×32×1 grey-scale image associated with a binary label y, which represents the student’s gender.
• FMoW: Machine learning models can be used to analyze satellite imagery and aid humanitarian and policy efforts by monitoring croplands and predicting crop yield and poverty levels. Due to human activity, satellite imagery changes over time, requiring models that are robust to temporal distribution shifts. We study this problem on the Functional Map of the World (FMoW) dataset, adapted from the WILDS benchmark. Given a satellite image, the task is to predict the type of land usage. The FMoW dataset consists of 141,696 examples from 2002 − 2017.
• MIMIC-IV: Many machine learning healthcare applications have emerged in the last decade However, changes in healthcare over time, such as the emergence of new treatments and changes in patient demographics, are an obstacle in deploying machine learning-based clinical decision support systems. We study this problem on MIMIC-IV, one of the largest public healthcare datasets that comprises abundant medical records of over 40,000 patients. In MIMIC-IV, we treat each admission as one record, resulting in 216,487 healthcare records from 2008 − 2020. Specifically, we consider two classification tasks:
  • MIMIC-Readmission aims to predict the risk of being readmitted to the hospital within 15 days.
  • MIMIC-Mortality aims to predict in-hospital mortality for each patient.
• Huffpost: In many language models which deal with information correlated with time, temporal distribution shifts cause performance degradation in downstream tasks such as Twitter hashtag classification or question-answering systems. Performance drops across time reflect changes in the style or content of current events. We study this temporal shift on the Huffpost dataset. The task is to identify tags of news articles from their headlines. Each input feature x is a news headline, and the output y is the news category
• arXiv: Due to the evolution of research fields, the style of arXiv pre-prints also changes over time, reflected by the change in article categories. For example, “neural network attack" was originally a popular keyword in the security community, but gradually became more prevalent in the machine learning community. We study this temporal shift in the arXiv dataset [18], where the task is to predict the primary category of arXiv pre-prints given the paper title as input. The entire dataset includes 172 pre-print categories from 2007 − 2022.