Research Exposes Hidden Data Splitting in Sequential Recommendation Models, Questioning SOTA Claims

What Happened

A new preprint on arXiv, "Pay Attention to Sequence Split: Uncovering the Impacts of Sub-Sequence Splitting on Sequential Recommendation Models," delivers a critical audit of a common but often undisclosed practice in AI research for recommender systems. The paper investigates Sub-Sequence Splitting (SSS), a technique used to mitigate data sparsity by splitting a user's long interaction history (e.g., clicks, views, purchases) into multiple shorter sequences. While previous work has shown SSS can boost performance, this research reveals a more troubling reality: many recent papers claiming state-of-the-art (SOTA) results for Sequential Recommendation (SR) models are secretly using SSS during data preprocessing without reporting it.

The core findings are threefold:

1. SSS interferes with fair model evaluation. The authors discovered that when they removed the unmentioned SSS operation from several recent SOTA models, their performance "significantly declined, even falling below that of earlier classical SR models." This suggests the reported advancements may be attributable more to data manipulation than to superior model architecture.
2. SSS is not a universal booster. Its effectiveness is highly contingent on a specific combination of the splitting method (e.g., sliding window, random split), the training target strategy, and the loss function. An inappropriate combination can actually harm model performance.
3. SSS works by altering data distributions. The analysis indicates that SSS improves performance primarily by creating a more balanced training data distribution and increasing the variety of items that serve as prediction targets during training, rather than by capturing more nuanced user intent.

The paper concludes with a call to action for the research community to adopt more transparent and rigorous evaluation protocols, providing code to help others audit their own models.

Technical Details

Sequential Recommendation (SR) is the task of predicting a user's next likely interaction (e.g., the next product to buy) based on their historical sequence of actions. Training data sparsity—where users have few interactions—is a perennial challenge.

SSS is a form of data augmentation. A raw sequence like [A, B, C, D, E] (five interactions) might be split, using a sliding window of length 3, into sub-sequences [A, B, C] -> D, [B, C, D] -> E. This artificially creates more training samples from limited data. The paper meticulously tests SSS across different dimensions:

• Splitting Methods: Sliding window, time-based, and random splits.
• Target Strategies: Whether the model predicts the very next item or a future item within the sub-sequence.
• Loss Functions: Common choices like Bayesian Personalized Ranking (BPR) and Binary Cross-Entropy (BCE).

The key technical insight is that the benefit of SSS is not inherent to the model's ability to understand sequence dynamics. Instead, it's a statistical effect: by creating more (and shorter) sequences, SSS ensures that a wider array of items appear as the "next item" target during training, which can improve the model's overall item coverage and reduce overfitting to frequent items. However, this can come at the cost of losing the context of very long-term user patterns.

Retail & Luxury Implications

For retail and luxury companies investing in next-product-to-buy or next-content-to-view algorithms, this research is a crucial reminder to scrutinize the provenance and evaluation of the models they consider deploying or building in-house.

1. Vendor & Model Evaluation: If an AI vendor or research team claims a new SR model delivers breakthrough accuracy, technical leaders must ask: Was sub-sequence splitting used? If so, how? The paper shows that performance gains from an undisclosed SSS pipeline may not translate to real-world deployment where the model must predict on complete, unsplit user histories. A model that excels on split data may fail on holistic user journeys.

2. In-House R&D Rigor: Internal data science teams building recommendation engines must adopt the transparent benchmarking practices advocated by this paper. Before declaring a new model architecture successful, teams should run ablation studies with and without SSS to understand the true source of performance deltas. This prevents wasted effort optimizing a data trick rather than fundamental model capabilities.

3. Application-Specific Suitability: The finding that SSS effectiveness depends on the specific combination of techniques is critical. A luxury retailer modeling a customer's multi-year journey toward a high-consideration purchase (like a handbag or watch) may rely on understanding long, coherent sequences. Blindly applying a sliding-window split could destroy the long-horizon intent signals the business needs to capture. The choice of splitting strategy must be intentional and aligned with the business context.

Ultimately, this paper doesn't invalidate SSS as a tool—it can be a legitimate technique for dealing with sparse data. The warning is against its unreported use, which creates an uneven playing field and obscures true model innovation. For practitioners, the mandate is clear: demand transparency and validate claims on evaluation methodologies, not just final metrics.