SOHET: Sequence Of Heterogeneous Events Transformer with Self-Supervised Pre-Training

The article introduces SOHET (Sequence Of Heterogeneous Events Transformer), a hierarchical model designed for processing heterogeneous event streams using event-type-specific tabular encoders and temporal embeddings, supported by a causal or bidirectional transformer architecture. It presents three self-supervised pre-training objectives, achieving a 5.8% performance improvement over existing models like FlexTPP on a large-scale fraud detection task with 17 event types, along with a 2.6% gain from pre-training and 2.4% faster convergence. For practitioners, SOHET's architecture and performance enhancements highlight its potential for improving predictive accuracy in complex event-driven applications.