Matching Tasks to Objectives: Fine-Tuning and Prompt-Tuning Strategies for Encoder-Decoder Pre-trained Language Models

This study introduces the Match Task to Objective (MTO) framework, which optimizes encoder-decoder pre-trained language models by aligning pre-training objectives with specific tasks, enhancing performance in generation and question answering tasks, particularly in commonsense knowledge retrieval. The framework employs automated methods for unsupervised data preparation and novel fine-tuning templates, achieving over 120% performance improvement in few-shot settings compared to conventional methods. The findings provide critical insights for practitioners on model customization and prompt-tuning strategies, with the accompanying code available for implementation.