MixedPEFT: Combining Multiple PEFT Methods with Mixed Objectives for Unsupervised Domain Adaptation

The study introduces MixedPEFT, a novel parameter-efficient strategy for unsupervised domain adaptation that integrates multiple PEFT architectures with mixed-objective training. By leveraging a combination of invertible adapters and Low-Rank Adaptation (LoRA), the method optimizes both classification performance on labeled source data and masked language modeling on unlabeled target data, achieving significant improvements on the MNLI dataset with only 7% of the model's trainable parameters. This approach sets new benchmarks for parameter-efficient domain adaptation, offering practitioners a more effective means to adapt pre-trained language models to new domains without incurring the costs of full fine-tuning.