Impact of Fine-Tuning on Model Training: A Comprehensive Study Using the SQuAD Benchmark

Abstract

Purpose - This study investigates the effect of fine-tuning on model training, and the impact of model training on the environment in terms of carbon footprint emissions. As language modeling evolves, we are obtaining complex and large models with a large number of trained parameters. However, only large enterprises are able to train these large models using large datasets. It is not possible for small and mid-size enterprises due to their computation resources capability. This study proposes a fine-tuning model training pipeline for such enterprises to get the same model capability as large enterprises with less number of model parameters training using SQuAD benchmark. The pre-trained BERT base model is incorporated in combination with fine-tuning methods in this study.
Findings - The results explain us that fine-tuned models need less computation time and less number of trainable parameters without losing too much model capabilities compared to the fully trained model, which indirectly helps us to protect the environment as it emits less carbon footprints.