FINE-TUNING TRANSFORMER-BASED NATURAL LANGUAGE GENERATION ALGORITHMS FOR USDA GRAINS REPORTS FOR FARMERS, PRODUCERS, AND SMALL BUSINESSES

Abstract

The Transformer architecture is a deep learning model that adopts the mechanism of self-attention, the ability of a Transformer model to attend to different parts of the input sequence when making predictions. The Transformer differentially weighs the significance of each part of the input data, learning context and meaning by tracking relationships in sequential data, such as the words in a sentence. For Natural Language Generation (NLG), the application of the Transformer was nothing short of a breakthrough. Self and multi-head attention models have proven their efficacy in a variety of textual tasks, including classification, translation, summarization, and generation. However, incorporating numerical information into textual tasks is a relatively novel frontier, especially when said information is vital for assessing the efficacy of automating the summary generation process. In this research, I focus on evaluating pre-trained Transformer-based NLG models, such as the widely popularized and open-source BERT and T5, on USDA grain reports to produce high-quality, information-rich summaries. I conduct a comparative analysis of these models’ performances on a small collection of numerically-oriented commodities reports, and compute two simple yet informative metrics to judge which models perform the best at relaying numerical information in their respective summaries.