Text Simplification Using Transformer and BERT  

作　　者：Sarah Alissa Mike Wald 

机构地区：[1]College of Computer Science and Information Technology,Imam Abdulrahman Bin Faisal University,Dammam,Saudi Arabia [2]School of Electronics and Computer Science,University of Southampton,Southampton,United Kingdom

出　　处：《Computers, Materials & Continua》2023年第5期3479-3495,共17页计算机、材料和连续体（英文）

摘　　要：Reading and writing are the main interaction methods with web content.Text simplification tools are helpful for people with cognitive impairments,new language learners,and children as they might find difficulties in understanding the complex web content.Text simplification is the process of changing complex text intomore readable and understandable text.The recent approaches to text simplification adopted the machine translation concept to learn simplification rules from a parallel corpus of complex and simple sentences.In this paper,we propose two models based on the transformer which is an encoder-decoder structure that achieves state-of-the-art(SOTA)results in machine translation.The training process for our model includes three steps:preprocessing the data using a subword tokenizer,training the model and optimizing it using the Adam optimizer,then using the model to decode the output.The first model uses the transformer only and the second model uses and integrates the Bidirectional Encoder Representations from Transformer(BERT)as encoder to enhance the training time and results.The performance of the proposed model using the transformerwas evaluated using the Bilingual Evaluation Understudy score(BLEU)and recorded(53.78)on the WikiSmall dataset.On the other hand,the experiment on the second model which is integrated with BERT shows that the validation loss decreased very fast compared with the model without the BERT.However,the BLEU score was small(44.54),which could be due to the size of the dataset so the model was overfitting and unable to generalize well.Therefore,in the future,the second model could involve experimenting with a larger dataset such as the WikiLarge.In addition,more analysis has been done on the model’s results and the used dataset using different evaluation metrics to understand their performance.

关 键 词：Text simplification neural machine translation TRANSFORMER 

分 类 号：TP391.2[自动化与计算机技术—计算机应用技术]