Embedding Extraction for Arabic Text Using the AraBERT Model  

作　　者：Amira Hamed Abo-Elghit Taher Hamza Aya Al-Zoghby 

机构地区：[1]Faculty of Computers and Information,Department of Computer Sciences,Mansoura University,Mansoura,35516,Egypt [2]Faculty of Computers and Artificial Intelligence,Department of Computer Sciences,Damietta University,Damietta,34517,Egypt

出　　处：《Computers, Materials & Continua》2022年第7期1967-1994,共28页计算机、材料和连续体（英文）

摘　　要：Nowadays,we can use the multi-task learning approach to train a machine-learning algorithm to learn multiple related tasks instead of training it to solve a single task.In this work,we propose an algorithm for estimating textual similarity scores and then use these scores in multiple tasks such as text ranking,essay grading,and question answering systems.We used several vectorization schemes to represent the Arabic texts in the SemEval2017-task3-subtask-D dataset.The used schemes include lexical-based similarity features,frequency-based features,and pre-trained model-based features.Also,we used contextual-based embedding models such as Arabic Bidirectional Encoder Representations from Transformers(AraBERT).We used the AraBERT model in two different variants.First,as a feature extractor in addition to the text vectorization schemes’features.We fed those features to various regression models to make a prediction value that represents the relevancy score between Arabic text units.Second,AraBERT is adopted as a pre-trained model,and its parameters are fine-tuned to estimate the relevancy scores between Arabic textual sentences.To evaluate the research results,we conducted several experiments to compare the use of the AraBERT model in its two variants.In terms of Mean Absolute Percentage Error(MAPE),the results showminor variance between AraBERT v0.2 as a feature extractor(21.7723)and the fine-tuned AraBERT v2(21.8211).On the other hand,AraBERT v0.2-Large as a feature extractor outperforms the finetuned AraBERT v2 model on the used data set in terms of the coefficient of determination(R2)values(0.014050,−0.032861),respectively.

关 键 词：Semantic textual similarity arabic language EMBEDDINGS AraBERT pre-trained models regression contextual-based models concurrency concept 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]