Research on risk URL detection based on Boosting ensemble learning
Li Yun 1,2，Jiang Bing 1,2，Wang Lisong 1,2，Liu Chunbo3，Chen Wei1,2
1. Operation Center,TravelSky Technology Limited; 2. IT Infrastructure Localization Adaptation Engineering Technology Research Center,TravelSky Technology Limited 3. Information Security Evaluation Center, Civil Aviation University of China
Abstract： With the continuous development of the Internet and the growing number of websites, URL, as the only access to websites, has become the focus of web attacks. The traditional URL detection method mainly targets malicious URLs, based on feature values and black-and-white lists, but it is prone to false positives and lacks detection capability for complex URLs. To resolve the appeal issue, a hybrid model for risk URL detection in business access is proposed based on the Boosting concept in ensemble learning. In the early stage of this model, the URL is treated as a string, and natural language processing techniques are used to segment and vectorize it. Then, a two-step approach is adopted. Firstly, the GBDT algorithm is used to construct a binary classification model to determine whether the URL is at risk. Then, the original string of the risk URL is input into a multi classification model, and the XGBoost algorithm is used to perform multi classification judgment on it, clarifying the specific risk types of the risk URL and providing reference for security analysts. During the model construction process, parameter optimization was continuously carried out, and the AUC value and F1 value were used to evaluate the binary classification model and the multi classification model, respectively. The evaluation results showed that the AUC value of the binary classification model was 98.91%, and the F1 value of the multi classification model was 0.993, indicating good performance. Applying it to practical environments and comparing it with existing detection methods, it was found that the detection rate of the model is higher than that of existing WAF and APT detection devices, and its detection results make up for the missed reports of existing detection methods.
Key words : web attacks; ensemble learning; regularization; stepwise modeling method
