Big Data Analysis using CUR Algorithm

Abstract

In this thesis, we present the results of two well known analyses, Principal Component Analysis (PCA) and CUR algorithm, conducted on two data sets: one related to clinical test such as COVID-19 test and the other related to electroencephalography (EEG). In some cases, PCA and other dimensionality reduction methods based on the truncation of the Singular Value Decomposition (SVD), could have a problem with the interpretation of their results. It happens because PCA creates new variables that are linear combination of the original variables and, in applications, these not necessarily have a physical interpretation. For that reason we used the CUR Algorithm and applied it to our selected data sets to show that it is easier to interpret the reduced matrix it returns. We also developed the mathematics behind PCA and the CUR Algorithm. For example, we have proven that the CUR Matrix Decomposition (from which the CUR Algorithm is based) exists only using linear algebra. Also we proved that the CUR Matrix Decomposition is as good approximation as the truncation of the SVD. Since the data sets we used are of different dimension, it was useful to see the behavior of the CUR Algorithm when it works with small an large data matrices. For the clinical tests (COVID-19) data set we analyzed the effectiveness of CUR Algorithm over PCA and found out that, for this data matrix, CUR Algorithm is more effective than PCA whenever the control parameters of the CUR algorithm c and k are equal. Furthermore, the results of CUR algorithm suggest that the laboratory tests D-dimer, ferritin and CRP are the most important variables. For the EEG data set, we concluded that the CUR Algorithm is more effective for this data matrix if its input parameters hold: c = 8 and k ≤ 57, c = 16 and k ≤ 50, c = 24 and k ≤ 43, c = 32 and k ≤ 36, c = 40 and k ≤ 29, c = 48 and k ≤ 22, c = 56 and k ≤ 15, c = 64 and k ≤ 8.