PCA (Dimensionality reduction with variance preservation)

PCA (Principal Component Analysis) is a statistical method that compresses high-dimensional data, leaving only the most informative directions. It automatically finds new axes along which the spread of values is maximized and discards insignificant dimensions, simplifying analysis without substantial loss of structure.

The method is widely used in bioinformatics for visualizing genetic markers, in finance for constructing yield curves and reducing portfolio risk dimensionality. In computer vision, PCA is used to extract key features from images, and in marketing for customer segmentation. The algorithm is also indispensable at the data preprocessing stage before training complex models, eliminating multicollinearity among variables.

The main problem is the linearity of the transformation, which makes the method poorly suited for describing nonlinear dependencies on curved manifolds. The result is highly sensitive to outliers and requires careful feature scaling: differences in measurement units distort the directions of the principal components. Moreover, the resulting axes are orthogonal and often lack direct physical interpretation, turning into abstract linear combinations of the original variables that are difficult to explain to a business client.

How PCA works

The working principle is based on eigenvalues and eigenvectors. First, the data is centered by subtracting the mean, then the covariance matrix is computed, showing pairwise relationships between features. Its spectral decomposition reveals eigenvectors, new orthogonal directions in space, and the corresponding eigenvalues determine the amount of explained variance along these directions. Unlike factor analysis, PCA does not model measurement error and aims to maximize variance rather than uncover latent structures. Compared to t-SNE or UMAP, which focus on preserving local neighborhoods for visualization, PCA excellently preserves global structure and is a deterministic algorithm without random initialization, guaranteeing reproducibility of the result. The choice of the number of components is usually made based on cumulative variance, keeping the minimum dimensions for a 90–95% information threshold.

PCA functionality

1. Mathematical formulation of the problem. PCA performs an orthogonal linear transformation of the data into a new coordinate system. The first principal component defines the direction of maximum variance in the sample. Each subsequent component is orthogonal to all previous ones and maximizes the residual variance under the imposed constraints.
2. Centering and standardization of features. Before applying PCA, subtraction of the mean value for each feature is mandatory to shift the origin to the center of the data cloud. When features have different dimensions, additional standardization to unit variance is performed to neutralize the influence of variable measurement scale on the result.
3. Computation of the covariance matrix. After centering, a p×p covariance matrix is calculated, where p is the number of original features. The elements on the main diagonal represent the variances of the variables, and the off-diagonal elements reflect the pairwise linear relationships between the corresponding columns of the object-feature matrix.
4. Eigenvalues and variance. Solving the characteristic equation of the covariance matrix yields a set of eigenvalues λ_i. Each λ_i quantitatively equals the variance of the data along the direction of the corresponding principal component. The sum of all eigenvalues equals the total variance of the original multidimensional data array.
5. Eigenvectors as axes. The eigenvectors corresponding to the computed λ_i define the directions of the principal components in the original feature space. These non-random unit-length vectors form an orthonormal basis. The coefficients of an eigenvector are interpreted as the weights with which the original variables enter the linear combination of the component.
6. Dimensionality reduction. The rank of the data matrix is limited by the number of objects and variables. To reduce dimensionality, the first d principal components are kept, where d is significantly smaller than p. Projecting the centered data onto the truncated basis of d eigenvectors forms a score matrix of minimal dimensionality while preserving maximum information.
7. Scree plot criterion. A graphical method for choosing the number of components consists of analyzing the ordered plot of eigenvalues. The inflection point is sought, where the decline in variance sharply slows down, forming a plateau of noise factors. Components located before the inflection point are considered meaningful and are retained for further analysis.
8. Cumulative explained variance. The threshold selection method involves keeping the minimum number of principal components that provide a predetermined percentage of the total variance of the original data. In practice, a threshold of 90-95% is often used, guaranteeing a negligibly small loss of information with radical space compression.
9. Loadings matrix. Factor loadings are calculated as the correlation between the original variables and the principal components. The loadings matrix allows meaningful interpretation of the latent factors, determining the degree of influence of each original feature on the formation of the new coordinate axis.
10. Biplot of projections. Visualization in the coordinates of the first two principal components allows displaying both observations as points and variables as loading vectors. The cosine of the angle between vectors approximates the correlation coefficient of the features, and the vector length indicates the degree of representation of the variable in the displayed plane.
11. Outliers on score charts. Plotting the Q-statistic scores against Hotelling’s T² statistic allows the detection of anomalous observations. Points with a high T² value indicate an extreme position of the object in the model space, whereas a large Q-statistic value signals a poor description of the object by the chosen number of components.
12. Singular value decomposition. A computationally stable implementation of PCA is based on the SVD decomposition of the centered data matrix. The right singular vectors of the matrix V correspond to the eigenvectors, and the squared singular values divided by the number of degrees of freedom equal the eigenvalues of the covariance matrix.
13. SVD (Matrix factorization into three components)
14. Inverse reconstruction. The compressed score matrix and the transposed loadings matrix allow approximate reconstruction of the original data matrix through multiplication. The difference between the original values and the reconstructed predictions forms the residual matrix, the analysis of which is critical for assessing the adequacy of the PCA model.
15. Scatter ellipsoid. Geometrically, the principal components are interpreted as the axes of a p-dimensional scatter ellipsoid approximating the data point cloud. The directions of the ellipsoid axes are given by the eigenvectors, and the half-lengths of these axes equal the square root of the corresponding eigenvalues.
16. Proportion of explained variance for a feature. The communality of a variable is determined by the sum of the squares of its loadings across all retained principal components. This metric shows what proportion of the variance of a specific original feature is successfully reproduced by the constructed factor model.
17. Degenerate covariance matrix. When the number of variables significantly exceeds the number of observations, the covariance matrix becomes singular. In such high-dimensional settings, standard computation of eigenvectors is impossible, and one resorts to truncated SVD decomposition or methods of covariance matrix regularization.
18. Sparse principal components. A modification of the algorithm with L1-regularization (a LASSO-like penalty) on the loading vectors leads to the zeroing of some coefficients. Sparse representation enhances the interpretability of the model by forming components that are a linear combination of only a small subset of the original features.
19. Probabilistic formulation of PCA. The latent factor model assumes that the data is generated by a linear combination of hidden Gaussian variables with additive isotropic noise. Parameter estimation using the maximum likelihood method via the EM algorithm allows working with missing values in the training sample.
20. Application to multicollinearity. Principal component regression solves the problem of poor conditioning of the design matrix in the presence of multicollinearity. Orthogonalized predictors, which are the principal components, eliminate the instability of regression coefficient estimates and reduce the variance inflation of parameter estimates.
21. Kernel generalization of the method. Kernel PCA performs nonlinear projection of the data into a high-dimensional feature space using a kernel function, where standard linear PCA is then applied. The result is the discovery of nonlinear structures and manifolds in the original space that are inaccessible to the classical linear method.
22. Uncorrelatedness of components. By construction, the covariance matrix of the score vectors is diagonal. Statistical independence is achieved only under the assumption of multivariate normality of the original data. This property is used for feature decorrelation in preprocessing before feeding into other machine learning algorithms.

Comparisons

• PCA vs Factor Analysis. The principal component method aims to maximize the explained variance of the original variables without considering the structure of measurement errors. Factor analysis, in contrast, models latent variables, separating common and unique variance. PCA performs an orthogonal decomposition of the covariance matrix, whereas FA seeks a solution with minimal communality, making it preferable for uncovering hidden theoretical constructs.
• PCA vs t-SNE. PCA is a linear dimensionality reduction method that guarantees the preservation of the global data structure by maximizing variance in new axes. t-SNE is a nonlinear stochastic algorithm optimized for visualization that preserves local neighborhoods of points but distorts distances between distant clusters. PCA works faster and deterministically, while t-SNE requires tuning of the perplexity hyperparameter and is not suitable for feature preprocessing before modeling.
• PCA vs Truncated SVD. Both methods perform linear dimensionality reduction via singular value decomposition, yielding identical results on pre-centered data. The key technical difference lies in the processing of the input matrix: PCA requires mean subtraction to work with the covariance structure, whereas Truncated SVD can be applied to non-centered sparse matrices. This makes SVD more memory- and speed-efficient when working with large text data in latent semantic analysis tasks.
• PCA vs Autoencoder. Linear PCA projects data onto a hyperplane defined by the eigenvectors of the covariance matrix, forming a unique orthogonal transformation. Autoencoders with activation functions form a nonlinear generalization of PCA, capable of modeling complex manifolds in data. If only linear layers and mean squared error are used, the weights of the trained encoder will converge to the space of the principal components, but the autoencoder allows unpredictable rotation of this space without orthogonal constraints.
• PCA vs ICA. Principal component analysis generates uncorrelated features, maximizing variance and relying only on second-order covariances. Independent component analysis solves the stricter problem of statistical independence of signals, using high-order moments and maximizing the non-Gaussianity of the source distributions. PCA serves as a typical preprocessing step for ICA, whitening the signal and reducing dimensionality; however, the resulting ICA components directly uncover the hidden mixing sources rather than the directions of maximum variation.

OS and driver support

The PCA algorithm is implemented at the level of application software and mathematical libraries, so its operation does not depend on device drivers and is cross-platform: computations are performed identically on Windows, Linux, and macOS through linear algebra (BLAS/LAPACK). Data loading functions are abstracted from the file system, and floating-point operations use standard CPU instructions (SSE/AVX) without direct hardware access, eliminating the need to write low-level drivers for graphics cards or specialized controllers.

Data security

When performing PCA, the main threat vector is not code injection but statistical information leakage: knowing the principal components and eigenvalues, an attacker can partially reconstruct the structure of the original data, so the model training function must run in an isolated environment with restricted permissions to read raw samples. To prevent the reconstruction of confidential features, differential privacy (adding calibrated noise to the covariance matrix) and secure aggregation are applied, and serialized model objects are signed with a cryptographic signature to protect against the substitution of loading vectors.

Logging

The logging system in PCA implementations records not the intermediate matrix multiplications but the macro-stages of the pipeline: the start of standardization, the dimensionality of the input tensor, the method of computing eigenvectors (full SVD decomposition or the iterative Arnoldi method), as well as the proportion of explained variance and the data centering time. In the event of safeguards triggering (a singular covariance matrix or exceeding the number of iterations during computation), the error stack with the identifier of the non-converged feature is written to the log, while debug mode additionally saves the determinant and condition number of the matrix without exposing the data itself.

Computational limitations

The fundamental limitation of PCA lies in the cubic computational complexity of naive decomposition (O(n·m² + m³)), because of which the standard method works only with datasets that fit into the RAM of a single node, and an attempt to process an ultra-wide matrix (hundreds of thousands of features) leads to RAM shortage during covariance computation. To overcome this, truncated singular value decomposition or approximate randomized methods are applied; however, the guarantee of strict component orthogonality is lost, and there is a risk of discarding sparse informative peaks in the tail of the variance distribution.

History and development

The principal component method was mathematically formulated by Karl Pearson in 1901 as the problem of finding a line that minimizes the sum of squared deviations, after which Harold Hotelling in 1933 introduced the modern multivariate representation via eigenvectors of the covariance matrix. The development of computing technology in the 1970s led to the emergence of SVD algorithms (Golub, Reinsch), and in the 21st century the focus shifted to scalability: stochastic and kernel versions emerged, allowing the processing of streaming data and nonlinear manifolds without explicit centering in the original space.