In this thesis, we consider a class of regularization techniques, called thresholding, which assumes a certain transform of the parameter vector is sparse, meaning it has only few nonzero coordinates. The parsimony assumption is natural in high-dimensional data where the number of features of the model can be dramatically larger than the sample size. These techniques are indexed by a nonnegative tuning parameter which governs the sparsity level of the estimate. We first introduce the quantile universal threshold, a tuning parameter selection methodology which follows the same paradigm in various domains. We then propose a new class of testing procedures in linear models, thresholding tests, which are based on thresholding estimators. Finally, we derive necessary and sufficient conditions for the existence of regularized estimators in generalized linear models when some parameters are left unpenalized, since they are assumed a priori to be nonzero.