Confirmatory factor analysis (CFA) is a data analysis procedure that is widely used in social and behavioral sciences in general and other applied sciences that deal with large quantities of data (variables). The underlying model links a set latent factors, that are supposed to correspond to latent concepts, to a larger set of observed (manifest) variables through linear regression equations. With CFA, it is not necessary that all manifest variables are linked to all latent factors, and is particularly useful for the construction of so-called measurement scales like depression scales in psychology. The classical estimator (and inference) procedures are based either on the maximum likelihood (ML) or generalized least squares (GLS) approaches. Unfortunately these methods are known to be non robust to model misspecification, which in the case of factor analysis in general, and in CFA in particular, is misspecification with respect to the multivariate normal model. A natural robust estimator is obtained by first estimating the (mean and) covariance matrix of the manifest variables and then "plug-in" this statistic into the ML or GLS estimating equations. This two-stage method however doesn't fully take into account the covariance structure implied by the CFA model. In this paper, we propose an S-estimator for the parameters of the CFA model that is computed directly from the data. We derive the estimating equations and an iterative procedure. The two estimators have different asymptotic properties, in that their asymptotic covariance matrix is not the same, and they both depend on the model and the parameters values. We perform a simulation study to compare the finite sample properties of both estimators and find that the direct estimator we propose is more stable (smaller MSE) than the two-stage estimator.