Several applications in optimization, image, and signal processing deal with data belonging to matrix manifolds. These are manifolds in the sense of classical Riemannian geometry, where variables are matrices. This thesis is divided into four main parts, and in all of them, we make extensive use of Riemannian geometry. In the first part, we deal with the problem of finding the distance between two points on the Stiefel manifold. We describe and specialize shooting methods to the Stiefel manifold, discuss their limitations, and provide some numerical examples. In the second part, we study another method for finding geodesics: the leapfrog algorithm introduced by L. Noakes. We propose a convergence proof of leapfrog as a nonlinear Gauss-Seidel method. In the third part, we give an overview of the Hager-Zhang line search, and we generalize this technique to Riemannian manifolds. In the final part, we use the manifold of fixed-rank matrices in the context of certain large-scale variational problems arising from the discretization of elliptic PDEs, where the optimization variable is rank-constrained.