The problem of high-dimensional and large-scale representation of visual data is addressed from an unsupervised learning perspective, where an emphasis is put on discrete representations. The algorithmic infrastructure is developed based on the synthesis and analysis prior models whose rate-distortion properties are carefully optimized. These are then extended to multi-layers, namely the RRQ and the ML-STC frameworks, where the latter is further evolved as a powerful deep neural network architecture with fast and sample-efficient training. For these frameworks, three important applications are considered. First, large-scale similarity search in retrieval systems is addressed, where a double-stage solution is proposed leading to faster query times and smaller storage. Second, the problem of learned image compression is targeted, where the proposed models can capture more redundancies within images than the conventional compression codecs. Finally, the proposed algorithms are used to solve ill-posed inverse problems with promising results in image denoising and compressive sensing.