We present an optical setup for spectroscopic measurements in the infrared and of Raman shift under high pressure and at low temperature. Using a membrane-driven diamond anvil cell, the pressure can be tuned in situ up to 20 GPa and the temperatures ranges from room temperature down to 18 K in transmission mode and 13 K in reflection mode. In transmission, the setup is entirely working under vacuum to reduce the water absorption features and obtain a higher spectral stability. Since the infrared throughput obtained with a thermal source is limited, the use of a synchrotron source allowed to enhance the performance, as illustrated with results obtained with various materials. The analysis of the reflectivity is adapted so that it benefits from ambient pressure data and produces quantitative optical conductivity curves that can be easily compared to the results at ambient pressure.