Inferring emotions based on accessible signals is a tendency that we have both as social individuals and as scientists. Academia and industry have developed methods and devices aimed at detecting specific emotions (e.g., joy, anger or fear) based on physiological or behavioral signals. The current opinion paper argues that this is currently a risky path to be taken in terms of scientific validity. We argue that using measures to test hypotheses concerning emotions is efficient, but that going backward - using measures to infer emotions - is risky. We also argue that ways to circumvent this reverse inference issue include making use of converging evidence across the five components of emotion (cognitive appraisal, action tendency, expression, physiological reaction and feelings), and investing even more in methodological developments.