Challenging players is a fundamental element when designing games, but finding the perfect balance between a frustrating and boring experience can become a challenge in itself. This paper proposes the usage of deep data-driven affective models capable of detecting anxiety and boredom from raw human physiological signals to adapt the fluctuation of difficulty in a game of Tetris. The first phase of this work was to construct several emotion detection models for performance comparison, where the most accurate model achieved an average accuracy of 73.2%. A user evaluation was subsequently conducted on a total of 56 participants to validate the efficiency of the most accurate model obtained using two diverging difficulty adaptation types. One method adapts difficulty according to raw values directly outputted by the affective model (ABS), while another compares the current value with the previous one to influence difficulty (REA). Results show that the model using the ABS adaptation was capable of effectively adjusting the overall difficulty based on a player's physiological state during a game of Tetris.