Parasite environments are heterogeneous at different levels. The first level of variability is the host itself. The second level represents the external environment for the hosts, to which parasites may be exposed during part of their life cycle. Both levels are expected to affect parasite fitness traits. We disentangle the main and interaction effects of variation in the immediate host environment, here the diatom Asterionella formosa (variables host cell volume and host condition through herbicide pre-exposure) and variation in the external environment (variables host density and acute herbicide exposure) on three fitness traits (infection success, development time and reproductive output) of a chytrid parasite. Herbicide exposure only decreased infection success in a low host density environment. This result reinforces the hypothesis that chytrid zoospores use photosynthesis-dependent chemical cues to locate its host. At high host densities, chemotaxis becomes less relevant due to increasing chance contact rates between host and parasite, thereby following the mass-action principle in epidemiology. Theoretical support for this finding is provided by an agent-based simulation model. The immediate host environment (cell volume) substantially affected parasite reproductive output and also interacted with the external herbicide exposed environment. On the contrary, changes in the immediate host environment through herbicide pre-exposure did not increase infection success, though it had subtle effects on zoospore development time and reproductive output. This study shows that both immediate host and external environment as well as their interaction have significant effects on parasite fitness. Disentangling these effects improves our understanding of the processes underlying parasite spread and disease dynamics.