In this paper we discuss the problems faced when trying to design an evaluation protocol for a multimodal system using novel input modalities and in a new domain. In particular, we focus on the problem of trying to minimize bias towards certain modalities and interaction patterns. Such bias might be introduced by experimenters in the instructions given to users which explain how the system can be used.