This article describes a framework for distributed multimedia retrieval which permits the connection of compliant user interfaces with a variety of multimedia retrieval engines via an open communication protocol, MRML (Multi Media Retrieval Markup Language). It allows the choice of image collection, feature set and query algorithm during run-time, permitting multiple users to query a system adapted to their needs, using the query paradigm adapted to their problem such as query by example (QBE), browsing queries, or query by annotation. User interaction is implemented over several levels and in diverse ways. Relevance feedback is implemented using positive and negative example images that can be used for a best-match QBE query. In contrast, browsing methods try to approach the searched image by giving overviews of the entire collection and by successive refinement. In addition to these query methods, Long term off line learning is implemented. It allows feature preferences per user, user domain or over all users to be learned automatically. We present the Viper multimedia retrieval system as the core of the framework and an example of an MRML-compliant search engine. Viper uses techniques adapted from traditional information retrieval (IR) to retrieve multimedia documents, thus benefiting from the many years of IR research. As a result, textual and visual features are treated in the same way, facilitating true multimedia retrieval. The MRML protocol also allows other applications to make use of the search engnes. This can for example be used for the design of a benchmark test suite, querying several search engines in the same way and comparing the results. This is motivated by the fact that the content-based image retrieval community really lacks such a benchmark as it already exists in text retrieval.