There has recently been a significant interest in the organization and content-based querying of large images databases. Most frequently, the underlying hypothesis is that image similarity can be characterized by low-level image features, without further abstraction. This assumes that there is sufficient agreement between machine and human measures of image similarity for the database to be useful. We wish to assess the veracity of this assumption. To this end, we develop measures of the agreement between two partitionings of an image set; we show that it is vital to take chance agreements into account. We then use these measures to assess the agreement between human subjects and a variety of machine clustering techniques on a set of images. The results can be used to select and refine image distance measures for querying and organizing image databases.