In this paper we employ human judgments of image similarity to improve the organization of an image database. We first derive a statistic, $\kappa_B$ which measures the agreement between two partitionings of an image set. $\kappa_B$ is used to assess agreement both amongst and between human and machine partitionings. This provides a rigorous means of choosing between competing image database organization systems, and of assessing the performance of such systems with respect to human judgments. Human partitionings of an image set are used to define an similarity value based on the frequency with which images are judged to be similar. When this measure is used to partition an image set using a clustering technique, the resultant partitioning agrees better with human partitionings than any of the feature-space-based techniques investigated. Finally, we investigate the use multilayer perceptrons and a Distance Learning Network to learn a mapping from feature space to this perceptual similarity space. The Distance Learning Network is shown to learn a mapping which results in partitionings in excellent agreement with those produced by human subjects.