Temporal lobe epilepsy has been usually associated with progressive brain atrophy due to neuronal cell loss. However, recent animal models demonstrated a dual effect of epileptic seizures with initial enhancement of hippocampal neurogenesis followed by abnormal astrocyte proliferation and neurogenesis depletion in the chronic stage. Our aim was to test for the hypothesized bidirectional pattern of epilepsy-associated brain remodeling in the context of the presence and absence of mesial temporal lobe sclerosis. We acquired MRIs from a large cohort of mesial temporal lobe epilepsy patients with or without hippocampus sclerosis on radiological examination. The statistical analysis tested explicitly for common and differential brain patterns between the two patients' cohorts and healthy controls within the computational anatomy framework of voxel-based morphometry. The main effect of disease was associated with continuous hippocampus volume loss ipsilateral to the seizure onset zone in both temporal lobe epilepsy cohorts. The post hoc simple effects tests demonstrated bilateral hippocampus volume increase in the early epilepsy stages in patients without hippocampus sclerosis. Early age of onset and longer disease duration correlated with volume decrease in the ipsilateral hippocampus. Our findings of seizure-induced hippocampal remodeling are associated with specific patterns of mesial temporal lobe atrophy that are modulated by individual clinical phenotype features. Directionality of hippocampus volume changes strongly depends on the chronicity of disease. Specific anatomy differences represent a snapshot within a progressive continuum of seizure-induced structural remodeling.