Objective: There is increasing evidence that epileptic activity involves widespread brain networks rather than single sources and that these networks contribute to interictal brain dysfunction. We investigated the fast-varying behavior of epileptic networks during interictal spikes in right and left temporal lobe epilepsy (RTLE and LTLE) at a whole-brain scale using directed connectivity. Methods: In 16 patients, 8 with LTLE and 8 with RTLE, we estimated the electrical source activity in 82 cortical regions of interest (ROIs) using high-density electroencephalography (EEG), individual head models, and a distributed linear inverse solution. A multivariate, time-varying, and frequency-resolved Granger-causal modeling (weighted Partial Directed Coherence) was applied to the source signal of all ROIs. A nonparametric statistical test assessed differences between spike and baseline epochs. Connectivity results between RTLE and LTLE were compared between RTLE and LTLE and with neuropsychological impairments. Results: Ipsilateral anterior temporal structures were identified as key drivers for both groups, concordant with the epileptogenic zone estimated invasively. We observed an increase in outflow from the key driver already before the spike. There were also important temporal and extratemporal ipsilateral drivers in both conditions, and contralateral only in RTLE. A different network pattern between LTLE and RTLE was found: in RTLE there was a much more prominent ipsilateral to contralateral pattern than in LTLE. Half of the RTLE patients but none of the LTLE patients had neuropsychological deficits consistent with contralateral temporal lobe dysfunction, suggesting a relationship between connectivity changes and cognitive deficits. Significance: The different patterns of time-varying connectivity in LTLE and RTLE suggest that they are not symmetrical entities, in line with our neuropsychological results. The highest outflow region was concordant with invasive validation of the epileptogenic zone. This enhanced characterization of dynamic connectivity patterns could better explain cognitive deficits and help the management of epilepsy surgery candidates.