Medical image segmentation is a vital procedure for clinicians to make speedy and correct diagnoses. However, present strategies encounter challenges in simultaneously capturing global context and local details, reserving abundant spatial semantic information and reducing the semantic gap between the encoder and the decoder. To overcome these problems, we propose a dual-path transformer with element-wise attention and group cross-aggregation network (DPEG-Net) for medical image segmentation. Firstly, a dual-path visual transformer (DPVT) with global semantic paths and pixel-level paths is presented to extract global context and local details in lesions through global semantic paths and pixel-level paths. Secondly, an element-wise multiplication-based attention mechanism (EW-attention) is developed, in which 2D images with sufficient long-range dependencies is directly constructed without being segmented into 1D sequences, emphasizing on global context and spatial semantic information. Finally, a group cross aggregation module (GCA) is designed to effectively merge multi-scale features and decrease the semantic gap between the encoder and decoder by grouping the deep features of the decoder and the shallow features of the encoder. Extensive experiments on abdominal multi-organ segmentation, cardiac diagnosis, and skin lesion segmentation demonstrate that our DPEG-Net achieves remarkable performance without the utilization of pre-trained weights. In the primary multi-organ segmentation experiment, the mean Dice Similarity Score, mIoU score and HD95 score for the eight organs attain 83.41 %, 73.96 % and 14.20 %, respectively, demonstrating superior performance compared to state-of-the-art methods. Therefore, our study has the potential to positively impact clinical practice. Our code is available at https://github.com/ai-JIE/DPEG-Net