Tracking and lineage tracing are widely needed tasks in biological image analysis. For cells that grow and divide, tracking is challenging because cells change in number, shape, and size throughout a recording. Longer intervals between images make tracking more difficult. Consequently, tracking has to be performed between consecutive or temporally close images, which leads to exponentially decreasing tracking accuracy and high sensitivity to error rates. For budding yeast, this challenge is further heightened by the similarity of cells in colonies, their dense packing, asymmetric cell divisions, and movement due to colony growth. A related task, lineage tracing, is similarly challenging without fluorescent markers since a new daughter cell can be surrounded by multiple potential mother cells. Here, we present neural networks for budding yeast tracking and lineage tracing, named LYN-track and LYN-trace, respectively, which leverage fine geometric features of cells and their neighborhoods. To train and test the algorithms, we recorded and annotated budding and fission yeast timelapse microscopy movies (78,852 frame-to-frame tracklets, 2,512 images), which we make available. On these and existing datasets, our neural network-based methods demonstrate robust, above state-of-the-art performance. Both tools are integrated into graphical user interfaces (GUIs) and can be retrained with custom data.