X-ray binaries (XRBs) are some of the brightest X-ray sources in the Universe. The emission from XRBs is produced by the transfer of material from a star onto a compact object serving as its companion. The brightest of XRBs are ultra-luminous X-ray sources (ULXs), with X-ray luminosities greater than 10³⁹ erg s^-1 (which is about the Eddington limit of a 10M_sun black hole). Observations of coherent X-ray pulsations in a subset of ULXs indicate the presence of an accreting neutron star. Considering that the Eddington limit of a neutron star is ~10³⁸ erg s^-1, it makes these pulsating ULXs highly super-Eddington. This thesis aims at studying the formation and evolution of ULXs, and by extension XRBs as a whole. We investigate ULXs in the form of XRBs undergoing super-Eddington mass-transfer phases and include super-Eddington disc models to help explain their luminosities. In doing so, we also investigate the stability of mass transfer in XRBs and formulate a method of constraining the mass-transfer stability in semi-detached binaries, using the overflow from the second Lagrange point as a constraint. We study the demographic of ULX populations at different ages of stellar populations, and the effect of different assumptions of accretion physics on ULXs. In order to carry out these studies, we used population synthesis techniques in combination with detailed stellar and binary evolutionary calculations. Given the uncertainties that currently exist in our understanding of binary interactions and assumptions that need to be made for evolutionary phases (for instance, the efficiency in the common-envelope phase), we carried out a parameter study with the aim of constraining the said physics. We base our studies of ULX populations on the results of the parameter study and draw conclusions from the inferred information, finding that observations of ULXs could be used to constrain the treatment of kicks in binaries and that NS-ULXs are almost always geometrically beamed whereas BH-ULXs are beamed in ~50% of the cases. Future studies, including a wider range of metallicities, will aid us better in our understanding of sources like ULXs that are more abundant in sub-solar metallicities.