In a previous Letter, we showed that physical scattering observables for compact spinning objects in general relativity can depend on additional degrees of freedom beyond those described by the spin vector, which can be packaged into an unconstrained spin tensor. In this paper, we provide further details on the physics of these additional degrees of freedom, whose commutation relations and Poisson brackets are inherited from the underlying Lorentz symmetry, and on their consequence on observables. In particular, we give the waveform at leading order in Newton’s constant and up to second order in the components of the spin tensor, and the conservative impulse, boost and spin kick, exhibiting spin magnitude change, through next-to-leading-order in Newton’s constant and third order in the components of the spin tensor. We provide explicit examples — a Newtonian two-particle bound system and a certain black-hole solution in an exotic matter-coupled gravitational theory — that exhibit these degrees of freedom and are described by our four-dimensional and worldline field theories. We also discuss connections between these degrees of freedom and dynamical worldline multipole moments. We construct effective two-body Hamiltonians, we demonstrate explicitly that the extra degrees of freedom beyond the spin vector are necessary to describe the complete dynamics, and we explicitly remove certain unphysical singularities. Moreover, we show that the previously proposed eikonal (or radial action) formula correctly captures observables derived from the classical Hamiltonian. Finally, we comment on possible descriptions of the additional degrees of freedom from the perspective of Goldstone’s theorem.