Apixaban and rivaroxaban are the two most prescribed direct factor Xa inhibitors. With the increased use of DOACs in real-world settings, safety and efficacy concerns have emerged, particularly regarding their concomitant use with other drugs. Increasing evidence highlights drug–drug interactions with CYP3A/P-gp modulators leading to adverse events. However, current recommendations for dose adjustment do not consider CYP3A/P-gp genotype and phenotype. We aimed to determine their impact on apixaban and rivaroxaban blood exposure. Three-hundred hospitalized patients were included. CYP3A and P-gp phenotypic activities were assessed by the metabolic ratio of midazolam and AUC_0–6h of fexofenadine, respectively. Relevant CYP3A and ABCB1 genetic polymorphisms were also tested. Capillary blood samples collected at four time-points after apixaban or rivaroxaban administration allowed the calculation of pharmacokinetic parameters. According to the developed multivariable linear regression models, P-gp activity (p < 0.001) and creatinine clearance (CrCl) (p = 0.01) significantly affected apixaban AUC_0–6h. P-gp activity (p < 0.001) also significantly impacted rivaroxaban AUC_0–6h. The phenotypic switch (from normal to poor metabolizer) of P-gp led to an increase of apixaban and rivaroxaban AUC_0–6h by 16% and 25%, respectively, equivalent to a decrease of 38 mL/min in CrCl according to the apixaban model. CYP3A phenotype and tested SNPs of CYP3A/P-gp had no significant impact. In conclusion, P-gp phenotypic activity, rather than known CYP3A/P-gp polymorphisms, could be relevant for dose adjustment.