Interferon-inducible GTPases of the Immunity Related GTPase (IRG) and Guanylate Binding Protein (GBP) families provide resistance to intracellular pathogenic microbes. IRGs and GBPs stably associate with pathogen-containing vacuoles (PVs) and elicit immune pathways directed at the targeted vacuoles. Targeting of Interferon-inducible GTPases to PVs requires the formation of higher-order protein oligomers, a process negatively regulated by a subclass of IRG proteins called IRGMs. We found that the paralogous IRGM proteins Irgm1 and Irgm3 fail to robustly associate with "non-self" PVs containing either the bacterial pathogen Chlamydia trachomatis or the protozoan pathogen Toxoplasma gondii. Instead, Irgm1 and Irgm3 reside on "self" organelles including lipid droplets (LDs). Whereas IRGM-positive LDs are guarded against the stable association with other IRGs and GBPs, we demonstrate that IRGM-stripped LDs become high affinity binding substrates for IRG and GBP proteins. These data reveal that intracellular immune recognition of organelle-like structures by IRG and GBP proteins is partly dictated by the missing of "self" IRGM proteins from these structures.