The species composition of coral communities has shifted in many areas worldwide through the relative loss of important ecosystem engineers such as highly branched corals, which are integral in maintaining reef biodiversity. We assessed the degree to which the performance of recently recruited branching corals was influenced by corallivory, competition, sedimentation, and the interactions between these factors. We also explored whether the species-specific influence of these biotic and abiotic constraints helps to explain recent shifts in the coral community in lagoons of Moorea, French Polynesia. Population surveys revealed evidence of a community shift away from a historically acroporid-dominated community to a pocilloporid- and poritid-dominated community, but also showed that the distribution and abundance of coral taxa varied predictably with location in the lagoon. At the microhabitat scale, branching corals grew mainly on dead or partially dead massive Porites (“bommies”), promontories with enhanced current velocities and reduced sedimentation. A demographic study revealed that growth and survival of juvenile Pocillopora verrucosa and Acropora retusa, the two most common branching species of each taxon, were affected by predation and competition with vermetid gastropods. By 24 months of age, 20–60% of juvenile corals suffered partial predation by corallivorous fishes, and injured corals experienced reduced growth and survival. A field experiment confirmed that partial predation by corallivorous fishes is an important, but habitat-modulated, constraint for branching corals. Competition with vermetid gastropods reduced growth of both branching species but unexpectedly also provided an associational defense against corallivory. Overall, the impact of abiotic constraints was habitat-specific and similar for Acropora and Pocillopora, but biotic interactions, especially corallivory, had a greater negative effect on Acropora than Pocillopora, which may explain the local shift in coral community composition.