Plant defense against herbivory comes at a cost, which can be either direct (reducing resources available for growth and reproduction) or indirect (through reducing ecological performance, for example intraspecific competitiveness). While direct costs have been well studied in theoretical models, ecological costs have received almost no attention. In this study we compare models with a direct trade-off (reduced growth rate) to models with an ecological trade-off (reduced competitive ability), using a combination of adaptive dynamics and simulations. In addition, we study the dependence of the level of defense that can evolve on the type of defense (directly by reducing consumption, or indirectly by inducing herbivore mortality (toxicity)), and on the type of herbivore against which the plant is defending itself (generalists or specialists). We find three major results: First, for both direct and ecological costs, defense only evolves if the benefit to the plant is direct (through reducing consumption). Second, the type of cost has a major effect on the evolutionary dynamics: direct costs always lead to a single optimal strategy against herbivores, but ecological costs can lead to branching and the coexistence of non-defending and defending plants; however, coexistence is only possible when defending against generalist herbivores. Finally, we find that fast-growing plants invest less than slow-growing plants when defending against generalist herbivores, as predicted by the Resource Availability Hypothesis, but invest more than slow-growing plants when defending against specialists. Our results clearly show that assumptions about ecological interactions are crucial for understanding the evolution of defense against herbivores.