Chapter 10 Speciation

Our considerations thus far have dealt with the effects of different evolutionary forces on populations of the same species; we have not explored how lineages split to form new species. In this chapter, we will take a close look at the speciation processes. We will first clarify what a species actually is, and then discuss how the different evolutionary forces—mutation, selection, genetic drift, and migration—impact the emergence of reproductive isolation between diverging lineages.

10.1 Species and Species Concepts

Species are a basic evolutionary unit. Species typically consist of multiple populations, each connected by gene flow but independently shaped by mutation, selection, and genetic drift. Hence, species form the boundary for the spread of alleles. While this notion of a species is in principle simple, determining those boundaries is not. As a consequence, biologists have developed a wide variety of species concepts that use more straightforward criteria to determine the species status of different populations (see below). All of these concepts have different strengths, weaknesses, and applicabilities; but rather than dissecting these, I will just leave it with Darwin’s slightly facetious insight, because nothing has really changed since 1859:

I shall not discuss the various definitions which have been given of the term species. No one definition has as yet satisfied all naturalists; yet every naturalist knows vaguely what he means when he speaks of a species.”

Charles Darwin

I will say, however, that the biological species concept is the only species concept that recognizes and treats species as real natural entities. Classifications based on other species concepts are inevitably proxies, and as such, human constructs. This is not necessarily bad, because it provides the practicality needed for working with organisms—whether that is in biomedical research, argiculture, wildlife management, or conservation. It is just important to remember that species classifications do not always correspond precisely to actual biological species, a point we will return to later in this chapter.

I will briefly provide more background information on the biological species concept. As the only species concept that recognizes species as real natural entities, the biological species is critical if we want to understand the speciation process; it sets the criteria for when the process is completed and a single lineage has split into two.

Species Concepts: A Selection

A wide variety of concepts have been proposed to recognize, describe, and classify biodiversity. To learn more about the history and debates surrounding different species concepts, check out the article “Species Concepts” by Kevin Zelnio.

A selection of species concepts can be found below. Note that there have been concerted efforts in the recent past to unify different species concepts in an attempt to make species delimitation more consistent across taxonomic groups (De Queiroz 2007).

  • Biological Species Concept: A species is a group of actually or potentially interbreeding natural populations, which are reproductively isolated from other such groups.

  • Cohesion Species Concept: A species is the most inclusive group of organisms having the potential for genetic and/or demographic exchangeability; a group of organisms having the potential for phenotypic cohesion through intrinsic mechanisms.

  • Competition Species Concept: A species is the most extensive units in the natural economy such that reproductive competition occurs among their parts.

  • Ecological Species Concept: A species is a set of organisms exploiting or adapted to a single niche (or adaptive zone).

  • Evolutionary Species Concept: A species is a lineage (an ancestral-descendant sequence of populations) evolving separately from others and with its own unitary evolutionary roles, tendencies, and historical fate.

  • Geneological Species Concept: A species is an exclusive group of organisms whose members more closely resemble one another than members of any outside group.

  • Genetic Cluster Species Concept: A species is a group of individuals forming a distinct genotypic cluster without intermediates between other such clusters.

  • Isolation Species Concept: A species is a system of populations, the gene exchange between which is prevented by absence of interbreeding between heterospecific organisms based on intrinsic properties.

  • Morphological (or Phenetic) Species Concept: A species is a set of organisms that look similar to each other and is distinct from other sets.

  • Phylogenetic Species concept: A species is the smallest monophyletic group distinguishable by shared derived (synapomorphic) characteristics.

  • Recognition Species Concept: A species is is a group of individuals with a shared specific mate recognition or fertilization system.

  • Typological Species Concept: A species is a group of organisms conforming to a common morphological plan, emphasizing the species as an essentially static, non-variable assemblage.

10.1.1 The Biological Species Concept

The biological species concept was first proposed by Ernst Mayr, who defined a species as a group of actually or potentially interbreeding natural populations, which are reproductively isolated from other such groups. Hence, species are defined by reproductive compatibility, which is meaningful from an evolutionary perspective because it hinges on the lack of gene flow.

It is important to stress some nuances in this definition. First of all, Mayr explicitly emphasized natural populations in his definition, because the ability of two species to hybridize—especially outside of their typical ecological context—does not negate their species status. As we shall see, interspecific matings between many species can lead to perfectly viable and fertile offspring, yet such matings rarely occur in their natural environment due to the reproductive barriers that prevent individuals from different species from meeting or recognizing each other as potential mates. Thus, evidence of two species hybridizing outside of their ecological context, especially in captivity, does not mean that such hybridization would occur naturally.

The term “potentially interbreeding” also creates frequent misconceptions. It does not refer to potential interbreeding in artificial contexts, but to potential interbreeding in the face of physical barriers that prevent individuals from different populations meeting one another. Mayr argued that we are unable to determine the species status of allopatric populations, where interbreeding is currently impossible due to physical separation. What would happen if a physical barrier was removed? For example, consider the large ungulates of the genus Cervus that we know as wapiti (elk) and that Europeans know as red deer. There are three disjunct populations: one in North America, one in Europe and western Asia, and one in East Asia (Figure 10.1). Would animals from these different populations interbreed if their ranges reconnected, or would they remain separate, reproductively isolated species? Obviously there is no way to test this in practice, and our inability to determine the species status of allopatric populations is in fact one of the biggest shortcomings of the biological species concept. Hence, taxonomist have resorted to alternative species concepts in the case of wapiti and red deer. For a long time, the different populations were considered the same species (C. elaphus) based on their morphological similarity and shared evolutionary history. However, recent genetic analyses have restricted C. elpahus to the European and western Asian forms, and the east Asian and North American forms are classified as C. canadensis. Which is right? Well, it’s simply a matter of perspective, because such classifications are human constructs. Note that even in populations with adjacent or overlapping ranges, determining species status based on the biological species concept can be costly and time consuming, which is why in most cases biological classification operationally still hinges on other concepts.

Distribution of wapiti and red deer in North America and Eurasia. The species occurs in three disjunct populations. North American wapiti picture by Membeth (CC0); red deer picture by the [Wasp Factory](https://www.flickr.com/photos/thewaspfactory74/) ([CC BY-NC-SA 2.0](https://creativecommons.org/licenses/by-nc-sa/2.0/)); Asian wapiti picture by Membeth (CC0).

Figure 10.1: Distribution of wapiti and red deer in North America and Eurasia. The species occurs in three disjunct populations. North American wapiti picture by Membeth (CC0); red deer picture by the Wasp Factory (CC BY-NC-SA 2.0); Asian wapiti picture by Membeth (CC0).

Finally, it is important to mention that the biological species concept does not apply in contexts where reproductive compatibility cannot be assessed or simply does not exist. For example, we cannot determine reproductive isolation for extinct species that we only know from the fossil record. This contributes to a long-standing debate about how many species of humans roamed this planet in prehistoric times. Human fossils are classified based on morphological traits, and twelve species are recognized in the genus Homo alone. However, we just don’t know whether the different phenotypic variants represent a single, morphologically diverse species, or whether there were in fact multiple, reproductively isolated species of Homo coexisting at one time. Similarly, the biological species concept does not apply to organisms that reproduce asexually, because different asexual individuals—by definition—are reproductively isolated from all other individuals. If there is no sex and recombination, the spread of alleles is strictly limited from mothers to daughters, and there is no other exchange of genetic material among individuals or populations.

10.2 Speciation as a Gradual Process

If species are groups of actually or potentially interbreeding natural populations, which are reproductively isolated from other such groups, then speciation is the process by which reproductive isolation evolves between populations. The evolution of reproductive isolation is typically a gradual process, starting from a single variable population, continuing to differentiated populations that are still connected by gene flow, and finally ending in distinct and reproductively isolated species. This gradient is known as the speciation continuum (Figure 10.2). Note that movement along this speciation continuum can be bidirectional; lineages can split by accruing more and more reproductive isolation, but lineages can also merge together through hybridization.

Just like the evolution of other traits, changes in the degree of reproductive isolation tend to evolve gradually through time, and we can consequently observe populations at different stages along the speciation continuum in natural systems. Such systems with ongoing speciation are valuable study objects for examining how and why reproductive isolation evolves.

Speciation is not typically an instantaneous process. Rather, species evolve gradually along a speciation continuum that ranges from a single population, to differentiated populations connected by gene flow, to distinct and reproductively isolated populations.

Figure 10.2: Speciation is not typically an instantaneous process. Rather, species evolve gradually along a speciation continuum that ranges from a single population, to differentiated populations connected by gene flow, to distinct and reproductively isolated populations.

Note that the gradual and dynamic nature of speciation is one reason for why taxonomic species may not align well with biological realities. Some natural populations can be strongly differentiated and diagnosable based on a number of other species concepts. At the same time, there may still be a lot of gene flow with other populations, and even slight changes in environmental conditions could counteract existing reproductive barriers and reverse the speciation process. In other words, not all aspects of nature can be neatly classified into different categories, and we have to contend with the fact that variation in nature is continuous because speciation is a gradual and not an instantaneous process.

10.3 Mechanisms of Reproductive Isolation

If speciation is the evolution of reproductive isolation between populations, we first need to understand what mechanisms can mediate reproductive isolation. Only then can we consider how those mechanisms actually evolve. Broadly speaking, mechanisms of reproductive isolation can be categorized into three groups, depending on whether they prevent mating between individuals from different populations (pre-mating isolation), whether they prevent successful fertilization and the formation of a zygote (post-mating, pre-zygotic isolation), or whether they prevent the success of interpopulation hybrids (post-zygotic isolation). In this section, we will discuss some common mechanisms of reproductive isolation using illustrative examples from empirical studies.

10.3.1 Pre-Mating Isolation

Pre-mating isolation describes reproductive barriers that reduce the frequency or success of interpopulation mating. This can happen because members of different populations do not meet, do not recognize each other, or are not compatible with each other during mating.

Ecological Isolation

Ecological isolation reduces the probability that members of different populations encounter each other, for example as consequence of adaptation to different niches that causes spatial segregation of population. Such spatial segregation can occur at large spatial scales (e.g., adaptation to different habitat types) or at small spatial scales (e.g., adaptation to different host plants that grow in the same habitat patch). Two mechanisms can maintain spatial segregation between populations:

  1. Spatial segregation can be a consequence of selection against migrants, where individuals that disperse across habitats have a reduced survivability, and, thus, a lower likelihood of encountering potential mates. Such selection against migrants is evident in desert spiders (Agelenopsis aperta) that have adapted to different habitat types. Woodland spiders are limited by web-sites and prey, and are selected for bold and aggressive behaviors toward prey and conspecifics alike. In contrast, riparian spider populations are released from competition but suffer from increased predation by birds, favoring more shy and reclusive behaviors (Riechert & Hall 2000). Experimental translocation of spiders between woodland and riparian habitats have indicated selection against migrants, as individuals moved across habitats had significantly lower survival rates than individuals translocated within habitat types (Figure 10.3A). Hence, any spiders that would naturally move across habitat boundaries would be selected against, reducing the likelhood of cross-population mating.

  2. Spatial segregation can also be a consequence of differential habitat preferences. When individuals from locally adapted populations choose habitats based on suitability, interpopulation encounter rates can be low even in absence of strong selection against migrants. For example, stickleback fish (Gasterosteus aculeatus) locally adapted to either lake or stream habitats have a strong affinity to their particular habitat type (Bolnick et al. 2009). Over 80% of marked individuals that were released at the confluence of a stream and a lake were eventually recaptured in their habitat of origin (Figure 10.3B). They segregated spatially simply because of divergent behavioral preferences.

A. Translocation experiments in desert spiders (*Agelenopsis aperta*) across habitats reveals selection against migrants; translocated spiders exhibit significantly lower survival than residents in each habitat. [Data](data/11_habitat-iso2.csv) from Riechert and Hall (2000). B. Sticklebacks (*Gasterosteus aculeatus*) adapted to lake and stream habitats exhibit a strong behavioral preference for their habitat of origin. [Data](data/11_habitat-iso1.csv) from Bolnick et al. (2009).

Figure 10.3: A. Translocation experiments in desert spiders (Agelenopsis aperta) across habitats reveals selection against migrants; translocated spiders exhibit significantly lower survival than residents in each habitat. Data from Riechert and Hall (2000). B. Sticklebacks (Gasterosteus aculeatus) adapted to lake and stream habitats exhibit a strong behavioral preference for their habitat of origin. Data from Bolnick et al. (2009).

Segregation of populations can also occur temporally, when members of different populations are (reproductively) active during different times of the day or year. Evidence for temporal isolation comes from many flowering plants, insects, and amphibians. For example, three species of frog in the Rana pipiens complex have largely allopatric distributions, but their ranges overlap in parts of Texas and Oklahoma. In allopatric populations, the breeding seasons of all three species also overlap: all three species have coincident peaks in their breeding activity in the spring, and two species (R. sphenocephala and R. berlandieri) again in the fall (Figure 10.4). However, at sites where the three species occur sympatrically, each species has evolved a unique temporal niche, and peak breeding activities do not overlap: R. sphenocephala breeds in early spring, R. blairi in late spring, and R. berlandieri late summer and into fall. The secondary peaks in breeding activity of allopatric R. sphenocephala and R. berlandieri have disappeared completely in sympatry (Figure 10.4). So, even as members of the three species occupy the same habitats, the likelihood of interbreeding is significantly reduced because they are not thinking about reproduction at the same time.

Three species in the *Rana pipiens* complex have largely overlapping breeding seasons when they occur in allopatry. However, when they coexist in the same location, breeding seasons shift such that each species is temporally isolated. [Data](data/11_temporal-iso.csv) from Hillis (1981).

Figure 10.4: Three species in the Rana pipiens complex have largely overlapping breeding seasons when they occur in allopatry. However, when they coexist in the same location, breeding seasons shift such that each species is temporally isolated. Data from Hillis (1981).

Finally, a form of ecological isolation that is important for reproductive isolation in plants is pollinator isolation. Pollinator isolation occurs when animal pollinators do not transfer pollen between plants of different populations, usually as a consequence of coevolution between floral traits and pollinator preferences. For example, two closely related species of columbines (Aquilegia formosa and A. pubescens) coexist in the same habitat, but they have different flower colors and morphologies that attract different species. A. formosa has red, hanging flowers, while A. pubescens has white, erect flowers (Figure 10.5A and B). Since the red flowers are primarily visited by hummingbirds and the white flowers by hawkmoths, pollen is primarily distributed from A. formosa to A. formosa and from A. pubescens to A. pubenscens, while interspecific fertilization is rare 10.5C). Other groups of plants have diverged in flower coloration along a yellow-red gradient, exploiting differences in the visual systems of insect pollinators (with biases for yellow and UV) and bird pollinators (with biases for red).

A. *Aquilegia formosa*. Photo: Steve Berardi, [CC BY-SA 2.0](https://creativecommons.org/licenses/by-sa/2.0). B. *Aquilegia pubescens*. Photo: Dcrjsr, [CC BY-SA 3.0](https://creativecommons.org/licenses/by-sa/3.0). C. Due to different pollinator preferences in flower color and morphology, cross-pollination between species of *Aquilegia* is relatively rare. [Data](data/11_pollinator-iso.csv) from Hodges and Arnold (1994).

Figure 10.5: A. Aquilegia formosa. Photo: Steve Berardi, CC BY-SA 2.0. B. Aquilegia pubescens. Photo: Dcrjsr, CC BY-SA 3.0. C. Due to different pollinator preferences in flower color and morphology, cross-pollination between species of Aquilegia is relatively rare. Data from Hodges and Arnold (1994).

Behavioral isolation

Behavioral isolation occurs when individuals from different populations do not recognize each other as potential mates. For example, populations may differ in species-recognition mechanisms, where changes in a particular signal (e.g., the chemical composition of a pheromone or the color for a visual signal) alters the perception of individuals as con- vs. heterospecific, or their attractiveness during mate choice. Behavioral isolation is evident in many species of birds (especially based on song), fishes (especially based on coloration), and insects (especially based on chemical signals).

Heliconius butterflies are an example of behavioral isolation. Sympatric species are able to produce fertile offspring, but they typically do not hybridize because of assortative mating preferences. Males of the sympatric H. cydno (Figure 10.6A) and H. melpomene (Figure 10.6B), for example, almost exclusively court females of their own species (Figure 10.6C). Hence, the existence of species-specific signals and matching mating preferences can significantly reduce the probability of gene flow among closely related species.

A. *Heliconius cydno*. Photo: Greg Hume, [CC BY-SA 3.0](https://creativecommons.org/licenses/by-sa/3.0). B. *Heliconius melpomene*. Photo: Charles J. Sharp, [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0). C. Males of both species have assortative mating preferences and almost exclusively court females of their own species. [Data](data/11_heliconius.csv) from Jiggins et al.(2001).

Figure 10.6: A. Heliconius cydno. Photo: Greg Hume, CC BY-SA 3.0. B. Heliconius melpomene. Photo: Charles J. Sharp, CC BY-SA 4.0. C. Males of both species have assortative mating preferences and almost exclusively court females of their own species. Data from Jiggins et al.(2001).

Mechanical isolation

Mechanical isolation occurs when individuals of two populations meet, recognize each other as potential mates, and attempt to copulate, but copulations are unsuccessful because of some mechanical incompatibility. Mechanical incompatibilities can be caused by size-mismatches between populations, or because of genital incompatibilities that arise through genital coevolution. Coevolution of male and female genital traits can arise through a variety of mechanisms, including sexual conflict (see Chapter 10). As consequence, genitals in many animal groups evolve and diversify rapidly, with stark differences even among closely related species. Compatibility of male and female genitalia have been likened to a lock-and-key mechanisms, where the males’ “key” will only fit species-specific female “locks.” Accordingly, heterospecific matings would not lead to successful sperm transfer and fertilization even when copulation is attempted. Mechanical isolation can also occur in plants, when closely related species have stamens and styles in different configurations. For example, black sage (Salvia mellifera) have stamens and style positioned along the upper lip of the flower; hence, pollen is transferred from flower to flower on the backs of small pollinating bees. In contrast, the syntopic white sage (S. apiana) exhibits two long stamens and a long, forked style that extends out and away from the flower. This configuration primarily transfers pollen to the wings of larger carpenter and bumble bees. So even if the same pollinator visits flowers of both species, successful transfer of pollen from one species to the style of the other species is unlikely.

Another fascinating example of mechanical isolation comes from evolution experiments using feather lice. These avian parasites must be small enough to fit between the feather barbs of their host to avoid being removed during preening. On the other hand, there is selection for larger body size due to the higher fecundity of larger individuals. Since larger hosts also have a larger distance between the barbs of their feathers, there is a correlation between host and parasite body size. Scott Villa and his colleagues investigated the evolution of feather lice (Columbicola columbae) on domesticated pigeons (Columba livia) of different body size over a period of four years (60 louse generations). As expected, lice that lived on large hosts evolved a larger body size over this period of time. Interestingly, this also led to the evolution of mechanical isolation between lice from large hosts and lice from small hosts. While males were still eager to mate with females of the other strain, size mismatched individuals were unable to copulate effectively with each other (check out the videos of a mating of a size-matched pair, a male of the large strain trying to copulate with a female of the small strain, and a male of the small strain trying to copulate with a female of the large strain). These mismatches resulted in reduced copulation times, a reduced probability of egg production, and a reduced number of offspring produced in interlineage matings (Figure 10.7). Hence, mechanical incompatibility has the capacity to reduce gene flow between populations adapted to different host sizes.

Even though male feather lice (*Columbicola columbae*) of different sized lineages attempt to copulate with females that are too large or too small (as indicated by points outside of the gray shaded area), those copulations are not necessarily successful. A. Lice within the typical range of sexual size dimorphism exhibited longer copulation duration compared to matings between individuals with large size disparities. [Data](data/11_lice-copulation.csv) from Villa et al. (2019). B. Lice within the typical range of sexual size dimorphism were more likely to produce eggs when kept in experimental vials. [Data](data/11_lice-eggs.csv) from Villa et al. (2019). C. Lice within the typical range of sexual size dimorphism were more likely to produce offspring on a natural host. [Data](data/11_lice-reprod.csv) from Villa et al. (2019).

Figure 10.7: Even though male feather lice (Columbicola columbae) of different sized lineages attempt to copulate with females that are too large or too small (as indicated by points outside of the gray shaded area), those copulations are not necessarily successful. A. Lice within the typical range of sexual size dimorphism exhibited longer copulation duration compared to matings between individuals with large size disparities. Data from Villa et al. (2019). B. Lice within the typical range of sexual size dimorphism were more likely to produce eggs when kept in experimental vials. Data from Villa et al. (2019). C. Lice within the typical range of sexual size dimorphism were more likely to produce offspring on a natural host. Data from Villa et al. (2019).

10.3.2 Post-Mating, Pre-zygotic Isolation

Post-mating, prezygotic isolation describes reproductive barriers that reduce the frequency or success of inter-population fertilization, if mating between members of different populations has occurred. Post-mating prezygotic isolation mechanisms can be associated with sperm competition and cryptic female choice, or—most importantly—with gametic incompatibilities.

Gametic Isolation

Gametic isolation occurs when male and female gametes cannot merge to form a zygote. Such gametic incompatibilities are frequently mediated by interactions of proteins on the surface of eggs and sperm. Co-evolution of sperm and egg surface proteins can create a lock-and-key mechanism, just like co-evolution of male and female genitalia. Gametic isolation is particularly well-studied in aquatic broadcast spawners (e.g., in Porifera, Cnidaria, and Echinodermata), where individuals release their gametes directly into the water, which of course limits the possibility for pre-mating isolation. For example, there are strong gametic incompatibilities among populations of a tropical sea urchin, Echinometra mathaei (Palumbi & Metz 1991). Mixing eggs and sperm from different populations clearly indicates high fertilization rates when eggs are encountering sperm from the same population, irrespective of sperm concentration (Figure 10.8). In contrast, only 2-9% of eggs were fertilized when sperm and eggs of different populations were combined. Hence, chemical interactions between eggs and sperm can be critical for determining the degree of reproductive isolation between populations. It is important to note that gametic isolation can also contribute to reproductive isolation between populations of species with other reproductive modes—including some with internal fertilization; however, mechanisms mediating gametic incompatibilities in those taxa are not as thoroughly investigated.

At all test sperm concentrations, more eggs were fertilized when they encountered sperm from the same population compared to when they received sperm from a different population. [Data](data/11_sea-urchin.csv) from Palumbi and Metz (1991).

Figure 10.8: At all test sperm concentrations, more eggs were fertilized when they encountered sperm from the same population compared to when they received sperm from a different population. Data from Palumbi and Metz (1991).

10.3.3 Post-Zygotic Isolation

Post-zygotic isolation occurs when individuals from different populations mate successfully and their gametes form a zygote that starts to develop, but the resulting hybrids have reduced fitness compared to either of the parents. Reduced hybrid fitness can be caused by both intrinsic and extrinsic factors.

Intrinsic Incompatibilities

Intrinsic hybrid incompatibilities are reductions in hybrid fitness that are independent of environmental conditions. Intrinsic barriers primarily arise due to epistatic effects and are also known as genetic incompatibilities. Such incompatibilities can be severe and cause hybrid inviability, where the development of hybrids ceases at early stages, long before such individuals are born. In other cases, genetic incompatibilities cause hybrid sterility (e.g., sterility in mules, which are hybrids between horses and donkeys), or quantitative reductions in hybrid survival or reproduction.

Genetic incompatibilities can arise through coevolution of different genes that contribute to the same protein complex or the sample physiological pathway. For example, enzymes involved in the mitochondrial respiratory chain are protein complexes composed of amino acid chains derived from both the nuclear and mitochondrial genomes, and different protein subunits need to be compatible and work together to form a functional protein. Evolutionary change in one subunit may consequently be accompanied by corresponding changes in other subunits. Breaking up such co-evolved gene complexes during hybridization, where an individual might inherit the mitochondria from one parent but nuclear subunits from another, can cause issues with protein complex assembly or function, ultimately leading to mitochondrial, cellular, and organismal dysfunctions. These cases of genetic incompatibilities are also known as mito-nuclear incompatibilities.

Another well-studied example of a genetic incompatibility occurs through the coevolution of tumor and tumor-suppression genes in swordtail fishes of the genus Xiphophorus. X. malinche has a tumor gene with low activity and a matching tumor-suppressor gene with low activity, while X. birchmanni has a tumor gene with high activity and a matching tumor suppressor gene with high activity (Powell et al. 2020). Hybridization can disrupt the balance between tumor genes and their suppressors. Particularly, X. malinche and X. birchmanni hybrids that inherit at least one high activity tumor allele from X. birchmanni (on chromosome 21) and only low activity tumor-suppressor genes from X. malinche (on chromosome 5) have an increased chance of developing invasive melanocytes, which cause melanoma (a form of skin cancer; Figure 10.9A). Mismatches of tumor and tumor-suppressor genes in hybrids have clear consequences for hybrid phenotypes and fitness. For example, hybrids exhibit larger melanistic spots on their tails (Figure 10.9B), and—consistent with the higher invasiveness of melanocytes—their caudal spots grow faster and larger over time (Figure 10.9C). In addition, the frequency of fish with a caudal spot deceases across age classes in hybrid populations, indicating that individuals with spots are selected against (Figure 10.9D). This decrease is particularly evident in hybrid populations with a higher caudal spot frequency (populations hybrid 2 and 3 in Figure 10.9D), where the high-activity tumor gene derived from X. birchmanni is more common.

A. Frequency of invasive melanocytes in *Xiphophorus* fish with different genotypes at a tumor suppressor locus (on chromosome 5) and tumor locus (on chromosome 21). Only fish that are homozygous for the low activity tumor suppressor allele from *X. malinche* and have at least one tumor allele from *X. birchmanni* develop a high frequency of invasive tumor cells. [Data](data/11_xipho1.csv) from Powell et al. (2020). B. Caudal spot frequencies in parental *X. birchmanni* and *X. malinche* populations, and natural populations with hybridization between the two species (Hybrid 1-3). [Data](data/11_xipho2.csv) from Powell et al. (2020). C. Caudal spots grow faster and larger in hybrids compared to *X. birchmanni*. [Data](data/11_xipho3.csv) from Powell et al. (2020). D. The decrease in caudal spot frequency from juveniles to adults in two populations of *Xiphophorus* hybrids indicate an increased mortality of fish with a caudal spot. [Data](data/11_xipho4.csv) from Powell et al. (2020).

Figure 10.9: A. Frequency of invasive melanocytes in Xiphophorus fish with different genotypes at a tumor suppressor locus (on chromosome 5) and tumor locus (on chromosome 21). Only fish that are homozygous for the low activity tumor suppressor allele from X. malinche and have at least one tumor allele from X. birchmanni develop a high frequency of invasive tumor cells. Data from Powell et al. (2020). B. Caudal spot frequencies in parental X. birchmanni and X. malinche populations, and natural populations with hybridization between the two species (Hybrid 1-3). Data from Powell et al. (2020). C. Caudal spots grow faster and larger in hybrids compared to X. birchmanni. Data from Powell et al. (2020). D. The decrease in caudal spot frequency from juveniles to adults in two populations of Xiphophorus hybrids indicate an increased mortality of fish with a caudal spot. Data from Powell et al. (2020).

Extrinsic Incompatibilities

Extrinsic incompatibilities occur when hybrids face reduced fitness because of a mismatch between their phenotype and their abiotic or biotic environment. In this case, hybrids are generally viable and fertile under laboratory conditions, but are ecologically inviable under natural conditions— either because they cannot tolerate the stressful environmental conditions that periodically arise in nature, or because they are competitively inferior to either parent in their respective niche. For example, some North American postglacial lakes harbor two species of stickleback (Gasterosteus aculeatus): a smaller limnetic species that lives in the open water and primarily feeds on zooplankton, and a larger benthic species that lives in the littoral zone and feeds on macroinvertebrates (Schluter 1995). Hybrids between the two species are viable and fertile. However, cage experiments in natural habitats revealed that hybrids are ecologically inferior compared to either parental species in the parents’ native habitat; hybrids have a lower growth rate than benthic stickleback in the littoral zone and also have a lower growth rate than the limnetic parents in open water (Figure 10.10A). Hence, ecological selection limits hybrid success in the natural context.

Hybrids may also be behaviorally sterile. Behavioral sterility may arise by hybrids not recognizing members of either parental species as potential mates, or because they themselves are not recognized as potential mates. In other words, behavioral sterility in hybrids is a consequence mismatched signals involved in species recognition, mate choice, and associated mating preferences. An example of reduced hybrid fitness caused by mate choice comes from the same Heliconius butterflies that we already discussed in the context of behavioral isolation. Heliconius melpomene females—and to a lesser degree those of H. cydno—exhibit mating preferences for males of their own species, as opposed to F1 hybrid males stemming from a cross between the two species (Figure 10.10B). Notably, however, females of both species also prefer to mate with hybrids over males from the opposite species, indicating that behavioral sterility is not complete.

A. Ecological selection against hybrids as evidenced in stickleback (*Gasterosteus aculeatus*). Hybrids between benthic and limnetic stickleback have lower growth rates than either parent in their native habitat. [Data](data/11_hyb-sel.csv) from Schluter (1995). B. Sexual selection against hybrids as evidenced in *Heliconius* butterflies. Females of *H. cydno* and especially *H. melpomene* tend to avoid hybrid males during mate choice. [Data](data/11_heliconius-hybrid.csv) from Naisbit et al. (2001).

Figure 10.10: A. Ecological selection against hybrids as evidenced in stickleback (Gasterosteus aculeatus). Hybrids between benthic and limnetic stickleback have lower growth rates than either parent in their native habitat. Data from Schluter (1995). B. Sexual selection against hybrids as evidenced in Heliconius butterflies. Females of H. cydno and especially H. melpomene tend to avoid hybrid males during mate choice. Data from Naisbit et al. (2001).

10.3.4 Reproductive Isolation as a Complex Trait

It is important to note that different mechanisms of reproductive isolation are not mutually exclusive. Between any population pair, many mechanisms may coincide such that total reproductive isolation is a composite of mechanisms that reduce the probability of mating between different populations, reduce the probability of fertilization should mating happen, and reduce the success of hybrids should fertilization happen. More over, as we will see below, the evolution of one reproductive isolation mechanism can profoundly impact the evolution of other reproductive barriers. Hence, it many natural systems it is critical not to just examine different mechanisms in isolation, but to also consider how different mechanisms interact to keep gene pools separated from one another.

Different mechanisms of reproductive isolation often act in concert and can influence each other evolutionarily.

Figure 10.11: Different mechanisms of reproductive isolation often act in concert and can influence each other evolutionarily.

10.4 The Evolution of Reproductive Isolation

Now that we have examined what mechanisms can cause reproductive isolation between populations, we can ask how those isolating barriers actually arise. Understanding how different evolutionary forces impact the evolution of reproductive isolation ultimately sheds light on the speciation process. We will consider three distinct scenarios: (1) Polyploidization and the evolution of instantaneous reproductive isolation, (2) allopatric speciation that includes periods of geographical separation between populations, and (3) ecological speciation, where reproductive isolation between populations can arise as a byproduct of adaptation even in the presence of ongoing gene flow.

10.4.1 Polyploidization and Instantaneous Speciation

Polyploidization is a form of mutation that leads to the duplication of entire genomes, typically due to meiotic errors leading to unreduced gametes. Fertilization of unreduced gametes with other unreduced gametes causes the formation of fertile polyploid lineages (i.e., from diploid to tetraploid, from tetraploid to octoploid, etc.) (Figure 10.12A). Polyploid lineages are immediately isolated from their ancestral lineage because of dysfunctional chromosome complements in crosses between individuals of different ploidy. Such auto-polyploid speciation is particularly common in plants and other organisms for which self-fertilization is possible, because the combination of two unreduced gametes is a strong limiting factor in the formation of polyploids. Accordingly, up to 4 % of plant species are thought to have arisen through auto-polyploid speciation (Figure 10.12B), while this phenomenon is rare in animals.

Polyploid speciation can also occur through the combination of two genomes derived from different species, a process called allo-polyploid speciation (Figure 10.12C). In this case, an unreduced gamete of one species is fertilized with a normal gamete from another, creating a hybrid with an uneven number of chromosomes. When unreduced gametes from that hybrid are then fertilized by a normal gamete from the second species, a fertile hybrid with the full genome set of each parent arises. Allo-polyploidy is extremely common in plants, and about 80 % of plant species may be allo-polyploids by some estimations. The common bread wheat (Triticum aestivum), for example, is an allo-hexaploid that contains three distinct sets of chromosomes from three different species of grass in the genus Aegilops. Similarly, various vegetables in the cabbage family (mustard, collard greens, cauliflower, kale, bok choi, rape seed, etc.) represent different allo-polyploid combinations of three species, Brassica nigra, B. rapa, and B. olarecea. In contrast, allo-polyploids are rare among animals, but some cases have been documented in insects, fish, amphibians, and reptiles.

A. Auto-polyploidization occurs through the combination of two unreduced gametes, which usually occurs through self-fertilization. The resulting ployploids are instantaneously isolated from ancestral lineages with lower ploidy. B. Blackberries of the genus *Rubus* represent a polyploid species complex, including species with 2, 4, 8, 16, and 32 sets of chromosomes. Without genetic or cytological analyses, species with different ploidy levels are almost impossible to distinguish. C. Allo-polyploidization oocurs through the combination of two genomes derived from different species. It is a multi-step process that first involves the production of hybrids with uneven chromosome sets.

Figure 10.12: A. Auto-polyploidization occurs through the combination of two unreduced gametes, which usually occurs through self-fertilization. The resulting ployploids are instantaneously isolated from ancestral lineages with lower ploidy. B. Blackberries of the genus Rubus represent a polyploid species complex, including species with 2, 4, 8, 16, and 32 sets of chromosomes. Without genetic or cytological analyses, species with different ploidy levels are almost impossible to distinguish. C. Allo-polyploidization oocurs through the combination of two genomes derived from different species. It is a multi-step process that first involves the production of hybrids with uneven chromosome sets.

10.4.2 Allopatric Speciation and Hybridization

The allopatric speciation model was first introduced by Ernst Mayr in his seminal book Systematics and the Origin of Species. This model postulates that the speciation process unfolds in three distinct steps: (1) The subdivision of an initial population through a geographic barrier, (2) population divergence in isolation, and (3) secondary contract of incipient species. In the following sections, we will examine these three steps in detail, along with pertinent examples.

Explore More: Systematics and the Origin of Species

Ernst Mayr’s book “Systematics and the Origin of Species” is available for free through archive.org.

Geographic Isolation

An initial stage of geographic isolation is central to the allopatric speciation model. Geographic isolation drives reproductive isolation and limits the spread of alleles within the separate populations. Two mechanisms can create geographically isolated populations from an initial ancestor: dispersal and vicariance.

Dispersal occurs when individuals of an original population overcome an existing geographic barrier and colonize a new and previously unoccupied habitat (Figure 10.13A). Classic examples of dispersal occur during the colonization of new oceanic islands. For example, the Hawai’ian archipelago arose as Earth’s crust moved over a volcanic hot spot from east to west. Accordingly, the western most island (Kaua’i; ~4.2–5 million years old) is the oldest, and islands get progressively younger towards the east; Hawai’i, as the youngest island, rose out of the ocean only 0.3–1.0 million years ago. Drosophila fruitflies that colonized the archipelago early during its formation provide a formidable example of dispersal. For two independent clades within the genus, the sequence of phylogenetic divergence corresponds to the sequence in which different islands arose from the ocean. In the D. planitibia subgroup (green clade in Figure 10.13B), dispersal occurred from O’ahu to Maui to Hawai’i; in the D. mitchelli, subgroup dispersal occurred from O’ahu to Moloka’i to Hawai’i. This pattern is consistent with the idea that species diversified through step-wise colonization of new habitats as they became available. Similar processes also gave rise to the diversity we observe in other oceanic archipelagos, like the Galapagos. It is important to emphasize that dispersal across geographic barriers and colonization of novel habitats is important in other ecological contexts as well. For example, lakes are the terrestrial equivalent of oceanic islands. Species may also overcome mountain ranges that separate suitable habitats, or valleys that separate mountain tops. Furthermore, island-like situations occur in the apparently well-connected habitats within oceans. For example, patches of coral reefs or deep-sea hydrothermal vents are separated by vast stretches of unsuitable habitats, which migrants must overcome.

A. Dispersal is the colonization of a new habitat across a geographic barrier. B. Patterns of diversification in *Drosophila* fruitflies are consistent with dispersal. In two clades (green and orange), the sequence of species divergence corresponds to colonization of younger islands in the archipelago (adopted from Obbard et al. 2012). The inset photo shows *D. silvestris* (KarlM, [CC BY-SA 3.0](https://creativecommons.org/licenses/by-sa/3.0)).

Figure 10.13: A. Dispersal is the colonization of a new habitat across a geographic barrier. B. Patterns of diversification in Drosophila fruitflies are consistent with dispersal. In two clades (green and orange), the sequence of species divergence corresponds to colonization of younger islands in the archipelago (adopted from Obbard et al. 2012). The inset photo shows D. silvestris (KarlM, CC BY-SA 3.0).

Vicariance occurs when the distributional range of a species is subdivided by an emerging geographic barrier (Figure 10.14A). For example, islands—or even entire continents—may be divided as a consequence of rising sea levels associated with climate change; as water levels flood low laying valleys, regions with higher elevations become separated. Similarly, volcanic activity and plate tectonics can shift land masses and create novel barriers of unsuitable habitat across which movement of organisms becomes limited. For example, the ancestor of ratites, which includes a number of flightless birds, had a Gondwanian distribution. As the supercontinent broke up into smaller pieces and landmasses drifted apart, geographic isolation gave rise to today’s flightless birds in South America, Africa, Australia and New Guinea, and New Zealand (Figure 10.14B). Other classic examples of vicariance include paired species of crustaceans, fish, and other marine animals that occur in either side of the Central American land bridge (i.e., sister species in the Pacific and the Caribbean) and in the Atlantic and the Mediterranean Sea.

A. Vicariance is the subsetting of an original population by an emerging geographic barrier B. Patterns of diversification in ratites are a consequence of vicariance. The ancestral, contiguous range of this group on Gondwana was broken apart as a consequence of continental drift.

Figure 10.14: A. Vicariance is the subsetting of an original population by an emerging geographic barrier B. Patterns of diversification in ratites are a consequence of vicariance. The ancestral, contiguous range of this group on Gondwana was broken apart as a consequence of continental drift.

Population Divergence

The second step of the allopatric speciation model is population divergence in isolation. Such population divergence was long thought to be driven primarily by mutation and genetic drift; different sets of mutations arise and drift to fixation in each of the isolated populations, ultimately causing them to diverge. However, it is now well-recognized that natural and sexual selection also play a critical role in driving the differentiation of isolated populations. Differences in abiotic and biotic environmental conditions can drive local adaptation, potentially causing substantial phenotypic and genetic differences between populations. Thus, divergent selection can accelerate population divergence, though it is not requisite for allopatric speciation to occur.

Secondary Contact and Reproductive Isolation

The fate of populations that diverged in isolation is determined in the third stage of the allopatric speciation model. When populations come into secondary contact after the initial period of isolation, there is an entire range of potential outcomes. Divergent populations are often referred to as incipient species at this stage.

If divergence between incipient species was relatively minor and does not affect their ability to interbreed, secondary contact may simply cease the speciation process. In this case, the two lineages hybridize and fuse back together into a single panmictic population.

Alternatively, at least some degree of reproductive isolation may have accrued between incipient lineages. Consistent with the idea that drift was critical in driving population divergence, genetic incompatibilities were long thought to be a primary mechanism of reproductive isolation. In particular, secondary contact and hybridization between incipient species brings together alleles that previously drifted to fixation in the isolated lineages, and some of the new allelic combinations can be deleterious due to negative epistatic interactions (Figure 10.15). Such genetic incompatibilities that reduce hybrid fitness upon secondary contact are also known as Dobzhansky-Muller incompatibilities.

There are two more important points to make here: (1) Dobzhansky-Muller incompatibilities do not only arise as a consequence of population differences that evolved through genetic drift. Alternate alleles that rise to high frequencies in different populations through divergent selection can also be subject to negative epistatic interactions and cause reductions in hybrid fitness. So, while the Dobzhansky-Muller model was primarily developed under the assumption that mutation and genetic drift are the primary drivers of population divergence, the model also applies when other evolutionary forces contribute to population differentiation. (2) Reproductive isolation between incipient species can not only be mediated by genetic incompatibilities (intrinsic post-zygotic isolation), but by any other mechanism of reproductive isolation as well. Evolution in isolation may have inadvertent consequences on ecological, behavioral, and morphological traits, which then affect the probability of mating and successful fertilization between incipient species. In addition, though there may be no genetic incompatibilities that limit hybrid success, hybrids may still be selected against by natural or sexual selection.

Dobzhansky-Muller incompatibilities arise as a consequence of evolution in geographic isolation. Specifically, new sets of mutations arise in geographic isolation and eventually drift to fixation. Upon secondary contact, recombination of these new alleles in hybrids can be subject to negative epistatic interactions. The novel alleles a and b have no evolutionary history of interacting, and if the alleles are incompatible, hybrid fitness is reduced.

Figure 10.15: Dobzhansky-Muller incompatibilities arise as a consequence of evolution in geographic isolation. Specifically, new sets of mutations arise in geographic isolation and eventually drift to fixation. Upon secondary contact, recombination of these new alleles in hybrids can be subject to negative epistatic interactions. The novel alleles a and b have no evolutionary history of interacting, and if the alleles are incompatible, hybrid fitness is reduced.

Irrespective of what mechanisms mediate reproductive isolation between incipient species, reproductive isolation is rarely complete upon secondary contact, as evidenced by the commonality of hybridization in nature. Hybridization, in these cases, is not evidence that incipient species are not species, but rather that the speciation process is still ongoing and has yet to be completed. Ultimately, there are three alternative outcomes upon secondary contact besides complete fusion of the the incipient species: completion of the speciation process through reinforcement, stable hybrid zones of contact, and hybrid speciation.

Reinforcement and the Completion of the Speciation Process

Reductions of hybrid fitness upon secondary contract—whether caused by genetic incompatibilities or by natural or sexual selection against hybrids—should exert direct selection on assortative mating preferences in both incipient species. Individuals that exhibit preferences for mating partners from their own incipient species and discriminate against heterospecifics ultimately have higher fitness, because they avoid producing unfit hybrid offspring. The process of natural selection increasing the degree of reproductive isolation between incipient species is known as reinforcement. In practice, reinforcement often leads to reproductive character displacement, where differences in reproductive traits—both in terms of signals and the corresponding preferences—are accentuated whenever closely related species coexist. A particular trait may be similar between two species in allopatry, but starkly differentiated in sympatry (Figure 10.16). In the end, reproductive character displacement strengthens pre-zygotic reproductive isolation and can lead to the completion of the speciation process.

Reinforcement can lead to reproductive character displacement, where differences between species are accentuated in sympatry.

Figure 10.16: Reinforcement can lead to reproductive character displacement, where differences between species are accentuated in sympatry.

The importance of reinforcement in maintaining species boundaries upon secondary contact has been studied in Ficedula flycatchers, a group of small, Old World songbirds. Allopatric populations of the collared flycatcher (F. albicollis) and the pied flycatcher (F. hypoleuca) both exhibit black and white plumage and look almost identical to the untrained eye (Figure 10.17A and B). However, the two species are very easy to tell apart when they co-occur in sympatry, because the pied flycatchers are actually brown and white (Figure 10.17C). A series of elegant behavioral experiments has revealed how these changes in plumage coloration are a consequence of reinforcement (Sætre et al. 1997). It turns out that plumage similarities not only make it difficult for humans to distinguish allopatric populations of collared and pied flycatchers; one third of matings actually occur between species when females from sympatric populations have a choice between the allopatric color morphs of the two species. This finding is in stark contrast to the strong assortative mating observed when females can choose between males of sympatric color morphs (Figure 10.17D). Consequently, differences in plumage coloration reduce the probability of hybridization, as predicted by the reinforcement hypothesis. Moreover, experiments with female pied flycatchers have revealed evidence for the evolution of mating preferences between sympatric and allopatric populations. While sympatric females prefer males with the brown plumage of sympatric males, allopatric females prefer to mate with males that exhibit the characteristically black plumage of allopatric males (Figure 10.17E). Hence, reinforcement has not only acted on a signal for species recognition and caused character displacement in male plumage coloration, but it has also acted on the corresponding female preference to increase pre-zygotic reproductive isolation.

A. Collared flycatcher (*Ficedula albicollis*). Photo: [Andrej Chudy](https://www.flickr.com/photos/andrej_chudy), [CC BY-SA 2.0](https://creativecommons.org/licenses/by-sa/2.0). B. Allopatric pied flycatchers (*Ficedula hypoleuca*) are black and white like collared flycatchers. Photo: Francesco Veronesi, [CC BY-SA 2.0](https://creativecommons.org/licenses/by-sa/2.0). C. Sympatric pied flycatchers (*Ficedula hypoleuca*) are brown and white. Photo: Steve Garvie, [CC BY-SA 2.0](https://creativecommons.org/licenses/by-sa/2.0). D. While there is strong assortative mating when female flycatchers have a choice between males of the two species that exhibit the sympatric color combination, nearly a third of matings are heterospecific when females are given a choice between males from the two species that exhibit the allopatric color combination. [Data](data/11_flycatcher_spec_rec.csv) from Sætre et al. (1997). E. Evidence for the evolution of female mating preferences in pied flycatchers. Females from sympatric populations prefer to mate with males that exhibit the brown plumage typical for sympatric males; females from allopatric populations prefer males with the black plumage typical for allopatric males. [Data](data/11_flycatcher_mate_pref.csv) from from Sætre et al. (1997).

Figure 10.17: A. Collared flycatcher (Ficedula albicollis). Photo: Andrej Chudy, CC BY-SA 2.0. B. Allopatric pied flycatchers (Ficedula hypoleuca) are black and white like collared flycatchers. Photo: Francesco Veronesi, CC BY-SA 2.0. C. Sympatric pied flycatchers (Ficedula hypoleuca) are brown and white. Photo: Steve Garvie, CC BY-SA 2.0. D. While there is strong assortative mating when female flycatchers have a choice between males of the two species that exhibit the sympatric color combination, nearly a third of matings are heterospecific when females are given a choice between males from the two species that exhibit the allopatric color combination. Data from Sætre et al. (1997). E. Evidence for the evolution of female mating preferences in pied flycatchers. Females from sympatric populations prefer to mate with males that exhibit the brown plumage typical for sympatric males; females from allopatric populations prefer males with the black plumage typical for allopatric males. Data from from Sætre et al. (1997).

Hybrid Zones

Secondary contact between incipient species can lead to stable hybrid zones in geographically restricted areas where lineages meet and cross-breed. Stable hybrid zones can either be maintained in so-called “tension zones,” or in areas of bounded hybrid superiority.

Tension zones are geographic regions where continuous dispersal and hybridization between parental species is balanced by selection against hybrids. Although hybrids are continuously produced, the integrity of species boundaries is largely maintained, because hybrids have lower fitness than either parental species and are continuously selected against. It is important to note, however, that backcrossing between hybrids and parentals can lead to the the introgression of neutral and adaptive alleles between species.

In contrast, the bounded hybrid superiority hypothesis postulates that hybrids have a higher fitness than either of the parental species in environments that are intermediate to those typically occupied by the parentals. In this case, hybrid zones are expected particularly along environmental gradients that separate the niches of the parental species. For example, hybrids between the two swordtail fishes X. malinche (a high elevation species) and X. birchmanni (a low elevation species) primarily occur at intermediate elevations. Hybrids have a higher heat tolerance than X. malinche and a higher cold tolerance than X. birchmanni, which allows them to outcompete either of the parental species at intermediate elevations, where it gets too hot for X. malinche in the summer and too cold for X. birchmanni in the winter.

Hybrid Speciation

Last but not least, hybridization upon secondary contact can lead to homoploid hybrid speciation; i.e., the formation a new species, reproductively isolated from both parentals but without a change in ploidy levels. While not common, such hybrid speciation has been documented in insects, fishes, amphibians, birds, and even a marine mammal (the clymene dolphin, Stenella clymene, which arose through hybridization between spinner dolphin, S. longirostris, and striped dolphin, S. coeruleoalba).

Another notable example of hybrid speciation includes the Darwin’s finches on Daphne Major Island in the Galapagos Archipelago, which we studied in detail in Chapter 2 and Chapter 3. In the 1980s, a male Española cactus finch (Geospiza conirostris) dispersed to Daphne Major and bred with female medium ground finches (G. fortis) that were already there. This hybridization event gave rise to a new lineage (also known as big bird lineage due to its size), which is endemic to Daphne Major and reproductively isolated from the other species that occur on the island.

Explore More: Big Bird and Hybrid Speciation

To learn more about the fantastic tale of hybrid speciation in Darwin’s finches as directly observed by humans, check out Jordana Cepelewicz’ article “New Bird Species Arises from Hybrids, as Scientists Watch” in Quanta Magazine.

10.4.3 Ecological Speciation

The allopatric speciation model dominated our understanding of the speciation process for over half a century. But with the rise of genetic tools, there was increasing evidence for sister species that occur in the same location without apparent periods of geographic separation. New species—it seemed—were also arising without physical separation, a process known as sympatric speciation. The big question, however, was how reproductive isolation can evolve between diverging populations when ongoing gene flow keeps homogenizing gene pools.

Both theoretical models and empirical evidence have since revealed that speciation with gene flow is possible if there is strong divergent selection driving sympatric populations apart. When selection is sufficiently strong, it can overcome the homogenizing effects of gene flow, leading to significant population differentiation (Figure 10.18). Hence, speciation becomes a mere consequence of adaptation to different environmental conditions. Speciation with gene flow in response to divergent selection is also known as ecological speciation.

It is important to note that ecological speciation does not only happen in sympatry, but also in allopatry and other geographical contexts (peri- and parapatry). Hence, the geographic context is less of a focus in ecological speciation research. Rather, research primarily focuses on how exactly adaptation in response to divergent selection translates to reproductive isolation between incipient species.

Divergence between populations connected by gene flow is a reflection of the balance between divergent selection and the rate of migration. As selection gets stronger, the homogenizing effect of migration gets smaller. Each simulation conists of two populations (represented in red and blue) that are connected by migration at the rate of 0.05 per generation. However, similations differ in the strength of divergent selection. A. *w*<sub>migrants</sub>=0.95, *w*<sub>hybrids</sub>=0.975; B. *w*<sub>migrants</sub>=0.8, *w*<sub>hybrids</sub>=0.9; C. *w*<sub>migrants</sub>=0.6, *w*<sub>hybrids</sub>=0.8; D. *w*<sub>migrants</sub>=0.25, *w*<sub>hybrids</sub>=0.5.

Figure 10.18: Divergence between populations connected by gene flow is a reflection of the balance between divergent selection and the rate of migration. As selection gets stronger, the homogenizing effect of migration gets smaller. Each simulation conists of two populations (represented in red and blue) that are connected by migration at the rate of 0.05 per generation. However, similations differ in the strength of divergent selection. A. wmigrants=0.95, whybrids=0.975; B. wmigrants=0.8, whybrids=0.9; C. wmigrants=0.6, whybrids=0.8; D. wmigrants=0.25, whybrids=0.5.

Reproductive Isolation as a Byproduct of Adaptation

A wide variety of divergent sources of selection have been identified as driving forces in ecological speciation. For example, divergent selection can be caused by environmental differences in habitat structure, climate, or resource availability, as well as different communities of competitors, predators, or parasites. In addition, divergent selection may be driven by sexual selection, particularly when environmental variation causes selection on secondary sexual traits or on sensory systems.

Reproductive isolation during ecological speciation evolves incidentally, as a byproduct of adaption caused by divergent selection. Adaptation to different environmental conditions simply reduces the likelihood that individuals from diverging populations meet, that they recognize each other as potential mates if they do meet, that they are compatible during mating if they attempt to, and that they produce competitive hybrid offspring if eggs get fertilized. With the exception of gametic isolation, ecological speciation has been associated with every possible mechanism of reproductive isolation described above.

A classic example of ecological speciation stems from mosquitofish (Gambusia hubbsi) that inhabit blue holes on the Bahamas islands. Blue holes are water-filled sinkholes that were created when underground caverns collapsed. Some of the Bahamas islands are dotted with isolated blue holes (Figure 10.19A), and most have been colonized by mosquitofish. These blue holes, however, differ in whether other fish species have also arrived. In some, the mosquitofish live by themselves; in others, they have to share their habitat with bigmouth sleepers (Gobiomorus dormitor), a large predatory fish. Mosquitofish in different blue holes have adapted to the local ecological conditions. In habitats without predators, mosquitofish have convergently evolved streamlined bodies that allow for energy efficient swimming, while those in habitats with predators have evolved bodies that allow for rapid acceleration during predator escape movements. How has adaptation to the different predation regimes impacted the emergence of reproductive isolation? Brian Langerhans and colleagues (2007) conducted mate choice experiments with mosquitofish from different ecological settings. They first quantified the strength of female preference for a male of their own population, as opposed to a male from a different blue hole with the same predator regime, or a male from a different blue hole with the opposite predator regime. The results indicated that assortative mating (i.e., a preference for the male from the female’s own population) was much stronger when females were confronted with a foreign male derived from different environmental conditions (Figure 10.19B). So, females from no-predation habitats preferred to mate with males from no-predation habitats, while those from blue holes with bigmouth sleeper preferred to mate with males from other high-predation habitats. Females made mating decisions based on male traits that are directly related to adaptation to different predation environments, as evidenced by the negative correlation between morphological dissimilarity and the relative strength in mating preference (Figure 10.19C). In other words, females from both environments prefer to mate with males exhibiting a body shape that matches their specific ecological context. Adaptive evolution of a morphological trait consequently had inadvertent repercussions for mate choice, and ultimately caused behavioral isolation between diverging lineages.

A. Watling's Blue Hole, southwestern San Salvador Island, eastern Bahamas. Photo: [James St. John](https://www.flickr.com/people/47445767@N05), [CC BY 2.0](https://creativecommons.org/licenses/by/2.0). B. Assortative mating preferences of female mosquitofish that could choose beteen a male from their own population or a male that either came from a different blue hole with the same predation environment or a different blue hole from a different predation environment. Assortative mating with the male from the female's own population is much strong when she is confronted with a foreign male from a different environment. [Data](data/11_gambusia-assortative-mating.csv) form Langerhans et al. (2007). C. Female mating preferences are primarily based on body shape, and relative mating preferences decrease with increasing dissimilarity of a male's body shape to the locally adapted form. [Data](data/11_gambusia-assortative-mating2.csv) form Langerhans et al. (2007).

Figure 10.19: A. Watling’s Blue Hole, southwestern San Salvador Island, eastern Bahamas. Photo: James St. John, CC BY 2.0. B. Assortative mating preferences of female mosquitofish that could choose beteen a male from their own population or a male that either came from a different blue hole with the same predation environment or a different blue hole from a different predation environment. Assortative mating with the male from the female’s own population is much strong when she is confronted with a foreign male from a different environment. Data form Langerhans et al. (2007). C. Female mating preferences are primarily based on body shape, and relative mating preferences decrease with increasing dissimilarity of a male’s body shape to the locally adapted form. Data form Langerhans et al. (2007).

The Role of Sexual Selection

Even though ecological speciation is primarily driven by natural selection, sexual selection can also play a critical role in speciation processes. Specifically, ecological speciation can be facilitated when traits under divergent selection (i.e., traits involved in adaptation) also contribute to non-random mating (i.e., behavioral isolation). Such traits are often referred to as “magic traits,” because a single trait is associated with functions usually attributed to separate traits—those mediating adaptation and those mediating assortative mating.

Magic traits appear to be a common theme in ecological speciation, as many adaptive modifications are also reliable cues for mating decisions. One of my favorite examples comes again from Darwin’s finches, where adaptive evolution of beak size has inadvertently affected the sounds males can produce when singing (Podos 2001). Adaptation to different food resources caused the diversification of beak sizes in finches from the genera Camarhynchus and Geospiza; species with smaller beaks tend to forage insects and other soft prey items, while those with larger beaks primarily feed on seeds. Changes in beak size have also impacted the vocal range individuals can produce. Birds with larger beaks have evolved songs with lower rates of syllable repetition and narrower frequency bandwidths, which ultimately reduces song complexity (Figure 10.20). Adaptation to different food resources was thus inadvertently correlated with the vocal communication essential in songbird mate recognition and courtship, facilitating the emergence of reproductive isolation in conjunction with adaptive trait evolution.

Beak size evolution in Darwin's finches in correlated with song complexity, as measured by vocal deviation. Higher vocal deviation in this case corresponds to lower song complexity. [Data](data/11_beaksound.csv) from Podos (2001).

Figure 10.20: Beak size evolution in Darwin’s finches in correlated with song complexity, as measured by vocal deviation. Higher vocal deviation in this case corresponds to lower song complexity. Data from Podos (2001).

Magic traits can also arise from selection on sensory systems that impact mating decisions, not just mating signals themselves. For example, cichlids in Lake Victoria have adapted their eye physiology to optimize visual perception at the different depths where they live (Seehausen et al. 2008). Since ambient light becomes more red-shifted at greater depths, cichlids have evolved retinal pigments that are more apt to detect red light. As a consequence of chooser biases (Chapter 10), the changes in visual physiology are correlated with red shift in male nuptial coloration, such that adaptation to different light regimes has caused speciation between a population with blue nuptial coloration inhabiting shallow waters and a population with red nuptial coloration inhabiting deeper waters. Interestingly, speciation into blue and red forms has independently occurred multiple times within the lake.

10.5 References