Capers are actually flower buds from the capparis spinosa (aka the "caper bush"), which grows throughout the Mediterranean.

Capparis spinosa, the caper bush, also called Flinders rose, is a perennial plant that bears rounded, fleshy leaves and large white to pinkish-white flowers. The plant is best known for the edible ...