There has been some doubt about the grouping of the given ingredients.
Some comments and answers have inspired me to look into different plant families and chemical compounds, which has been an interesting discovery.
Unfortunately, I didn't find the answers to be sufficiently self-contained so I felt a need to compile my findings in an answer. But I would like to thank everyone who helped me understand this grouping: Thank you!
Now, I want to start with making it clear that I was asking for the flavor and not the heat.
Stephie linked to an answer for another question that asked for heat and not flavor but the category for mustard and horseradish led to
Mustard and horseradish (and to a lesser extent radishes, cress and other plants) contain glucosinolates, which we percieve as pungent, sharp or hot.
An extreme example for glucosinolate-hotness is wasabi.
-- Stephie [highlights mine]
Looking into glucosinolates, we see that the flavor of it is caused by
mustard oils, which contain
allyl isothiocyanate as mentioned by Eugene Welker.
Now, this explains the compound that groups mustard, horseradish, and wasabi, but not yet for capers and jalapeños.
Capers are in the family
capparaceae and are in the order
The Capparaceae (or Capparidaceae), commonly known as the caper family, are a family of plants in the order Brassicales.
-- Wiki (capparaceae)
Capers are in the same family as mustard and contain glucosinolate.
The Capparaceae have long been considered closely related to and have often been included in the Brassicaceae, the mustard family (APG, 1998), in part because both groups produce glucosinolate (mustard oil) compounds.
-- Wiki (capparaceae) [highlights mine]
I can't find any sources that says that jalapeños contains glucosinolate (isothiocyanate).
Maybe I'm wrong on this?
It's just peculiar because I found the taste to be specific to jalapeños and no other chilis.
Addressing Joe's answer, the group being pungent is maybe incorrect but definitely too vague because it covers other things like curry.
I'm writing maybe because pungency is mostly associated with having heat but different definitions are a bit contradictory.
However, I found the terminology section to be an interesting read: terminology on pungency.
Checking the definition of pungency by Merriam Webster, we get
- 1: sharply painful
- 4b: having an intense flavor or odor, example: a pungent chili
but capers don't have an intense flavor to be compared with
sharply painful or a chili.