I think the issue is primarily linguistic, but there may also be a mismatch between your experience of Japanese food and the average Japanese experience of Japanese food.
Let's start with the experience itself. Wasabi is generally used in moderation in Japanese cuisine, and when real, fresh wasabi is used, instead of the mustard/western horseradish mix that's more common, it's more pungent than spicy. That's a fairly nuanced distinction, and you may find both Japanese and non-Japanese that would use the word "spicy" to describe what amounts to a nasal reaction, instead of the more direct tongue stimulation that say capsaicin, or glutamates trigger. In Japanese, you might say "piri" or "piri-tto" to refer to an abrupt sensation of pungency that doesn't linger, like (real) wasabi offers, or "tsuun" to refer to the tingling sensation in a more visceral onomatopea. "Karai" is used to describe spicy foods (and, in some cases, to describe salty foods, typically soups, but let's ignore that for now).
In any event, wasabi isn't really used as heavily in everyday Japanese cooking as its popularity in the US would suggest. Additionally, the US has latched on to spicy tuna rolls and complex, multi-ingredient gooey "rolls" as representative of sushi, even though in Japan most "makimono" are minimalist creations involving little more than some cucumber, or gourd, and aren't even the reason you go out to a sushi restaurant. The multi-ingredient ones with say egg and pickled vegetables are still simpler in taste than what most Americans would get excited about.
For many Japanese, seeing the ridiculous amount of reconstituted wasabi served with their little plate of nigiri-sushi or the sriracha augmented rolls comes as a bit of a surprise when they visit the US. Our culinary preferences tend to be adventure-seeking, whereas Japanese tend to have more interest in "sappari" (refreshing) or "asari" (light/subtle) flavors and are more focused on texture contrasts than intense flavors.
To some degree, wasabi is a regional food (Shizuoka prefecture grows much of it), even though it's found around the country thanks to modern distribution. Sushi is not an everyday experience for most people, either, and it's not seen as a "spicy" thing when it is consumed, because most people don't eat it with loads of wasabi; they want to taste their fish.
From a culinary perspective, mustard is one of few "spices" that wouldn't actually be referred to as an herb that's really used in Japanese cuisine. (It's also a major component in mass market wasabi). Ginger is an exception, though it's also mostly used sparingly, and generally fresh, so it is only arguably "spice".
The "spicy" flavors that are popular in Japan are probably the Japanese interpretation of English-style stews called "curry". These use Indian blends of spices adapted to Japanese tastes, but most versions are sweeter and milder than they are "hot". It's somewhat common, but not necessary, for people to enjoy extra spicy curries. But curries have a status that is vaguely foreign, like tikka masala or mulligatawny soup in England, even if both are really "local" innovations. Even if you're Japanese, you may not quite consider curry as a spicy "Japanese" food.
Additionally, you may notice even in English, the notion of "spice" isn't perfectly attached to the notion of "spicy." If I use cloves or ginger in something, it might be "spiced" with spices, but perhaps isn't considered spicy.