There's a bit of confusion here because there are two senses of the word "fruit": the botanical sense with a more precise definition (plant parts with seeds), and the common everyday culinary usage with no precise definition (mostly sweet plant things).
In a culinary sense, i.e. talking about food, among chefs or just in common speech, there is no strict definition of fruit, and no completely precise delineation between fruit and vegetables. Sweetness and how it's eaten play a big part in defining fruit, as you've noted, and culinary fruits are mostly botanical fruits that meet those criteria, but that doesn't form a perfect definition. See rumtscho's excellent answer for more on how categories like this are (and aren't) defined. Vegetables, then, are basically everything from plants besides the fruit, also excluding grains.
This is the sense of "fruit" you'll generally hear. If you ask someone to organize a grocery store produce section, to go to the store and buy some fruit, or to put fruit in each lunch bag, they'll tend to use this sense of fruit/vegetable. So apples, oranges, and bananas will be fruits, while spinach, peppers, cucumbers, and beets will be vegetables.
But other things are less clear. For example, if you cook green plantains (starchy, not sweet, can be used much like potatoes) and ask someone if they're a fruit or a vegetable, you're pretty likely to get a response along the lines of "well, they're like bananas, so they must be fruit, but they're not sweet." And you could hear differing opinions on whether fresh corn is a vegetable or a grain.
Then there's the botanical sense, i.e. talking about parts of plants. In that sense, there's a strict definition: fruit is the seed-bearing structure of the plant. Other parts of the plant then have their own names; you'd talk about stems and leaves and roots, not vegetables.
You'll hear this sense in botany, such as "the fruit of the maple tree are called samaras or whirlybirds". It's similarly used in gardening, as in "my zucchini and tomato plants are bearing plenty of fruit".
Note that while it makes plenty of sense to talk about fruit in this context, it doesn't really make a lot of sense to talk about fruit vs vegetable in this context. You're not going to look at a tomato plant and say "it has a lot of fruit, and the vegetable parts look healthy too." You might talk about your garden as having fruit and vegetable sections, but then you're coming back to the culinary sense.
Unfortunately, when the two senses come together, people can sometimes make pretty wild claims. Perhaps because the botanical definition is scientific and precise, while the culinary one is unscientific and fuzzy, people sometimes portray the botanical definition as the "real" one. This leads to statements like "technically, fruit has seeds, so tomatoes are fruit."
But it's best to simply consider the context. Generally you'll be clearly talking about food, or clearly talking about plant anatomy. And if you're talking about food, most foods have uncontroversial categorizations, even if explaining why is difficult.
So if you want to be a smartass, by all means, when someone asks for a piece of fruit hand them a tomato and say "technically...". But if you want to just enjoy food and not get into pointless arguments, just use a bit of common sense and avoid trying to force the two senses of "fruit" together.