The differences will be down to the strain or variety of the yeast that you cultivate from the source. Generally baker's yeast is considered to be Saccharomyces cerevisiae, which is a eukaryotic organism with a large genome. Many of the genes produced are involved with metabolism of various sources of energy for different environments and disposing of waste compounds from metabolism. In addition, the yeast that you "isolate" is very unlikely to be a single strain or even a single species, all of which taste and smell subtly different to each-other. For instance another yeast Schizosaccharomyces pombe is also present on grapes and is commonly used in wine making. If you used grapes as your starter source, you are likely to get this yeast too or maybe instead of baker's yeast.
Some of the waste compounds involved are things like esters which are produced as by-products of metabolic activity of different carbon sources. Esters are often short-chain volatile compounds with distinctive smells/flavours - if you like candy/lollies flavoured as "banana", the flavour is often the ester amyl acetate. These are the different flavours that you get from different yeast strains.
As to which flavours they are exactly - well that will depend on exactly which environment you have got them from - grapes will be different to tomatoes, which will be different to apples etc. Now you might think "well, I'll just get some [insert favourite yeast source here]" and that each time you did this you would get the same result, but it isn't that simple - the yeast that you get will also be dependent on where the yeast source came from, the time of year, the stage of fruit ripeness, how the grower treated their plants, what's floating around in the air at the time... etc. and finally exactly which mix of yeast strains are present at your location.
Some of this complexity is seen in wine and beer manufacture - not all locations are equal in terms of wine flavours because of the wild yeasts that are present on the grapes or barley when they are crushed/malted to get the fermentation started, so most vineyards and breweries now add bulk stocks of single-source yeast to overcome this problem and standardize the flavours their vineyard/brewery produces.