Often times I find myself swishing, dunking, or moving a teabag around to affect the steeping time. Does it have any effect at all on the tea infusion, or am i actually hurting the flavor?
You are not hurting the flavor. In theory, it will shorten the steeping time a miniscule (probably imperceptable) amount, but the effect is more psychologial perception than reality. I pour boiling hot water over my teabag and start my timer for five minutes and don't touch or swirl the tea bag. If the bag sits in the hot water 20% longer (6 minutes instead of 5), I would be lying if I tried to pretend that I could preceive a taste difference from the previous cuppa I'd sipped. I cannot. Costs you nothing - so try it and see if you can taste any difference.
When you move water around the tea you are making the chemical reaction faster. With black teas, it's easier to see that the color becomes darker more quickly. Steeping extracts various compounds - caffeine, Polyphenols, Tannins etc. It is possible that first there has to be some sort of reaction within the tea leaves themselves that will then steep into the water, meaning that some compounds won't be released into the liquor within the first 2 minutes no matter what you do. Regardless, moving the teabag around will shorten the steeping time. I believe most instructions refer to leaving the teabag as it is.
As Cynetta put it, try to test it - move one around for 1 minute and leave the other one put for one minute. You'll probably taste the difference.