The Yolk and White are very different and I would expect to see a significant difference in result.
That isn't to say it will be bad, just different.
The Yolk is the source of fat. It is going to impart flavor to the dish and provide a creamier texture.
The White is mostly protein and so will set more, have more of a texture impact.
If it called for several whole eggs and you wanted to take a Yolk or two out for less fat, that would have less of a variance, but switching several Yolks to several Whites should definitely alter the results.