The short answer is that it's probably safe, but here are some considerations:
What Does Yeast Do?
Yeast eats sugar and converts it to carbon dioxide (also alcohol, but that's not a consideration in most bread baking). "Sugar" doesn't necessary mean sucrose (table sugar) - it also means fructose and glucose, like you'd find in honey or agave syrup.
If you don't provide the yeast with anything to eat, it won't produce any carbon dioxide.
Can Yeast Eat Stevia?
Stevia is not any kind of sugar, nor can it be converted into sugar (more on that in a moment).
Does Leavened Bread Need Sugar (glucose/fructose/sucrose/lactose/maltose)?
Many (most?) bread recipes do not contain any sugar - and yet they can rise impressively. How does the yeast produce carbon dioxide if you don't add sugar? The flour can feed the yeast.
Flour is largely starch, which is a chain of sugar (glucose) molecules. Our digestive systems are powerful enough to break starch down into sugar, but yeast cannot do so. Fortunately, the wheat berry itself contains the solution. An enzyme present in wheat berries (amylase) will break down the starch into sugar. Once this happens, the yeast can eat the sugar and happily bubble away.
1 I had a note here about proofing active dry yeast, but this information is now out of date.
Active dry yeast has been reformulated. It's now in smaller particles and does not need to be dissolved in water. It can be added directly to the dry ingredients. If your active dry yeast is nearing its expiration date, you can proof it by mixing it with warm water and a small amount of sugar. This doesn't do much to affect the bread, but it does allow you to avoid baking with dead yeast.
Will Anything Else Happen If There's No Sugar?
Possibly - that is, some things will change, but you may not detect the change.
(Most of the rest of this information comes from this article - "How to reduce sugar in yeast bread". Feel free to skip my summary and go straight to the source!)
The bread may be drier
when you reduce sugar in yeast bread, the only textural difference you might see is a tendency towards dryness.
Why is this? Sugar is hygroscopic; that means it attracts and holds moisture. Without sugar, moisture evaporates from bread during baking, creating a drier loaf. The more sugar you cut from a sweet yeast bread recipe, the more you’ll notice this effect.
Counter intuitively, it might speed up the dough's rise
Remember, sugar is hygroscopic. And in yeast dough, this means it can deprive yeast of the moisture it needs to grow. Ever waited impatiently for your sweet bread to rise? Blame the “arid” atmosphere
The bread may brown more slowly
Some of the sugar in yeast dough rises to the surface and caramelizes as bread bakes, yielding rich brown color
A Note about Acacia
A comment asks about acacia, otherwise known as acacia gum, or gum arabic. As the Wikipedia article says, "Gum arabic is a complex polysaccharide," which makes it chemically similar to starch (also a polysaccharide). But can amylase (or some other enzyme already present in the dough) break down this polysaccharide? I would guess so, but here's where we come up hard against the limits of my high school chemistry education.