Yes, a very dense dough will not be able to rise. The gases will not have the strength to push the hard dough apart. There are types of bagels which are supposed to be quite dense, but American bagels are seldom made this way. This recipe is quite low hydrated at 52%, so even small errors can push it into problematic territory.
Individually packaged instant yeast keeps for a very long time. Big packs keep well if refrigerated after opening. Your yeast is not likely to be the problem. There are three possible problems you may have encountered:
- You measured your dry ingredients by volume, not weight
- You used the wrong type of flour
- The recipe gives an insufficient amount of time
I suspect that you probably measured by volume. This is the most common mistake which leads to a wrong dough consistency. You should measure by weight because measuring by volume is not precise enough: depending on your technique and on the humidity in your pantry, you can have up to 50% measuring error. The recipe gives you a weight; use it. If you find a recipe which doesn't give you a weight, consider changing the recipe - the best sources always use weight. If you insist on using a recipe specified in volume units, use a converter and measure everything in weight. This ensures that you use the ratio given in the recipe and eliminates technique errors on your part. If the recipe author didn't make technique errors while creating the recipe, you will get the right output. (For example, Corriher calculates her recipes by weight ratio, but she lists them in volume because her readers prefer it that way. If you convert back by the factor she gives, you will never be wrong). If you absolutely must measure by volume, then pour your flour into the measuring cup and level it by shaking. Never scoop flour. Still, prepare for botched doughs from time to time, even with pouring.
The second likely reason is the wrong type of flour. Or maybe not "wrong" per se, but different from what the author had in mind. In different regions, flour labeled as "bread flour" has different amounts of gluten. If in doubt, look at your nutrition label; it should be around 12% for bread flour. If it's not, adjust the amount of water slightly (you have to go by feel here).
The time given in the recipe is also shady. At 4.5 g dry yeast and 453g flour, it is equivalent to 3% fresh yeast (even less if you overmeasured the flour). This is a good percentage for a slow, long rise. But saying to ferment it in the fridge "for at least one hour" is nonsense. This amount of yeast needs more than an hour at room temperature, or something like 6 hours in the fridge, even for a rich wet dough - probably more for the lean dense one. If you only kept it for an hour before shaping, you didn't have any primary rise. Next time, just go by volume. Primary fermentation should double the dough volume. The change in volume in secondary fermentation depends on the process, and I don't remember the correct one for bagels.
Bottom line: your diagnosis sounds very likely - not enough fermentation for the amount of flour. Measure correctly, and allow enough time for fermentation, to get your bread right.