In my opinion 2-3 days is really just a guide to remind people that fish is highly perishable, more so than most animal proteins. Do not toss it in the refrigerator and expect to wait a week to get around to cooking it and expect it to still be edible.
From time out of water to cooking, it will greatly depend on variety, size, etc., but if it is properly cared for you will normally have longer than 2-3 days from catch. But, the longer it is from catch to cleaning, time is reduced, that is offal will deteriorate faster than flesh but damage the flesh if not promptly removed. Poor cleaning deteriorates the flesh. Any time out of optimal temperature storage, reduces useful time. You have no idea really how long any of these periods are, unless you harvested the fish yourself, so you assume reasonable handling and that from the time the merchant makes it available, you probably have 2-3 days before you risk spoilage and try to be familiar enough with your merchant that they have not already had it sitting for 2-3 days.
All of this though is non-optimal. Fish, and all seafood deteriorates fast. It is not like beef that often benefits from aging. Seafood quality goes down by the hour after harvest. If any seafood that begins to smell like anything other that fresh is past its prime at best. Sliminess, dullness of the surface, any loss of firmness, weeping of fluids, these are almost always signs that the fish is either past its prime or simply beyond usefulness. Some fish are soft to begin with such as salmon, but comparing two fish of the same variety, the firmer fish will almost always be get fresher or better handled fish.