While it is usual to remove the organs before eating in many countries, there are many else where it belongs to the fish and is always eaten.
Some organs like the eggs from female fish might even be considered a delicacy in those countries, where people normally don't eat the organs. A good recipe is to fry them together with onions, vinegar and some salt.
All organs are edible (from commonly consumed fishes, there are fishes that have highly toxic organs! The liver of pufferfish isn't edible), just like from most animals. It's rather a question of taste. With small fishes, mostly the organs aren't removed at all because it would be too much work. Anchovies are an example.
You should consider, that fish organs have the highest amount of environmental pollution, microplastics e.g. and heavy metal accumulation! It is highly recomendable to not eat the organs at all! The least heavy metal accumulation happens in the muscles. Other tissues have more, like organs, skin, lungs, brain etc.
Nevertheless, the organs are at the same time those parts with the highest nutritional value. Like FuzzyChef said, organs are also used for preparing fermented food.
Before I aware of the pollution, I ate all organs (apart from the lungs) from sardines, mackerel, horse mackerel, and anchovy. They taste totally different than anything else. Not bad not really good but interesting.
Deep frying the bones from fish is extremely delicious and supplies high amounts of potassium and calcium. The bones get crisp and you can easily eat them. Just keep the pollution in mind and hold the balance.