I have been eating fresh and brined olives for years. I like the olives with the pits and usually consume the whole pit without removing it. It was recently pointed out to me that this is not a healthy practice, and even though I have been doing this for some time, I began to wonder whether it is 1)abnormal to eat the whole olive, pit and all, and 2) will the pits have a long term adverse effect? I have never met anyone who ate olive pits, and I find the with-pit olives to be the best, so I thought I would ask. Note that I do not grind up and chew the pits, as I am sure that my teeth would not like that!
I'd avoid this practice.
The olive pit husk is very hard and should not be eaten, however, as it is indigestible and might injure the intestinal wall if sharp edges are present. Also, do not chew the husk, as it is so hard it could break a tooth. Swallowing the husk is also a bad idea, as it has sharp ends that could be dangerous. It is also quite indigestible and will come out just as it went in. (source)