Try granite.
I have used water-smoothed granite cobbles (~ 1 kilo) as key ingredients for making 'stone soup' (AKA Minestrone ala Lithos) with no ill effect to myself, family or guests. However, this use keeps the stone's temperature low, geologically speaking. Granite is used for countertops; most types have little I worry about, but your milage may vary.
I suggest that whatever you use, you:
Identify the type of rock, and check for any worrisome constituents. (No peacock copper {arsenic}, Galena {lead}, or Pitchblende (radioactives.) Corolary: if you can't identify it, don't muck with it (much like wild mushrooms.)
Sterilize it well. I boil mine for at least 15 minutes.
If going for high heat, do several heat/thermal-shock tests, wearing some sort of eye protection, before using the stone around anybody else.
Hope it turns out well!