I've been told that ghee has no lactose in it, so that might be one alternative to butter, but not to the milk, as I find you need some water-based liquid to get the consistency I like -- you're rupturing starch into the liquid, which gives a creamy mouthfeel.
I personally use stock, not milkuse stock, not milk to adjust the liquid in my potatoes. You have to adjust the salt, but I personally prefer it. (and it has nothing with my not being able to eat dairy.) You can add some olive oil to change the mouth feel, although too much and it's pretty obvious it's olive oil in there.
As milk is sweet, you might also consider adding a bit of other root vegetables like carrots or sweet potatoes in, but that'll significantly change the color, which the kids may not like (or they may love ... you never know). If you use carrots, you'll want to start them cooking while you're prepping the pototes, so they get a head start on the cooking, and it doesn't hurt to cut them a bit thinner.