I'm creating a service for changing people nutrition habits and convincing them to eat more healthy. As a first step I would like to develop an app to recognize what's on the plate.

I'm aware about apps which can recognize if a hot-dog on my plate is a hot-dog or a salad - however it's not exactly what I want to or need to achieve. For starters I need a website or a digital book where I can find:

  • Recipes with listed ingredients names and weights
  • Pictures of food which was created out of those ingredients - it can be a photo of just one serving or entire meal, however I need to know for 100% it's a picture of outcome of that recipe with info if I have entire meal or just a serving on that picture.

For example: allrecipes.com Spicy lime avocado soup.

I'm cooking it in a pot where I can fit 4 servings, then I divide it to 4 plates, so the photo which would work in this case would be:

  • Pot with 4 servings
  • Plate with serving
  • 4 plates with one serving each

allrecipes.com has list of ingredients together with their weights, however photos not always reflect the size/weight of the created meal. To make my app working I need a reliable resource where I could get many of such recipes without checking if each of them has a good photo or not.

I'm asking here, as I cook only basic stuff and I use only 1-2 websites where unfortunately there is no such information. Maybe by any chance you use/saw/create a website where I can find the information which I need.

  • 2
    Oskar, welcome! Maybe your question would be better received if you would add a bit of background - for example why do you need such a picture? What’s the actual goal you are trying to achieve? Most cooks will likely just prepare the recipe and split it in n equal portions - or maybe not equally, depending on who’s sharing the meal.
    – Stephie
    Mar 28 '19 at 13:33
  • 2
    Oskar, you still haven't explained what the purpose of those pictures would be. If the recipe tells you the number of servings, why do you need pictures of what it looks like? What do the pictures tell you that you don't get from the instructions?
    – user141592
    Mar 28 '19 at 14:25
  • 1
    Hi @Johanna I just rephrased my question - I need those pictures to create an AI to recognize/extract information, I cannot do it without a reliable resource
    – Oskar
    Mar 28 '19 at 14:29
  • 5
    Suppose I produced soup, one batch with full-fat dairy and lots of salt, and the other batch with dairy substitute and no salt. These would look visually the same and so there is nothing your AI could use to tell what it was looking at. This problem more generally (that properties like ingredients, weight, servings and calories do not map bijectively to visual appearance) means your project is doomed to fail.
    – dbmag9
    Mar 28 '19 at 15:10
  • 1
    @Oskar 120 pieces over 2 weeks is about 10/day. Reasonable if they're bite size. Do you also want the recipe authors to photograph on calibrated plates? I may have to post some pictures in an answer tonight to illustrate the point some of use are trying to make.
    – Chris H
    Mar 28 '19 at 16:32

This should illustrate some of the issues we point out in the comments.

Here's a pizza on a plate. It fills the plate, a reasonable portion by the look of things.

enter image description here

Actually no. That's a mini pizza on a side plate. Here's the same pizza on a dinner plate.

enter image description here

Not quite so generous now. How would you take that into account?

The cheese was mozarella and cheddar, but the cheddar was reduced fat. I did that because they'd sold out of ready-grated full-fat, but someone on a diet (presumably your target market) could be expected to make that substitution and expect your app to deal with it -- after all what proportion of the calories comes from the dough, and what proportion from the cheese? Of course I didn't weigh the cheese, or measure it by volume (or the sauce for that matter, but that's home made with hardly any calories in).

I happen to be sitting next to a pile of cookery books. Some do have pictures of the entire dish, to serve n people -- for a few recipes, but with nothing to give an idea of scale. Some have pictures of individual servings, usually with an unspecified side dish which would account for most of the calories. Photos are often artfully shot so you couldn't extract a size from them; a few are shot from directly above so you don;t know the depth. My pizza above was shot to try and show its size - but it's not a very appealing photo (to be fair those bases are nowhere near as good as home made and I won't be buying them again).

Veering off topic: I've played around with myfitnesspal in the past. That's the sort of thing you're up against. For packaged foods (and restaurants that list nutrition information) it works quite well - but you or your users would need to recreate their database. For recipes it works quite well - by entering the ingredients as numbers and dividing them up, or entering the nutrition information from a recipe book. Actually you can get a good feel for what your home-cooking is going to come out like in terms of calories, so after a little while you don't need to keep entering dishes but can use something similar.

Properly off-topic: These are issues that have plagued machine-vision systems designers for years. AI isn't a magic bullet to deal with that. It's easily fooled. Even if we could provide a training dataset that wouldn't help with your end-users' dishes.

  • 2
    I won't +1 because this isn't an answer, but I want to say I like this explanation as to why this question is difficult to answer/why this project will be difficult to pull off. Amount/volume of ingredients should be more than sufficient to estimate serving size and nutrition. Maybe a more generic graphical representation of the estimated size based on a standard dinnerware measurement, such as a plate 10" in diameter and a medium soup bowl, compared to a universally recognized measurement, such as Amazon does with some products.
    – Jorgomli
    Mar 28 '19 at 21:13
  • Hi Chris, thank you for taking your time and answering my question. I really appreciate it. People in comments asked about the context so I provided it - now I regret that. Believe me I considered all the pros and cons, I won't explain exactly what I want to achieve as it would take too much time. I'm aware of ML/DL, or any other optimization algorithms' capabilities. Calories counting isn't perfect, all the labeled information can vary by 20%! (by FDA: alturl.com/ehexo)
    – Oskar
    Mar 29 '19 at 3:27

Not the answer you're looking for? Browse other questions tagged or ask your own question.