Recipe Recommender System Using Image Recognition of Food Ingredients

We employed image recognition of food ingredients, specifically vegetables, to recommend recipes based on the recognized ingredients.

1 month

Introduction to Machine Learning

Mona Mishra(Me)
Responsibilities: Creating user interface to interact with the system, research and testing machine different machine learning models on large datasets.

Yifan Gong
Responsibilities: Research, dataset creation and testing different machine learning models on those datasets

Microsoft Azure
Jupiter Notebook for python
Flask framework for front end development
ImageNet for Dataset creation


There are several cooking apps or websites available today, that are used to find recipes based on some keyword, like name of the food ingredient or type of cuisine, etc. These apps are mindful of the needs and interests of their users, but they fail at identifying their user's constraints, i.e., limited number of food ingredients. In such cases, users find themselves shopping for ingredients or they decide to substitute the missing ingredient with something else. To help users avoid such adjustments, image recognition can be employed to identify food ingredients that are already available at their disposal and recommend them recipes based on those ingredients.


Currently the system is a web application that performs image recognition on the uploaded images and recommends recipes that contains the recognized ingredients. We built a convolutional neural network model for image recognition to identify five categories of food ingredients and achieved, 62.9% accuracy rate. The recommendation system uses the labels of the identified images to display a list of recipes that contains most of the identified ingredients.

System Flow

Accuracy Comparison

Loss Comparison

Please follow this link to view the concept video

What I learned

Through this project, I learnt to build different machine learning models to improve upon the accuracy for object recognition. How we build the dataset affects the model greatly and hence it is extremely important to clean the data before using it. We also found that as we increased the number of categories of the food ingredients, the accuracy dropped significantly. Hence, as we realized that in order to increase the number of categories, it is important to increase the number of images for all other categories to achieve best possible result.