Building Struggle Chef: When AI Looks Inside Your Fridge

I wanted to investigate how well AI could detect things from images. Not in a lab setting with perfect lighting and curated datasets - in messy, real-world conditions. And what's messier than the inside of a fridge?

A fridge is actually a surprisingly good test case for computer vision. You've got varied items with different shapes and sizes, partial occlusion (that yogurt hiding behind the milk), inconsistent lighting, reflective surfaces, and items at different depths. If AI can reliably identify what's in a fridge, it can probably handle most real-world object detection scenarios.

Plus, there's an obvious practical application: if the AI knows what food you have, it can suggest what to cook. That's how Struggle Chef was born.

How It Works

The concept is dead simple:

1. Take a photo of your fridge
2. GPT-4o Vision analyses the image and identifies the contents
3. The app generates recipe suggestions based on what you actually have
4. Optionally, it roasts your food choices

That last point deserves explanation. During development, I was testing with my own fridge and thought it'd be funny to add a "roast me" toggle. You know those moments where you open the fridge and think "I really need to go shopping"? The roast feature is the AI saying that out loud. It looks at your sad collection of condiments and half-empty containers and gives you honest, slightly brutal feedback.

It turned out to be the feature everyone remembers. The recipes are useful, but the roasts are what people share.

The AI Detection: How Well Does It Actually Work?

This was the whole point of the project, so let me be honest about the results.

Pretty good, but not perfect. GPT-4o Vision correctly identifies most common items - fruits, vegetables, bottles, cartons, containers with visible labels. It's surprisingly good at reading text on packaging, even at angles or in dim fridge lighting.

Where it struggles:

• Items hidden behind other things - if only a corner of something is visible, the AI either misses it or guesses wrong
• Unlabelled containers - a plain tupperware container could be anything, and the AI knows it. It'll usually say "container with food" rather than guess
• Similar-looking items - different types of cheese, various sauces in similar bottles, different greens. It gets the category right but might miss the specific variety
• Items in the door shelves - the angle and lighting tend to be worse here, and items are often partially obscured

Overall, I'd say it correctly identifies about 80-85% of visible items. That's more than good enough to generate useful recipe suggestions - you don't need a perfect inventory to figure out you could make a stir-fry with those vegetables and that bottle of soy sauce.

The Tech Stack

I kept this one deliberately simple:

• Next.js 14 - the app framework. Server-side API routes handle the OpenAI calls, keeping the API key secure
• React - the frontend, with a camera capture component and a results display
• TypeScript - type safety throughout, especially useful for the structured responses from GPT-4o
• Tailwind CSS - rapid styling. The UI is clean and mobile-first since most people will photograph their fridge with their phone
• OpenAI API (GPT-4o Vision) - the brain of the operation. Handles both the image analysis and the recipe generation

Deployed on Netlify, which handles the Next.js server-side routes seamlessly. The whole thing is a single deployment with no external services beyond OpenAI.

The Prompt Engineering

Getting useful, structured output from GPT-4o Vision took more iteration than I expected. The naive approach - "what's in this fridge?" - gives you a paragraph of text that's hard to parse programmatically.

I ended up with a two-step process:

1. Identification step: The image goes to GPT-4o with a structured prompt that asks for a JSON list of identified items, each with a name, confidence level, and category (produce, dairy, protein, condiment, etc.)
2. Recipe step: The identified items list goes back to GPT-4o (text-only this time) with a prompt asking for recipe suggestions that primarily use the available ingredients, noting any common pantry staples that might be needed

The roast feature is a separate prompt path entirely. It gets the same identified items but with instructions to be witty and slightly judgmental about the fridge contents. The tone calibration took a few rounds - too mean isn't funny, too gentle isn't memorable. The sweet spot is "friend who's honest with you."

What I Learned

The main takeaway is that vision AI is genuinely useful for real-world detection tasks, even without fine-tuning or custom models. GPT-4o out of the box, with good prompting, handles the fridge detection task well enough to build a useful product on top of it.

The other insight is about product design: the "fun" feature (the roast) drives more engagement than the "useful" feature (the recipes). People open the app to get roasted, and the recipes are a bonus. If I were building this as a real product, I'd lean hard into the personality and humour angle rather than trying to compete with dedicated recipe apps on utility.

Struggle Chef is live and deployed. Give it a try - open your fridge, take a photo, and see what the AI thinks of your food situation. Just be prepared for some honest feedback if you hit that roast button.