← All Posts

Building a Better Nutrition Label Scanner

Building a Better Nutrition Label Scanner

I released FoodNoms 2020.6 just a few hours ago. This was a smaller release, but it included some major enhancements to FoodNoms' signature feature: the nutrition label scanner.

A New Algorithm

The previous label scanning algorithm was quite elegant and written in a very functional way: it boiled down to a series of pure array and string transformations. This ended up working good enough for most scenarios, but it had one fatal flaw.

Here's an example of a label that would not scan easily with FoodNoms prior to 2020.6:

See how the "Calories" text is on a separate baseline from the "130"? Previously, FoodNoms would process this as two lines: ["Calories"], and ["per serving", "130"]. Then it wouldn't ever pick up the value for calories, which is the most important fact to detect!

Prior to 2020.6, the nutrition label scanner algorithm assumed that labels and values would exist on the same baseline and have similar heights. Turns out this isn't always the case.

These types of labels are quite common here in the USA. Going forward, it's actually becoming the standard. The FDA recently published new guidelines for nutrition labels in the United States, and they dictate that the calories amount should be displayed in a larger, bolder font like this:

Even thought the "Calories" text and the value are on the same baseline, FoodNoms would still sometimes get confused with labels like this.

I now take more advantage of the geometry data provided by the Vision framework. The new algorithm detects the nutrients label keywords, then scans horizontally to find candidates for the value.

Unfortunately, this change necessitated that I discard my previous solution and start from scratch. It ended up taking longer than I had originally planned, but I think the end result was totally worth it.

A New Progress Indicator

The most visible change amount all of these improvements is a new visual progress indicator. It fills up based on the percentage of the nutrients scanned and turns green when it is confident that it's detected everything that can be detected.

Here's my announcement tweet which includes a video capture of the progress indicator in action:

This started as a fun experiment and ended up being a huge UX win. One of the biggest problems with the nutrition label scanner has been manually validating that what you've scanned is complete and correct. Now, the new progress indicator lets you keep your focus on the camera and feel more confident that you've captured everything.

Technical side note: this progress bar is built entirely in SwiftUI. It uses a subtle gradient animation, which ended up being a challenge to build. I ended up getting it to work by repurposing some of my animation code I wrote for the first onboarding screen.

Other Improvements

I've also added support for vitamins and minerals you'd find on some labels: Vitamin A, Vitamin D, Vitamin E, niacin, calcium, iron, potassium, phosphorus, magnesium, zinc, copper, and manganese.

Currently, FoodNoms will only pick up values with absolute measurements, not percentages of recommended daily intake. Potential future versions may introduce support for percentages.

I've also added improved language support. Users with Spanish and French labels should benefit the most. In fact, I've found that Mexican and Canadian labels now perform on-par with standard USA labels!

A New "Help & Tips" Screen

This release also introduces a new FAQ-style screen in the app. There are a lot of useful tips and bits of info in this screen. If you regularly use this the nutrition label scanner, I highly recommend you check it out. You can find via the "…" button when scanning.

Functional Testing

As part of the algorithm rewrite, I started a functional test suite with real-life labels. This will enable me to make more changes to the algorithm with confidence.

Goals for this Release

This release is less about moving some metrics, and more about preparing for crowdsourcing food data. These improvements should make it easier and faster to scan nutrition labels, which will be the source for much of the data that will get crowdsourced.

As always, I'll be listening to customer feedback. If you have any complaints or questions about these changes, please let me know.