Continuous testing and model selection with Lakera and Voxel51

This article was originally posted on our company website. Lakera’s developer platform enables ML teams to ship fail-safe computer vision models.

We are excited to announce the release of our first integration with FiftyOne by Voxel51.

Lakera is a leader in evaluating and testing computer vision models and data–to ensure safe, fair, and robust AI. FiftyOne by Voxel51 is a fantastic tool to visualize your data and curating high-quality datasets.

Now we’ve made it possible to benefit from FifyOne’s powerful visualization features to dig into the insights generated by Lakera’s MLTest. This is particularly convenient if you are already a FiftyOne user, as everything will fit right into your workflow.

What are your benefits?

Easier and more thorough model comparisons.

Every ML engineer has to constantly compare and select the best models for deployment. Either to compare a release candidate to the current production version, to test for regressions when quantizing models, or while doing a hyperparameter search. MLTest provides in-depth functionality to evaluate and compare models that go beyond standard ML metrics and include robustness, fairness, generalizability, and more. It also gives a neat diff-like comparison of one model against another, enabling you to make your decisions quickly.

Model comparison for two computer vision models. Showing a diff similar to code review tools.

MLTests model comparison feature to help select the right model for production.

Our new integration with FiftyOne now enables you to go deeper–for those times when you need to compare models on a prediction-by-prediction basis. To that end, we export all model predictions for individual images to easily load and inspect them with FiftyOne. This makes it easy to quickly compare the predictions for a specific class and filter by false positives/negatives.

Images and predictions from MLTest imported to FiftyOne for closer inspection.

Deep failure analysis/debugging.

MLTest automatically finds and clusters images on which your models don’t perform as expected. This happens by analyzing your model’s performance and grouping the images based on image embeddings, metadata you provide, and metadata MLTest automatically adds to your images. You can see an example below:

Clusters of images on which the ML model does not perform well. Here showing cluster where precision is lower than expected.

MLTest automatically finds clusters of images on which a model doesn’t perform well.

Getting started with Lakera and FiftyOne is easy. Head over to our docs on the integration. It includes an executable example as a Jupyter Notebook to get you up and running as quickly as possible. We’ve also previously reviewed the onboarding experience and features of FiftyOne, here.

You can get access to MLTest here. If you have any questions or would like to continue the discussion around other ways to integrate FiftyOne, please reach out to Santiago at santiago@lakera.ai.