{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", " \n", " \n", " \n", " \n", "
\n", " \n", " \n", " Try in Google Colab\n", " \n", " \n", " \n", " \n", " Share via nbviewer\n", " \n", " \n", " \n", " \n", " View on GitHub\n", " \n", " \n", " \n", " \n", " Download notebook\n", " \n", "
\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Getting Started with FiftyOne\n", "\n", "Welcome to this walkthrough of [FiftyOne](https://voxel51.com/fiftyone), a powerful package for dataset curation, analysis, and visualization.\n", "\n", "We designed FiftyOne to help CV/ML engineers curate better datasets and train better models. Based on our own experience and other CV teams we've had the pleasure of collaborating with, we found that the key to better data and better models is to _get closer to our datasets_ and understand our data and models at a deeper level.\n", "\n", "However, without the right tools, this process can be time-consuming, requiring endless custom scripting and data wrangling that doesn't translate between tasks or scale over time.\n", "\n", "We built FiftyOne to solve these challenges, enabling you to:\n", "\n", "- Rapidly experiment with your datasets\n", "- Visualize your datasets and their annotations/model predictions\n", "- Understand the strengths and weaknesses of your models" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Overview\n", "\n", "In this walkthrough, we'll cover:\n", "\n", "- Installing FiftyOne\n", "- Loading a dataset\n", "- Adding model predictions\n", "- Using the FiftyOne Brain to index your dataset by visual uniqueness\n", "- Using the App to explore the dataset\n", "\n", "FiftyOne has rich documentation [available online](https://voxel51.com/docs/fiftyone).\n", "\n", "There you'll find installation instructions, a user guide that covers the basic FiftyOne concepts, as well as short recipes and in-depth tutorials that cover the most common FiftyOne workflows in detail.\n", "\n", "The best part is that FiftyOne is open source! Check out the project [on GitHub](https://github.com/voxel51/fiftyone) where you can leave feedback, feature requests, bug reports, or even get involved and contribute to the library!\n", "\n", "You can also\n", "[join our Slack community](https://join.slack.com/t/fiftyone-users/shared_invite/zt-gtpmm76o-9AjvzNPBOzevBySKzt02gg) to chat 1-1 with us and the other FiftyOne users." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Installation\n", "\n", "Installing FiftyOne is easy via `pip`:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install fiftyone" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For this walkthrough, we'll also install PyTorch, so that we can use a pretrained model to add predictions to our dataset:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install torch torchvision" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "FiftyOne is designed to integrate naturally into your existing CV/ML workflows. It plays well with TensorFlow, PyTorch, and your other ML tools such as cloud storage, cloud compute, experiment tracking, and more." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Load a dataset\n", "\n", "Working with datasets in different formats can be a pain. That's why FiftyOne provides extensive support for loading datasets in many common formats with just a single line of code. The [user guide](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/index.html) provides detailed instructions on this.\n", "\n", "In this case, we'll load a public dataset from the [FiftyOne Dataset Zoo](https://voxel51.com/docs/fiftyone/user_guide/dataset_zoo/index.html):" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Split 'validation' already downloaded\n", "Loading 'coco-2017' split 'validation'\n", " 100% |███████████████| 5000/5000 [43.0s elapsed, 0s remaining, 111.5 samples/s] \n", "Dataset 'coco-2017-validation' created\n" ] } ], "source": [ "import fiftyone as fo\n", "import fiftyone.zoo as foz\n", "\n", "# Load COCO validation split from the zoo\n", "dataset = foz.load_zoo_dataset(\"coco-2017\", split=\"validation\")" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Name: coco-2017-validation\n", "Media type: image\n", "Num samples: 5000\n", "Persistent: False\n", "Info: {'classes': ['0', 'person', 'bicycle', ...]}\n", "Tags: ['validation']\n", "Sample fields:\n", " filepath: fiftyone.core.fields.StringField\n", " tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n", " metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)\n", " ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n" ] } ], "source": [ "print(dataset)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Datasets](https://voxel51.com/docs/fiftyone/user_guide/basics.html) are the core data structure in FiftyOne. A dataset is composed of one or sample objects, which can contain arbitrary fields, all of which can be dynamically created, modified, and deleted. Samples can store the path to the source data, metadata about it, one or more sets of labels (detections, classifications, etc.), and other data such as numeric fields, dictionaries, arrays, and more.\n", "\n", "FiftyOne uses a lightweight non-relational database to store all information about the dataset **except** the raw data (images, videos, etc), so you can easily scale to datasets of any size without worrying about RAM constraints on your machine." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Launch the App\n", "\n", "Data management and storage is only part of the story. When working with image/video datasets, you also need visualization tools that enable you to **see** the quality of your datasets and models.\n", "\n", "Let's [launch the FiftyOne App](https://voxel51.com/docs/fiftyone/user_guide/app.html) so we can visually explore the dataset we loaded above.\n", "\n", "Right away you will see that because we are in a notebook, an embedded instance of the App with our dataset loaded has been rendered in the cell's output. The [Session object](https://voxel51.com/docs/fiftyone/api/fiftyone.core.session.html?highlight=session#fiftyone.core.session.Session) created below is a bi-directional connection between your Python kernel and the FiftyOne App, as we'll see later." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Launch the App\n", "session = fo.launch_app(dataset)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Automatic screenshots as you work\n", "\n", "Notebooks are great for many reasons, one of which is the ability to share your work with others. FiftyOne is designed to help you write notebooks that capture your work on visual datasets, using a feature we call **automatic screenshotting**.\n", "\n", "Whenever you open a new App instance in a notebook cell, e.g., by updating your [Session](https://voxel51.com/docs/fiftyone/api/fiftyone.core.session.html?highlight=session#fiftyone.core.session.Session) object, any previous App instances will be automatically replaced with a static screenshot. In fact, that's what you're seeing below; screenshots of the Apps we opened when we created this notebook!\n", "\n", "The cell below issues a [session.show()](https://voxel51.com/docs/fiftyone/api/fiftyone.core.session.html#fiftyone.core.session.Session.show) command, which opens a new App instance in the cell's output. When you run the cell for yourself, notice that the App instance in the previous cell is automatically replaced with a screenshot of its current state. You can reactivate old App instances by hovering over them and clicking anywhere.\n", "\n", "After running the cell below, try double-clicking on an image in the grid to expand the sample." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Launch a new App instance\n", "session.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using the App\n", "\n", "With the App, you can visualize your samples and their fields either in image grid view, or by double-clicking an image to enter an expanded sample view, where you can study individual samples in more detail.\n", "\n", "The [view bar](https://voxel51.com/docs/fiftyone/user_guide/app.html#using-the-view-bar) allows you to search and filter your dataset to study specific samples or labels of interest.\n", "\n", "With FiftyOne, you can seemlessly transition between the App and Python.\n", "\n", "For example, create a search using the `Shuffle()` and `Limit()` stages in the view bar:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "session.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can access the current view back in your Python shell at any time:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dataset: coco-2017-validation\n", "Media type: image\n", "Num samples: 10\n", "Tags: ['validation']\n", "Sample fields:\n", " filepath: fiftyone.core.fields.StringField\n", " tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n", " metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)\n", " ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n", "View stages:\n", " 1. Shuffle(seed=51)\n", " 2. Limit(limit=10)\n" ] } ], "source": [ "# Access the current view in the App\n", "print(session.view)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can also select samples in the App and access the currently selected samples from Python:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "session.show()" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['5ff5dc5adec2f443622378bf', '5ff5dc51dec2f44362232ce9', '5ff5dc59dec2f44362236a22', '5ff5dc48dec2f4436222e71a']\n" ] } ], "source": [ "# The currently selected samples\n", "print(session.selected)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dataset: coco-2017-validation\n", "Media type: image\n", "Num samples: 4\n", "Tags: ['validation']\n", "Sample fields:\n", " filepath: fiftyone.core.fields.StringField\n", " tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n", " metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)\n", " ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n", "View stages:\n", " 1. Select(sample_ids=['5ff5dc5adec2f443622378bf', '5ff5dc51dec2f44362232ce9', '5ff5dc59dec2f44362236a22', ...])\n" ] } ], "source": [ "# Create a view containing the currently selected samples\n", "selected_view = dataset.select(session.selected)\n", "\n", "print(selected_view)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Manipulating your view in Python\n", "\n", "You can also manipulate the App view from Python! For example, you may want to construct a complex view from code and then open it in the App!" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dataset: coco-2017-validation\n", "Media type: image\n", "Num samples: 5000\n", "Tags: ['validation']\n", "Sample fields:\n", " filepath: fiftyone.core.fields.StringField\n", " tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n", " metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)\n", " ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n", "View stages:\n", " 1. FilterDetections(field='ground_truth', filter={'$eq': ['$$this.label', 'person']}, only_matches=False)\n", " 2. SortBy(field_or_expr={'$size': {'$ifNull': [...]}}, reverse=True)\n" ] } ], "source": [ "from fiftyone import ViewField as F\n", "\n", "# Hide all detections that do not have label `person`, and sort the samples to\n", "# show those with the most people\n", "person_view = (\n", " dataset\n", " .filter_labels(\"ground_truth\", F(\"label\") == \"person\")\n", " .sort_by(F(\"ground_truth.detections\").length(), reverse=True)\n", ")\n", "\n", "print(person_view)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Show the view in the App\n", "session.view = person_view" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Indexing images by uniqueness\n", "\n", "FiftyOne includes a `fiftyone.brain` package that provides a collection of algorithms to help you gain insight into your datasets and models. For more information, [check out the user guide](https://voxel51.com/docs/fiftyone/user_guide/brain.html).\n", "\n", "Let's use the `compute_uniqueness()` function to index the samples in our dataset according to their visual uniqueness:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loading uniqueness model...\n", "Preparing data...\n", "Generating embeddings...\n", " 100% |███████████████| 5000/5000 [1.8m elapsed, 0s remaining, 47.8 samples/s] \n", "Computing uniqueness...\n" ] } ], "source": [ "import fiftyone.brain as fob\n", "\n", "fob.compute_uniqueness(dataset)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Inspecting the dataset shows that a numeric `uniqueness` field has been added to each sample, which measures its visual uniqueness with respect to the other samples in the dataset:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Name: coco-2017-validation\n", "Media type: image\n", "Num samples: 5000\n", "Persistent: False\n", "Info: {'classes': ['0', 'person', 'bicycle', ...]}\n", "Tags: ['validation']\n", "Sample fields:\n", " filepath: fiftyone.core.fields.StringField\n", " tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n", " metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)\n", " ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n", " uniqueness: fiftyone.core.fields.FloatField\n" ] } ], "source": [ "print(dataset)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "print(dataset.select_fields(\"uniqueness\").first())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's visualize this information in the App by showing the most visually unique samples first:" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Explore most unique samples\n", "session.view = dataset.sort_by(\"uniqueness\", reverse=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sorting by **least unique** can help us identify near duplicate samples in our dataset. This can be useful in situations where you need to send a dataset for annotation and need to select a diverse set of images." ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Explore the least unique samples\n", "session.view = dataset.sort_by(\"uniqueness\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Adding model predictions\n", "\n", "Now let's add some model predictions to our dataset.\n", "\n", "First, let's select some samples to process:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dataset: coco-2017-validation\n", "Media type: image\n", "Num samples: 15\n", "Tags: ['validation']\n", "Sample fields:\n", " filepath: fiftyone.core.fields.StringField\n", " tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n", " metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)\n", " ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n", " uniqueness: fiftyone.core.fields.FloatField\n", "View stages:\n", " 1. SortBy(field_or_expr='uniqueness', reverse=False)\n", " 2. Limit(limit=15)\n" ] } ], "source": [ "# Select the 15 least unique samples to process\n", "predictions_view = dataset.sort_by(\"uniqueness\").limit(15)\n", "\n", "print(predictions_view)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can access the source image at inference time via the `filepath` attribute of the samples in the view:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/Users/Brian/fiftyone/coco-2017/validation/data/001147.jpg\n", "/Users/Brian/fiftyone/coco-2017/validation/data/003969.jpg\n", "/Users/Brian/fiftyone/coco-2017/validation/data/002645.jpg\n", "/Users/Brian/fiftyone/coco-2017/validation/data/000035.jpg\n", "/Users/Brian/fiftyone/coco-2017/validation/data/004965.jpg\n", "/Users/Brian/fiftyone/coco-2017/validation/data/004510.jpg\n", "/Users/Brian/fiftyone/coco-2017/validation/data/000641.jpg\n", "/Users/Brian/fiftyone/coco-2017/validation/data/004329.jpg\n", "/Users/Brian/fiftyone/coco-2017/validation/data/000868.jpg\n", "/Users/Brian/fiftyone/coco-2017/validation/data/001348.jpg\n", "/Users/Brian/fiftyone/coco-2017/validation/data/004341.jpg\n", "/Users/Brian/fiftyone/coco-2017/validation/data/001614.jpg\n", "/Users/Brian/fiftyone/coco-2017/validation/data/004546.jpg\n", "/Users/Brian/fiftyone/coco-2017/validation/data/000648.jpg\n", "/Users/Brian/fiftyone/coco-2017/validation/data/001154.jpg\n" ] } ], "source": [ "for sample in predictions_view:\n", " print(sample.filepath)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "FiftyOne integrates naturally with any ML framework. You simply load and perform inference with your model using your existing code, and then simply add the predictions to the corresponding samples in the dataset using FiftyOne's [Label type](https://voxel51.com/docs/fiftyone/user_guide/using_datasets.html#labels) corresponding to your task (classification, detection, etc).\n", "\n", "In this case, we'll load a pretrained Faster R-CNN model made available by PyTorch and store the predictions in a `faster_rcnn` field of our samples:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 100% |███████████████████| 15/15 [1.7m elapsed, 0s remaining, 0.1 samples/s] \n" ] } ], "source": [ "from PIL import Image\n", "import torch\n", "import torchvision\n", "from torchvision.transforms import functional as func\n", "\n", "\n", "def run_inference(sample, model, device, classes):\n", " \"\"\"Performs inference on the image at `sample.filepath` and returns the\n", " predictions in a `fiftyone.core.labels.Detections` instance.\n", " \"\"\"\n", " # Load iamge\n", " img = Image.open(sample.filepath)\n", " img_tensor = func.to_tensor(img).to(device)\n", " _, height, width = img_tensor.shape\n", "\n", " # Perform inference\n", " predictions = model([img_tensor])[0]\n", " labels = predictions[\"labels\"].cpu().detach().numpy()\n", " scores = predictions[\"scores\"].cpu().detach().numpy()\n", " boxes = predictions[\"boxes\"].cpu().detach().numpy()\n", "\n", " # Convert detections to FiftyOne format\n", " detections = []\n", " for label, score, box in zip(labels, scores, boxes):\n", " # Convert to [top-left-x, top-left-y, width, height] format with\n", " # relative coordinates in [0, 1] x [0, 1]\n", " x1, y1, x2, y2 = box\n", " rel_box = [\n", " x1 / width, y1 / height, (x2 - x1) / width, (y2 - y1) / height\n", " ]\n", "\n", " detections.append(\n", " fo.Detection(\n", " label=classes[label],\n", " bounding_box=rel_box,\n", " confidence=score,\n", " )\n", " )\n", "\n", " return fo.Detections(detections=detections)\n", "\n", "\n", "# Load pre-trained model\n", "model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)\n", "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n", "model.to(device)\n", "_ = model.eval()\n", "\n", "# Add model predictions to dataset\n", "classes = dataset.info[\"classes\"]\n", "with fo.ProgressBar() as pb:\n", " for sample in pb(predictions_view):\n", " sample[\"faster_rcnn\"] = run_inference(sample, model, device, classes)\n", " sample.save()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can verify that predictions were added to our dataset by checking for the `faster_rcnn` field in the view's schema:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dataset: coco-2017-validation\n", "Media type: image\n", "Num samples: 15\n", "Tags: ['validation']\n", "Sample fields:\n", " filepath: fiftyone.core.fields.StringField\n", " tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n", " metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)\n", " ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n", " uniqueness: fiftyone.core.fields.FloatField\n", " faster_rcnn: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n", "View stages:\n", " 1. SortBy(field_or_expr='uniqueness', reverse=False)\n", " 2. Limit(limit=15)\n" ] } ], "source": [ "# The `faster_rcnn` field of the sample contains the model predictions\n", "print(predictions_view)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can recover the samples with predictions at any time by using the [Exists stage](https://voxel51.com/docs/fiftyone/api/fiftyone.core.stages.html?highlight=exists#fiftyone.core.stages.Exists)\n", "in the view bar in the App to select samples with a value in their `faster_rcnn` field:\n", "\n", "As usual, we can create the same view via Python code:" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Only show samples with a value in their `faster_rcnn` field\n", "session.view = dataset.exists(\"faster_rcnn\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Evaluating models\n", "\n", "With the FiftyOne App, you can easily visualize the predictions and qualitatively compare them with the ground truth:" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "session.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "However, FiftyOne also provides a powerful [evaluation framework](https://voxel51.com/docs/fiftyone/user_guide/evaluation.html) that you can use to analyze your models.\n", "\n", "Although the evaluation framework provides support for common quantiative evaluation measures such as confusion matrices, mAP, and PR curves, aggregate metrics alone don’t give the full picture of a model's performance. In practice, the limiting factor of a model is often data quality issues that you need to **see** to address. FiftyOne is designed to make it easy to do just that.\n", "\n", "For example, the snippet below uses the [evaluate_detections()](https://voxel51.com/docs/fiftyone/api/fiftyone.core.collections.html?highlight=evaluate_detections#fiftyone.core.collections.SampleCollection.evaluate_detections) method available on all datasets and views to perform COCO-style evaluation of the predictions, including per-sample true positives (TP), false positives (FP), and false negatives (FN):" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Evaluating detections...\n", " 100% |███████████████████| 15/15 [499.0ms elapsed, 0s remaining, 30.1 samples/s] \n" ] } ], "source": [ "# Retrieve the samples with predictions\n", "predictions_view = dataset.exists(\"faster_rcnn\")\n", "\n", "# Evaluate the predictions in the `predictions` field\n", "# w.r.t. the ground truth labels in the `ground_truth` field\n", "results = predictions_view.evaluate_detections(\n", " \"faster_rcnn\",\n", " gt_field=\"ground_truth\",\n", " eval_key=\"eval\",\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Refreshing the view in the App, we see that additional fields have been added to each sample to tabulate the evaluation metrics:" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Update the view in the App\n", "session.view = predictions_view" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Both visually and quantitatively, we see that the model is generating too many false positive predictions:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "session.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's create a view into the dataset with FiftyOne that contains only predictions with a confidence score of at least 0.8:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dataset: coco-2017-validation\n", "Media type: image\n", "Num samples: 15\n", "Tags: ['validation']\n", "Sample fields:\n", " filepath: fiftyone.core.fields.StringField\n", " tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n", " metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)\n", " ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n", " uniqueness: fiftyone.core.fields.FloatField\n", " faster_rcnn: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n", " eval_tp: fiftyone.core.fields.IntField\n", " eval_fp: fiftyone.core.fields.IntField\n", " eval_fn: fiftyone.core.fields.IntField\n", "View stages:\n", " 1. Exists(field='faster_rcnn', bool=True)\n", " 2. FilterLabels(field='faster_rcnn', filter={'$gt': ['$$this.confidence', 0.8]}, only_matches=True)\n" ] } ], "source": [ "from fiftyone import ViewField as F\n", "\n", "# Only keep predictions whose confidence is at least 0.8\n", "high_conf_predictions_view = predictions_view.filter_labels(\n", " \"faster_rcnn\", F(\"confidence\") > 0.8\n", ")\n", "\n", "print(high_conf_predictions_view)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Don't worry, the lower confidence predictions have not been deleted! They are just being excluded from the view.\n", "\n", "Let's re-run the evaluation method to update the metrics for the high-confidence-only predictions:" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Evaluating detections...\n", " 100% |███████████████████| 15/15 [378.3ms elapsed, 0s remaining, 39.6 samples/s] \n" ] } ], "source": [ "# Re-evaluate\n", "results = high_conf_predictions_view.evaluate_detections(\n", " \"faster_rcnn\",\n", " gt_field=\"ground_truth\",\n", " eval_key=\"eval\",\n", ")" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Reload view in App\n", "session.view = high_conf_predictions_view" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we can now see, both visually and quantitatively, the false positive rate of\n", "the model has been decreased!" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "session.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Sharing notebooks\n", "\n", "To make a notebook ready for sharing, you'll need to screenshot the currently active App by calling [Session.freeze()](https://voxel51.com/docs/fiftyone/api/fiftyone.core.session.html#fiftyone.core.session.Session.freeze):" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": [ "session.freeze()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now when you share this notebook, publish it online, etc., all of your App outputs will be available for readers to see when they first open the notebook." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Next steps\n", "\n", "This walkthrough provided just a glimpse into the possibilities of using FiftyOne. If you'd like to learn more, check out these [tutorials](https://voxel51.com/docs/fiftyone/tutorials/index.html) and [recipes](https://voxel51.com/docs/fiftyone/recipes/index.html).\n", "\n", "And did we mention that FiftyOne is open source? Check out the project [on GitHub](https://github.com/voxel51/fiftyone) and [leave an issue](https://github.com/voxel51/fiftyone/issues/new/choose) if you think something is missing.\n", "\n", "Thanks for tuning in!" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.13" } }, "nbformat": 4, "nbformat_minor": 4 }