The ability to track machine learning experiments in real time has become essential in AI. DVC makes it possible for you to visualize the progress of the experiment, share it with team members, and find out early if the experiment isn't progressing as planned. You can then stop the experiment if needed. Collaboration, time, and resource savings are yours with Iterative Studio.
Computer vision is a complex field requiring much experimentation and trial and error to achieve optimal results. However, managing and tracking the progress of these experiments has not been easy. You can't see it once you've sent it to the server for training. Keeping an eye on its progress over (often) days makes it possible to miss something. This makes it difficult to effectively manage your time and reduce unnecessary resource use. Moreover, a team working on the same project needs to be able to easily share their results with colleagues. This can be challenging with existing (or non-existent) tooling.
That's where DVCLive and Iterative Studio come in. These tools offer live experiment tracking and efficient result sharing, making it easy to optimize your experimentation process and streamline the workflow with your team.
See experiment results in real-time in Iterative Studio
DVCLive is a Python library connected to DVC that provides a real-time experiment logger that allows machine learning engineers to track the metrics and parameters of their experiments. It is beneficial for long-running experiments, which can take hours or even days to complete.
Iterative Studio is a SaaS platform that displays logged experiments with their metrics, parameters, and plots all tied together and tracked using DVC and Git under the hood. It allows for rich, visual, real-time tracking and sharing of the results, making it easy to collaborate with others and be production-ready efficiently.
Real-time, nested experiment tracking in Iterative Studio
In this computer vision project (see repo here), we embark on an exciting journey to uncover swimming pools, often obscured from street-level views, right in the middle of our neighborhoods and cities. Using ResNet-18 and Fast.ai, we will be able to accurately identify and segment pools from satellite images.
Images and ground truth segmentation of BH-Pools Dataset (Source link)
It's worth noting that the experiment in this example is beyond a toy project by design. It may take around one hour to run on an ordinary laptop, and the time may vary depending on the specific configuration and settings. However, you can use a GPU to speed up the process.
We will use a modified version of the BH-Pools dataset, which consists of high-resolution 4K images of various neighborhoods in the city of Belo Horizonte, Brazil. These images were captured through Google Earth Pro and come pre-annotated with swimming pools and water tanks. For this project, we will focus on just the swimming pools.
We have made the dataset more manageable with some pre-processing to crop the images into smaller tiles of 1024x1024 pixels.
When using DVCLive in Iterative Studio, we will be able to see the progress of our experiments. Let’s get started!
Follow the initial setup instructions in the
README. Next, we
need to run
dvc pull in our root directory to fetch the dataset from our
remote. This command retrieves the data from the remote storage and makes it
available locally for our experiments. Once the download is complete, we will
create a data loader using the label function with
fastai library. This data loader allows us to easily load and
preprocess the images (e.g. resizing the images to the desired resolution). You
can dig deeper into the code
Sample of Belo Horizonte Pools Dataset from
After creating the data loader and resizing the images, we train a ResNet-18 model with unet_learner with varying hyperparameters and utilizing the DVCLiveCallback. The DVCLiveCallback is a built-in logger provided by DVCLive that allows us to track the intermediate results of the training process, such as the loss and accuracy of the model, in real-time. By logging these metrics, we can easily monitor the progress of our model and make adjustments as needed to optimize the training process and improve the performance of the model.
learn = unet_learner( data_loader, arch=getattr(models, params.train.arch), metrics=DiceMulti ) learn.fine_tune( **params.train.fine_tune_args, cbs=[DVCLiveCallback(dir="results/train", report="md", dvcyaml=False)], )
Additionally, we can also use Studio to analyze and visualize the results of our experiments, making it easy to share and collaborate with others. By providing the STUDIO_TOKEN, DVCLive will automatically post the results of the experiment to Studio. To do this, first, let’s obtain an individual token from the user profile page in Studio.
Generating Studio Access Token in the Iterative Studio Profile page (Source link)
By providing this token as an environment variable, we can access the results of our experiments in an Iterative Studio project. The project lets you compare them with previous experiments, helps you find insights to improve our model and share it with others.
Compare with previous experiments in Iterative Studio (Source link)
To export the token run the command below with the token obtained from your Studio profile:
Running an experiment locally using DVC will now automatically live-update the Studio project(s) associated with your git remote (the one named "origin")
You may want to change the parameters and run the experiment again.
dvc exp run -S train.fine_tune_args.epochs=16 -S train.img_size=512
Real-time Experiment tracking in Iterative Studio (Source link)
As you can see, the change to the epochs and image size brought improvement to the metrics.
It's safe to say that if you provide the model with a satellite image of any neighborhood, it will pretty accurately identify all swimming pools in that image! And by using DVCLive and Studio, we were able to track and efficiently control the model training process, without squandering expensive training resources on unfruitful training runs.
Our work has produced a model which is able to accurately identify and segment swimming pools from satellite images! With the help of DVCLive and Iterative Studio, we've been able to visualize results in real-time to make resource-saving decisions. And finally, this work is readily visible for the entire team to review!
We’d like to express our gratitude to the creators of the incredible BH-Pools dataset, without which there would have been less fun and less impressive results!
Do you have any use case questions or need support? Join us in Discord!
Head to the DVC Forum to discuss your ideas and best practices.