23.11.2020 — Tobias Schaffrath Rosario
We need "Agile" in machine learning
It’s time for a non-linear, iterative approach that builds highly performant vision AI applications.
It’s time for a non-linear, iterative approach that builds highly performant vision AI applications.
The challenges facing vision AI projects due to escalating costs and failure to deliver to production are well documented by now. This post outlines a methodology that addresses 90% of the cited challenges - namely data preparation and solution design.
Current machine learning (ML) teams’ approaches to developing deep learning models are primarily linear and resemble old-fashioned waterfall software development rather than agile methodologies. As a result, it takes months until a first testable model is ready. This first model’s performance is subject to blind assumptions made at the beginning of the process rather than insights gained from ongoing testing.
Here at Hasty, we are a team of ML practitioners who ran into these problems over and over. So we created a next-gen annotation tool that enables an agile approach so you don’t have to deal with the problems anymore. Our users achieve higher levels of automation, iterative workflows, and exponential speed due to us training the models while you’re labeling.
Are you an ML engineer? Have you ever spent weeks on developing an application? When you finally deployed, you noticed that the models don’t perform like expected? If this didn't happen to you yet, you’re the exception (or a user of Hasty).
Generally speaking, to train deep learning models, ML engineers have to go through the following phases:
Many ML engineers follow a linear approach and view all these tasks as separate, sequential tasks. Once the problem is scoped out, a significant amount of time is spent preparing the data or outsourcing this. It takes weeks, if not months, to finish preparing the data.
Only then, are the first models built and real insights gained about how different architectures perform. Often, the performance falls behind expectations – as is typical for early versions. One common reason is that the annotation strategy wasn't the optimal choice, e.g., instance segmentation rather than bounding boxes should have been used. This is valuable feedback but comes late in the process when a lot of money and time has already been spent. Sometimes so much so that the project gets killed.
you will bounce between the stages of development as you refine your approach
Reality dictates that ML teams will iterate, especially when building something novel, and this should be expected and built into the process – you will bounce between the stages of development as you refine your approach. The shortcoming with a linear approach is that the lateness of insights derail development rather than improving it. This is why we believe feedback-loops and agile methodologies are critical for success.
Here at Hasty, we view the different phases as neither separate nor sequential. Rather, we train the model while you label. This allows us to build insights on the relationship between the training data and the resulting model and bring that closer together by giving the engineer actionable feedback.
The holy grail for vision AI is a virtuous cycle of feedback where ML engineering teams create a data-flywheel – a self-reinforcing loop. This is where neural-networks constantly improve with each new case they see and therefore are able to understand more and handle more cases.
With Hasty, the model is improved with every tiniest task during development and every single action taken by the person annotating. Already during the initial data-labeling, you get feedback on your annotation strategy, establishing the first feedback-loop. Additionally, testing different architectures and deployment is made easy and intuitive, so teams can prototype and iterate quickly.
Once a model is deployed to the final use-case, real-world, relevant feedback by end-users is fed back to close the loop. Images, on which the models performed poorly, can be sent back into the annotation-platform for re-training. By this, the performance of the application can be improved continuously – even after deployment.
On average, ML engineers spend 80% of our time on data preparation, which is why we started with focussing on automating labeling with a bespoke model.
For an agile workflow to work, data preparation should be fast so you can rapidly test different annotation strategies.
Furthermore, you should already be able to get feedback while annotating and re-adjust your strategy after just a few images – not only when the dataset is completed. This is only possible if there is a model trained during the labeling process.
Finally, as an ML engineer, you benefit massively from doing the initial labeling yourself, when the model is trained as part of the process, as you gain invaluable insight on model performance and overall solution design. For that, tools that take the hassle out of annotation are needed.
Hasty automates 70% of the data preparation work, making you fast and empowering you to annotate the first data yourself without lengthy configurations, set-up or instead of outsourcing this to BPOs (we aren’t hating on BPOs, but feel they are most effective at the scale stage of a project).
The critical part for your project’s success is: you get instant feedback with every single annotation. By this, you can test data quality in real-time as we take you from images to your first model in a single day. (Remember what we said above? This usually takes weeks.)
We have AI-Assistants that observe while you annotate and make suggestions for labels after a few images only. By correcting these suggestions, you’ll train the models behind the assistants and get instant feedback about how well your annotation strategy is working. This gives the neural network a learning curve as it learns on the project while you label.
Further, we built models that identify errors while you work, so you don’t have to go through your data manually to find errors. This allows you to spend your time fixing errors instead of looking for them and helps you to build confidence in your data quickly.
Importantly, you can even export all models that are powering our assistants directly to your applications. If you want to understand how Hasty works in detail and how you can start developing computer vision applications in an agile manner, go here.
Only 13% of vision AI projects make it to production. With Hasty, we boost that number to 100%.
Our comprehensive vision AI platform is the only one you need to go from raw data to a production-ready model. We can help you with:
All the data and models you create always belong to you and can be exported and used outside of Hasty at any given time entirely for free.
You can try Hasty by signing up for free here. If you are looking for additional services like help with ML engineering, we also offer that. Check out our service offerings here to learn more about how we can help.
80% of vision AI teams don’t make it to production because of bad or insufficient data. Hasty solves removes that risk.
