We are opening up our Model Playground, the complete no-code solution for all your model development needs
Learn more

Blog

We need "agile" in machine learning

It’s time for a non-linear, iterative approach that builds highly performant vision AI applications.

By Tobias Schaffrath Rosario
Published on November 23rd, 2020

The challenges facing vision AI projects due to escalating costs and failure to deliver to production are well documented by now. This post outlines a methodology that addresses 90% of the cited challenges - namely data preparation and solution design.

Current machine learning (ML) teams’ approaches to developing deep learning models are primarily linear and resemble old-fashioned waterfall software development rather than agile methodologies. As a result, it takes months until a first testable model is ready. This first model’s performance is subject to blind assumptions made at the beginning of the process rather than insights gained from ongoing testing.

Introducing Hasty’s Next Generation Annotation

Here at Hasty, we are a team of ML practitioners who ran into these problems over and over. So we created a next-gen annotation tool that enables an agile approach so you don’t have to deal with the problems anymore. Our users achieve higher levels of automation, iterative workflows, and exponential speed due to us training the models while you’re labeling.

Blogpost Agile Graph

ML engineering today is missing rapid feedback-loops.

Are you an ML engineer? Have you ever spent weeks on developing an application? When you finally deployed, you noticed that the models don’t perform like expected? If this didn't happen to you yet, you’re the exception (or a user of Hasty).

Generally speaking, to train deep learning models, ML engineers have to go through the following phases:

  • Design phase: define a problem and choose an annotation strategy and a model architecture
  • Data preparation: collect and label training data
  • Building the model: build the model architecture and train the model
  • Deployment: preperation, optimization, and finally, the actual deployment
  • Evaluation: check the model's performance

Many ML engineers follow a linear approach and view all these tasks as separate, sequential tasks. Once the problem is scoped out, a significant amount of time is spent preparing the data or outsourcing this. It takes weeks, if not months, to finish preparing the data.

Only then, are the first models built and real insights gained about how different architectures perform. Often, the performance falls behind expectations – as is typical for early versions. One common reason is that the annotation strategy wasn't the optimal choice, e.g., instance segmentation rather than bounding boxes should have been used. This is valuable feedback but comes late in the process when a lot of money and time has already been spent. Sometimes so much so that the project gets killed.

you will bounce between the stages of development as you refine your approach

Reality dictates that ML teams will iterate, especially when building something novel, and this should be expected and built into the process – you will bounce between the stages of development as you refine your approach. The shortcoming with a linear approach is that the lateness of insights derail development rather than improving it. This is why we believe feedback-loops and agile methodologies are critical for success.

Bringing the data-flywheel to computer vision.

Data Flywheel for Blog

Here at Hasty, we view the different phases as neither separate nor sequential. Rather, we train the model while you label. This allows us to build insights on the relationship between the training data and the resulting model and bring that closer together by giving the engineer actionable feedback.

The holy grail for vision AI is a virtuous cycle of feedback where ML engineering teams create a data-flywheel – a self-reinforcing loop. This is where neural-networks constantly improve with each new case they see and therefore are able to understand more and handle more cases.

With Hasty, the model is improved with every tiniest task during development and every single action taken by the person annotating. Already during the initial data-labeling, you get feedback on your annotation strategy, establishing the first feedback-loop. Additionally, testing different architectures and deployment is made easy and intuitive, so teams can prototype and iterate quickly.

Once a model is deployed to the final use-case, real-world, relevant feedback by end-users is fed back to close the loop. Images, on which the models performed poorly, can be sent back into the annotation-platform for re-training. By this, the performance of the application can be improved continuously – even after deployment.

Data preparation should be fast, iterative, and only outsourced once the approach is confirmed.

On average, ML engineers spend 80% of our time on data preparation, which is why we started with focussing on automating labeling with a bespoke model.

For an agile workflow to work, data preparation should be fast so you can rapidly test different annotation strategies.

Furthermore, you should already be able to get feedback while annotating and re-adjust your strategy after just a few images – not only when the dataset is completed. This is only possible if there is a model trained during the labeling process.

Finally, as an ML engineer, you benefit massively from doing the initial labeling yourself, when the model is trained as part of the process, as you gain invaluable insight on model performance and overall solution design. For that, tools that take the hassle out of annotation are needed.

Hasty brings automation and agile methodology

Hasty automates 70% of the data preparation work, making you fast and empowering you to annotate the first data yourself without lengthy configurations, set-up or instead of outsourcing this to BPOs (we aren’t hating on BPOs, but feel they are most effective at the scale stage of a project).

The critical part for your project’s success is: you get instant feedback with every single annotation. By this, you can test data quality in real-time as we take you from images to your first model in a single day. (Remember what we said above? This usually takes weeks.)

We have AI-Assistants that observe while you annotate and make suggestions for labels after a few images only. By correcting these suggestions, you’ll train the models behind the assistants and get instant feedback about how well your annotation strategy is working. This gives the neural network a learning curve as it learns on the project while you label.

Be agile

Further, we built models that identify errors while you work, so you don’t have to go through your data manually to find errors. This allows you to spend your time fixing errors instead of looking for them and helps you to build confidence in your data quickly.

Importantly, you can even export all models that are powering our assistants directly to your applications. If you want to understand how Hasty works in detail and how you can start developing computer vision applications in an agile manner, go here.

There’s much more.

Hasty’s starting point is our next-gen annotation tool, but we are continually working on new features and solutions for the entire data-flywheel. For example, we already provide API-support, so you can upload images into the annotation-tool for re-training.

The next big feature to be released, our Model Lab, will allow you to deploy models and experiment with different architectures in-app – getting you to flywheel status even faster.

Also, we just started a community for ML engineers in computer vision. We don’t believe in silver bullets and many challenges still remain in vision AI – our community is there for skilled and novice ML practitioners to ask questions, share their experience and learn from each other.

We invite you to join us today and benefit from a community of people like you, building the tech and tools you need today. Join our community now.