How to Run a Machine Learning Project?

Today, Machine Learning has the ability to solve complex business tasks with the help of intelligent algorithms. Many businesses are looking for ways to make use of the data to make more profit or gain competitive advantages. 

As developers of ML solutions since 2016, we’ve had to deal with a lot of inquiries for ML models. We’ve had tоns of meetings with potential clients discussing their ideas and brainstorming numerous hypotheses. We feel like it’s time to share our experience. Controversially, this experience shows that not every company seeking to implement machine learning technologies actually needs them. It may sound provocative, but let us share our findings. After reading this guide, you will actually understand if you should reserve your time and resources on finding the right Data Science/ML team.

Who is it for?

If you are thinking of using ML algorithms in your business, you are already aware of the possibilities that they bring. You understand that there exists a class of smart algorithms capable of computing vast quantities of data to perform complex tasks. You are also very likely to have already faced custom software development and hiring dedicated teams, so you may think that Machine Learning development goes the same way. And this is actually the first pitfall you may stumble upon.

What you should be prepared for

The first thing you have to understand is Machine Learning projects are very risky. It is never possible to estimate the work hours required for the project, simply because there is no clear understanding of which way the project will go in the very beginning. After elaborating an idea, we will set some paths and we won’t understand which one we should take until we’ve tried each of them. We will see the right direction after we make the first estimate, which is already a mini-research requiring a certain number of work hours.

Another thing to realize is that in the case of ML, it’s not the end result you should be ready to invest in, but the R&D process. As opposed to traditional software development, having invested one-fifth of the total budget, you will not receive one-fifth of the product that could already bring you some profit. Instead, you will probably get proof if your hypothesis is right or wrong together with understanding, where to go next.

Usually, the first R&D step will last from 1 week up to 4 months. This first step may well result in the Client realizing that the data have not been collected properly and there’s absolutely no way we can build an effective model based on this data. Sadly, one of our clients had to pay around $30,000 just to get this insight, there were 4 FTEs working on the research for 4 months. Having paid this amount they received recommendations on how they should collect data and for how long, in order to build a good model on its basis. So, understandably, not every company is ready to invest this much in a 15-page report. It is only the ones who are really seeing the growth opportunities and the value that ML technologies can bring them.

Why do you need ML?

Machine learning is used in different sectors: starting from retail and finance, through health care, to education and charity. Each of them adjusts it to its needs. In healthcare, ML initiative is driven by the lack of qualified personnel and the desire to exclude human error in making a diagnosis. The education sphere is thrilled by the idea of producing personalized tracks to better serve students who have different goals and needs. Industrial companies may realize they have accumulated large massives of data that can help them optimize processes, reduce costs, and increase operational safety.

Anyway, most companies want to invest in ML to gain competitive advantages in the market. The leading players will probably want to get into complex R&D on purpose, simply because they know their competition does not have the resources required to obtain similar features. This will win them some time to come up with more competitive advantages. In fact, according to a research by Microsoft, companies using AI are outperforming by 5% those which have no AI strategy.  If you are determined to become one of those, you should start by answering these simple questions:

  1. Do you really need this feature? Make sure you are not following the hype. Can your business function without ML features? Maybe there could be an automation solution that will bring 80% of the effect for 20% of the budget that you’d spend on ML? If there is any chance that your feature can be made using traditional software engineering, you should go for it. You should make sure that the problem you are trying to solve is important enough.
  2. Do you have the data? When speaking about ML algorithms, we don’t always mean we need lots of data. But usually, we do. And sometimes we don’t need a lot, but the data must be appropriate, and it should exist somewhere in the world. Otherwise, there is no way we can help you. We have expertise in Data Engineering, but this is really challenging task that requires tons of resources and may not be the most efficient.
  3. Do you have expertise? You should clearly see which intellectual functions you are trying to replace with AI. If you are not an expert in this certain field, neither are we. Subject matter experts (SMEs) are the true authorities who lay down the foundational knowledge upon which your model is built. And they must be on board with what you are trying to achieve. Otherwise, you’ll face a lot of resentment. For example, not all math teachers are ready to cooperate in building an adaptive online math tutor, feeling threatened that AI will take their jobs away.
  4. Do you have the finances? The Client should realize that we are working with an iceberg. The question is, which part of it we see: is it 1/10 or 1/100? After the first R&D stage it may turn out that the data is excellent, but you could end up spending 10 times the amount than you have planned. The good thing, it will be probably worth it.

So these are the things we recommend you keep in mind before starting an ML project. If you feel that some of the parts are missing, do not go there. Consider traditional software engineering instead.

If you feel it might be worth risking, this means you have all the components in place, and each of them is your huge competitive advantage. Other companies in your industry are likely to lack at least one of them. They might have the data, but not the expertise. They might be experts in their field, but not be ready to invest resources. Reports say that the majority of companies are actively working on a roadmap for handling data (68 percent), yet only 11 percent of these companies have completed this task. If you have the life-changing idea, the data, SME, and a budget, you are likely to fall into those 11 percent. If you are not sure, you can always reach out for help and ENBISYS experts will figure out the most suitable solution for your needs. 

Dmitry Bubnov, CEO ENBISYS