Some people call it the holy grail in data science. It is a large dataset that looks ready to go. There is a clear problem to solve with it. The leaders are also willing to pay for the work. Very few ever get beyond the pilot stage. The industry anecdotal figures put the failure rate of AI and data science projects at over 50% (and the reasons are rarely caused by poor algorithms) They come down to bad questions — or questions that were never asked at all.
Before you write a single line of code or sign anything with a vendor, answer these eight questions in a real way. They cannot promise good results. Still, if you skip them, you are setting yourself up for failure.
1. What Business Problem Are We Actually Solving?
It sounds obvious, but it’s the single most common point of failure: teams start with a technology (“we should use machine learning”) instead of a problem (“we’re losing customers and don’t know why”).
Before scoping anything technical, get specific:
- What decision or process will this project change?
- Who is the end user — an executive, a support agent, a customer?
- What happens today without this system, and what will happen differently once it exists?
If you cannot explain the business problem in a single sentence without bringing up any technology, you should not begin. Always define the project in terms of a business outcome first (“reduce churn by 15%) and then in terms of a technical methodology (the model, the pipeline, the platform).
It is important here for internal alignment, too. Often data science projects get stuck not because of model failure, but because different constituencies have had different definitions of “success” all along.
2. Do We Have the Right Data — and Enough of It?
Data availability: where ambition and reality collide. It is not unusual for a project to be approved on the basis of data assumptions, only to discover these assumptions are invalid when the source systems are examined.
Ask:
- Is the data that we require being captured in the first place or should it be built form the ground up?
- How clean, consistent, and complete is it? Duplicate records? Empty fields? Cross,system inconsistency in formats?
- Do we have sufficient historical information to fit a trusted model, or are we hitting against a sparse and bare data set? Who can we request this data from internally, and with what lead,time?
Worth conducting a quick data audit before establishing a timeline. It’s much less expensive to invest two weeks in data analysis at the beginning than 3 months into a project to realize the data is not suitable for the use case.
3. What Does Success Look Like — and How Will We Measure It?
“Better predictions” or “smarter automation” are not success metrics. They’re aspirations. A data science project needs measurable, agreed-upon criteria before development begins, not after the first model is trained.
Good success metrics are usually a mix of:
- Model performance metrics — accuracy, precision/recall, F1 score, or whatever fits the problem type.
- Business metrics — revenue impact, cost savings, time saved, error reduction.
- Adoption metrics — will people actually use this system, and how will you know?
It allows to anticipate a minimum viable threshold: at which value should we go from pilot to production, at which value should we stop or turn? I set it since I saw the value, and I set it mostly because I wanted to match what I already believed. One can look at the new findings and still have a justification to keep going, even though the data points in another direction. They translate their previous works in as much of an indicator as of a motivation to justify their decision, instead of acknowledging it.
4. Should We Build In-House or Bring in Outside Expertise?
Some groups do not really need their own data science staff. Also, every project does not have to be fully given to outside vendors. The correct approach depends on a few considerations:
- Do you have the in,house skill for the type of problem you want to tackle? For NLP, computer vision, forecasting and agentic AI they are all quite different.
- Is this a one, off project; or a capability you want to own and develop within your business?
- How urgently do you need to see results? Building your own team will take a few months. An experienced external partner could you give you the first iteration much faster.
- What’s the risk of getting it wrong? If your project is in a regulated environment or mission critical your need experience more than haste.
Many organizations land on a hybrid model: an external partner such as InData Labs Get outside the hardest technical problemsarchitecture, model development, executionand leave those to someone else, while you retain ownership of the data, the business context, and the ongoing preservation of the model. That’s not a compromise; it may be the most sustainable arrangement, where outside pattern recognition is combined with intimate domain knowledge.
5. What’s the Realistic Timeline and Budget?
Data science timelines are notoriously optimistic, largely because they’re often based on the “happy path” — clean data, no scope changes, a model that works on the first few tries. In practice, expect:
- Data preparation and cleaning previously took longer than building the model. Now, this step usually represents 60,80% of the total project time.
- At least one round of re-scoping once the data reveals its actual quality and structure.
- A gap between “the model works in a notebook” and “the model works reliably in production” — this transition is where many budgets and timelines break down.
Make a real plan with checkpoints. First, do a data feasibility check. Third, establish a proof,of,concept stage. Fourth, establish a go/no,go decision before proceeding to the complete build.Ask vendors, or your own team, for a phasedapproach. Do not ask for one single end-to-end number right away. A phased plan is a better way to handle unknowns in the budget.
6. How Will This Move From Prototype to Production?
A prototype and a production system are two very different engineering challenges. A model that worked reliably in a Jupyter notebook on a carefully compiled dataset may react very differently to live data, edge cases and scale.
Questions worth raising early:
- What infrastructure will host the model — cloud, on-premises, hybrid?
- How will the model be integrated into existing systems and workflows?
- What’s the plan for retraining as data drifts over time?
- Who is responsible for MLOps — versioning, monitoring, rollback procedures?
Projects that treat “getting to production” as an afterthought tend to stall indefinitely in pilot purgatory — technically successful, but never actually deployed. Planning the production path from the start, even at a high level, dramatically increases the odds the project ships.
7. Who Owns Data Governance, Security, and Compliance?
As soon as a project involves customer data, financial data, or anything that would put you in a regulated business, governance ceases to be a “later” problem. Particularly, it is true for companies working in healthcare, finance or any other industry covered by legislation like GDPR, HIPAA and such.
Before launch, clarify:
- Where lies the accountability for data privacy and security management during the project life span?
- Are there any mandates that specify what kind of data can be held,processed or used in model training?
- How will the system handle bias, explainability, and auditability — particularly if its outputs affect people (loan approvals, hiring, medical triage)?
- What’s the data retention and deletion policy once the project is complete?
Getting this wrong does not only mean there is a legal risk – it means they lose trust with their customers and stakeholders, which is so much harder to rebuild than the technical system.
8. What Happens After Launch?
Shiping the first version is a milestone, not the end of the road. Data science systems tend to decay over time because the world they trained on drift away, this is called model or data drift. A model that works well when launch can slowly get out of accuracy over time, if not properly monitored.
Plan for:
- Ongoing monitoring — dashboards or alerts that flag performance degradation before it affects business outcomes.
- A retraining cadence — will the model be updated monthly, quarterly, or triggered by specific performance thresholds?
- Feedback loops — a way for end users to flag incorrect predictions, which should feed back into future iterations.
- Ownership — who is responsible for the system six months from now, especially if the original team (internal or external) has moved on?
Projects that treat launch as the end of the process tend to see their initial ROI quietly erode. Projects that treat launch as the beginning of a maintenance cycle tend to compound their value over time.
Bringing It All Together
None of these questions are technically difficult to answer. What makes them hard is that they require honesty — about your data, your timeline, your internal capabilities, and what “success” really means to your organization. Most data science projects don’t fail because the math was wrong. They fail because these conversations happened too late, or not at all.
Asking them upfront won’t make a project risk-free. But it will make the risk visible, manageable, and worth taking — which is the difference between a pilot that quietly disappears and a system that actually changes how your business operates.
Read the full article here