The Data Science Workflow from Data to Decision

Data science combines statistics, programming and domain knowledge to answer questions with data. Projects fail far more often from a poorly defined question or bad data than from the wrong algorithm. A clear workflow keeps the work focused on the decision it is meant to support.

Download diagram (WebP)

1. Define the problem

Start with the decision, not the data. What will someone do differently because of the result? Turn a vague goal (“use our sensor data”) into a measurable question (“predict which compressors will need unplanned repair in the next 30 days, so maintenance can be scheduled”).

It also helps to ask whether data science is needed at all. Some questions are answered by a simple query, a better report or an engineering calculation. Others need a model because the pattern is too complex or the data too large to inspect by hand.

Agree on success criteria early: the metric, the target value, the cost of false alarms versus missed events, and who will use the output.

2. Collect data

Identify the sources: databases, historians, ERP and maintenance systems, spreadsheets, APIs, sensors or surveys. Data usually arrives through data pipelines built by data engineers. Check:

• Is there enough history and enough examples of the events of interest?
• Is the data representative of the conditions where the model will be used?
• Are there legal, privacy or access restrictions?

3. Clean and prepare

Real data is messy. Typical work:

• Missing values: remove, fill with a sensible value, or flag them.
• Outliers: investigate before removing; some are sensor faults, others are the events you care about.
• Units and scaling: convert to consistent units and handle mixed formats.
• Duplicates and inconsistent labels: “Pump-01”, “PUMP 1” and “P01” may be the same asset.
• Time alignment: match time zones and sampling rates when joining sources.

This step often takes more than half of the project time, and it has the biggest effect on the result.

4. Exploratory data analysis

Exploratory data analysis (EDA) builds understanding before modelling. Calculate summary statistics such as mean, median and standard deviation, plot distributions, look at trends over time, and check correlations between variables. EDA often reveals data problems and sometimes answers the question without any model.

A common screening tool is the z-score, which measures how many standard deviations a value is from the mean:

$$z = \frac{x - \mu}{\sigma}$$

5. Feature engineering

Features are the inputs a model learns from. Good features carry domain knowledge: rolling averages, rates of change, time since last service, ratios such as power per unit flow, and day of week or shift. Categorical values are encoded as numbers, and numeric features may be scaled. Well chosen features often improve results more than a more complex algorithm.

6. Build models

Choose the approach to fit the question: supervised learning for predicting a known target, unsupervised learning for finding groups or anomalies. Begin with a simple baseline, such as always predicting the average or a linear model, then try stronger methods like gradient boosted trees or neural networks. Split data into training, validation and test sets, and for time series always test on later data than you train on.

7. Evaluate against the goal

Technical metrics (accuracy, precision, recall, mean absolute error) must be translated into business value. A model with 95% accuracy can be useless if the event of interest is only 2% of cases.

8. Deploy and monitor

A model only creates value when it is used. Deployment can mean a dashboard, a scheduled report, an alert in a maintenance system or a real time API. After deployment:

• Monitor performance, because data drifts as processes, equipment and behaviour change.
• Retrain on new data at a planned interval or when performance drops.
• Document the data, assumptions and limitations so others can trust and maintain the model.
• Collect feedback from users to refine the problem definition, closing the loop.

Tools for the data science workflow

• Languages: Python (pandas, scikit-learn, matplotlib) and R are the most common; SQL is essential for data access.
• Notebooks such as Jupyter for exploration; scripts and version control (Git) for anything that will be reused.
• Reproducibility: fix random seeds, record data versions and keep code in a repository.
• Collaborate: review analysis and code with colleagues, and involve domain experts at every stage, not only at the end.
• Communicate: a clear chart and a one paragraph conclusion usually matter more than the model details.

The data science workflow is not a straight line but a cycle. Teams that keep the decision in view, invest in clean data and monitor results after deployment get far more value than those that focus only on algorithms. For large data volumes, see the five Vs of big data.