Misreading Charts: How Correct Numbers Lead to Wrong Decisions in Machine Learning

Imagine a manufacturer deciding whether to deploy a new equipment-failure warning system. The operations lead wants fewer shutdowns, the maintenance manager worries about unnecessary inspections, and the data team uses a simplified teaching example to compare two candidates. On 500 test cycles, the first reports 87.2% accuracy and the second 85.4%. If the meeting stops at those numbers, the first model appears to have won.

The maintenance manager asks a different question: how many failures did each model miss? The higher-accuracy model always predicts normal and misses all 64 failures. The second model raises more false alarms but catches 44 failures. Under the illustrative assumption that a missed failure costs five times an unnecessary inspection, the lower-accuracy model has the lower weighted misclassification cost. Neither 87.2% nor 85.4% is wrong. The problem is allowing one metric to define “better.”

This article calls that mistake a chart misreading: the numbers can be correct while the conclusion exceeds the evidence. Misreading usually starts with a broken evaluation boundary, an over-interpreted algorithmic output, or a summary metric that does not match the decision. The six examples examine these problems first in data and representation, then in models and evaluation. Except for the public Wine dataset, the examples use fixed-seed synthetic teaching data. The figures illustrate diagnostic methods; they do not establish general results for industrial or retail systems.

1. Ask what the chart can support #

When a set of results arrives, the first instinct is often to choose a plotting function. Yet the choice of chart matters less than the decision the review must support.

A confusion matrix answers “how many errors occurred, and which kind?” A precision–recall curve answers “how do precision and recall change with the threshold?” Both charts can be correct, but neither can answer the other question.

Before plotting, identify the object under review, the output produced by the algorithm, and the decision the reader must make. A chart can support only a limited set of conclusions. Misreading begins when readers use it to answer a different question.

To make those questions concrete, this article follows scikit-learn’s six task groups: preprocessing, dimensionality reduction, clustering, classification, regression, and model selection[1]. The first three concern data and representation; the last three concern models and evaluation. The table offers a reading map: what can each overview hide, and what evidence should accompany it?

Task	What the overview can hide	Diagnostic evidence
Preprocessing	Leakage, time drift	Train-only fitting, time axis, missingness
Dimensionality reduction	Scale dominance, parameter sensitivity	Loadings, explained variance, stability
Clustering	A continuum forced into groups	Silhouette distribution, sizes, resampling
Classification	Missed positives, alert burden	Count matrix, PR, calibration, cost
Regression	Local bias, lag-following	Residuals, business slices, naive baselines
Model selection	Fold reversals, selection bias	Paired folds, learning curves, held-out test

This table is not another scoring system. It provides a reading order: decide what the chart answers, then ask what it leaves out. A count confusion matrix reports failures and false alarms at one threshold. A precision–recall curve shows the threshold trade-off but not the operational number of false alarms or their cost.

The next six examples come from different algorithms, but they repeat the same mistake: a chart displays an algorithmic output, and the reader treats it as a complete operational conclusion. We begin with the evaluation boundary because every later score depends on it.

2. Six common chart misreadings #

2.1 Data and representation: transformation is not structure #

The data have already changed before a predictive model makes its first estimate. Preprocessing changes the scale, dimension reduction changes the viewpoint, and clustering assigns group labels. If these steps cross a train–test boundary, or if scale and algorithmic partitions are mistaken for business structure, later scores can lose their meaning.

Preprocessing: tidy distributions can still hide leakage #

Start with sensor readings that contain a clear shift over time. The training period has 300 observations and the test period has 150; readings are generally higher in the test period, as they might be when equipment conditions change. More precisely, the training values are drawn from \(N(50,8^2)\) and the test values from \(N(60,9^2)\). These four numbers define the generating distributions; they are not statistics computed from the finite samples.

The data team wants to put the readings on a common scale. StandardScaler estimates a mean \(\mu\) and standard deviation \(\sigma\), then computes \(z=(x-\mu)/\sigma\). Other preprocessors may estimate quantiles or imputation values. In every case, fitted statistics must come from the training data.

Two scaled distributions can both look neatly centred even when one transformation has already “seen” the test set. A clean histogram shows that the transformation ran; it does not show that the evaluation boundary remained intact. If the team calls fit_transform on the complete table before the train–test split, information about the test distribution enters the training reference point.

With the fixed random seed, the realised training mean is 49.67, the test mean is 60.23, and the combined mean is 53.19. Those values make the difference between the two procedures visible. In the upper panel, the higher test readings move the global reference and push the transformed training mean down to -0.37. The lower panel fits the scaler on training data only. Its transformed training mean is 0.00, while the test sample sits about +1.42 training standard deviations above it.

The formula is not the problem. The misreading is to treat a distribution made to look centred by global scaling as evidence that no drift exists, even though test information has already contaminated the evaluation boundary.

Two related warning signs deserve the same scrutiny:

• A random split can hide time drift. A random sample from a trending sales or sensor series may make training and test distributions look similar. A deployment-aligned time split exposes the difference between the historical training period and the future test period.
• A post-event field can masquerade as a strong predictor. If one feature has univariate AUC near 0.99, or dominates a feature-importance chart, check whether it is created after the event. A maintenance work-order number, reset flag, or post-event replenishment quantity may record the outcome rather than predict it.

What to check: split by prediction time or entity before fitting transformations. Call fit only on training data and transform on validation and test data. Put preprocessors and estimators in the same Pipeline so that every validation fold estimates its own statistics[2][3]. For unusually dominant features, inspect data lineage and confirm that the value exists at prediction time.

Dimensionality reduction: high explained variance may reflect scale #

This example is not synthetic. It uses scikit-learn’s public Wine dataset: 178 observations with 13 chemical and physical measurements. Although the measurements sit in the same table, their numeric scales differ sharply. proline ranges from 300 to 1680 and has sample variance above \(9.9\times10^4\), while features such as nonflavanoid_phenols and hue lie between about 0.1 and 1.7 and have variance below 0.06.

The data team wants to compress those 13 measurements with principal component analysis (PCA). PCA finds orthogonal directions of maximum variance after centring the data. A loading measures how strongly an original feature contributes to a component, and the explained variance ratio reports the component’s share of total variance. PCA centres features by default, but it does not automatically give them comparable scales[4]. For component \(j\),

\[ \mathrm{EVR}_j = \frac{\lambda_j}{\sum_{r=1}^{p} \lambda_r}, \]

where \(\lambda_j\) is the component’s eigenvalue and \(p\) is the number of features.

“PC1 explains 99.8% of the variance” is the kind of result that readily becomes a slide headline. It sounds as though PCA has found the most important business structure. Here, however, “important” means only that the variance is large. On the unscaled data, proline receives an absolute PC1 loading of 0.9998, and PC1 explains 99.8% of total variance. After standardising all 13 features, PC1 explains 36.2%; its loadings are shared across features including flavanoids (0.42), total_phenols (0.39), and od280/od315 (0.38). This example isolates the effect of scale and does not define a predictive train–test split.

The direction in the chart can be correct while its interpretation is not. Misreading begins when a maximum-variance direction or a two-dimensional layout is interpreted as the dominant business structure without checks on scale, loadings, or projection stability.

For t-SNE or UMAP, visually separated islands are conditional on the method, hyperparameters, and random seed. These methods prioritise local neighbourhoods; the area of an island or the gap between islands does not directly measure distance in the original high-dimensional space.

What to check: state whether features were standardised and which observations fitted the scaler. For PCA, show loadings and cumulative explained variance[5]. For t-SNE or UMAP, compare several defensible settings and retain checks in the original feature space rather than treating a two-dimensional map as proof of natural groups.

Clustering: algorithmic partitions are not natural groups #

This dataset deliberately contains no true groups. It has 450 observations, each with eight independent standard-normal features. The result is one continuous eight-dimensional Gaussian cloud with no preset classes or cluster labels.

If the team still asks the algorithm for three groups, KMeans(n_clusters=3) will return three clusters in the original eight-dimensional space. PCA is then used only to display the observations on PC1 and PC2. KMeans minimises within-cluster squared distances, so it returns the requested three Voronoi regions even when the data form a continuum.

Once the projected points receive three colours, readers naturally look for three kinds of customer or machine. The three panels show the uncoloured projection, the same projection coloured by the fitted labels, and the sample-level silhouette coefficients.

Colour shows how the algorithm divided the observations, but not whether the groups are well separated. The silhouette coefficient provides that second check. For sample \(i\), let \(a_i\) be its average distance to other observations in its assigned cluster, and let \(b_i\) be the smallest average distance to another cluster. Its silhouette coefficient is

\[ s_i = \frac{b_i-a_i}{\max(a_i,b_i)}. \]

Values near 1 indicate a closer fit to the assigned cluster than to neighbouring clusters. Values near 0 place the observation around a boundary, while negative values suggest that another cluster may be closer. In a silhouette plot, each horizontal bar represents an observation; the vertical span of a group shows cluster size, not cluster quality.

The mean silhouette coefficient in this example is only about 0.09, and many observations sit close to zero. The three colours in the projection therefore do not establish three natural populations.

The colours correctly show the assigned labels, but labels are not natural groups. The misreading is to treat a partition produced for a requested cluster count as a structure already present in the data.

A low silhouette score also has limits. It says that the data are not compactly separated under the current feature set, Euclidean distance, and KMeans shape assumptions. It does not rule out structure under another distance measure or on a local manifold.

What to check: label the scatter plot as a low-dimensional projection. Report cluster sizes, the full distribution of sample-level silhouette coefficients, and the global mean[6]. Then resample the data or change the random seed and use a measure such as the adjusted Rand index (ARI) to test whether the assignments remain stable.

2.2 Models and evaluation: look beyond the aggregate score #

Protecting the data boundary only prevents the model from seeing answers too early. Misreading can still occur when a valid overall score hides the errors that matter to the decision. Accuracy, \(R^2\), and cross-validation means each compress a different pattern of variation.

Classification: high accuracy can hide costly errors #

This section returns to the fixed-count teaching data from the opening: 500 equipment cycles, of which 436 are normal and 64 are failures. It does not retrain an end-to-end model. Instead, two predetermined sets of predictions isolate how class imbalance and error cost can change a deployment decision.

Why can a candidate with 87.2% accuracy still be the worse choice? Accuracy combines performance on the positive and negative classes. If the positive-class prevalence is \(\pi\), the true-positive rate is \(\mathrm{TPR}\), and the true-negative rate is \(\mathrm{TNR}\), then

\[ \mathrm{Accuracy} = \pi\cdot\mathrm{TPR} + (1-\pi)\cdot\mathrm{TNR}. \]

The test set contains 64 failures and 436 normal cycles, so \(\pi=12.8\%\). An always-normal baseline has \(\mathrm{TPR}=0\) and \(\mathrm{TNR}=1\), yet it still achieves \(436/500=87.2\%\) accuracy. Its score hides the fact that all 64 failures are false negatives.

The warning model records 383 true negatives, 53 false positives, 20 false negatives, and 44 true positives. Its accuracy is lower at \((383+44)/500=85.4\%\). Under the stated illustrative assumption \(c_{\mathrm{FN}}=5\) and \(c_{\mathrm{FP}}=1\), however, its weighted misclassification cost is \(20\times5+53=153\), compared with \(64\times5=320\) for the always-normal baseline—a reduction of 52.2%.

Accuracy answers how often the prediction is right. It does not answer who bears each error or what that error costs. Treating the first answer as business value creates the misreading.

When the positive class is very rare—perhaps 0.1% to 1% in a wafer-defect or fraud setting—the false-positive rate can remain small even when the absolute number of false alarms is operationally large. A precision–recall (PR) curve compares two quantities tied directly to the positive class:

\[ \begin{aligned} \mathrm{Precision} &= \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}, \\ \mathrm{Recall} &= \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}} . \end{aligned} \]

Precision asks how many alerts are true failures, so false alarms affect it directly. Recall asks how many actual failures the model catches, so missed failures affect it directly. Class imbalance does not guarantee a high ROC AUC, and a high ROC AUC does not guarantee useful precision at the required working point. Inspect the PR curve and average precision alongside the positive-class rate[7], then report the actual alert count at the chosen threshold.

What to check: report positive and negative sample counts and the raw confusion-matrix counts. For an imbalanced task, include the PR curve and its random baseline at prevalence \(\pi\). If error costs differ, choose the threshold from out-of-fold training predictions or a separate validation set, never by repeatedly tuning on the final test set.

Regression: high R² can hide local bias #

This example uses one operating measure \(x\) to predict continuous equipment load \(y\). It contains 300 training observations and 220 independent test observations, with \(x\) sampled uniformly from 10 to 90. Load generally rises with \(x\), but the relationship also contains a modest bend and random noise.

For reproducibility, the target is generated as \(y=20+1.8x+0.0285(x-50)^2+\varepsilon\), where \(\varepsilon\sim N(0,6.2^2)\). We deliberately fit ordinary univariate linear regression. It can capture the overall rise, but not the full curvature.

Engineers report repeated underestimation at peak load, while the evaluation report shows \(R^2=0.88\). Both observations can be correct. The coefficient of determination and mean absolute error (MAE) are

\[ \begin{aligned} R^2 &= 1-\frac{\sum_{i=1}^{n}(y_i-\hat{y}_i)^2} {\sum_{i=1}^{n}(y_i-\bar{y})^2}, \\ \mathrm{MAE} &= \frac{1}{n}\sum_{i=1}^{n}\lvert y_i-\hat{y}_i\rvert . \end{aligned} \]

Here, \(n\) is the number of test observations and \(\bar{y}\) is the mean observed test value. \(R^2\) compares the model’s squared error with a mean baseline, while MAE reports the average absolute error. Both reduce all test errors to one number, so neither shows which operating region contains the errors or reveals a local systematic pattern on its own.

In the teaching example, a linear regression reaches \(R^2=0.88\) and \(\mathrm{MAE}=13.1\) on an independent test set of 220 observations. The actual-versus-predicted scatter lies close to \(y=\hat y\), which makes the fit look adequate. The residual plot tells a different story. With residual \(e=y-\hat y\), the smoothed curve forms a clear U shape:

• for \(\hat y\in[80,140]\), residuals stay below zero, so the model systematically overpredicts;
• for \(\hat y>160\), residuals rise above +15, so the model substantially underpredicts peak load.

The curved pattern suggests that the linear model may have missed a nonlinear relationship, such as a quadratic term. It is evidence of possible misspecification, not proof that the true process is quadratic[8].

The overall fit is real, but it cannot guarantee every operating region. Extending it into that guarantee hides the systematic error at high loads.

Time-series forecasts can create a related illusion. If a series is strongly autocorrelated, a lag-one naive forecast \(\hat y_t=y_{t-1}\) can track the observed line closely without providing useful lead time. Compare the model with that baseline and, where seasonality matters, with a seasonal naive baseline. Evaluate errors by forecast horizon and consider a scaled measure such as MASE.

What to check: on the independent test set, show both actual versus predicted values and residuals versus predictions, with an \(e=0\) reference and a smooth trend line[8]. Break errors down by operationally important regions such as peak load, promotion periods, or equipment batches. For time series, include naive baselines at the deployment horizon.

Model selection: the highest mean is not a stable lead #

Residuals show where one model fails. Once several candidates enter the comparison, a leaderboard can hide another layer of uncertainty.

This example uses the same end-to-end synthetic dataset as the worked example later in the article: 1,500 training cycles, 500 independent test cycles, and a 12.8% failure rate in each split. Every observation contains eight informative core features, 256 noise features, and deliberately varied feature scales. This section isolates one task: using five-fold cross-validation on the training set to choose the logistic regression regularisation parameter \(C\).

The Notebook uses scikit-learn 1.9.1 and does not explicitly set the solver, penalty type, class weights, or sample weights. In this example, LogisticRegression therefore uses the default lbfgs solver with L2 regularisation and unit sample weights. Under these conditions, each fit is equivalent to minimising the following objective[9]:

\[ \begin{aligned} \min_{w,w_0}\quad & \frac{1}{n}\sum_{i=1}^{n} \left[-y_i\log p_i-(1-y_i)\log(1-p_i)\right] \\ &\quad +\frac{1}{2nC}\lVert w\rVert_2^2, \\ p_i &= \frac{1}{1+\exp\left[-\left(x_i^\top w+w_0\right)\right]} . \end{aligned} \]

Here, \(x_i\) is the feature vector transformed by the StandardScaler fitted within the current training fold, \(y_i\in\{0,1\}\) is the class label, \(w\) contains the feature coefficients, \(w_0\) is the intercept, and \(n\) is the number of training observations used by that fit. The L2 penalty applies to \(w\), not to \(w_0\). Each fit in this five-fold comparison trains on four folds, so \(n=1{,}200\); after selecting \(C\), the final refit uses all 1,500 training observations.

The coefficient on the penalty is \(1/(2nC)\), so a smaller \(C\) produces stronger shrinkage. Internally, scikit-learn’s lbfgs path first computes the sum of sample weights as sw_sum, then sets l2_reg_strength = 1.0 / (C * sw_sum)[10]. Because this example does not set sample or class weights, sw_sum equals \(n\). The displayed objective is the mathematically equivalent form for this configuration, not a line-by-line transcription of the source. Class weights, sample weights, or a different penalty would require a corresponding change to the objective.

A leaderboard makes it easy to forget that each mean summarises several validation runs whose results are not identical. Five-fold cross-validation produces five paired scores for each candidate, not just one average. A mean-only chart can create two distinct errors:

1. It hides fold variation and pairing. Two candidates may differ by 0.007 in their means while one candidate has a fold-to-fold standard deviation of 0.025. Without paired fold scores, readers cannot see whether the lead is consistent or driven by one split.
2. It treats selection evidence as generalisation evidence. Choosing the highest mean from several candidates introduces selection bias. The selected validation result will often be optimistic, so the final estimate must come from a test set that played no role in selection[11].

The example compares five regularisation settings, \(C\in\{0.01,0.03,0.1,0.3,1.0\}\). The truncated bar chart reports means of 0.784, 0.831, 0.859, 0.866, and 0.862, making their differences look larger than they are. The diagnostic panel restores paired fold trajectories and \(\pm1\,\mathrm{SD}\) bands, then marks the independent test result after validation selects \(C=0.3\). Individual fold scores range from about 0.740 to 0.901, far more than the roughly 0.004 gap between \(C=0.3\) and \(C=1.0\). The held-out test ROC AUC is 0.841, below the selected candidate’s validation mean of 0.866.

The highest validation mean is evidence for selection, not proof of a stable lead or final generalisation. Compressing those three judgments into one rank makes a small mean difference look more certain than it is.

Learning curves and validation curves answer another question that two isolated training and test bars cannot. If training and validation scores remain low together, the model may lack useful features or capacity. If a persistent gap separates them, stronger regularisation or more data may help. These curves diagnose model capacity; they do not replace the independent test.

What to check: show paired scores by fold and a dispersion measure rather than a truncated mean-only bar chart. State whether validation is stratified (StratifiedKFold), grouped (GroupKFold), or time-ordered (TimeSeriesSplit). Once validation selects the hyperparameters, lock the candidate set and evaluate exactly once on the untouched test set[11].

3. Before delivery: check presentation and evidence #

After these six examples, one pattern is clear: misreading does not always begin with bad arithmetic. The evaluation design or metric may already have created a blind spot, and a truncated axis, unexplained colour scale, or mean-only leaderboard can magnify it.

Before someone else uses the chart to make a decision, review it twice. First ask whether the visual design exaggerates the conclusion. Then ask whether the experiment behind the chart can withstand scrutiny. The four safeguards below address presentation; the six questions that follow address evidence.

3.1 Four presentation safeguards #

1. Use colour only for comparable quantities. A confusion matrix must say whether its scale represents raw counts or normalised proportions. Accuracy, latency, and model size should not share one raw-value heatmap because their units and preferred directions differ. Use directionally aligned scores or faceted dot plots, and do not rely on red–green contrast alone[12].
2. Keep a zero baseline for bar charts. Readers interpret bar length as magnitude. Showing only [0.77, 0.88] makes the gap between 0.784 and 0.866 look larger than it is. For small differences, use a labelled dot or line plot with uncertainty[13].
3. Show variation as well as rank. First and second place may differ by 15 percentage points or by 0.001. When results span several folds, seeds, or time windows, report the mean with dispersion or paired trajectories so that sampling variation does not look like a stable lead.
4. Apply engineering constraints before optimising a composite score. Latency, memory, and minimum failure recall can determine whether a model is deployable. A model that exceeds the production cycle time remains infeasible even if its validation mean is 0.005 higher; a composite score should not hide that constraint.

These four rules govern how the chart speaks. The next checklist asks whether the experiment has earned the right to say it. The first avoids visual amplification; the second avoids evidential overreach.

3.2 Six questions: is the evidence sound? #

Check	Question	Delivery standard
Data boundary	Were all fitted transformations restricted to training data?	Scaler, Imputer, and PCA stay inside the Pipeline; each fold calls fit on training data only.
Split design	Does the split reproduce deployment?	Time series hold out the future; repeated entities use group-aware splits rather than random shuffling.
Representation	Do projections and clusters have supporting evidence?	PCA states scaling and shows loadings; cluster plots include sample-level silhouettes and their mean.
Decision cost	Do classification metrics match prevalence and error cost?	Report prevalence, a PR curve, raw confusion counts, and a threshold selected without the test set.
Residual baseline	Does regression test local error and naive alternatives?	Show residual structure and operational slices; time series include lag-one or seasonal naive baselines.
Selection isolation	Are validation choices isolated from final testing?	Preserve paired fold scores; open the independent test set only after models and thresholds are locked.

4. Worked example: reviewing an anomaly-warning system #

Each safeguard is straightforward on its own. The challenge is making all of them hold in the same review. The earlier classification chart uses fixed counts only to isolate the accuracy-versus-cost question. The cross-validation chart in the model-selection section and the dashboard below come from the same end-to-end synthetic experiment. We now follow the team’s decision sequence through training, selection, thresholding, and testing. The dashboard therefore does not reproduce the earlier fixed counts, but it does continue the model-selection results.

The dashboard reuses the data from the model-selection section: 1,500 synthetic training cycles and 500 independent test cycles. Both sets have a 12.8% failure rate: 192 failures in training and 64 in testing. Each observation contains eight informative core features, including temperature, pressure, common-mode vibration, differential vibration, and high-frequency channels, plus 256 features unrelated to the target. The features also use different numeric scales, so preprocessing and regularisation must remain inside the correct evaluation boundary. The vertically stacked six-panel dashboard is not intended to prove that regularised logistic regression beats deep learning. It shows how a team can build a chain of evidence step by step.

1. Lock the evaluation boundary (panel 1). The team first decides which data can support training and which must remain sealed until the end. The example contains 1,500 training cycles and 500 independent test cycles from the same feature-generating process, with a 12.8% positive rate in each set. Absolute counts also expose the 87.2% always-normal baseline. Real equipment data would usually require a future time holdout, while evaluation on unseen machines would require group isolation.

2. Select the model without touching the test set (panel 2). The team then compares candidates only within the training set. One Pipeline combines StandardScaler and LogisticRegression, and stratified five-fold validation compares five regularisation settings. Paired fold scores select \(C=0.3\), with a mean ROC AUC of about 0.866. Production time series would use rolling validation; unseen-machine evaluation would use grouped validation.

3. Translate error costs into a decision threshold (panel 3). Assume the model returns a calibrated failure probability \(p\), and correct classifications have zero cost. Raising an alert then has expected cost \(c_{\mathrm{FP}}(1-p)\), while not raising one has expected cost \(c_{\mathrm{FN}}p\). The model should raise an alert when the first cost is no greater than the second:

   \[ \begin{aligned} c_{\mathrm{FP}}(1-p) &\le c_{\mathrm{FN}}p, \\ \tau^* &= \frac{c_{\mathrm{FP}}}{c_{\mathrm{FP}}+c_{\mathrm{FN}}} = \frac{1}{1+5} = \frac{1}{6}\approx 0.17 . \end{aligned} \]

   A five-to-one cost ratio therefore moves the theoretical threshold from 0.50 towards \(1/6\). This result depends on the calibration and cost assumptions; it does not fix the empirical threshold by itself. To reflect the observed probability distribution, the team scans [0.05, 0.85] using out-of-fold training probabilities:

   \[ \widehat{R}(\tau)=\frac{c_{\mathrm{FN}}\,\mathrm{FN}(\tau)+c_{\mathrm{FP}}\,\mathrm{FP}(\tau)}{N}. \]

   Here, \(N\) is the number of out-of-fold predictions, and \(\mathrm{FN}(\tau)\) and \(\mathrm{FP}(\tau)\) are the error counts at threshold \(\tau\). The empirical out-of-fold cost scan fixes \(\tau=0.18\) without touching the test set. The chart rescales average risk to the expected cost per 500 cycles so that it matches the test-set scale; this linear conversion does not change the optimal threshold.

4. Open the test set and inspect the operating point (panel 4). Only after the model and threshold are locked does the team evaluate the 500 test cycles. ROC AUC is 0.841, average precision is 0.585, and at \(\tau=0.18\), precision is 0.468 with failure recall of 0.562.

5. Settle the final error bill (panels 5 and 6). The calibration curve checks whether the earlier calibration assumption is broadly plausible, but 64 failures are too few for that curve alone to establish adequate calibration. The final matrix contains TN = 395, FP = 41, FN = 28, and TP = 36. Accuracy is 86.2%, below the 87.2% baseline, while weighted misclassification cost falls from 320 to 181, a 43.4% reduction.

The dashboard produces different counts from the fixed confusion-matrix example, but it supports the same judgment: accuracy alone cannot decide which errors matter more. With prevalence at 12.8%, the threshold fixed at \(\tau=0.18\), and the false-negative to false-positive cost ratio set to 5:1, the end-to-end model has slightly lower accuracy than the majority baseline but lower weighted misclassification cost. If prevalence or error costs change, the team must evaluate the threshold and comparison again rather than carry this conclusion forward.

5. Conclusion: do not let the chart decide for you #

Looking back across the six examples, the arithmetic was not the problem. In each case, local evidence was asked to carry a conclusion it could not support. Accuracy can summarise how often predictions match labels, but it cannot decide whether a missed failure or false alarm matters more. A two-dimensional projection can show one representation, but it cannot prove that natural business groups exist. A cross-validation mean can guide selection, but it cannot replace an independent test.

The next time a chart looks impressive, ask three questions before accepting its conclusion. What evaluation boundary produced it? What do its colours, distances, or scores actually represent? If someone acts on it, who bears each kind of error? Those questions often contribute more than another composite score. A chart should expose the evidence and its limits, not merely make a conclusion look persuasive.

Experiment dashboards and model reports are becoming more automated, so charts arrive faster and look more polished. Teams still need to preserve the prediction-level records, split rules, threshold rationale, and cost assumptions behind them. A chart supports judgment only while that evidence remains open to review; otherwise, automation merely creates an impression of certainty more efficiently.

If you only need the decision framework, you can stop here. The appendix below provides a reference for code and common data shapes.

Appendix: common data shapes for six tasks #

The following shapes provide a compact bridge between the discussion and a scikit-learn implementation. Here, \(n\) is the number of observations, \(p\) the original feature count, \(p'\) the transformed feature count, \(G\) the number of classes, \(K\) the number of clusters, \(k\) the retained dimension, \(o\) the number of regression outputs, \(m\) the number of candidates, and \(F\) the number of validation folds.

Layer	Task	Common input or output shapes	Metrics and diagnostics
Data and representation	Preprocessing	Raw (n, p); transformed (n, p'); fitted statistics (p,)	Split-specific quantiles, missingness, and reference-point shifts
Data and representation	Dimensionality reduction	Coordinates (n, k); PCA loadings (k, p); variance ratios (k,)	Cumulative variance, loadings, reconstruction or neighbourhood stability
Data and representation	Clustering	Labels (n,); centroids (K, p); silhouettes (n,)	Mean silhouette, cluster sizes (K,), ARI across resamples
Models and evaluation	Classification	Labels (n,); positive scores (n,); multiclass probabilities (n, G)	Accuracy, ROC AUC, AP, PR and calibration curves, confusion matrix (G, G)
Models and evaluation	Regression	Targets and predictions (n,); multi-output (n, o); residuals (n,)	MAE, RMSE, \(R^2\), residual trend, errors by operational slice
Models and evaluation	Model selection	Candidate-by-fold scores (m, F); cv_results_	Paired folds, mean and SD, fit time, learning curves, held-out test

Displays such as ConfusionMatrixDisplay, PrecisionRecallDisplay, CalibrationDisplay, and PredictionErrorDisplay turn sample-level predictions into diagnostic views[14]. Save prediction-level records and out-of-fold probabilities, not only a single best_score_; the detailed records make later review possible.

References #

[1] scikit-learn developers. scikit-learn: Machine Learning in Python. https://scikit-learn.org/stable/

[2] scikit-learn developers. Getting Started: Transformers and pre-processors. https://scikit-learn.org/stable/getting_started.html

[3] scikit-learn developers. sklearn.preprocessing.scale. https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.scale.html

[4] scikit-learn developers. sklearn.decomposition.PCA. https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html

[5] scikit-learn developers. Importance of Feature Scaling. https://scikit-learn.org/stable/auto_examples/preprocessing/plot_scaling_importance.html

[6] scikit-learn developers. Selecting the number of clusters with silhouette analysis on KMeans clustering. https://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_silhouette_analysis.html

[7] T. Saito and M. Rehmsmeier. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLOS ONE, 10(3): e0118432, 2015. https://doi.org/10.1371/journal.pone.0118432

[8] scikit-learn developers. Visual evaluation of regression models. https://scikit-learn.org/stable/modules/model_evaluation.html#visual-evaluation-of-regression-models

[9] scikit-learn developers. Logistic regression. scikit-learn 1.9.1 User Guide. https://scikit-learn.org/1.9/modules/linear_model.html#logistic-regression

[10] scikit-learn developers. sklearn.linear_model._logistic source code. scikit-learn 1.9.1. https://github.com/scikit-learn/scikit-learn/blob/1.9.1/sklearn/linear_model/_logistic.py#L466-L580

[11] scikit-learn developers. Validation curves: plotting scores to evaluate models. https://scikit-learn.org/stable/modules/learning_curve.html

[12] F. Crameri, G. E. Shephard, and P. J. Heron. The misuse of colour in science communication. Nature Communications, 11, 5444, 2020. https://doi.org/10.1038/s41467-020-19160-7

[13] Office for National Statistics. Simple comparisons: Bar chart. Data visualisation service manual. https://service-manual.ons.gov.uk/data-visualisation/chart-types/bar-charts

[14] scikit-learn developers. Metrics and scoring: quantifying the quality of predictions. https://scikit-learn.org/stable/modules/model_evaluation.html