A high number is not evidence

Running code and running an experiment look the same from the outside. Same notebook, same keystrokes, same number at the end. The difference is whether anyone can say why that number should be believed.

I spent five months as a junior teaching assistant on the data-mining labs at Nile University. The point of the course was that data mining is a chain rather than a modelling exercise: data preprocessing, exploratory analysis, classification and clustering, feature selection and model evaluation, in Python and Power BI.

Most of the students had already done the basics of programming and statistics. Writing the code was not where the gap was. It was in why each step was there, and what the result in front of them meant.

A high number is the first thing to doubt

A notebook prints 99% accuracy and the student is delighted.

The delight is not unreasonable. A high number looks like the thing the work was for. Nothing threw an error, and nothing in the output says otherwise.

The right first reaction is still not celebration. It is suspicion. Usually one of five things has happened, and not one of them is a merit of the model.

The target has leaked into the features — a column recorded after the outcome was known, looking no different from the ones beside it. Dropping a column you do not understand feels more reckless than keeping it.

Preprocessing was run across the whole dataset before the split, because cleaning once and dividing afterwards is how anyone would organise a piece of work. It looks like housekeeping, not a decision.

Training and test share duplicated records. Nobody removed the repeats, because more rows read as more data.

The classes are imbalanced and the model predicts the majority every time, which on data that is 99 to 1 scores 99%. Accuracy is the one number that hides it.

Or nothing has gone wrong at all: one honest feature separates the classes almost by itself, and the problem was never as hard as it looked.

A high number is not evidence that the model is good. Sometimes it is the first evidence that the experiment is broken.

Which algorithm is the second question

The thing I corrected most often was a student starting from the model before understanding the data. For many of them the opening question was which algorithm to use. The opening question is what is wrong with this data, and what we are trying to predict or discover.

The cost of that order is not the time. You can try five models and never meet the column that leaks the answer. None of the five will mention it. They will all return a number.

Every step in the recipe is a decision

Preprocessing gets treated as a fixed list. Drop the nulls, encode, scale, run the model. Every item on it is a decision that can change what the data means. Dropping rows with missing values can remove a whole category of user — the one that tends to leave the field blank. Filling with the mean can invent a value that never occurred. Turning categories into integers can leave the model treating them as ordered when no order exists, reading a 3 as somehow more than a 1.

The problem was never that students did not know the commands. It was that the command was run before anybody asked what it would do to the data.

The exam was handed out before the term

A test set is not just the part of the data you have not used yet. It is a simulation of the future.

The way I put it was this. The exam paper is handed out at the start of term, before any of the material has been taught. Nobody so much as opens it. The lecturer collects every copy back, locks them away, and sits the same class in front of the same paper at the end of term.

It is not a fair exam, and locking the papers away does not make it one. The paper was in the room while the term was arranged around it. Those questions decided what the class revised: which topics were gone over twice and which were skimmed, which examples got practised, what everyone chose to memorise the night before. Nobody had to cheat for the exam to stop measuring what it was for.

A test set stops being independent the same way. The whole dataset gets cleaned, or the best features chosen using all of it, and only then split. On paper there is a split. In practice information from the test went into building the experiment. Whether the model literally trained on those rows is beside the point, in the same way the locked drawer is.

The quieter version is a random split the data does not support. Time-ordered records, or several rows from the same user, get shuffled as though they were independent draws, and the test set ends up closer to the training data than reality will be.

The metric follows the error you cannot afford

Many students would write the same sentence: "The model achieved high accuracy, precision and recall." Which of the three matters most in this problem, and why, is a different question, and nothing in the notebook asks it.

The gap is not in calculating the metrics. It is in connecting them to the cost of an error. In medical screening, missing a positive case can be worse than sending a healthy person for a second look. That makes recall the number to defend, whatever accuracy says. In fraud detection, accuracy can sit above 99% while the model has caught almost no fraud, because almost nothing in the data is fraud.

A metric means something only once you can say which error you cannot afford. Before that it is a number the notebook printed.

Changing the doctor without running a test

When the result came out bad, the usual response was to change the model. Random Forest did not work, so try XGBoost. Still flat, so try a neural network. It is the one move the tools make easy: a single line changes and everything else stays where it is.

Changing the algorithm before examining the data is like changing your doctor without running a test. First you look at the data. Are the labels right? Is anything in these features actually informative about the target? Is there enough data? Are the classes imbalanced? Is the problem learnable from this data at all? And where exactly are the errors falling?

Not one of those is answered by a different algorithm.

A bad result is not always a failure. It can be the first honest thing the data has told you about the problem.

It is not only a student problem

It would be easy to read all of that as a list of things students get wrong. It is not one.

When data mining is taught as a sequence of ready-made commands, a student learns that success means running the notebook and reaching a high number. That is a rational thing to learn from a course marked that way, and anyone optimising for what gets marked is doing what was asked of them.

A syllabus can reward the final result and still not reward the quality of the experiment. Whether leakage was avoided. Whether the metric suited the problem, and whether the model was measured against a baseline. Whether the cases it got wrong were examined, and whether the student can explain the decisions they made. Every one of those can be marked, and marking them changes what the week is spent on. Not one of them needs a better model.

The work that impresses me is not the one with the highest number. It is the one whose author notices a result that does not make sense and goes back to the data instead of putting the figure in the presentation.

If I were designing a data-mining course, I would cut the number of algorithms a student passes through and give them far longer with one dataset. Three models understood deeply are worth more than ten run without understanding.

Teaching exposed the questions I had been skipping

Understanding an idea and using it turn out to be two different things. I could carry out a step in minutes. Having to explain to a student why it was there brought up the questions I had been stepping over. The reason for this metric rather than another. The reason this split suited this data. The assumption the model was resting on, and what happened when it did not hold.

It left me less interested in remembering tools and more interested in the question that comes before reaching for one. The mistakes above are what happens when that question is skipped, and I had been skipping it too.

Three things before the first cell

The habit worth having from the first day costs a minute. Before running any cell in the notebook, write down three things. The question you are trying to answer. The result you expect, and why. And the thing that, if it appeared, would make you doubt the experiment.

The second has an answer you can compute before you start: what a model that always predicts the majority class would score. That is the number the 99% has to beat.

The third does most of the work. Deciding in advance what would look wrong is your defence against a number that arrives looking like success, because by then you will want to believe it. Written down beforehand, 99% is a reason to stop. Found afterwards, it is already in the slides.

Nothing else about the notebook changes. The keystrokes are identical.

The biggest mistake in data mining is not choosing a weak algorithm. It is arriving at a strong result and not being able to say why you trust it.