A detector that says "I don't know"

The riskiest code in Bhoomi is nine lines that decide whether to subtract 1000 from a pixel value. Sentinel-2 scenes processed after a certain baseline carry a reflectance offset; earlier ones don't. Subtract it when it isn't there and NDVI shifts by about 0.24 — silently, with every value still inside its valid range. Nothing crashes. You just get a number that's wrong by an amount large enough to change the conclusion.

Three metadata fields claim to answer the question. All three are unreliable. On the catalogue Bhoomi actually reads, one field came back meaning "offset present" on one scene and "offset absent" on another, and a second reported the same value regardless.

Attempt one: measure it from the pixels

The offset is exactly 1000 DN, and reflectance can't be meaningfully negative. So a scene carrying the offset should have essentially no pixels below a dark floor. Measured across seven scenes near full resolution:

offset present      0.00 %  of pixels below 700 DN
offset absent       3.48 – 8.17 %

Clean separation. A threshold at 1% splits them with room to spare.

Attempt two: the calibration was right, the code wasn't

Those percentages were measured near full resolution. The detector shipped sampling the tile at decimation 32 — and overviews are built by averaging, which pulls dark pixels up toward their bright neighbours.

At decimation 32 the same offset-absent scenes measure 0.74–1.42%. The threshold sits at 1.0. Four of eight scenes landed on the wrong side. One missed by 0.024 of a percentage point, and subtracting an offset that wasn't there produced 93% negative reflectance and a median NDVI of +1.703 — a number that is not merely wrong but impossible.

The lesson people usually take from this is "test at the resolution you ship at," which is true and insufficient.

Attempt three: it was measuring terrain

The fix looked correct. Then I widened the sample from ten scenes on one tile to 48 scenes across 8 regions, deliberately including desert and salt flat — places where dark pixels barely exist at all.

The rule misclassified 36% of offset-absent scenes. Every Thar Desert and Delhi scene read below the threshold, not because it carried the offset but because those tiles contain almost no dark ground.

The statistic was measuring terrain. It worked on Kolkata because Kolkata is wet.

The shape of an honest answer

What replaced it is one-sided:

floor below 800 DN   ->  offset is absent      (proven)
high floor           ->  unknown               (fall back to metadata,
                                                and say so in the output)

A low floor proves the offset absent. A high floor proves nothing, because a bright desert tile and an offset-bearing scene are genuinely indistinguishable from pixels alone. Where the pixels can't decide, the decision falls back to metadata and the output carries a warning.

Across those 48 scenes: zero misclassified, against 17 for the rule it replaced.

Three things worth keeping:

1. A calibration is only valid for the sampling that produced it — and, more sharply, for the population that produced it.
2. Some questions aren't answerable from the data you have. A detector that returns "I don't know" is worth more than one that always answers.
3. The guard caught the first failure. The index function raises when values leave [−1, 1] rather than logging, which is the only reason this surfaced as a failed job instead of a plausible-looking raster with a systematic bias.

That third one is the whole argument for strict invariants in numerical code. The bug wasn't found by a test. It was found because the code refused to continue.