Does it actually work?
"Fix hotspots first and you get the best return" is a hypothesis. This page is the attempt to falsify it, run on public repositories you can check.
The setup
Eight mature TypeScript repositories, 202,604 commits in total.
| Repository | History | Files | Fixed files in the test window |
|---|---|---|---|
| vuejs/core | 2018– | 478 | 185 |
| jestjs/jest | 2014– | 931 | 111 |
| vitejs/vite | 2020– | 560 | 220 |
| microsoft/playwright | 2019– | 1398 | 657 |
| nrwl/nx | 2017– | 4712 | 2200 |
| angular/angular | 2014– | 5604 | 371 |
| nestjs/nest | 2017– | 1676 | 21 (excluded) |
| storybookjs/storybook | 2016– | 3986 | 22 (excluded) |
Method. Train on the 12 months before a split date (dowsing's default window), then measure against the 12 months after. The split matters: measure both on the same period and the fixes themselves inflate the churn you are scoring with, so the metric predicts its own input.
A file counts as defective if a commit touching it has a conventional-commit fix: prefix and references an issue. nest and storybook are excluded from scoring — both squash-merge without enforcing the convention, leaving fewer than 30 labelled fixes each. Excluding them is a limitation, not a result.
Where it loses: overall ranking
Restricting the population to files touched during the test window (otherwise the metric just rediscovers which files exist), ranked by AUC:
| Repository | hotspot | churn | LoC |
|---|---|---|---|
| vuejs/core | 0.641 | 0.589 | 0.685 |
| jestjs/jest | 0.605 | 0.594 | 0.625 |
| vitejs/vite | 0.679 | 0.648 | 0.719 |
| microsoft/playwright | 0.637 | 0.636 | 0.722 |
| nrwl/nx | 0.641 | 0.629 | 0.684 |
| angular/angular | 0.700 | 0.674 | 0.724 |
Lines of code alone beat hotspot in seven of eight repositories. This is not a surprise to the defect-prediction literature, where file size shows up repeatedly as a baseline that is hard to beat. Widening the training window to twenty years did not change it. Loosening the labels did not change it.
If you only ever look at a whole-repository ranking, wc -l is a competitive tool and dowsing is not obviously worth installing.
Where it wins: the top of the list
But AUC scores the entire ordering, and nobody uses the tool that way. The actual use is "show me the top 10 or 20." Base rates differ wildly between these repositories (0.4% to 41%), so the comparison is by lift — how many times the base rate the top 20 achieved.
| Repository | Base rate | hotspot | churn | LoC |
|---|---|---|---|---|
| vuejs/core | 38.7% | 2.3x | 1.9x | 2.2x |
| jestjs/jest | 11.9% | 4.2x | 4.6x | 4.2x |
| vitejs/vite | 37.1% | 2.6x | 2.4x | 2.7x |
| microsoft/playwright | 37.8% | 2.6x | 2.6x | 2.4x |
| nrwl/nx | 41.2% | 2.3x | 2.4x | 1.2x |
| angular/angular | 5.7% | 10.4x | 10.4x | 5.2x |
| Average | 4.1x | 4.1x | 3.0x |
At the top of the ranking, hotspot clearly beats LoC (4.1x vs 3.0x). The gap is widest on nx (2.3x vs 1.2x — LoC is close to random there) and angular (10.4x vs 5.2x). Size pushes large-but-stable files to the top; multiplying by change frequency drops them.
The correlation is global, the value is local. That is why the right measure here is precision@K, not AUC.
What complexity actually contributes
On this data, churn alone matches hotspot exactly (4.1x vs 4.1x). Multiplying by complexity appears to add nothing — which would make the central formula pointless.
It turned out to be the definition of a defect. Changing what counts as a positive case:
| Positive case | hotspot | churn | LoC |
|---|---|---|---|
| Fixed at least once | 4.1x | 4.1x | 3.0x |
| Fixed 3+ times (chronic) | 14.4x | 12.2x | 7.7x |
| Fixed 5+ times | 16.3x | 15.5x | 8.4x |
For chronically re-fixed files, hotspot beats churn in four of five repositories (vue +2.0, vite +1.4, playwright +0.4, angular +8.5; the loss is nx at −1.0).
The reading: complexity does not predict whether a file will break. It predicts whether one fix will be enough. A simple file gets fixed once and stays fixed. A complex one gets fixed in the same place over and over. That is a specific, falsifiable claim about what churn × complexity is for, and it is the case where the tool earns its place.
What this means when you use it
- Read the top of the list, not the ranking. The evidence supports "the top 10–20 are worth your attention," not "file #400 is safer than file #380."
- The target is chronic files. If a hotspot has been fixed repeatedly and is still being fixed, that is where complexity is costing you. A file that broke once is not the argument for this tool.
- A whole-repository correlation is not why you would install this. LoC gets you most of that for free. What it cannot do is find the files that keep coming back, or packages that change together with no dependency between them.
Limits of this validation
- Six repositories in the scored comparison, all popular OSS libraries and frameworks. Application codebases and private monorepos may behave differently.
fix:+ an issue reference is a proxy for "a defect existed here." It misses defects fixed without the convention and counts fixes that were not defects.- One split date per repository. No cross-validation across multiple windows.
- Nothing here measures whether acting on the ranking improved anything. That requires measuring before and after a change, which is what
dowsing diffis for.
Every threshold and weight in dowsing is a hypothesis, labelled as one. If you reproduce this and get a different answer, that is a useful result — open an issue.