Image settings are walked as a ladder sorted by expected output weight: scale squared multiplied by JPEG quality. The first step that measures under your target wins. If the result is comfortably below the target, the engine tries one step better to give you a sharper file, and keeps whichever fits.
Rasterisation uses a separate ladder from 160 DPI down to 36 DPI. Reducing resolution is the fastest way to shrink a scan, so the ladder trades resolution and JPEG quality together rather than dropping either to an extreme.
The honesty rule: success means outputBytes <= targetBytes, measured on the finished file. If nothing fits, the tool shows the smallest real result and how far above the target it is. It never rounds down, estimates, or labels an overshoot as a success.