Git limits its tree diff by the pathspec before it looks for renames, so
a directory only ever gets one end of a rename whose other end is outside
it. Nothing is left to pair up, and the file turns into an addition or a
deletion that the commit doesn't contain.
Pass the other end along with the directory. This is bounded by the
number of renames that cross the directory's boundary, so it costs
nothing at all for the vast majority of commits.