- A detailed understanding of the connections of fabrication and processing to structural and thermal properties of low-dimensional nanostructures is essential to design materials and devices for phononics, nanoscale thermal management, and thermoelectric applications. Silicon provides an ideal platform to study the relations between structure and heat transport since its thermal conductivity can be tuned over 2 orders of magnitude by nanostructuring. Combining realistic atomistic modeling and experiments, we unravel the origin of the thermal conductivity reduction in ultrathin suspended silicon membranes, down to a thickness of 4 nm. Heat transport is mostly controlled by surface scattering: rough layers of native oxide at surfaces limit the mean free path of thermal phonons below 100 nm. Removing the oxide layers by chemical processing allows us to tune the thermal conductivity over 1 order of magnitude. Our results guide materials design for future phononic applications, setting the length scale at which nanostructuring affects thermal phonons most effectively.