Earth’s inner core acquires texture as it solidifies within the fluid outer core. The size, shape and orientation of the mostly iron grains making up the texture record the growth of the inner core and may evolve over geologic time in response to geodynamical forces and torques1. Seismic waves from earthquakes can be used to image the texture, or fabric, of the inner core and gain insight into the history and evolution of Earth’s core2,3,4,5,6. Here, we observe and model seismic energy backscattered from the fine-scale (less than 10 km) heterogeneities7 that constitute inner core fabric at larger scales. We use a novel dataset created from a global array of small-aperture seismic arrays—designed to detect tiny signals from underground nuclear explosions—to create a three-dimensional model of inner core fine-scale heterogeneity. Our model shows that inner core scattering is ubiquitous, existing across all sampled longitudes and latitudes, and that it substantially increases in strength 500–800 km beneath the inner core boundary. The enhanced scattering in the deeper inner core is compatible with an era of rapid growth following delayed nucleation.