Photonic switching technologies show potential for transforming communication networks across diverse markets from long-haul to short-reach distance scales due to their large bandwidth density, high energy efficiency, and potential for low cost. In recent years, numerous outstanding advancements have been made in scaled silicon photonic switching fabrics: spanning a variety of manufacturing platforms and packaging methods, relying on different switching mechanisms, and assembled on-chip in a diverse mixture of loosely related architectures. This paper reviews the current approaches employed by leading researchers in this area, and surveys the state of the art in achieved performance at both the technological and the architectural level. Specifically, we consider thermo-optic, electro-optic, and MEMS-based switch actuation embedded in Mach-Zehnder interferometer, ring resonator, and directional coupler based silicon photonic switches. We define common metrics and compare performances. We outline critical requirements for constructing scaled switch fabrics from elementary cells. We investigate similarities and differences between a number of commonly utilized topologies. And, we survey recent accomplishments in scaled switch fabrics at the chip and package level. Moving these demonstrations from research to product will require many further advancements, and we highlight areas that we believe will be critical for market adoption.