We discuss silicon photonic switch fabric designs that target data-intensive computing networks, reviewing recent results, and projecting future performance goals. We analyze the achievements of demonstrated hardware in terms of switching time, footprint, crosstalk, and power consumption, concluding that the most crucial metric to improve upon is net loss. We propose integrating semiconductor optical amplifiers into the switch fabric using either flip-chip or wafer-bonding technology, and investigate its potential merits alongside several challenges in implementation. Furthermore, we explore the dominant causes of crosstalk, and discuss manners for reducing it. We perform switch simulations that project a 7-dB reduction in crosstalk, when using a push-pull, rather than a single-ended phase shifter drive scheme. We also evaluate crosstalk effects on transmission performance using a full-link model that incorporates multiple crosstalk-accumulating photonic switch hops. The study demonstrates the degree to which crosstalk may degrade signal integrity after just a few occurrences. Finally, a comparison of four topologies highlights tradeoffs in physical-layer design and scheduling complexity, illustrating the scales that may be accomplished with the simplest topologies, and the device improvements required to achieve the more robust architectures.