We demonstrated a non-volatile photonic matrix computation core which contains a 3×3 photonic phase change in-memory computing matrix to carry out matrix vector multiplication on a silicon-based substrate. We established an optical computing core model and hardware implementation based on our photonic phase change material (PCM) devices and fabricated a 3×3 matrix using the silicon photonic foundry. We demonstrated the functionality of this matrix as the linear convolution layer in a convolutional neural network (CNN) and demonstrated simple pattern recognition. Finally, we simulated scaling up matrix limits using experimental data from smaller matrices. This on-chip, non-volatile, photonic computation matrix transfers optical computing from single device to matrix, paving the way for practical wide usage of optical computing systems.