Geopolymer binders are a promising alternative to ordinary Portland cement (OPC) because they can significantly reduce CO2 emissions. However, to apply geopolymer in concrete, it is critical to understand the compatibility between the coarse aggregate and the geopolymer binder. Experimental studies were conducted to explore the effect of the size of the coarse aggregate on the mechanical properties and microstructure of a metakaolin-based geopolymer (MKGP) concrete and ordinary concrete. Three coarse aggregate size grades (5–10 mm, 10–16 mm, and 16–20 mm) were adopted to prepare the specimens. The microstructure of the concretes was investigated with scanning electron microscopy/energy-dispersive X-ray spectroscopy (SEM/EDS) and mercury intrusion porosimetry (MIP). Results showed an opposite coarse aggregate size effect between OPC and MKGP specimens in terms of compressive strength. SEM/EDS analysis indicated that the MKGP concrete has a weaker microstructure compared to OPC concrete induced by a higher porosity. The differences in mechanical properties and pore structure between the MKGP and OPC concrete are attributed to the greatly differing shrinkages triggered by the large surface area and penny-shaped particles of metakaolin. The findings in this work help tailor the mechanical properties and microstructure of MKGP concrete for future engineering applications.