A Building Block for Coarse-Grain Optimizations in the On-Chip Memory Hierarchy