Your observation is right.
Whenever a change is made in the schematic, PSoC Creator undertakes the entire process of HDL Generation, Synthesis, Place and Route, Analog Placement, Analog Routing, Digital Placement, Digital Routing, Bitstream Generation, API Generation and Dependency Generation.
Hence it takes longer time for build.
But whenever there is a change only in the code, then the Compilation of the code is done upon build and the Routing (Analog and Digital) remains intact without any changes. This takes small amount of time.