From the logs, I see that in the pass case with other hubs, the VDM with SVID 0x17EF is sent only once to which there is no response, and following that, notebook sends VDMs with standard SVID 0xFF00 to which the hub responds normally and enters display port mode. However, in the fail case, the VDM with SVID 0x17EF is sent multiple times, even after standard SVID 0xFF00 is sent and CCG4 is able to respond normally and enter display port mode.
As per the USB PD specification, if a UFP receives a structured VDM for a SVID it doesnt support, it shall respond with a Responder NAK command and the handling of the NAK is left to the Initiator. So, I think CCG4 is correctly responding here but the notebook keeps retrying.
-Please check on the notebook side as to why the the VDMs are sent multiple times in case of CCG4 but not other hubs.
-You could also add support for CCG4 to not respond for that SVID in the code as shown in the example below and check if it helps change the behavior.
Function: void eval_vdm(uint8_t port, const pd_packet_t *vdm, vdm_resp_cbk_t vdm_resp_handler)
case 0x12: //as per the reserved command value in VDM
if(vdm->dat[VDM_HEADER_IDX].std_vdm_hdr.svid == 0x17EF)
app_stat->vdm_resp.no_resp = VDM_AMS_RESP_NOT_REQ;