What app note are you referring to ? That is a very large CapSense.
AN64846 "Getting Started with CapSense", section 3.8.14 Shield Electrode and Guard Sensor...
"Provide directionality to proximity sensing: The electric field of a proximity sensor is omnidirectional and can detect proximity in all directions. In most of the applications, it is required to detect proximity from only one direction. In such cases, you can use a shield electrode to make the proximity sensor sense the target object in a single direction."
"To make the proximity sensing unidirectional, place a hatch fill of 0.17 mm (7 mil) trace and 1.143 mm (45 mil grid) in between the sensor and the direction in which proximity detection should be avoided and drive the hatch fill with the driven shield signal."
AN92239 "Proximity Sensing with CapSense", Section 10 describes using a 17cm diameter copper trace (or wire loop) for large Proximity sensing distance
My main concern is that the proximity sensing distance is reduced by roughly half when I enable the shield electrode. I can't think why that would be.
Measure the raw counts and peak to peak noise of the raw counts with hatch pattern connected to ground /shield. With shield the peak to peak noise on the sensors will be more and noise immunity of the sensor will be decreased. I hope this may be the reason for very low SNR when shield is connected.
Apply IIR filter with different filter coefficients (start from 128 and decrease) and check the SNR, peak to peak noise and signal value at different heights. Please go through the following application note downloadable from the link below which tells you various steps to design a proximity based application.
If still you are not getting good SNR, you should go with Manual Tuning of CapSense. The guide for manual tuning of CapSense is attached here. In this document please go through the Lab examples 1 and 2 which has detailed steps for manual tuning.