..as soon as I mount the antenna close to the espgw board I get significant noise.
Ooh, that's not good. Only on receive or on tx as well?
1.) I decided to use a balun with the dipole to try and get better noise immunity. I settled on the HHM1522A7 (https://product.tdk.com/info/en/documents/data_sheet/rf_balun_hhm1522a7_en.pdf) for 868 mhz. The reason I think it might help is that without it the outer surface of the coax and the ground plane of the board acts as third leg of the antenna, distorting the radiation pattern and providing an easier path for ground noise pickup.
You got it. The GND tracks of your espgw board WILL form part of the antenna, the trick is to a) minimise the amount of noise the antenna picks up from your board when it's receiving and b) minimise the effect of a radiating antenna on your board.
Also consider 'unintentional' antennas such as powersupply leads and other wires. At these frequencies almost anything acts as an antenna. It would be interesting to see if you can get even better immunity by adding small toroids to wires and by metal-shielding your espgw board.
The balun you found adds some losses but definitely helps to isolate common mode current caused by the (hopefully small) asymmetry of the dipole. I think you're on the right path, I just chose not to include this component in my design. However, I'd love to see some numbers when you've finished testing with the balun. It could be a great addition for those who want to experiment with antenna designs and I think it's a worthwhile path of investigation.
2.) I'm going to mount the dipole perpendicular to the PCB to keep the orientation of the fields the dipole can pick up perpendicular to any noise the esp might generate. To do so I have a cutout at the end of the PCB into which I'll insert the dipole. Then I'll solder onto the SMA pads on both sides. So I appreciate their presence in your design.
Are you saying that initially the coax feedline was not straightened out but maybe somewhat curled up? Then what you're seeing is expected behavior. Dipoles are very prone to picking up noise within a short distance of it. Not sure if this is purely within the near-field or just within the first half-wavelength.
Another point of interest is that, even with an isolation transformer, if your coax is not straight and perpendicular to the dipole for at least a half wavelength, it will very easily pick up rf on the outside of the coax when the antenna is transmitting.
However, if you're going to mount your espgw right at the feedoint of the dipole you may be asking for trouble. Without any shielding, you're putting your board right in the 'hottest' field of the antenna. I'm not saying it can't be done but if you're already having noise problems *with* a feedline, putting the gw board right on top of the dipole may make it worse. For your sake I hope you can prove me wrong! haha.. :-)