yeah you need to mess around with snappiness to avoid that
(so either increase it to make it unload stuff to hdd earlier, leaving more ram for critical stuff, or decrease it to like 5 in order to make it only use swap when absolutely needed) basically just mess around with it and see what value works best. i my case 10-15 works more or less okay-ish under full ram load