You worry too much. 
32 GB will be just fine. As for Windows using half the RAM even with not much running, that’s what it does, and is what I have also found. This might help explain it - from Reddit - The heart of the internet
Hey OP, it’s normal for PCs to use around half of the RAM when in idle mode, even when nothing is currently running. That’s because Windows uses Superfetch, a program that increases the performance of Windows by pre-loading apps you frequently use into RAM before you open them. This is essentially a free performance boost, as otherwise, the extra RAM would be wasted. Don’t worry, the cache will empty itself out if the RAM is needed elsewhere.
The amount of RAM used by this cache can scale up or down depending on how much RAM you have, so adding more RAM will result in Windows using more. If you want to troubleshoot SuperFetch, follow these instructions to disable it.