This kind of gets into the whole question of secrets management. At $WORK we use software that distributes secrets to hosts that have permissions for them; thus, we can set a policy like “Please ensure the CI builder has access to foo.pem”, then set up a job there that grabs the key file from wherever the software dropped it. If your CI builder is managed via Chef, you might use encrypted data bags to do this.
One thing we’re doing: in order to avoid writing secrets to disk, we’re using /dev/shm on our CI builder to hold the files in memory – no persistence on the hard drive. That’s a Linux-ism, of course, and I don’t know what the Windows equivalent would be. In case of a reboot, we need to wait for the secret distribution service to sync up, but for us that’s an acceptable tradeoff.