Something I apparently never knew (or maybe this is just a newer feature), is that Godot defaults Audio playback to "Sampled" for web exports- which for some reason prevents audio files from looping properly. For anyone else running into this issue, changing it to "Stream" fixes the issue. (Relevant setting is audio/general/default_playback_type.web).