Extreme Lag When Using Xbox Controller

Whenever I use my Xbox One controller to play the game on Steam, it causes extreme FPS lag. Sliding, tearing, audio distortions, skipped frames, the whole kaboodle. It doesn't matter where I am or what I'm doing, the lag is the same degree of awful. I have Ultra Performance Mode turned on and am playing Nightingale in Borderless Windowed mode. However, everything is fine if I simply switch back to mouse and keyboard. I greatly prefer using a controller to play a game if it's an option, but the lag that comes with using my controller is outrageous. I can still play, but not comfortably; mouse and keyboard causes a lot of pain in my wrists.

The odd thing is that when I started the game I got very reasonable frames (80+) on controller. Now, however, it seems if I lag with every single controller action.

Attached is a short clip illustrating the problem. The lagging parts are only when I pick up and use my controller.