My Emacs muscle memory is too stubborn, so that I always find myself pressing C-x o to switch between terminal and editor window. The pain ended today. To switch focus on VSCode with C-X o, add the following to your JSON config:

{
    "key": "ctrl+x o",
    "command": "workbench.action.terminal.focus"
},
{
    "key": "ctrl+x o",
    "command": "workbench.action.focusActiveEditorGroup",
    "when": "terminalFocus"
}

Note: This only switches focus between terminal and editor. Switching focus to other elements in the screen will require another keybinding.