How to intercept key presses in the Visual Studio text editor

One common question from add-in/package authors is how they can hook on to keystrokes performed at the editor. Everyone wants this for a different reason (a shortcut typing editor enhancement, acting upong the position where the cursor is, myself I wanted it in order to emulate vi/vim within Visual Studio).

The short answer is: VS has no built-in support for this. The long answer is: if you really want to, you can.

Editor windows convert keypresses they receive into OLE commands, and the commands are sent to the IVsTextView COM object that handles the editing window. IVsTextView does have a mechanism by which you can intercept any OLE command (these command involve anything from ECMD_TYPECHAR, which is issued when you press an alphanumeric key, to ECMD_RIGHT, which is processed when you press the cursor-right key, and all the way to ECMD_SHOWMEMBER_LIST which is called when you press Ctrl-Space.

The first question is how to get to a IVsTextView object. If you want to intercept all IVsTextViews in the environment, the best way is to register with the text manager in order to receive IVsTextManagerEvents::OnRegisterView() notifications. This will be called for every new view created: for example, when a new text file is opened, but also when the user splits a code view and a new IVsTextView is created for the split view.

Once you have the IVsTextView, the main "proper" interface that allows you to intercept stuff which happens to the view is IVsTextView::AddCommandFilter() to get a first-look at OLE commands. Then, when a key is pressed, a ECMD_TYPECHAR will be generated and passed through your command filter before getting to the actual code that inserts the character. You can block the message, not pass it through to the next handler, and then you will have prevented its operation. You can also perform some additional operation in addition to or instead of the original behavior.

One problem that you will find is that you don't really receive key presses, but OLE commands translated using the global and text-editor keyboard shortcuts table (Tools|Options|Keyboard). This may be a problem or not depending on your needs. Another problem is that you won't get mouse messages, etc...

So, your next option is to subclass the actual editor window - that way, you can get all WM_ messages and do anything with them. It seems really simple, as IVsTextView has a GetWindowHandle() method that returns the HWND of the window.

The first hurdle, though, comes when you try to get it in OnRegisterView(): it turns out the method returns 0, I guess because the window is yet not created or not stored with the IVsTextView.

But, what you can actually do is to add an OLE command filter to the IVsTextView object, and then use the filter to call GetWindowHandle(). And it works, as that function will return a valid HWND when processing later OLE commands. There is at leat one other much more hackish method to get the HWND at creation time, involving the Win32 API to find the top level window and then using FindWindow(), but the OLE command filter method seems quite reliable and only uses documented facts.

Once you do that, you are on your own to doing whatever you want with all types of messages. But this is only the beginning, as you will be learning about VS's internal message processing behavior as you try out things. You will find that the message proc doesn't receive many WM_KEYDOWNs and WM_CHARs you may be expecting, such as Ctrl-F, and the reason is in the way the input processing architure of VS works. Visual Studio, probably in the main message pump, checks keypresses against the global and current editor's shortcut table, and if it finds a match, it issues the OLE command from there. This means that messages such as WM_KEYDOWN with VK_DELETE never reach the editor window's message proc. You need to watch out for the OLE command guidVSStd97:cmdidDelete if you really need to know when the delete key has been pressed (many of the basic mappings are not visible in Tools|Options|Keyboard page).