The one such issue I'd like to talk about in this post is one that can be pretty hard to identify but not that hard to fix (or work around). My hope is that if developers are more aware of the problem they might have an easier time identifying it. The problem is that some applications (mostly user mode but it could happen for kernel drivers as well) occasionally open handles for asynchronous IO and then forget about this and perform IO on the handle as if it is synchronous (or maybe the developers aren't certain how to use a handle that was opened for asynchronous IO ). Anyway, what happens is that the application looks something like this:
HANDLE hFile;
hFile = CreateFile(szFileName,
GENERIC_READ,
0,
NULL,
OPEN_EXISTING,
FILE_FLAG_NORMAL | FILE_FLAG_OVERLAPPED,
NULL);
...
// use the handle for some asynchronous IO
…
// perform some operation on the handle
If (GetFileInformationByHandle( hFile, ..)) {
// do stuff with the information
}
Please note that I'm using GetFileInformationByHandle() here just as an example, any function that works on a file handle could be used instead. Anyway, the problem in some cases is that the function might not be working as expected for asynchronous handles. For our example, GetFileInformationByHandle() is simply a thin wrapper over NtQueryInformationFile and it simply calls it and if the status it gets is NT_SUCCESS(status) then it thinks the IO was successful. However, if the handle has been opened for asynchronous IO it is possible that NtQueryInformationFile will return STATUS_PENDING, which is a success status (so NT_SUCCESS(STATUS_PENDING) == TRUE) and so the caller of GetFileInformationByHandle() might think the operation has completed successfully even though it hasn't. The result of this in our case is the fact that the application might be working an uninitialized buffer.
Of course, most such bugs will be caught during testing but (as always) there is a twist. There are some operations for which the file system never returns STATUS_PENDING and instead it just completes the operation in-line. In fact the file system is generally pretty conservative about returning STATUS_PENDING because doing it introduces additional overhead (there must be another thread that will be used to complete the request) and so the scenario where some operations are completed inline is rather common. This means that during testing the application will never get STATUS_PENDING and so the code that calls GetFileInformationByHandle() always works well and so the potential problem is never found. Driver Verifier has a mode in which it will return STATUS_PENDING for most IO but then not many application developers run with Driver Verifier enabled. I'm not sure whether Application Verifier offers something like this, but it's worth looking into.
This is where file system filters enter the picture. File system filters might return STATUS_PENDING in cases where the file system won't do that. In fact, there is one file system in particular that marks IRPs pending all the time: FltMgr. Filter Manager has simplified the IO model for minifilters quite a lot. Pretty much all the complexity of passing IRPs down and dealing with the completion routine is hidden from minifilters, but there is a cost. In FltMgr's model a filter can decide that it needs to pend a request during a postOp callback. In the NT model however this is not possible, the IRP must have been pended already in order for a completion callback to be able to return STATUS_MORE_PROCESSING_REQUIRED. Since FltMgr doesn't make the filter decide during the preOp whether it will pend the request during the postOp callback (in fact FltMgr allows a filter to not even have a preOp callback), it proactively marks the IRP pending, in case the minifilter will want to return FLT_POSTOP_MORE_PROCESSING_REQUIRED.
If the explanation above seems convoluted, the short version is this: FltMgr returns STATUS_PENDING a lot more than the file system. So this is where the problem shows up. Someone installs an innocent looking minifilter and suddenly some application starts failing in a strange way (if you're lucky; the unlucky case is the one where application starts slowly corrupting data…). The investigation shows that the presence of the filter clearly has something to do with the problem, but it's impossible to pin down what the filter is doing wrong. In some cases even disabling all filter functionality still somehow manages to reproduce the problem (and of course the reason is that the bug is in the application and the mere presence of the filter makes FltMgr return STATUS_PENDING and expose the bug).
So how can this be fixed ? In most cases there is a very simple fix, the filter must replace FLT_PREOP_SUCCESS_WITH_CALLBACK with FLT_PREOP_SYNCHRONIZE for the operation that exhibits the problem (for our example, IRP_MJ_QUERY_INFORMATION). This has the effect of making the operation synchronous at FltMgr's level and so NtQueryInformationFile() won't return STATUS_PENDING anymore. Of course, from a filter perspective the problem is identifying the operation that is causing the problem, but that can be done by replacing FLT_PREOP_SUCCESS_WITH_CALLBACK with FLT_PREOP_SYNCHRONIZE one at a time until the problem no longer occurs. Please note that it's possible that there is more than one operation that needs this so there may be more than one replacement necessary for the app to start working. In my experience it's easiest to start by replacing all the return statuses and then reverting them one at a time. Of course, this is a hack around the real fix (which is fixing the application) but since in most cases the filter developer doesn't have the luxury to tell the app developer to fix their problem this approach can be quite handy.
Now, this problem has largely been addressed by now in most apps since FltMgr has been around for a while and since Vista there are some minifilters running by default on the system and so application developers have mostly fixed their stuff, but it's possible that one still runs into this issue.
Finally I'd like to show you how this might look in practice. This thread shows how this problem might appear to a developer. Also, see KB article 939767 and KB article 2009604 and KB article 970878, all of which are examples of applications doing this sort of thing.
Update: So I was a bit uncomfortable about not being able to remember if there are any specific Nt APIs that would leak STATUS_PENDING so I spent some time looking at some of the APIs. In most cases (including in the NtQueryInformationFile() function from my example) if the handle was opened for async IO and if the IO manager returns STATUS_PENDING the function will wait for the API to complete by waiting on a event. So the Nt APIs are well behaved, but applications might still try to perform reads and writes on a async handle treating it like a synchronous one. For example KB 970878 that I mentioned above states that the problem comes from using BackupRead() and BackupSeek() on an asynchronous handle (and the documentation for BackupWrite() states that it also requires a synchronous handle so it likely would exhibit the same behavior). KB article 939767 also explains how this could happen for a kernel driver (and in fact it's more likely to happen for a driver if they roll their own IRPs (or the minifilter equivalent of using the FLT_CALLBACKDATAs) instead of going through the Nt/Zw APIs since as we've seen those are careful about this sort of thing). Another thing i've been meaning to say but I forgot until Jeho pointed it out (thanks Jeho!) was that the easiest way I found to check for this scenario is to use the Passthrough sample. It filters all APIs and it has completion routines for all of them, so FltMgr will be doing a lot of pending. Moreover, it's easy to just hand it with the source and everything to the app developer (where possible) and tell them how to reproduce the problem and since it's also coming from Microsoft it carries potentially more weight. However, when using this approach make sure to change the altitude of the sample because if the problem exists in a filter in the file system stack that is lower than the sample it won't reproduce the problem.
Posts
