Tracking a minifilter's ActiveOpens files

I've recently done a bit of detective work that I thought might be an interesting thing to share, especially since we've been talking about contexts so much lately. The issue was how to find the files that were opened by a minifilter which prevent that minifilter from unloading. FltMgr actually keeps track of the files that were opened by a minifilter and if the minifilter gets unloaded it will wait for those files to be closed before it unloads the driver. You can see this in the debugger here:

1: kd> !fltkd.filter 94130008 

FLT_FILTER: 94130008 "luafv" "135000"
   FLT_OBJECT: 94130008  [02000000] Filter
      RundownRef               : 0x0000000a (5)
      PointerCount             : 0x00000001 
      PrimaryLink              : [922c75a4-92cdaa24] 
   Frame                    : 92cda9c8 "Frame 0" 
   Flags                    : [00000006] FilteringInitiated NameProvider
   DriverObject             : 9306bc50 
   FilterLink               : [922c75a4-92cdaa24] 
   PreVolumeMount           : 81fbe0cc  luafv!LuafvPreRedirect 
   PostVolumeMount          : 00000000  (null) 
   FilterUnload             : 00000000  (null) 
   InstanceSetup            : 81fca62b  luafv!LuafvInstanceSetup 
   InstanceQueryTeardown    : 00000000  (null) 
   InstanceTeardownStart    : 00000000  (null) 
   InstanceTeardownComplete : 00000000  (null) 
   ActiveOpens              : (941300dc)  mCount=1 
   Communication Port List  : (94130108)  mCount=0 
   Client Port List         : (94130134)  mCount=0 
   VerifierExtension        : 00000000 

So the task is to find that one file that ActiveOpens is tracking for LUAFV. Before we go any further, let's see what the chain of structures looks like, starting from the FILE_OBJECT. Some of the structures are documented while others are not. I've marked the undocumented structures with a '?' at the end of the name. We don't know (or care for this post) what the other members of the structures are.

Now, we need to walk the arrows backwards so the steps we need to follow are:

1. From fltmgr!_FLT_FILTER->ActiveOpens find fltmgr!FO_CONTEXT?
   
2. From fltmgr!FO_CONTEXT? find the nt!FILE_OBJECT_CONTEXTS_HEADER? Structure
   
3. From nt!FILE_OBJECT_CONTEXTS_HEADER? Structure find the nt!_IOP_FILE_OBJECT_EXTENSION pointing to it
   
4. From the nt!_IOP_FILE_OBJECT_EXTENSION find the nt!_FILE_OBJECT structure that points to it
   
5. ???
   
6. Profit!!!!
   

Starting with ActiveOpens, let's look at the structure. Please note that mCount appears to be shifted by 1. Also, please note that mList is a regular doubly linked list and we expect that it contains one entry (since mCount is 1):

1: kd> dt 941300dc _FLT_MUTEX_LIST_HEAD
fltmgr!_FLT_MUTEX_LIST_HEAD
   +0x000 mLock            : _FAST_MUTEX
   +0x020 mList            : _LIST_ENTRY [ 0x93712498 - 0x93712498 ]
   +0x028 mCount           : 2
   +0x028 mInvalid         : 0y0
1: kd> dl 0x93712498 
93712498  941300fc 941300fc 0421000e 706e5043
941300fc  93712498 93712498 00000002 00000001
1: kd> !pool 0x93712498 2
Pool page 93712498 region is Nonpaged pool
*93712430 size:   70 previous size:    8  (Allocated) *FMfc
  Pooltag FMfc : FLTMGR_FILE_OBJECT_CONTEXT structure, Binary : fltmgr.sys

So now we know that the structure that we've been calling fltmgr!FO_CONTEXT? is in fact called FLTMGR_FILE_OBJECT_CONTEXT. Step 1 is done and we're moving on to step 2. We also know the size isn't larger than 0x70. However, we don't know where the LIST_ENTRY is in there. The only other thing we know is that in that structure there must be a member that is of type FSRTL_PER_FILEOBJECT_CONTEXT (and we know this because that's how FILE_OBJECT contexts are implemented; see the documentation for FsRtlInsertPerFileObjectContext). Since the beginning of the _LIST_ENTRY is at address 0x93712498 and the pool block starts at 0x93712430, we can guess our LIST_ENTRY is towards the end of the structure, so we'll go back a bit and display the words. Then we'll look for something that looks like an FSRTL_PER_FILEOBJECT_CONTEXT. FSRTL_PER_FILEOBJECT_CONTEXT doesn't really contain much easily identifiable information but it does contain a LIST_ENTRY, which means we should find two words that look like kernel mode addresses next to each other. I've highlighted possible candidates and then we simply try "dl" on them (if anyone knows a better way I'd love to hear about it, please leave a comment). Of course, if you look at the number carefully you can see that it's very likely that there is a LIST_ENTRY at 93712480 if the list has only one element. But in most cases the list has more than one element so it's hard to tell at a quick glance. So we simply issue a "dl" on each candidate and hope that if they're not doubly linked lists they'll simply run into some invalid address sooner or later.

1: kd> dp 0x93712498-0x60 
93712438  48706345 00000000 00000000 00000000
93712448  00ac5851 45e4702b 2a5794ac 7e7ac1fe
93712458  00000000 00000047 00000068 96040c00
93712468  00000000 0034f110 001d0000 92cda9c8
93712478  94134ce8 93011008 9306b0c8 9306b0c8
93712488  938f1230 00000000 94130008 941307d8
93712498  941300fc 941300fc 0421000e 706e5043
937124a8  937395a8 937126b8 937699d8 93739ad8
1: kd> dl 92cda9c8
92cda9c8  0340f103 960409f8 960409f8 00000000
0340f103  00000000 00000000 00000000 00000000
1: kd> dl 94134ce8 
94134ce8  00800005 92f05788 92f06870 92fa4998
…
1: kd> dl 9306b0c8  
9306b0c8  93712480 93712480 00010006 e56c6946
93712480  9306b0c8 9306b0c8 938f1230 00000000
1: kd> !pool 9306b0c8 2
Pool page 9306b0c8 region is Nonpaged pool
*9306b0a0 size:   30 previous size:   10  (Allocated) *FOCX
  Pooltag FOCX : File System Run Time File Object Context structure, Binary : nt!fsrtl

It looks like we may have found our structure. Now we're starting our step 3 and we have a pointer into a structure of type nt!FILE_OBJECT_CONTEXTS_HEADER?, but since we don't know the type we don't know where the structure starts. Normally what I do is assume it starts right after the pool tag and search for that address, and if that fails search for the address at the next word boundary and so on. Let's see:

1: kd> db 9306b0a0 L0x20
9306b0a0  02 00 06 04 46 4f 43 58-01 00 00 00 00 00 00 00  ....FOCX........
9306b0b0  00 00 00 00 01 00 04 20-00 00 00 00 bc b0 06 93  ....... ........
1: kd> s -d 80000000 L?0x20000000 9306b0a8
9306ba78  9306b0a8 00000000 00000000 00000000  ................
1: kd> !pool 9306ba78  2
Pool page 9306ba78 region is Nonpaged pool
*9306ba60 size:   30 previous size:   10  (Allocated) *Io  
  Pooltag Io   : general IO allocations, Binary : nt!io

So actually this looks pretty good. In other cases I've found multiple random values that looked like references so I've just had to look at each one. But in this case it looks pretty clean. We expect that the pointer is in a structure that's allocated by the IO mgr, and that the structure size something about the size of nt!_IOP_FILE_OBJECT_EXTENSION. So this means we've completed step 3 and we have the nt!_IOP_FILE_OBJECT_EXTENSION structure. In our picture I've shown that in the _IOP_FILE_OBJECT_EXTENSION it is FoExtPerTypeExtension[3] that points to this structure that we just found. I've figured this out by experimenting. I simply got a FILE_OBJECT and added a context by calling FsRtlInsertPerFileObjectContext and watched which value changed. So now we know where the structure starts and we need to search a pointer to it. The pointer we would find here would be a FILE_OBJECT->FileObjectExtension so we expect the pool tag to be "File". Also, we expect the FILE_OBJECT to start 0x7c bytes before the pointer. Please note that the first two results returned by the search were invalid (!pool told me so):

1: kd> ? 0x9306ba78-0x10
Evaluate expression: -1828275608 = 9306ba68
1: kd> s -d 80000000 L?0x20000000 9306ba68
82b80008  9306ba68 9306ba68 00000000 abcddcba  h...h...........
82b8000c  9306ba68 00000000 abcddcba 00000001  h...............
94134d64  9306ba68 04530015 6661754c 00000000  h.....S.Luaf....
1: kd> !pool 94134d64  2
Pool page 94134d64 region is Nonpaged pool
*94134cc0 size:   a8 previous size:  2d8  (Allocated) *File (Protected)
  Pooltag File : File objects
1: kd> !fileobj 0x94134d64-0x7c  



Device Object: 0x92f05788   \Driver\volmgr
Vpb: 0x92f06870
Event signalled
Access: Read Write SharedRead SharedWrite SharedDelete 

Flags:  0x440008
 No Intermediate Buffering
 Handle Created
 Volume Open

FsContext: 0x92fa4998 FsContext2: 0x97e0cbc0
CurrentByteOffset: 0
Cache Data:
  Section Object Pointers: 92f0cd14
  Shared Cache Map: 00000000


File object extension is at 9306ba68:

So this is it, we know now that LUAFV opens the volume using FltCreateFile (if it didn't it would be on the ActiveOpens list). For the record, this is win7:

1: kd> vertarget
Windows 7 Kernel Version 7600 MP (2 procs) Free x86 compatible
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 7600.16617.x86fre.win7_gdr.100618-1621
Machine Name:
Kernel base = 0x82809000 PsLoadedModuleList = 0x82951810
Debug session time: Thu Feb 17 07:44:10.641 2011 (UTC - 8:00)
System Uptime: 0 days 0:02:42.733

Filter Layering and IO Targeting in FltMgr - part II

Let's look at how targeting in filter manager can fail and what it looks like when it happens. I'd like to say that while such things do happen and I've analyzed a couple of cases over the years, they are far less frequent with minifilters than with legacy filters. Anyway, layering violations in some cases can just go unnoticed, but if they do cause trouble most likely what will happen is an infinite recursion which will result in a bugcheck. Deadlocks can also happen, but the cases I've investigated so far were all infinite recursions.

One way layering can fail is when a FILE_OBJECT is used in postCreate by a minifilter and the minifilter is not using Flt APIs to perform the IO. For example, using the setup from the previous blog post, if minifilter1 in postCreate calls a Zw function or creates a handle for a user mode app and lets the user mode app use that handle to do something like scan the file or recall it or decompress it, what will happen is that the requests generated by the user mode service or the Zw calls will go to Frame0 because the FILE_OBJECT stores the information about the device hint, but FltMgr will not find its targeting structure and will show the requests all minifilters in the frame starting with Minifilter 3. However, this is a clear violation of the minifilter rules because minifilter1 broke the layering contract by sending IO to the top of the IO stack. If it wants to do this sort of thing it needs to call its own FltCreateFile and after that create succeeds it can either use a Zw API or create a handle for that FILE_OBJECT to be used by a user mode service.

Now, if filter manager would always use a device hint things wouldn't be too bad. However, there is a case where FltMgr violates the rule of never sending IO to the top of the stack. This case is in the naming path (and more precisely in the FltpExpandFilePathWorker function). When a minifilter calls FltGetFileNameInformation and requests a normalized name, FltMgr gets a name for the file and then proceeds to normalize it. It does so by opening folders along the path to the file and querying information that contains the long name for each component on the path. In this case however, FltMgr does use a targeting structure to identify which minifilters should see the create, but it does not use a DeviceHint for the IoCreateFileEx call. The reason, as far as I can tell, is that the request will fail if the name contains a reparse point that reparses to a different volume (remember that bit about IopCheckTopDeviceHint a couple of posts back ?) so FltMgr just sends it to the top of the stack.

So in this case the IRP_MJ_CREATE is not targeted in the IO manager (it will go to the top of the IO stack) but there will be FltMgr targeting information attached to the IRP_MJ_CREATE. Looking at our picture from the previous post, we can see that if the IRP_MJ_CREATE issued by (or on behalf of) minifilter2 is one of these creates then Frame1, Legacy Filter B and Legacy Filter A will all see the IRP_MJ_CREATE and the subsequent requests. Frame1 will find the targeting information and infer that no minifilter in that frame should see the request and just send it below. However, Legacy Filter B and Legacy Filter A will not be aware that they shouldn't see this request and will perform their usual functions. So, since the targeting information has not been attached to the FILE_OBJECT yet (it will only happen when IoCreateFileEx returns to FltMgr) and there is no DeviceHint on the FILE_OBJECT if Legacy Filter B issues any requests to the device below them, then not only will legacy filter A see that request (which is expected), but also when the request reaches Frame 0 all the minifilters will see it. But minifilter3 and minifilter2 should not have seen it an in fact they haven't seen the IRP_MJ_CREATE as well so there are quite a few things that can go wrong here.

Another interesting case I've seen was when a legacy filter (let's use Legacy Filter B again as an example) tried to implement something similar to FltReissueSynchronousIo in the minifilter world and in its postCreate, if the IRP_MJ_CREATE failed, it changed something in the request and sent it down again. This worked well on Vista and newer but in XP it failed. As you remember, in XP the targeting information is stored in an EA, and for whatever reason the EA mechanism was designed such that the structure is associated with an IRP_MJ_CREATE by storing the EaLength in the IO_STACK_LOCATION and the EA buffer in the IRP (Irp->AssociatedIrp.SystemBuffer). This is in my opinion a pretty poor design, because it suggests that the EaLength can be layered but the EA buffer cannot. It also means there is only one EA buffer per IRP and so FltMgr must use EA chaining. When FltMgr receives such a create it needs to remove its EA information from the buffer before it sends the request down. However, once the EA buffer has been changed if a legacy filter sends the request down again in the manner we explained then FltMgr (in Frame0 in our example) will not find the targeting information and will therefore show the information to all the minifilters in the frame, which can also result in a layering violation. Moreover, the EaLength and the EA buffer are now out of sync and the file system might not like this. For a very clear example of this issue looks like, please read this thread: http://www.osronline.com/showthread.cfm?link=187295. Please note that in this thread though we're not dealing with recursion but with the file system not being able to cope with the EaLength and the EA buffer being in an inconsistent state. Though infinite recursion could still have happened if there were more minifilters installed on the system.

Before ending this post, I'd like to point out that pretty much all layering issues require a legacy filter in the picture. In general FltMgr by itself with just minifilters is pretty good about it. Please note that even perfectly written legacy filters could trigger this issue (which is another way of saying that it isn't the legacy filter's fault), the real problem is that FltMgr breaks layering. What I would like FltMgr to do (in fact, what I wish it had done already) is to offer an API to legacy filters by which such a filter can tell whether a certain IRP or FILE_OBJECT is one they should ignore. Also, I think that FltMgr could address a large class of issues very easily by simply moving the targeting information to the FILE_OBJECT immediately after the IRP_MJ_CREATE completes in the file system.

Filter Layering and IO Targeting in FltMgr

I've been talking about layering quite a lot on this blog. I've also mentioned how FltMgr performs IO targeting when a minifilter calls FltCreateFile in this post and how after such a FILE_OBJECT is created, targeting works even when using Zw apis in this post. However, let's take a more closer look at how it actually is implemented in FltMgr and what are some of the implications of the design.

As it might be apparent from the previous links, there are two different kinds of targeting going on. The IO manager targeting that directs an IRP at the appropriate device and FltMgr's targeting, which identifies the appropriate minifilter for that operation. Please take a look at the following picture, where the blue blocks represent devices (FltMgr's Frame0 and Frame1 and the attachments for the two legacy filters are all devices) and the red blocks are minifilters. The picture shows how an FltCreateFile request goes to the IO manager, and how then it find the minifilter below the one issuing that call.

The steps involved in this are as follows:

1. Minifilter2 calls FltCreateFile
   
2. FltMgr allocates targeting information (fltmgr calls it TargetedIoControl) and inserts it into an ECP structure and then it calls IoCreateFileEx with the ECP and a device hint that points to the device for Frame0.
   
3. IoCreateFileEx goes through the usual steps in the OB manager, the OPEN_PACKET is initialized, and it eventually gets to IopParseDevice
   
4. IoMgr in IopParseDevice allocates an IRP_MJ_CREATE , attaches the ECP and sends it directly to the hint device, Frame0.
   
5. FltMgr get's the IRP_MJ_CREATE on the device for Frame0, looks for the targeting ECP and extracts the TargetedIoControl from it and then it analyzes it and figures out that the first minifilter that should see this request is Minifilter1.
   
6. Minifilter1's preCreate callback get's called.
   
7. the IRP_MJ_CREATE is processed further by the IO stack, file system and so on
   
8. the IRP_MJ_CREATE completes to the IO manager
   
9. IoCreateFileEx returns to FltMgr
   
10. the original call to FltCreateFile returns control to Minifilter2
    

Please note that things are a bit different in XP, for example instead of an ECP FltMgr uses an EA and instead of IoCreateFileEx FltMgr calls IoCreateFileSpecifyDeviceObjectHint. I will only focus on the behavior in Vista and newer releases but XP should be pretty similar anyway.

Let's take a look at how IO manager's targeting is implemented. Each FILE_OBJECT structure has something called a FileObjectExtension and there are some functions in the IO manager that can set things in the extension. Please note that this is not the same as the FILE_OBJECT context support added in Vista (and which is available throug APIs like FsRtlLookupPerFileObjectContext and friends).:

0: kd> dt nt!_FILE_OBJECT
   +0x000 Type             : Int2B
   +0x002 Size             : Int2B
   +0x004 DeviceObject     : Ptr32 _DEVICE_OBJECT
   …
   +0x07c FileObjectExtension : Ptr32 Void

0: kd> x nt!*Extension*
...
828b1d09 nt!IopAllocateFileObjectExtension = 
...
828a40e2 nt!IopGetFileObjectExtension = 
...
828c8fe7 nt!IopSetTypeSpecificFoExtension = 
...
82a686f0 nt!IopDeleteFileObjectExtension = 
82aa3238 nt!IopSymlinkSetFoExtension = 
...
82a6c1e7 nt!IopAllocateFoExtensionsOnCreate = 

So these extensions are of different types, for internal use by various OS components. The interesting function here is nt!IopAllocateFoExtensionsOnCreate which initializes some extensions whenever a FILE_OBJECT is initialized. For example, if a DeviceHint was specified then some specific extension is allocated and then the IO manager will always use that extension on the FILE_OBJECT to figure out which device an IO request needs to be sent to. So IO manager's targeting information is associated with the FILE_OBJECT immediately upon creation.

FltMgr takes a different approach. For one, it is not involved in FILE_OBJECT creation and so it doesn't know when the FILE_OBJECT is created. So the approach it takes is a bit more complex. Looking at the steps above associated with the picture above, in step 9 FltMgr now takes the TargetedIoControl structure that was associated with the IRP_MJ_CREATE and associates it with the FILE_OBJECT, before returning from FltCreateFile. In fact, the flow in FltCreateFile looks something like this:

1. Allocate TargetedIoControl
   
2. Call IoCreateFileEx with the DeviceHint pointing to the current FltMgr device and the TargetedIoControl
   
3. When IoCreateFileEx returns, if the create was successful, associate the TargetedIoControl with the FILE_OBJECT.
   

This algorithm is also employed in cases where FltMgr needs to open a file itself (mostly in the Naming code) because it doesn't internally call FltCreateFile and instead it simply follows these steps. In fact let's take one more look in the debugger at the functions in FltMgr that are associated with targeting:

0: kd> x fltmgr!*target*
96050f68 fltmgr!FreeTargetedIoCtrl = 
960394b6 fltmgr!FltpGetIoTargetFromFileObject = 
...
96051114 fltmgr!TargetedIOCtrlGenerateECP = 
9605132e fltmgr!TargetedIOCtrlAttachAsFoCtx = 

So as you can see, we have a function to add a TargetedIoControl as an ECP (undoubtedly for the IRP_MJ_CREATE case) and as a FILE_OBJECT context (after the IRP_MJ_CREATE is complete), as well as a function to get the target of an IO operation from the FILE_OBJECT context (fltmgr!FltpGetIoTargetFromFileObject). There doesn't seem to be a function that figures out the target from an ECP so that's probably only handled inline.

The really important thing to note here is that this mechanism is different from the IO manager mechanism in that the FILE_OBJECT doesn't have FltMgr's targeting information that should be associated with it until after IoCreateFile returns. So for a fair bit of time, between the moment when the IRP_MJ_CREATE is completed by the file system (and when the FILE_OBJECT becomes initialized) and the moment when the IoCreateFileEx call returns to FltMgr, the FILE_OBJECT is initialized but it doesn't have any FltMgr targeting information (it does however have IO manager's targeting information). We'll discuss the implications of this particular approach (and the whole class of issues it introduces) in the next blog post, as well as a couple of various different approach FltMgr could have used.

More contexts: tracking hardlinks

In one of the comments to my previous post, Lyndon pointed out that there is not a lot of support from either the OS or FltMgr when it comes to tracking hardlinks. So I figured I'd explain why this is so complicated and explain what a filter would need to do to implement this. I'm not going to describe what hardlinks are or how they operate, focusing instead on what FltMgr does and what a filter might need to do as well.

However, there is one specific particularity about hardlinks that i'll keep referring to. Once a file is opened the file system remembers which link was used to open the file and it will return that name when querying the file name. If that linked is renamed, the FS will of course return the new name.

So the problem with hardlinks is, like Lyndon pointed out, that the SCB model isn't granular enough. The SCB is associated with the stream and it doesn't really matter how the stream was opened (by which name), the SCB is the same. So a StreamContext is the same, regardless of how many hardlinks were used. On the other hand, StreamHandleContexts are too granular, in that they simply track the FILE_OBJECT and different opens even from the same link (using the same name for the file) will obviously get different FILE_OBJECTs and thus different StreamHandleContexts.

Filter manager doesn't offer an additional type of context. However, it does need to deal with hardlinks because it implements a name cache. The name cache is pretty simple to implement for files that only have one name, the name is stored in a structure associated with the stream. However, for hardlinks, clearly the structure needs to be different so that opens for the same name are cached properly. FltMgr solves this problem by not caching the file name in a structure associated with the SCB if the file has more than one link (as reported by the FileStandardInformation information class) and instead it caches the name per FILE_OBJECT.

If a filter wanted to keep track of hardlinks it would need to, as Lyndon indicated in his comment, look at the name that it gets from the file system (the FileNameInformation class) and from that deduce which link was used. This is complicated because a link can be renamed at any time so that must be taken into account. A possible implementation would need to keep some structure in a perStream context that would map each FILE_OBJECT to a link (possibly introducing an artificial concept like linkID or linkGuid or something) and in postCreate would map the newly opened FILE_OBJECT to the appropriate link (which requires looking at link names using the FileHardLinkInformation class) while disabling renames for that stream.

I was planning on writing more on this topic and playing with hardlinks some more, but I'm busy at work and it'll have to wait for a future post.