Thursday, February 11, 2010

Issuing IO in minifilters: Part 2 – Flt vs. Zw

Sorry about the frequency of my posts, i’m been really swamped with the IFS Plugfest preparations.

Anyway, let’s get down to business. So now the way the create path works should be clear. The basic idea is that FltMgr has some targeting information (in which it stores the minifilters below which altitude should see the operation). In the create path the information is stored in an ECP (or, pre-Vista, an EA).

After the create operation has completed successfully, the targeting information is taken from the ECP and moved to a FILE_OBJECT context (private to filter manager). This allows filter manager to always figure out where IO to a file object should be targeted. This is very important. It means that once a filter opens a file with FltCreateFile, filter manager will always target the IO properly, at the minifilter below the one that created the file.

Another piece of the puzzle that I should mention is that IO manager uses a similar mechanism to the one filter manager uses to remember the device that was specified as a hint when a FILE_OBJECT was created. This information is used in IoGetRelatedDeviceObject to figure out which device the IO should be targeted at.

Finally, I need to address one very important concept: where a minifilter is sitting on the IO path. There are a couple of cases that are best described by example:

  1. A minifilter, being a device driver after all, can register for some notifications such as IoRegisterShutdownNotification or PsSetLoadImageNotifyRoutine. When these callbacks are called, the minifilter is not on the IO path at all. It should behave like any other driver in the system. So if it needs to do stuff like write something into a log it should issue IO to the top of the stack, using APIs like ZwCreateFile, ZwWriteFile and so on.
  2. Expanding on the example above, a minifilter can filter some volume (like the system volume) and write something (like a log) to another volume. In this case, while the minifilter is on the IO path on the volume it filters, it is not on the IO path for the volume it wants to write to. So, like in the case above, it might issue IO to the top of the stack on that other volume using APIs like ZwCreateFile, ZwWriteFile. This is good because it doesn’t require the minifilter to attach an instance to that volume. The two device stacks are pretty much unrelated.
  3. However, there are cases where a minifilter wants to write to the same volume. As we’ve already established that reentering the IO stack at the top is evil and the wrongdoers should be sent to Azkaban, the minifilter writer has no choice but to issue the IO below itself on the same stack. This is where the minifilter has two options:
    1. Use the user’s FILE_OBJECT. I use the term “user’s FILE_OBJECT” to refer to the FILE_OBJECT in FltObjects->FileObject, which might in fact be a FILE_OBJECT created by a filter above, not necessarily coming from the user. Anyway, the only way to do this is to call APIs that take an instance, so that filter manager can target it properly: FltWriteFile, FltQueryDirectoryFile and so on. Alternatively, feel free to allocate a CALLBACK_DATA using FltAllocateCallbackData, set up the parameters and then call FltPerformSynchronousIo. Filter manager will show the IO to the minifilters below that instance and then, if none of them complete the IO, it will allocate an IRP and send the request below. IO manager does not get involved in this at all (i mean, filter manager will call IoAllocateIrp and IoCallDriver, but the targeting logic in IO manager will not be used).
    2. Call FltCreateFile and get its own FILE_OBJECT. Once this create completes successfully, the minifilter can use Flt APIs or Zw APIs. The reason Zw APIs work well is because IO manager will use IoGetRelatedDeviceObject which will send the IO directly to filter manager’s device and then filter manager will find the targeting information associated with the FILE_OBJECT and will send it to the proper instance.
  4. Finally, there can be a case where a minifilter might decide to issue IO below its instance on a different volume (so the minifilter is injecting IO into the IO path on a different volume). Since using the user’s FILE_OBJECT is clearly not an option, it will need to create a new FILE_OBJECT for that second volume. However, to target below its instance it needs to call FltCreateFile (which takes an instance). However, from that point on, it can call either the Zw APIs or the Flt APIs on that file object and the operations will be targeted correctly.

Of course, there are many ways to implement something in a minifilter and both the IO model and filter manager allow for a lot of flexibility. So the list below should be treated more as a guideline and not like strict rules. Finally, here are the guidelines:

  1. Anytime you find yourself trying to call an API that requires an instance and you don’t have one, don’t try to hack something up. You’re probably not supposed to call that API from in that context.
  2. If you want to use the user’s FILE_OBJECT, you must use send the IO below yourself and you must use Flt APIs. If there isn’t one you can build your own with FltAllocateCallbackData and FltPerformSynchronousIo. However, using Zw APIs in this context will cause reentrancy.
  3. If you have your own FILE_OBJECT (meaning it was created with FltCreateFile), you can use either Flt APIs or Zw APIs. You don’t have to stick with one set, you can mix & match, depending on which provides more value.
  4. If you want to issue IO on a different volume, you can either use ZwCreateFile or FltCreateFile, depending on your design (an important factor in the decision is whether you have an instance on that volume). However, once the FILE_OBJECT and handle are created, the logic in the rule above applies.
  5. If you are completely outside the IO stack in some cases, you might be better off sending IO to the top of the stack (in those cases). What I have in mind here is a minifilter that does a bunch of other stuff like registering for PsSetLoadImageNotifyRoutine.
  6. If you need to issue IO to a different volume and you don’t have an instance to call FltCreateFile, it’s probably not a good idea to attach to that volume just to have an instance to pass into FltCreateFile. ZwCreateFile might be what you need. This is not a rule, you can create an instance just to inject IO in a different stack, but think hard about it before you do. It might not be necessary.

I hope this makes sense, but feel free to post any questions.

15 comments:

  1. First let me say thanks for the blog, I found the info very useful.

    My mini-filter needs to write to an output file on a volume that is being filtered. So as you pointed out I need to open the file using FltCreateFile() and pass in the appropriate instance object. But how do I get that instance object? My mini-filter automatically attaches to all volumes at startup. The only solution I can think of is to call FltEnumerateInstanceInformationByFilter() and search for the instance that matches the volume where I want to create the file.

    Is there an easier way? I guess what I'm hoping for is FltGetInstanceFromFileName(). This API doesn't exist, but is there another simple way of doing this?

    ReplyDelete
  2. I'm not sure I understand. Do you intercept IO for a certain file (or any file) and want to write to your log file on the same volume where the file is ? In that case you can get the instance from the FLT_RELATED_OBJECTS structure that you get as a parameter in your preOp and postOp callback. In there you can find various objects related to the current operation (the volume, instance and FILE_OBJECT and so on).

    If you simply have a file name and want to figure out the instance attached to that volume, then there are some issues. From my other posts you have probably figured out that since a minifilter can virtualize the actual location of a file, some concepts like "file name" and "volume a file belongs to" are dependent on the altitude. However, assuming that that's not the case and you have a kernel path for the file then you can get the FLT_VOLUME structure for that volume using FltGetVolumeFromName and then call FltGetVolumeInstanceFromName and get the instance of your filter associated with that volume. Though if you decide to use this approach you should probably review your architecture and make sure there is no better approach...

    ReplyDelete
  3. Thanks for the quick reply. Let me try and explain better. I'm monitoring all disk activity on all volumes, for example let's say that's the volumes C: and D:. I'm then writing some output to a file, let's say C:\output.dat. When I'm calling FltCreateFile() to open C:\output.dat I need to pass the Instance handle for the C: volume.

    It sounds like I need to call FltGetVolumeFromName() for C:, then call FltGetVolumeInstanceFromName() to get the Instance handle, and finally call FltCreateFile().

    Later when I call FltWriteFile() I need the FILE_OBJECT. I believe I read somewhere I can get this by passing the handle from FltCreateFile() into ObReferenceObjectByHandle().

    ReplyDelete
  4. If you are following disk activity then you are disk filter and not a minifilter. So you should use the Zw APIs for this (ZwCreateFile, ZwWriteFile and so on). These operate with handles so you should have no need for a FILE_OBJECT. However, ObReferenceObjectByHandle is the right function to call for getting a FILE_OBJECT from a handle.

    This is the type of thing i meant when i said in the post above that "Anytime you find yourself trying to call an API that requires an instance and you don’t have one, don’t try to hack something up. You’re probably not supposed to call that API from in that context."

    Anyway, there is a discussion list that is more appropriate for this type of discussion, http://www.osronline.com/cf.cfm?PageURL=showlists.CFM?list=NTFSD.

    ReplyDelete
  5. Thanks for the link, I'll go check that link out. Forgive me if this is a dumb question, but what's the difference between a disk filter and a mini filter? According to Microsoft "A filter driver developed to the Filter Manager model is called a minifilter." Since I'm using the Flt APIs wouldn't that make my filter a "minifilter." In which case wouldn't I need to use FltCreateFile() instead of ZwCreateFile()?

    Again forgive me, I've been programming Windows for 15 years, but this is my first ever device driver so I'm learning a lot as I go. Thanks.

    ReplyDelete
  6. I've published a new post on what the windows storage filtering space looks like, i hope it will help answer your question. http://fsfilters.blogspot.com/2010/10/filtering-in-windows-storage-space.html.
    Basically, if you filter disk activity (by which i understand stuff like sectors being written, flushes, power state transitions, head movement and so on) then being a minifilter won't help you at all.

    The post above deals specifically with the misconception that a minifilter must always use Flt APIs instead of Zw ones. Flt APIs should only be used when necessary (again, as explained in the post above). Moreover, one can use Flt APIs without being a minifilter at all.

    I know where you're coming from, device drivers are a very different beast from regular windows programming. Good luck!

    ReplyDelete
  7. Thanks for the great post – lots of useful information!

    My driver intercepts open operations (pre-IRP_MJ_CREATE) and handles copying files down on-demand. I realize that the driver needs to use FltCreateFile() so we don’t send the open to the top of the stack. However, I’m not clear as to what’s the best approach to take regarding the actual file copy.

    I’ve already written a kernel-mode file copy routine but I would think this is better off being down by our user-mode service. However, I need to insure the service file i/o for this file doesn’t go to the top of the stack since the driver is already waiting on the open. I can’t send the file handle from the FltCreateFile() call as the handle is invalid in user-mode.

    Should we (driver and service) essentially use FltSendMessage and FilterReplyMessage to pass a buffer back and forth? -> Driver opens the local file for writing and the service opens the network source file for reading, but that would seem sloooow as it may require many messages depending on the size of the file. Plus, what’s a good size buffer to use? 1k buffer would be pretty small, imagine copying a 230MB (Or more) file!

    Or should the driver handle the copy and just send progress updates if necessary? Plus the user-mode service would store state information so the next request won’t require another copy.

    ReplyDelete
  8. The recommended approach in general is to do as little as possible in the kernel mode driver and use a helper user mode service for work whenever possible.
    You don’t HAVE to do this in a layered way. For example, you could configure your driver to not block any requests from your user mode service and so your user mode service could simply create the file and write into it without interference from your driver. This way all requests would traverse the full stack and you don’t need to worry about which layer sees what.
    Alternatively, you can share the handle you created with a user mode service by creating a handle in the user mode process (using the undocumented ZwDuplicateObject or the documented but more limited ObOpenObjectByPointer).
    In either case you won’t need to use the filter communication ports for anything other than a control channel.

    ReplyDelete
  9. Thanks for the quite reply!

    Presently we essentially do what you suggested as our driver ignores any file operations by our user-mode service - It sends an init msg that provides the driver with its PID and we ignore any operations from that PID.

    However, that's still going to lead to a deadlock I believe - which is what's causing this post to begin with.

    We'll be blocking an AV filter (We're lower in the stack) while we wait to hear back from our service. The AV filter will see our service I/O and the AV filter is blocking our service waiting for our driver!

    Also, I once tried passing along an access token handle using ObReferenceObjectByHandle along with ObOpenObjectByPointer. Even though I created a user-mode handle from the kernel-mode handle, when I send the message with the handle, the user-mode service receives it in a different process and thus the handle is invalid. I would think this would be the same with the file handle from FltCreateFile.

    ReplyDelete
  10. Yes, you are right, that is definitely going to deadlock.
    I have actually used the approach of creating a user mode handle in my user mode process and haven’t had any issues so far. You need to make sure you are in the right process when creating the user mode handle, of course. Perhaps you could try again?

    ReplyDelete
  11. Perhaps I didn't understand you correctly. In my driver I can call FltCreateFile() and get a handle to the file, then convert that to a user-mode handle and pass that to the service (FltSendMessage) and the service should be able to use that handle?

    Since I'm in a pre-IRP_MJ_CREATE the driver context will be whatever initiated the file i/o but that will surely be different then the service.

    How do I insure I'm in the right process? I must be missing something.

    Thanks in advance!

    ReplyDelete
  12. What i've done in the past was to send an IOCTL from the user mode process (my service) to the driver and that will be in the context of the user mode process so I can just create the handle then. But feel free to experiment other approaches (for example ZwDuplicateObject takes a handle to the target process and a handle to the source process, so all you need to do is a handle to your user mode process to pass in as the target process).

    ReplyDelete
  13. The IOCTL approach may work but then I'm missing the instance from the callback data to insure our open doesn't go to the top of the stack. I'll look into ZwDuplicateObject plus dig some more on OSR. I may make a post there to get further feedback as well.

    Your input has been great - If you're at Plugfest sometime this year I owe you a few beers!

    -Gary Weber

    ReplyDelete
  14. Couldn't I use FltCreateFileEx to open the file, getting a PFILE_OBJECT, then use a combination of ObReferenceObjectByHandle and ObOpenObjectByPointer to get a user-mode handle to the file object (FltCreateFileEx would also give me a handle but that's a kernel-mode one), then pass that to user land and their it could use DuplicateHandle?

    If this would work, is their an easier way to convert my kernel-mode file handle to a user-mode one?

    ReplyDelete
  15. Well, you can't really pass a kernel mode handle to user mode code or a user mode handle that belongs to a different process to a user mode process. You can only do this in kernel mode (because you can access the kernel handle table and also because you can attach to processes).
    You could do something like:
    1. call FltCreateFileEx and create a handle.
    2. send a message (via communication port) to your user mode process with something that identifies that handle (for example generate a GUID if you want).
    3. your user mode can send you a special IOCTL with that GUID, which you can then use to call ObOpenObjectByPointer (which should create a handle in the user mode process)
    4. use the communication port to send the newly created handle value (which is a user mode handle) to the user mode process which can now use it...

    This is a general plan, you can probably improve it in a number of ways.

    Let's take this offline.. send me an email at alexandru dot carp dot osr at google's email service (standard obfuscation against spam bots).

    ReplyDelete