## Thursday, March 31, 2011

### Names in Minifilters - Implementing Name Provider Callbacks

Since we're going to talk a bit more about names, one important aspect to cover is how to implement minifilters that change the namespace in some way. Of course, this is a very large topic so in this post I'm going to cover one particular aspect of that, how to implement the name provider callbacks. The "name provider" callbacks are in fact two members of the FLT_REGISTRATION structure, GenerateFileNameCallback and NormalizeNameComponentCallback (generally referred to as the generate callback and the normalize callback). There are two other callbacks related to this (NormalizeContextCleanupCallback and NormalizeNameComponentExCallback) but they should be pretty easy to figure out. Strangely enough Microsoft doesn't currently provide any sample on how these callbacks are supposed to look like even in a basic case so I'll do that in this post. Please note that this is written almost from scratch. It is very loosely based on some existing code I have but that was way too complicated for the purpose of this post. I just wanted to show what the callbacks are supposed to do in principle. So the code here hasn't been extensively tested, it might fail in unusual circumstances and so on. It should only used as a reference and not in a production environment (not that it does all that much anyway). With that said, if you do run this code and it fails under some circumstances or you spot an error just by looking at it, please let me know and I'll investigate and update it so that everyone benefits from it.

I'll be referring to minifilters that implement these name provider callbacks as name provider minifilters, or simply name providers.

First let's talk a bit about why these callbacks exist. As you may have read on this blog or you may know from experience, name generation is a pretty complicated business (I use name generation in the general sense, referring both to creating a name for a file and to normalizing that name). In preCreate it might require looking in the FileObject and RelatedFileObject, or it might require looking up a fileID into a table to get the name and so on. If there is support for links (hardlinks for example) then the name depends on the FILE_OBJECT that's asking for it. For filters, it also matters if the file object is going to be a target of some operation that changes the namespace (rename, hardlink), in which case the name during the preOp is different from the name in postOp. Then you have the tunnel cache which might change the name and so on. Legacy filters spent a lot of time (both during development and at runtime) trying to get names for files, and so the Filter Manager team at MS started the project with the intention to simplify things. In the process of doing that they wrote a lot of code (I've heard that about 20% of the initial FltMgr code was dedicated to this) and they've noticed a couple of problems. First, performance isn't all that great and second (and this is the real nasty discovery in my opinion), that there is still a need to violate the strict layering rules FltMgr tries hard to obey (I haven't given it enough thought to see if there may have been another way so I'll just take their word for it). So they tried to address both these issues by creating a model where only minifilters that actually need to be involved in the name generation path (namespace virtualization and such) need to see the ugliness, while the rest of them can just enjoy the rather simple abstraction of calling FltGetFileNameInformation and getting back a name that is ready to be (mis)used.

For name provider minifilters, FltMgr requires that they implement two callbacks that it will call when generating (or normalizing) a name. Moreover, the IO generated by FltMgr while trying to generate these names will only be shown to these types of minifilters (name providers). This is a pretty serious decision because the minifilter model is designed so that minifilters really do see all IO happening on a volume, so excluding a certain class of IO was not a decision to be taken lightly. However, the performance benefits were enough that it seemed justified and in terms of what IO is shown to minifilters, pretty much all minifilters that I know of work just fine without processing that FltMgr IO (in fact, it comes as a surprise to most people that there is some IO happening behind the scenes which shows that it is truly transparent). Another problem with that IO is that, as I said before, it might violate layering. Specifically, the callbacks might be called when minifilters below them call FltGetFileNameInformation, which means that the developers of these kinds of minifilters need to be extremely careful when implementing their filters. In conclusion, my advice is that unless a minifilter actually MUST implement a name provider it should really avoid doing so, no matter how much they want to process "all IO". In general a minfilter MUST implement a name provider if it implements some sort of name virtualization scheme where the name of a file below their layer is different than the one below or where they take over part of the namespace or if the verifier warning I was talking about in this post occasionally pops up.

Let's not waste more time on warnings and talk about how the callbacks need to be implemented and what they should do.

GenerateFileNameCallback

The generate callback should be thought of as "the function that gets called when someone asks for an opened name" (FLT_FILE_NAME_OPENED). Its purpose is to return a file name given a FILE_OBJECT and a FLT_CALLBACK_DATA structure. There are a couple of important points about this function:
• The FLT_CALLBACK_DATA structure might be missing if the request for the name does not come from a component that is involved in an IO operation (for example, if a minifilter is also registered to receive process creation notifications and it wants to call FltGetFileNameInformation from that callback it can't because it doesn't have a FLT_CALLBACK_DATA structure; in that case its only recourse is to call FltGetFileNameInformationUnsafe). However, the minifilter writer can safely assume that either the FILE_OBJECT is opened (FILE_OBJECT->FsContext is not NULL) or there is a FLT_CALLBACK_DATA structure. It is impossible to get an unopened FILE_OBJECT outside of the IRP_MJ_CREATE path and in that path a minifilter must never call FltGetFileNameInformationUnsafe anyway. This might not strike you as very important but it matters a lot for case sensitivity (that's what you get for changing names...). The information about whether operations on a file are to be case sensitive or not is stored in a file when the file is opened (FO_OPENED_CASE_SENSITIVE). However, if the file is not yet open then the information can be found in the IRP_MJ_CREATE request, in the IO_STACK_LOCATION->Flags for legacy filters and FLT_CALLBACK_DATA->Iopb->OperationFlags for minifilters (SL_CASE_SENSITIVE).
• The filter has the option to tell FltMgr whether the name it returns should be cached or not. This is very important because if FltMgr caches the wrong name it will run into very interesting issues that are pretty complicated to debug. I've had a lot of fun that way. On the other hand, if the minifilter always tells FltMgr not to cache the name then performance will suffer greatly. When to cache really depends on minifilter architecture so there isn't much more general advice I can give.
• In general a minifilter should never return a name it is not prepared to handle in the future. For example, I've seen cases where a minifilter (A) was doing something like returning a name that was only valid during preCreate (a GUID that the minifilter used as a key in a hash and that was used to get a real file name below the minifilter's layer, after which the GUID was discarded). The minifilter then got in trouble when some other minifilter (B) used that name later on to open their own handle to the same file and minifilter A no longer had the GUID in its internal hash and so it had no idea what the real file was. I guess my advice is to try no to do anything too fancy here.

NormalizeNameComponentCallback

The normalize callback is called when someone asks for a normalized name. FltMgr gets a regular name (using the generate callback) and then it looks at each component and if it thinks it might be a short name it calls the name providers with the parent directory path and the name of the component it is trying to normalize. Here are some of the important things to mention:
• This function has a lot of potential for recursion. Try calling FltGetFileNameInformation to get a normalized name in preCreate from your own name provider and you'll see what I'm talking about. Don't use a lot of stack and try to avoid recursion as much as possible. This will require quite a lot of ingenuity to work around.
• The name of a file can be different inside a transaction and so any IO you perform must be in the context of that transaction. Possibly other operations (registry lookups ?) will need to be transacted as well.
And finally, here is the code that can be plugged into the passthrough sample to make it filter name provider requests:
 BOOLEAN
PtDoRequestOperationStatus(
__in PFLT_CALLBACK_DATA Data
);  
NTSTATUS PtNormalizeNameComponentExCallback(
__in   PFLT_INSTANCE Instance,
__in_opt PFILE_OBJECT FileObject,
__in   PCUNICODE_STRING ParentDirectory,
__in   USHORT VolumeNameLength,
__in   PCUNICODE_STRING Component,
__out  PFILE_NAMES_INFORMATION ExpandComponentName,
__in   ULONG ExpandComponentNameLength,
__in   FLT_NORMALIZE_NAME_FLAGS Flags,
__inout PVOID *NormalizationContext
);
NTSTATUS PtNormalizeNameComponentCallback(
__in   PFLT_INSTANCE Instance,
__in   PCUNICODE_STRING ParentDirectory,
__in   USHORT VolumeNameLength,
__in   PCUNICODE_STRING Component,
__out  PFILE_NAMES_INFORMATION ExpandComponentName,
__in   ULONG ExpandComponentNameLength,
__in   FLT_NORMALIZE_NAME_FLAGS Flags,
__inout PVOID *NormalizationContext
);
NTSTATUS PtGenerateFileNameCallback(
__in   PFLT_INSTANCE Instance,
__in   PFILE_OBJECT FileObject,
__in_opt PFLT_CALLBACK_DATA CallbackData,
__in   FLT_FILE_NAME_OPTIONS NameOptions,
__out   PBOOLEAN CacheFileNameInformation,
__out   PFLT_NAME_CONTROL FileName
);
//
// Assign text sections for each routine.
//
#ifdef ALLOC_PRAGMA
#pragma alloc_text(INIT, DriverEntry)  
...
//
// This defines what we want to filter with FltMgr
//
CONST FLT_REGISTRATION FilterRegistration = {
sizeof( FLT_REGISTRATION ),     // Size
FLT_REGISTRATION_VERSION,      // Version
0,                 // Flags
NULL,                // Context
Callbacks,             // Operation callbacks
PtInstanceSetup,          // InstanceSetup
PtInstanceQueryTeardown,      // InstanceQueryTeardown
PtInstanceTeardownStart,      // InstanceTeardownStart
PtInstanceTeardownComplete,     // InstanceTeardownComplete  
PtGenerateFileNameCallback,     // GenerateFileName
PtNormalizeNameComponentCallback,  // NormalizeNameComponent
NULL,                // NormalizeContextCleanup
#if FLT_MGR_LONGHORN
PtNormalizeNameComponentExCallback, // NormalizeNameComponentEx
#endif // FLT_MGR_LONGHORN
 };  
...
NTSTATUS PtGenerateFileNameCallback(
__in   PFLT_INSTANCE Instance,
__in   PFILE_OBJECT FileObject,
__in_opt PFLT_CALLBACK_DATA CallbackData,
__in   FLT_FILE_NAME_OPTIONS NameOptions,
__out   PBOOLEAN CacheFileNameInformation,
__out   PFLT_NAME_CONTROL FileName
)
{
NTSTATUS status = STATUS_SUCCESS;
PFLT_FILE_NAME_INFORMATION belowFileName = NULL;
PT_DBG_PRINT( PTDBG_TRACE_ROUTINES,
("PassThrough!PtGenerateFileNameCallback: Entered\n") );
__try {

//
//  We expect to only get requests for opened and short names.
//  If we get something else, fail. Please note that it is in
//  fact possible that if we get a normalized name request the
//  code would work because it's not really doing anything other
//  than calling FltGetFileNameInformation which would handle the
//  normalized name request just fine. However, in a real name
//  provider this might require a different implementation.
//

if (!FlagOn( NameOptions, FLT_FILE_NAME_OPENED ) &&
!FlagOn( NameOptions, FLT_FILE_NAME_SHORT )) {

ASSERT(!"we have a received a request for an unknown format. investigate!");

return STATUS_NOT_SUPPORTED ;
}

//
// First we need to get the file name. We're going to call
// FltGetFileNameInformation below us to get the file name from FltMgr.
// However, it is possible that we're called by our own minifilter for
// the name so in order to avoid an infinite loop we must make sure to
// remove the flag that tells FltMgr to query this same minifilter.
//
ClearFlag( NameOptions, FLT_FILE_NAME_REQUEST_FROM_CURRENT_PROVIDER );
//
// this will be called for FltGetFileNameInformationUnsafe as well and
// in that case we don't have a CallbackData, which changes how we call
// into FltMgr.
//
if (CallbackData == NULL) {
//
// This must be a call from FltGetFileNameInformationUnsafe.
// However, in order to call FltGetFileNameInformationUnsafe the
// caller MUST have an open file (assert).
//
ASSERT( FileObject->FsContext != NULL );
status = FltGetFileNameInformationUnsafe( FileObject,
Instance,
NameOptions,
&belowFileName );
if (!NT_SUCCESS(status)) {
__leave;
}
} else {
//
// We have a callback data, we can just call FltMgr.
//
status = FltGetFileNameInformation( CallbackData,
NameOptions,
&belowFileName );
if (!NT_SUCCESS(status)) {
__leave;
}
}
//
// At this point we have a name for the file (the opened name) that
// we'd like to return to the caller. We must make sure we have enough
// buffer to return the name or we must grow the buffer. This is easy
// when using the right FltMgr API.
//
status = FltCheckAndGrowNameControl( FileName, belowFileName->Name.Length );
if (!NT_SUCCESS(status)) {
__leave;
}
//
// There is enough buffer, copy the name from our local variable into
// the caller provided buffer.
//
RtlCopyUnicodeString( &FileName->Name, &belowFileName->Name );
//
// And finally tell the user they can cache this name.
//
*CacheFileNameInformation = TRUE;
} __finally {
if ( belowFileName != NULL) {
FltReleaseFileNameInformation( belowFileName );
}
}
return status;
}
NTSTATUS PtNormalizeNameComponentCallback(
__in   PFLT_INSTANCE Instance,
__in   PCUNICODE_STRING ParentDirectory,
__in   USHORT VolumeNameLength,
__in   PCUNICODE_STRING Component,
__out  PFILE_NAMES_INFORMATION ExpandComponentName,
__in   ULONG ExpandComponentNameLength,
__in   FLT_NORMALIZE_NAME_FLAGS Flags,
__inout PVOID *NormalizationContext
)
{
//
// This is just a thin wrapper over PtNormalizeNameComponentExCallback.
// Please note that we don't pass in a FILE_OBJECT because we don't
// have one.
//
return PtNormalizeNameComponentExCallback( Instance,
NULL,
ParentDirectory,
VolumeNameLength,
Component,
ExpandComponentName,
ExpandComponentNameLength,
Flags,
NormalizationContext );
}
NTSTATUS PtNormalizeNameComponentExCallback(
__in   PFLT_INSTANCE Instance,
__in_opt PFILE_OBJECT FileObject,
__in   PCUNICODE_STRING ParentDirectory,
__in   USHORT VolumeNameLength,
__in   PCUNICODE_STRING Component,
__out  PFILE_NAMES_INFORMATION ExpandComponentName,
__in   ULONG ExpandComponentNameLength,
__in   FLT_NORMALIZE_NAME_FLAGS Flags,
__inout PVOID *NormalizationContext
)
{
NTSTATUS status = STATUS_SUCCESS;
HANDLE parentDirHandle = NULL;
OBJECT_ATTRIBUTES parentDirAttributes;
BOOLEAN isDestinationFile;
BOOLEAN isCaseSensitive;
IO_STATUS_BLOCK ioStatus;
#if FLT_MGR_LONGHORN
IO_DRIVER_CREATE_CONTEXT driverContext;
PTXN_PARAMETER_BLOCK txnParameter = NULL;
#endif // FLT_MGR_LONGHORN
PT_DBG_PRINT( PTDBG_TRACE_ROUTINES,
("PassThrough!PtNormalizeNameComponentExCallback: Entered\n") );
__try {
//
// Initialize the boolean variables. we only use the case sensitivity
// one but we initialize both just to point out that you can tell
// whether Component is a "destination" (target of a rename or hardlink
// creation operation).
//
isCaseSensitive = BooleanFlagOn( Flags,
FLTFL_NORMALIZE_NAME_CASE_SENSITIVE );
isDestinationFile = BooleanFlagOn( Flags,
FLTFL_NORMALIZE_NAME_DESTINATION_FILE_NAME );
//
// Open the parent directory for the component we're trying to
// normalize. It might need to be a case sensitive operation so we
// set that flag if necessary.
//
InitializeObjectAttributes( &parentDirAttributes,
(PUNICODE_STRING)ParentDirectory,
OBJ_KERNEL_HANDLE | (isCaseSensitive ? OBJ_CASE_INSENSITIVE : 0 ),
NULL,
NULL );
#if FLT_MGR_LONGHORN
//
// In Vista and newer this must be done in the context of the same
// transaction the FileObject belongs to.
//
IoInitializeDriverCreateContext( &driverContext );
txnParameter = IoGetTransactionParameterBlock( FileObject );
driverContext.TxnParameters = txnParameter;
status = FltCreateFileEx2( gFilterHandle,
Instance,
&parentDirHandle,
NULL,
FILE_LIST_DIRECTORY | SYNCHRONIZE,
&parentDirAttributes,
&ioStatus,
0,
FILE_ATTRIBUTE_NORMAL | FILE_ATTRIBUTE_DIRECTORY,
FILE_OPEN,
NULL,
0,
IO_IGNORE_SHARE_ACCESS_CHECK,
&driverContext );
#else // !FLT_MGR_LONGHORN
//
// preVista we don't care about transactions
//
status = FltCreateFile( gFilterHandle,
Instance,
&parentDirHandle,
FILE_LIST_DIRECTORY | SYNCHRONIZE,
&parentDirAttributes,
&ioStatus,
0,
FILE_ATTRIBUTE_NORMAL | FILE_ATTRIBUTE_DIRECTORY,
FILE_OPEN,
NULL,
0,
IO_IGNORE_SHARE_ACCESS_CHECK );
#endif // FLT_MGR_LONGHORN
if (!NT_SUCCESS(status)) {
__leave;
}
//
// Now that we have a handle to the parent directory of Component, we
// need to query its long name from the file system. We're going to use
// ZwQueryDirectoryFile because the handle we have for the directory
// was opened with FltCreateFile and so targeting should work just fine.
//
status = ZwQueryDirectoryFile( parentDirHandle,
NULL,
NULL,
NULL,
&ioStatus,
ExpandComponentName,
ExpandComponentNameLength,
FileNamesInformation,
TRUE,
(PUNICODE_STRING)Component,
TRUE );
} __finally {
if (parentDirHandle != NULL) {
FltClose( parentDirHandle );
}
}
return status;
}

Update 04/12/2011: added check and assert to PtGenerateFileNameCallback.
Update 03/08/2012: see this post.

1. Excellent post, it's about time someone puts some name provider sample-like code online. :)

So it seems to me that the way the filter manager might call these is something like:

GenerateNameCallback( .. &some_path );

for ( each directory in some_path )
NormalizeNameCallback( component );

However, when playing around with virtualizing namespaces, this doesn't seem to be the case. In particular, if you are using STATUS_REPARSE in pre-create to move accesses underneath c:\real into c:\virtual. So when one tries to redirect c:\real\foo\bar -> c:\virtual\foo\bar, when you call FltGetFileNameInformation when c:\real\foo\bar is in pre-create, then the GenerateNameCallback gets called [and returns c:\real\foo\bar as the path, since it will be reparsed later], but NormalizeNameCallback never gets called and FltGetFileNameInformation returns STATUS_OBJECT_PATH_NOT_FOUND. It appears that the filter manager tries to open up c:\real\foo (which of course doesn't exist) before calling any of the normalization callbacks. Is that how one is supposed to try to do that kind of virtualization?

Thanks for your excellent series of blog posts, they're most useful! :)

2. You're right, that's how FltMgr calls the callbacks. Please note however that this code path was changed quite a bit between OS releases so behavior might be different.
Still, to address your question, if your minifilter is a name provider it should also receive the IRP_MJ_CREATE for when FltMgr tries to open C:\real\foo and it should reparse it to C:\virtual\foo. Is this not what you are seeing ?
I've seen some minifilters that only reparse some requests and not others (on a per-process basis for example) get confused about this.

3. Ah, that might be it -- I was selectively reparsing requests as you said. Is that something that filter manager really doesn't like then?

4. I can't really say what the approach should be. This is very dependent on the design of your solution. For example, you may want some processes to not see the files and in that case FltGetFileNameInformation returning STATUS_OBJECT_PATH_NOT_FOUND might be good. Or you could always return the virtual path (the target of the reparse) above your minifilter and then FltGetFileNameInformation should return that.

5. Great read - Thanks for the info. I'm currently debugging a recurrsion issue with someone else's mini-filter. I was wondering why recursion was occuring when I only called FltGetFileNameInformation! This post helps me understand what the other driver is probably doing. Hopefully reducing my stack usuage will prevent the stack overflow.

6. I registed the two callback function , but the PtNormalizeNameComponentCallback never be called , and there isn't any reparse minifilter, how can i find the cause ?

7. I'm not sure i understand what the reparse minifilter has to do with anything. Anyway, the NormalizeNameCallback won't be called unless someone above your filter calls FltGetFileNameInformation to get a normalized name. So make sure when you test that you have a minifilter above your filter that's issuing these requests. For me the easiest way to test this was to modify the passthrough sample a bit to make it query for names.

8. Great Post. This blog is getting better all the time and my main resource for filter driver development.

Thank you Alex.

9. Great post, thanks. Hard to find good information on minifilters, your blog seems to be a never ending source of it :). However, I think you're missing a NULL check:

txnParameter = IoGetTransactionParameterBlock( FileObject );

I'm getting a fatal exception at least when that line's being run when FileObject is null. Or is it me that's doing something wrong?

1. Hmm, not sure what to say. According to the documentation for PFLT_NORMALIZE_NAME_COMPONENT_EX (http://msdn.microsoft.com/en-us/library/windows/hardware/ff551105(v=vs.85).aspx) the FileObject is an _In_ parameter, so it's not optional. I use the same code and haven't seen any issues. Could you perhaps send me a stack trace (offline, you should be able to find my email address in my contact information) ?

10. Alex, great article! Thank you! But what about the case when the FltGetFileNameInformationUnsafe routine is called at APC_LEVEL or inside the guarded region (and its description really claims IRQL: <= APC_LEVEL), for instance, as a part of processing LoadImageNotifyCallback? In this case calling FltCreateFile(Ex2) and FltQueryDirectoryFile will either hang the system or trigger the Driver Verifier. As I remember, the latter callback routine is always called at APC_LEVEL (on Windows versions prior Vista or Server 2003) or inside the guarded region (on Windows Vista and beyond). And here's the real case - Sysinternals ProcMon utility, but of course, there should be some minifilter with a name provider installed in the system.

11. Alex, thank you for the guide! I have a question: is there a way to separate "additional" IRP_MJ_CREATE, which I get as a name provider from "usual" ones? This will be helpful to reduce overhead from double checking the requests.

1. As far as I know there isn't a reliable way. You could implement some fancy heuristic I suppose. But i'd say don't bother, name normalization is much faster in Win8 and doesn't make much use of this path so it's probably not worth the effort.

2. Thanks for the answer. It seems that there is no 100% way, but as I could notice on Windows 7, fltmgr sets up FILE_OPEN_FOR_BACKUP_INTENT for name-construction creates.
Now I got into a new issue: it seems that the very presence of my name provider in the system somehow changes the fltmgr behaviour for minifilters "above" my name provider. If an upper minifilter asks for normalized name for a network path, it gets it in opened format (i.e. the mapped by net use share has the disk letter in th resulting path). Is it something with my name provider callbacks or just a part of how fltmgr works?

3. Well, you could implement something based on observed FltMgr behavior but you'll probably need some hacks (which you might be OK with, depending on your particular requirements).
If I remember correctly FltMgr will do things a bit differently when a name provider minifilter is present, however I'm not sure what is happing to trigger the behavior you observed. Does your minifilter attach to network filesystems ? In general there already is a name provider minifilter loaded on the system, LUAFV, so your minifilter shouldn't change behavior too much, except if attached to a network filesystem...

4. Yes, I forgot to point this out, it happens exactly for network filesystems. As I wrote before, the "normalized" name, which the upper filter gets, still contains the disk letter for a mapped share. Is there something special about name providers on network filesystems?

5. If I remember correctly there are several 'correct' network paths, some of which include the mapped drive letter. I don't have a list handy and I can't seem to be able to find them listed anywhere. Anyway, from what I remember a name that looks something like: "\Device\LanmanRedirector\;X:000000000000abce\foo\bar" is valid. So what name exactly are you seeing ?

If what bothers you is that the behavior seems to change when your filter is added into the mix, I'm afraid you'll probably have to debug it yourself. I've done this and it usually helps to print all the names that your filter sees and generates and such.