Thursday, April 26, 2012
Smart Reading and Storing of Files
I often run into tools that move files around (back-up solutions or cloud storage solutions or file copy utilities) that don't support some core NTFS features like alternate data streams. This lack of support for a very useful feature (just think about how many different file formats are out there that provide little value besides adding support for storing metadata along with the main data stream) is very frustrating.
Anyway, from looking at various applications that don't support this I see two main reasons for it. One is that they're multi-platform apps that only support the common feature set between all the platforms (which makes sense in some cases but doesn't really make much sense in other cases). The other reason is that I suspect people just don't want to have to deal with enumerating alternate data streams and then coming up with a mechanism to serialize them into a single file. On the other hand, most such solutions must read other file metadata and preserve it, such as file attributes, timestamps and so on, which I suspect they do on their own, using various Windows APIs that provide that functionality. Smarter solutions need to also deal with sparse files (it would be pretty stupid for a cloud storage solution to ignore the file system information that a large chunk of a file is all 0s and instead use bandwidth to transfer those) and possibly even hardlinks and reparse points.
Fortunately, this is simpler that it sounds, at least on Windows :). There is a set of Windows APIs that operate on a stream of data that describes a file complete with alternate data streams information and sparse blocks information and security information and so on. This stream is formatted in a way that allows the caller to understand what type of data each part of the stream represents. The API set even allows skipping certain types of file data that is not relevant to the caller. The APIs I'm referring to are referred to as the Backup APIs:
There is also the WIN32_STREAM_ID structure, which describes the information that follows the structure in the stream and allows for figuring out whether the stream is interesting for the caller, how long it is and so on.
These APIs allow the caller to read the file contents and metadata as a long stream of bytes, skip through the stream and write all the information back as one stream of bytes. Moreover, since the information is formatted it's also possible that when reading the data only a certain type of data is read. For example, let's discuss a solution that archives data to some cloud storage and then wants to read the data into a file on an OS different than Windows. It's quite easy to keep track in some database of the information of where the main data stream begins and how long it is so that it can only download that information for that platform.
There is quite a bit of documentation on these APIs and on what a backup solution should do with them and so on. There are some documents under the [MS-BKUP]: Microsoft NT Backup File Structure page and even a basic sample, Creating a Backup Application.
So please, next time you run across a solution that doesn't handle sparse files or alternate data streams or some other such features, feel free to point the developers to these APIs :)
Thursday, April 19, 2012
The Standby List and Storage Overprovisioning
This post is about an interesting issue I spent quite a bit of time debugging. As is often the case with very complex system, I knew most of the bits of information related to the issue but I didn't quite put everything together and so this scenario still surprised me.
It all started with me playing with file sizes and directory entries and so I was copying a large number of files to a VHD. Since I didn't have a lot of space for the VHD in my VM I decided to make all the files sparse so that they won't take any space on the VHD, which was rather small (2GB). I created about 20GB worth of sparse files on it and all was well. I've actually been using this setup for a while but when doing this in a Win8 VM I quickly ran into problems. The VHD ran out of free space. I knew there was no way for that to happen since all my files were sparse files and I didn't expect that I suddenly had that many directory entries that the file system metadata actually used most of the 2GB of the VHD. So figured this would be an interesting investigation.
The first thing I noticed was that if I dismounted the VHD and mounted it again the VHD look pretty much the way I expected: all the files were there and there were roughly 2GB worth of free space. So it seems it was a transient situation. I spent a bit of time looking at various NTFS counters (fsutil is a good tool for that), looking at ProcMon logs and poking the file system in various ways, but I couldn't find anything. I was about to embark on the next step, which was to try to figure out which blocks on the volume are owned by which file in the file system so that I can see why those files aren't sparse, but I was lucky enough to discover by chance that exactly the same behavior happens on a Win7 machine when running Microsoft Security Essentials. This was quite helpful because I stopped suspecting there was some new behavior in NTFS in Win8 and instead I could focus on Microsoft Security Essentials (MSE). Other AV products I had running in my VMs didn't seem to have the same effect so this was particular to MSE. One thing I knew that was rather unique to MSE (at least it was a some point in the past) was the fact that MSE uses mapped files (also known as sections in the NT world) to read file data so I started wondering if that had anything to do with it.
So using fsutil I created a new 1GB file and made it sparse. Then I opened it with FileTest, created a file mapping and then mapped a view for the whole file. Guess what: NTFS reserved space for the range I was reading (naturally I didn’t expect MSE to change the files it was scanning so I was just reading the files). This is necessary because in case something writes to the file using the section NTFS must be able to save that information to disk. When working with mapped files NTFS can't know in advance what data will be written (if any) and so it does a lot of preparation to be able to accommodate the scenario where everything is written. So it was pretty clear what was going on, the fact that MSE created sections for my sparse files made NTFS reserve blocks for the files. The one last thing I had to figure out was why MSE held on to the sections for so long. My files were pretty small (less than 1 MB on average) and so it took quite a lot of them to get NTFS to run out of free space. Initially I suspected a section leak in MSE, but while I was playing with FileTest I noticed that even when I closed FileTest (and so I could know for sure that all the file handles and memory mapping handles and mapped views and so on were released) the blocks still weren't returned to the free space pool. And at this point it hit me that it must have been MM that kept the section open and indeed using RamMap I could see that was the case.
Here is a quick recap about what the standby list is. When a file is used for memory mapped IO, when the pages are no longer used (the view is unmapped or the section or the file handle are closed and even when the whole process is terminated) the pages that are backed by the file are moved to the standby list. They will be moved out of the standby list to either the free list or the zero list (depending on whether there is memory pressure in the system and who's asking for what kind of pages) or they will be reused if the same file is used for memory mapped or cached IO. This last behavior is pretty much a file cache (not to be confused with the cache manager which has quite a different role). In my case since there was no memory pressure the pages would remain on the standby list for quite a while and so NTFS would not see the section being closed and so it kept the reserved blocks.
Please note that this is not unique to sparse files, any form of files that are overprovisioned (such as compressed files) have the same semantics. So it is quite easy to run out of space on a volume where the total logical size of all the files exceeds the volume's capacity even without writing anything to the volume.
Now, since this is a file systems and filters development blog I should mention that if you are a file system or a filter and you do any work with compressed files or sparse files or some such, you can actually tell MM to close a section using MmForceSectionClosed().
Update: I wanted to add some steps on how to reproduce this problem, in case you're interested.
It all started with me playing with file sizes and directory entries and so I was copying a large number of files to a VHD. Since I didn't have a lot of space for the VHD in my VM I decided to make all the files sparse so that they won't take any space on the VHD, which was rather small (2GB). I created about 20GB worth of sparse files on it and all was well. I've actually been using this setup for a while but when doing this in a Win8 VM I quickly ran into problems. The VHD ran out of free space. I knew there was no way for that to happen since all my files were sparse files and I didn't expect that I suddenly had that many directory entries that the file system metadata actually used most of the 2GB of the VHD. So figured this would be an interesting investigation.
The first thing I noticed was that if I dismounted the VHD and mounted it again the VHD look pretty much the way I expected: all the files were there and there were roughly 2GB worth of free space. So it seems it was a transient situation. I spent a bit of time looking at various NTFS counters (fsutil is a good tool for that), looking at ProcMon logs and poking the file system in various ways, but I couldn't find anything. I was about to embark on the next step, which was to try to figure out which blocks on the volume are owned by which file in the file system so that I can see why those files aren't sparse, but I was lucky enough to discover by chance that exactly the same behavior happens on a Win7 machine when running Microsoft Security Essentials. This was quite helpful because I stopped suspecting there was some new behavior in NTFS in Win8 and instead I could focus on Microsoft Security Essentials (MSE). Other AV products I had running in my VMs didn't seem to have the same effect so this was particular to MSE. One thing I knew that was rather unique to MSE (at least it was a some point in the past) was the fact that MSE uses mapped files (also known as sections in the NT world) to read file data so I started wondering if that had anything to do with it.
So using fsutil I created a new 1GB file and made it sparse. Then I opened it with FileTest, created a file mapping and then mapped a view for the whole file. Guess what: NTFS reserved space for the range I was reading (naturally I didn’t expect MSE to change the files it was scanning so I was just reading the files). This is necessary because in case something writes to the file using the section NTFS must be able to save that information to disk. When working with mapped files NTFS can't know in advance what data will be written (if any) and so it does a lot of preparation to be able to accommodate the scenario where everything is written. So it was pretty clear what was going on, the fact that MSE created sections for my sparse files made NTFS reserve blocks for the files. The one last thing I had to figure out was why MSE held on to the sections for so long. My files were pretty small (less than 1 MB on average) and so it took quite a lot of them to get NTFS to run out of free space. Initially I suspected a section leak in MSE, but while I was playing with FileTest I noticed that even when I closed FileTest (and so I could know for sure that all the file handles and memory mapping handles and mapped views and so on were released) the blocks still weren't returned to the free space pool. And at this point it hit me that it must have been MM that kept the section open and indeed using RamMap I could see that was the case.
Here is a quick recap about what the standby list is. When a file is used for memory mapped IO, when the pages are no longer used (the view is unmapped or the section or the file handle are closed and even when the whole process is terminated) the pages that are backed by the file are moved to the standby list. They will be moved out of the standby list to either the free list or the zero list (depending on whether there is memory pressure in the system and who's asking for what kind of pages) or they will be reused if the same file is used for memory mapped or cached IO. This last behavior is pretty much a file cache (not to be confused with the cache manager which has quite a different role). In my case since there was no memory pressure the pages would remain on the standby list for quite a while and so NTFS would not see the section being closed and so it kept the reserved blocks.
Please note that this is not unique to sparse files, any form of files that are overprovisioned (such as compressed files) have the same semantics. So it is quite easy to run out of space on a volume where the total logical size of all the files exceeds the volume's capacity even without writing anything to the volume.
Now, since this is a file systems and filters development blog I should mention that if you are a file system or a filter and you do any work with compressed files or sparse files or some such, you can actually tell MM to close a section using MmForceSectionClosed().
Update: I wanted to add some steps on how to reproduce this problem, in case you're interested.
- Create an empty 1GB file:
C:\>fsutil file createnew C:\TestFile.bin 1000000000 File C:\TestFile.bin is created
- Make the file sparse:
C:\>fsutil sparse setflag C:\TestFile.bin C:\>fsutil sparse setrange C:\TestFile.bin 0 1000000000
- open the file in FileTest.exe. Make sure to request GENERIC_WRITE access:
- create a read-only file mapping, map a view and read the whole file:
- unmap the view, close the section handle and then the file handle and then close FileTest.exe (we could have closed it directly as well).
- you now have 1 GB less free space on C:
C:\>dir Volume in drive C has no label. Volume Serial Number is 10FA-5C1D Directory of C:\ 06/10/2009 02:42 PM 24 autoexec.bat 06/10/2009 02:42 PM 10 config.sys 03/02/2010 06:31 PM >DIR> Far 07/13/2009 07:37 PM >DIR> PerfLogs 08/11/2011 11:13 AM >DIR> Perl 11/05/2010 09:34 AM >DIR> Program Files 04/20/2012 10:12 AM 1,000,000,000 TestFile.bin 11/04/2009 12:57 PM >DIR> Users 11/05/2010 09:40 AM >DIR> Windows 3 File(s) 1,000,000,034 bytes 6 Dir(s) 39,859,396,608 bytes free - Use RamMap and see all the pages for the file on the standby list:
- Empty the standby list (click on Empty->Empty Standby List).
- Finally check the free space on C: again:
C:\>dir Volume in drive C has no label. Volume Serial Number is 10FA-5C1D Directory of C:\ 06/10/2009 02:42 PM 24 autoexec.bat 06/10/2009 02:42 PM 10 config.sys 03/02/2010 06:31 PM >DIR> Far 07/13/2009 07:37 PM >DIR> PerfLogs 08/11/2011 11:13 AM >DIR> Perl 11/05/2010 09:34 AM >DIR> Program Files 04/20/2012 10:12 AM 1,000,000,000 TestFile.bin 11/04/2009 12:57 PM >DIR> Users 11/05/2010 09:40 AM >DIR> Windows 3 File(s) 1,000,000,034 bytes 6 Dir(s) 41,361,416,192 bytes free
Thursday, April 12, 2012
Useful Tools for File System Developers
In this post I'll go over some of the tools I use pretty frequently to investigate file system behavior and debug my filters. I've noticed that some of the tools I use are somewhat unknown to other developers and so I hope this might save someone some time.
- FileTest - I've already mentioned this tool many times in my previous posts. It's extremely useful to me for investigating file system behavior as well as trying to reproduce bugs in my filter. I really like the ability to generate a breakpoint right before a certain operation is issued, which is very helpful when debugging certain code paths where a regular breakpoint will just be too noisy. In those cases I enable the breakpoint in FileTest and then when I triggers I do a "bp /t @$thread
" and the breakpoint will only trigger in the context of that thread. It's also very useful to see how some of the structures for information classes and such are set by the file system. Overall, a great tool. - ProcMon - I'm sure everybody knows of ProcMon already. I use it a lot when I want to investigate the behavior of a certain application (how does an application write to a file or how does it use a log or some such). It's also the first thing I try when I get bug reports where "application X works without the filter but fails when the filter is present". For these debugging scenarios I also found it very useful for something else: finding the log file for the application. Most properly written applications have a mechanism to log errors and/or warnings but since there are so many ways to do that in Windows I found that ProcMon is quite useful to find the log file (if there is one). I'm sure everyone reading my blog is a power-user of ProcMon so I won't go into more detail about the use cases and such. However, there is one trick that I'd like to mention that makes is very useful for file system developers: changing the altitude of ProcMon's filter. For file system filter developers this is incredibly useful because it allows them to see what the IO their filter issues looks like. Also, for debugging interop issues it's very useful to be able to put ProcMon between the two instances and see what's going on. First I capture ProcMon's configuration like so (make sure to capture it again when upgrading ProcMon):
- Start procmon.
- Open the registry entry for the filter: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\PROCMON20 (for the current version on one of my test machines)
- Export the key into a human-readable format (I use *.REG files).
So now whenever I need to have ProcMon filter at a different altitude I open the file and change the Altitude to what I need it to be. After that the steps are:Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\PROCMON20] [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\PROCMON20\Instances] "DefaultInstance"="Process Monitor Instance" [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\PROCMON20\Instances\Process Monitor Instance] "Altitude"="385200" "Flags"=dword:00000000
- Load the REG file in the registry
- Open regedit and go to the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\PROCMON20\Instances\Process Monitor Instance key
- Add DENY for Everyone for Delete and Set Value.
- start ProcMon
- use fltmc to make sure that procmon is loaded at the right altitude.
- fsutil - I don't know exactly why but this very useful tool is not as well known as I would have expected. It is a Microsoft tool that allows command line access to many file system functions that are useful for file system developers and which would otherwise require writing an in-house tool to access. This is a list of the ones I've used over the years but there are many more:
- get file name by ID
- set file shortname
- query the allocated ranges for a file
- query all sorts of information about the file system on a volume, including file system capabilities, configuration options, statistics and so on.
- create and list hardlinks for a file
- display information for reparse points (symlinks, mountpoints, directory junctions and custom ones)
- work with sparse files (query and set the ranges, set the attribute and so on)
- manage the USN journal (create, delete, query the entries for a file and so on)
- mountvol - this is another Microsoft tool. I found it very useful for volume management. It's great to create mount points and change (or remove) drive letters, get familiar with using the NT names for volumes and so on.
- diskpart - another Microsoft tool useful for creating VHDs and formatting volumes with various unusual characteristics. Useful for file system developers that want to test their filters with various file systems (see my previous post on Testing a Minifilter on More Filesystems: UDF and ExFAT).
Thursday, April 5, 2012
Setting IoStatus.Information
I spent a couple hours yesterday investigating a bug that turned out to be a pretty silly mistake on my part, but I wanted to share some of the analysis with you. The problem was that an application was failing with a message of "There is not enough space on the disk", according to its logs.
Naturally the first step was to run procmon with and without my filter and compare the logs. After some filtering and searching and looking at timestamps (as an aside, any decent logging solution should have timestamps for the events, it makes it so much easier to find things) I managed to narrow things down to this:
And this is what the log looked like when the application worked:
The thing that I found most puzzling just by looking at these to screenshots was the fact that when the application was failing, the IRP_MJ_QUERY_INFORMATION call for the FilePositionInformation class was successful but still ProcMon didn't print the position like it did when the application was working. Anyway, the problem was obviously that the application was trying to set the EndOfFilte to a very large offset and it failed because the disk wasn't that large. I was curious to see how that file size was calculated and so I spent some time looking at the number, in the hope that maybe some high bit is set somewhere, messing up the number. But to no avail, the value looked positively random: 1,288,524,915,353 bytes is 0x12C0211E299…
Anyway, with that investigation leading nowhere (the next step was to debug the application to figure out how that value came about, something I wasn't very eager to do), I decided that I should take a break and instead investigate the other puzzling aspect, ProcMon not displaying the FilePositionInformation even thought the call was successful. So I spent a bit of time trying to figure it out and in my filter everything looked fine, I was setting the size properly:
case FilePositionInformation:
{
PFILE_POSITION_INFORMATION positionInfo = Data->Iopb->Parameters.QueryFileInformation.InfoBuffer;
// retrieve the current position from the file object
positionInfo->CurrentByteOffset = FltObjects->FileObject->CurrentByteOffset;
// then complete the operation
Data->IoStatus.Status = STATUS_SUCCESS;
callbackStatus = FLT_PREOP_COMPLETE;
}
break;
Compare this with the FastFat implementation:
VOID
FatQueryPositionInfo (
IN PIRP_CONTEXT IrpContext,
IN PFILE_OBJECT FileObject,
IN OUT PFILE_POSITION_INFORMATION Buffer,
IN OUT PLONG Length
)
/*++
Routine Description:
This routine performs the query position information function for fat.
Arguments:
FileObject - Supplies the File object being queried
Buffer - Supplies a pointer to the buffer where the information is to
be returned
Length - Supplies the length of the buffer in bytes, and receives the
remaining bytes free in the buffer upon return.
Return Value:
None
--*/
{
PAGED_CODE();
DebugTrace(+1, Dbg, "FatQueryPositionInfo...\n", 0);
//
// Get the current position found in the file object.
//
Buffer->CurrentByteOffset = FileObject->CurrentByteOffset;
//
// Update the length and status output variables
//
*Length -= sizeof( FILE_POSITION_INFORMATION );
DebugTrace( 0, Dbg, "*Length = %08lx\n", *Length);
DebugTrace(-1, Dbg, "FatQueryPositionInfo -> VOID\n", 0);
UNREFERENCED_PARAMETER( IrpContext );
return;
}
Not at all that different, right ? Anyway, I figured I should set an on-access breakpoint to see where the information that I set in the buffer was read:
So as you can see the buffer wasn't access until the memory got freed. I then decided to compare it with what happens when my filter is not in the picture:1: kd> ba r 4 937dd744 1: kd> g Breakpoint 0 hit nt!ExDeferredFreePool+0x2e3: 82931943 894604 mov dword ptr [esi+4],eax
And this was where it hit me. Since this was BUFFERED_IO, the IO manager was supposed to copy the data I provided back into the user's buffer. But as we can clearly see, with my filter there was no access to the buffer at all. This is because I didn’t set the IoStatus.Information properly. I did set it to 0 somewhere up the in the beginning of the function but forgot to set it to reflect the size of the data that I filled in when completing the request. The net result was that ProcMon was smart enough not to display anything because it got no data back, but the application had no idea this was going on and so it used whatever happened to be in the buffer as the actual position, and then proceeded to set that position as the EndOfFile. After setting the IoStatus.Information properly the application started working. For completeness, this is the code that worked:1: kd> ba r 4 94b6c874 1: kd> g Breakpoint 0 hit nt!memcpy+0x134: 8284f8f4 89448ffc mov dword ptr [edi+ecx*4-4],eax 1: kd> kn # ChildEBP RetAddr 00 9a037bf8 828b31c3 nt!memcpy+0x134 01 9a037c48 82a6e017 nt!IopCompleteRequest+0xa0 02 9a037d18 8285444a nt!NtQueryInformationFile+0x86c 03 9a037d18 779c64f4 nt!KiFastCallEntry+0x12a ...
case FilePositionInformation:
{
PFILE_POSITION_INFORMATION positionInfo = Data->Iopb->Parameters.QueryFileInformation.InfoBuffer;
// retrieve the current position from the file object
positionInfo->CurrentByteOffset = FltObjects->FileObject->CurrentByteOffset;
// then complete the operation
Data->IoStatus.Status = STATUS_SUCCESS;
Data->IoStatus.Information = sizeof(positionInfo->CurrentByteOffset);
callbackStatus = FLT_PREOP_COMPLETE;
}
break;
In closing I'd like to point to Doron's blog on How to return the number of bytes required for a subsequent operation that explains some more of the inner workings on IoStatus.Information and how the IO manager uses it.
Thursday, March 29, 2012
Directory Entries and File Properties
This post is about a pretty well-known behavior of NTFS that nevertheless seems to occasionally surprise people : the fact that the directory entries aren't an authoritative source of information when it comes to file properties. What I'm referring to here is that it's perfectly possible that doing a "dir" command will not return accurate information for the file. When hearing about this for the first time most people think of the situation where a file is open and actively being written to (appended for example) and so the dir command returns a file size that was the actual size at that time but since then the file has been modified and so the obviously the file size doesn't match what is the file size right at this moment.
What I've described above is actually a pretty straightforward case and I've not yet met anyone that was surprised or confused by it. However, following this train of thought leads us to pretty interesting places. If the situation I've described above can frequently happen, how is an application expected to get the actual file size so that it knows it can't change ? (For an example of where this might used think of a copy application that's trying to figure out whether there is enough space on a volume to copy some file there before it starts the copy…) Well, the right approach is to open the file and query the file size and not rely on the directory entry. However, it is possible that the file is modified even while the application has the file opened, so if being able to "know" the file size is really important then the application should not allow other applications write access to the file while it has it open (and this is done by using the sharing modes parameters of the create operation, where an application can not allow other handles to be opened for write, thus making sure the file size or contents can't change).
So let's recap. If the application must know the file size and must make sure it doesn't change, it must open a handle to the file. If there is no handle then the file size information can't be guaranteed to be accurate. So then why even bother to actually return the file size in a dir command ? It turns out that for a lot of cases knowing the file sizes without guaranteeing that they won't change is sufficient. After all, think back to all the cases where you've done a dir and looked at file sizes. I bet in most cases you didn't care if the file size changed a bit later.
So now we know that enumerating the files in a directory isn't by definition an operation that is expected to be 100% accurate and if an app needs those kinds of guarantees then the app must implement its own synchronization mechanism (that might involve opening all the files without sharing write and so on). So with this in mind, what should a file system do to implement the IRP_MJ_DIRECTORY_CONTROL with the IRP_MN_QUERY_DIRECTORY minor code ? One might be tempted to go through all the files in the directory and find their on-disk information and retrieve the file attributes and file information from there, but that would certainly be rather slow. So since the information isn't expected to be accurate, wouldn't it be better to have a sort of cache of the file information ? In fact that's how most file systems implement this. The directory actually stores information about the files it contains in a cache and it returns the information from that cache, which is much faster. One can even see this in action when comparing the time it takes to get a directory listing in CMD with the time it takes to open the directory in Explorer. Explorer displays additional information for each file (the icon) and so when it goes through a directory it will need to open each file and figure out what icon it should display. But please also note that Explorer implements an icon cache as well (there are many posts describing this icon cache, google it).
Now that we've established that a folder caches the file information, when does the cache get updated ? It would make sense that the cache gets updated when the file is closed, which is pretty much how most file systems do it. Incidentally, this explains why if you have a file that is constantly being written to in a thread and then you open it and close it from a different thread (or process) the attributes and file size get updated.
The really interesting thing happens for NTFS when you have hardlinks for a file. One might expect that the file system updates all the directories containing the file to show the new information, but that's not what happens. Instead, the link that is used to open the file is updated. It's not even about the directory that contains that link, it's that particular link. BTW, this is documented behavior: see Hard Links and Junctions and the msdn page for the CreateHardLink function. This is what this looks like (please note how the file size changes for the link I modify it from and then for the other link how it changes when I open the file even without modifying it):
Another such example that is very dear to me is file names. Most minifilters implement very complicated procedures to store names and cache them in file contexts and so on without taking advantage of the fact that in most cases names are meant to be transient information (and also without taking advantage of the fact that FltMgr's name cache is doing exactly that anyway). For more on this see my previous post on Names and file systems filters.
What I've described above is actually a pretty straightforward case and I've not yet met anyone that was surprised or confused by it. However, following this train of thought leads us to pretty interesting places. If the situation I've described above can frequently happen, how is an application expected to get the actual file size so that it knows it can't change ? (For an example of where this might used think of a copy application that's trying to figure out whether there is enough space on a volume to copy some file there before it starts the copy…) Well, the right approach is to open the file and query the file size and not rely on the directory entry. However, it is possible that the file is modified even while the application has the file opened, so if being able to "know" the file size is really important then the application should not allow other applications write access to the file while it has it open (and this is done by using the sharing modes parameters of the create operation, where an application can not allow other handles to be opened for write, thus making sure the file size or contents can't change).
So let's recap. If the application must know the file size and must make sure it doesn't change, it must open a handle to the file. If there is no handle then the file size information can't be guaranteed to be accurate. So then why even bother to actually return the file size in a dir command ? It turns out that for a lot of cases knowing the file sizes without guaranteeing that they won't change is sufficient. After all, think back to all the cases where you've done a dir and looked at file sizes. I bet in most cases you didn't care if the file size changed a bit later.
So now we know that enumerating the files in a directory isn't by definition an operation that is expected to be 100% accurate and if an app needs those kinds of guarantees then the app must implement its own synchronization mechanism (that might involve opening all the files without sharing write and so on). So with this in mind, what should a file system do to implement the IRP_MJ_DIRECTORY_CONTROL with the IRP_MN_QUERY_DIRECTORY minor code ? One might be tempted to go through all the files in the directory and find their on-disk information and retrieve the file attributes and file information from there, but that would certainly be rather slow. So since the information isn't expected to be accurate, wouldn't it be better to have a sort of cache of the file information ? In fact that's how most file systems implement this. The directory actually stores information about the files it contains in a cache and it returns the information from that cache, which is much faster. One can even see this in action when comparing the time it takes to get a directory listing in CMD with the time it takes to open the directory in Explorer. Explorer displays additional information for each file (the icon) and so when it goes through a directory it will need to open each file and figure out what icon it should display. But please also note that Explorer implements an icon cache as well (there are many posts describing this icon cache, google it).
Now that we've established that a folder caches the file information, when does the cache get updated ? It would make sense that the cache gets updated when the file is closed, which is pretty much how most file systems do it. Incidentally, this explains why if you have a file that is constantly being written to in a thread and then you open it and close it from a different thread (or process) the attributes and file size get updated.
The really interesting thing happens for NTFS when you have hardlinks for a file. One might expect that the file system updates all the directories containing the file to show the new information, but that's not what happens. Instead, the link that is used to open the file is updated. It's not even about the directory that contains that link, it's that particular link. BTW, this is documented behavior: see Hard Links and Junctions and the msdn page for the CreateHardLink function. This is what this looks like (please note how the file size changes for the link I modify it from and then for the other link how it changes when I open the file even without modifying it):
This is very interesting to think about from a filter perspective. This kind of behavior where the file system will return data without any guarantees that it will remain consistent is fairly common and identifying the pattern can make life a lot easier for filter developers. For example, let's say we have a filter that wants to make certain files appear in a directory. To make things harder, let's say that all the files are stored somewhere on a network with very expensive characteristics, for example in the cloud somewhere where there is a real dollar cost in terms of bytes of traffic. If the filter is written with the assumption that the directory entries must always reflect the actual file size in the cloud then on each IRP_MN_QUERY_DIRECTORY it might query the file size from the cloud, which generates traffic and so it has a real dollar cost associated with it. However, once the developer understands this particular contract of the file system they can get away with caching the file properties locally and only updating them when the file is actually opened.D:\templink>dir Volume in drive D is Data Volume Serial Number is 3817-6E24 Directory of D:\templink 03/29/2012 11:46 AM <DIR> . 03/29/2012 11:46 AM <DIR> .. 03/29/2012 11:46 AM 8 foo.txt 1 File(s) 8 bytes 2 Dir(s) 74,514,755,584 bytes free D:\templink>mklink /H bar.txt foo.txt Hardlink created for bar.txt <<===>> foo.txt D:\templink>dir Volume in drive D is Data Volume Serial Number is 3817-6E24 Directory of D:\templink 03/29/2012 11:47 AM <DIR> . 03/29/2012 11:47 AM <DIR> .. 03/29/2012 11:46 AM 8 bar.txt 03/29/2012 11:46 AM 8 foo.txt 2 File(s) 16 bytes 2 Dir(s) 74,514,755,584 bytes free D:\templink>echo hello world >foo.txt D:\templink>dir Volume in drive D is Data Volume Serial Number is 3817-6E24 Directory of D:\templink 03/29/2012 11:47 AM <DIR> . 03/29/2012 11:47 AM <DIR> .. 03/29/2012 11:46 AM 8 bar.txt 03/29/2012 11:47 AM 14 foo.txt 2 File(s) 22 bytes 2 Dir(s) 74,514,755,584 bytes free D:\templink>type bar.txt hello world D:\templink>dir Volume in drive D is Data Volume Serial Number is 3817-6E24 Directory of D:\templink 03/29/2012 11:47 AM <DIR> . 03/29/2012 11:47 AM <DIR> .. 03/29/2012 11:47 AM 14 bar.txt 03/29/2012 11:47 AM 14 foo.txt 2 File(s) 28 bytes 2 Dir(s) 74,514,755,584 bytes free
Another such example that is very dear to me is file names. Most minifilters implement very complicated procedures to store names and cache them in file contexts and so on without taking advantage of the fact that in most cases names are meant to be transient information (and also without taking advantage of the fact that FltMgr's name cache is doing exactly that anyway). For more on this see my previous post on Names and file systems filters.
Thursday, March 22, 2012
Some Limitations Using Files Opened By ID
The ability to open files by ID is a pretty nice feature of certain file systems, especially from the perspective of filters. The fact that file IDs are small and fixed in size makes them very suitable for things like storing in fixed-size records or allocating them from lookaside lists. Unfortunately the semantics for files opened by ID are a bit different from the semantics of the same files if they would have been opened by name.
As you can see we again end up talking about names. What is the relationship between a file and it's name ? Technically both the file's ID and a file's name are identifiers for the file. However, they belong to different namespaces, with different rules. The rules can be different between file systems and to keep things simple for the rest of this post I'll stick to talking about NTFS. The file name namespace for example allows multiple names for a file (hardlinks) while the ID namespace does not.
So let's get straight to the interesting bits. The different semantics of the different namespaces can make it so that some operations don't make sense. For example because NTFS allows multiple names for a file if the file is opened by ID and an operation that changes the namespace is attempted, which name should be affected ? To make this very clear, if file \Foo\f.txt and file \Bar\b.txt are hardlinks to the same file and I open the file by ID and I try to rename it, which name should change ? How about if I try a delete ?
Naturally NTFS will return some status codes that hardly describe what is going on and I've spent many hours reviewing code to figure out what I may have done wrong before figuring out that I'm working on a file that was opened by ID and that the particular operation wasn't supported. So here are some operations that I've found to not work when files are opened by ID:
- Rename (ZwSetInformationFile with FileRenameInformation) fails with STATUS_INVALID_PARAMETER.
- Delete (ZwSetInformationFile with FileDispositionInformation) also fails with STATUS_INVALID_PARAMETER.
- Opening a file with delete on close (ZwCreateFile with FILE_DELETE_ON_CLOSE | FILE_OPEN_BY_FILE_ID) doesn't fail, but it also doesn't delete the file.
- Creating a new hardlink (ZwSetInformationFile with FileLinkInformation) fails with STATUS_INVALID_PARAMETER (which I didn't expect).
- Querying the short name for a file (ZwQueryInformationFile with FileAlternateNameInformation) also fails with STATUS_INVALID_PARAMETER.
- Opening a file by ID with an ID that doesn't exist (ZwCreateFile with FILE_OPEN_BY_FILE_ID) will also fail with STATUS_INVALID_PARAMETER.
- First create a file (since you can't create a new file by ID) and then close it.
- Then get the file ID and click on the "Use" button.
- Then just open the file by ID (make sure to change DesiredAccess to match what you're trying to do).
- And then finally just try the operation to see if it can work on files opened by ID.
Thursday, March 15, 2012
Volume Names
In this post I want to talk about something that's not directly related to file system filters but that I've spent a lot of time fighting with. I'm specifically talking about volume names and the reason this is important to me is because these days I work on virtualization filters and in some cases when creating virtual files I need to make them feel the same as regular files on a real volume and the way some applications (both kernel mode and user mode) handle volume names is downright broken.
The most important point I want to make is that a volume name is NOT a drive letter. I read a lot of articles and attend a lot of presentations where volumes are identified by "drive letter" which, while is useful as a way to express a concept because everyone is familiar with drive letters, is actually wrong. Drive letters are a DOS concept, the NT concept is the volume name (and it looks like this '\\?\Volume{0d5759d1-429c-11df-8e0f-806e6f6e6963}'). Easiest way to see this is to use the "mountvol.exe" command line tool. This difference is very clearly expressed in the mountmgr.h file (%DDKPATH%\inc\ddk\mountmgr.h) where there are macros like 'MOUNTMGR_IS_VOLUME_NAME(s)' and 'MOUNTMGR_IS_DRIVE_LETTER':
As I mentioned in my previous post on Problems with STATUS_REPARSE - Part II, a lot of the times the problems come from user mode apps trying to build a path to a file and they expect to get a drive letter as the volume, which is just wrong. Even the MSDN example Obtaining a File Name From a File Handle falls into this trap by using drive letters all over:
Anyway, the best way to do this is to call GetFinalPathNameByHandle() and use the VOLUME_NAME_GUID flag to get used to using volume names. Unfortunately this is only available in Vista and newer OSes and so for XP one could still use the technique described in Obtaining a File Name From a File Handle but there is something that needs to be changed. The problem is that the volume APIs don't seem to have a way to convert a volume device name ('\Device\HarddiskVolume2') to a volume GUID name. In fact, none of the volume APIs offer an easy way to work with volume device names. The one way I've been able to do this in the general case was to use the MountMgr APIs directly. I don't have any user mode code that shows exactly what need to be done but I'll show the kernel mode code piece that queries the MountMgr:
The most important point I want to make is that a volume name is NOT a drive letter. I read a lot of articles and attend a lot of presentations where volumes are identified by "drive letter" which, while is useful as a way to express a concept because everyone is familiar with drive letters, is actually wrong. Drive letters are a DOS concept, the NT concept is the volume name (and it looks like this '\\?\Volume{0d5759d1-429c-11df-8e0f-806e6f6e6963}'). Easiest way to see this is to use the "mountvol.exe" command line tool. This difference is very clearly expressed in the mountmgr.h file (%DDKPATH%\inc\ddk\mountmgr.h) where there are macros like 'MOUNTMGR_IS_VOLUME_NAME(s)' and 'MOUNTMGR_IS_DRIVE_LETTER':
//
// Macro that defines what a "drive letter" mount point is. This macro can
// be used to scan the result from QUERY_POINTS to discover which mount points
// are find "drive letter" mount points.
//
#define MOUNTMGR_IS_DRIVE_LETTER(s) ( \
(s)->Length == 28 && \
(s)->Buffer[0] == '\\' && \
(s)->Buffer[1] == 'D' && \
(s)->Buffer[2] == 'o' && \
(s)->Buffer[3] == 's' && \
(s)->Buffer[4] == 'D' && \
(s)->Buffer[5] == 'e' && \
(s)->Buffer[6] == 'v' && \
(s)->Buffer[7] == 'i' && \
(s)->Buffer[8] == 'c' && \
(s)->Buffer[9] == 'e' && \
(s)->Buffer[10] == 's' && \
(s)->Buffer[11] == '\\' && \
(s)->Buffer[12] >= 'A' && \
(s)->Buffer[12] <= 'Z' && \
(s)->Buffer[13] == ':')
//
// Macro that defines what a "volume name" mount point is. This macro can
// be used to scan the result from QUERY_POINTS to discover which mount points
// are "volume name" mount points.
//
#define MOUNTMGR_IS_VOLUME_NAME(s) ( \
((s)->Length == 96 || ((s)->Length == 98 && (s)->Buffer[48] == '\\')) && \
(s)->Buffer[0] == '\\' && \
((s)->Buffer[1] == '?' || (s)->Buffer[1] == '\\') && \
(s)->Buffer[2] == '?' && \
(s)->Buffer[3] == '\\' && \
(s)->Buffer[4] == 'V' && \
(s)->Buffer[5] == 'o' && \
(s)->Buffer[6] == 'l' && \
(s)->Buffer[7] == 'u' && \
(s)->Buffer[8] == 'm' && \
(s)->Buffer[9] == 'e' && \
(s)->Buffer[10] == '{' && \
(s)->Buffer[19] == '-' && \
(s)->Buffer[24] == '-' && \
(s)->Buffer[29] == '-' && \
(s)->Buffer[34] == '-' && \
(s)->Buffer[47] == '}' \
)
As I mentioned in my previous post on Problems with STATUS_REPARSE - Part II, a lot of the times the problems come from user mode apps trying to build a path to a file and they expect to get a drive letter as the volume, which is just wrong. Even the MSDN example Obtaining a File Name From a File Handle falls into this trap by using drive letters all over:
…
if (GetLogicalDriveStrings(BUFSIZE-1, szTemp))
{
TCHAR szName[MAX_PATH];
TCHAR szDrive[3] = TEXT(" :"); <- this is wrong…
BOOL bFound = FALSE;
TCHAR* p = szTemp;
do
{
// Copy the drive letter to the template string
*szDrive = *p;
// Look up each device name
if (QueryDosDevice(szDrive, szName, MAX_PATH)) <- this is wrong...
{
size_t uNameLen = _tcslen(szName);
if (uNameLen < MAX_PATH)
{
bFound = _tcsnicmp(pszFilename, szName, uNameLen) == 0
&& *(pszFilename + uNameLen) == _T('\\');
if (bFound)
{
// Reconstruct pszFilename using szTempFile
// Replace device path with DOS path
TCHAR szTempFile[MAX_PATH];
StringCchPrintf(szTempFile,
MAX_PATH,
TEXT("%s%s"),
szDrive,
pszFilename+uNameLen);
StringCchCopyN(pszFilename, MAX_PATH+1, szTempFile, _tcslen(szTempFile));
...
Anyway, the best way to do this is to call GetFinalPathNameByHandle() and use the VOLUME_NAME_GUID flag to get used to using volume names. Unfortunately this is only available in Vista and newer OSes and so for XP one could still use the technique described in Obtaining a File Name From a File Handle but there is something that needs to be changed. The problem is that the volume APIs don't seem to have a way to convert a volume device name ('\Device\HarddiskVolume2') to a volume GUID name. In fact, none of the volume APIs offer an easy way to work with volume device names. The one way I've been able to do this in the general case was to use the MountMgr APIs directly. I don't have any user mode code that shows exactly what need to be done but I'll show the kernel mode code piece that queries the MountMgr:
#define MY_MOUNTMGR_MOUNT_POINT_TAG = 'mMyM'
typedef enum _MY_MOUNTMGR_BUFFER_TYPE {
//
// we'll query the MOUNTMGR using one of the three keys it supports..
//
MY_MOUNTMGR_SYMLINK = ' myS',
MY_MOUNTMGR_UNIQUE_ID = 'DIUU',
MY_MOUNTMGR_DEVICE = ' veD',
} MY_MOUNTMGR_BUFFER_TYPE, *PMY_MOUNTMGR_BUFFER_TYPE;
NTSTATUS
MyQueryMountMgr(
__in PVOID Buffer,
__in USHORT BufferLength,
__in MY_MOUNTMGR_BUFFER_TYPE BufferType,
__out PMOUNTMGR_MOUNT_POINTS * MountPoints
)
/*++
Routine Description:
Call MountMgr to get a names of a volume when knowing one of the
other names.
Arguments:
Buffer - the buffer that we want to send MountMgr to allow it to identify
the volume we're talking about.
BufferLength - the length of that buffer
BufferType - the type of information that the buffer describes.
MountPoints - this is a buffer that is allocated inside this function that
the caller must free which is the list of mount points that
MountMgr returned... if it's NULL then no buffer is returned..
This is NOT the standard convention (the caller should supply
the buffer) but it saves time.
Return Value:
an appropriate NTSTATUS value
--*/
{
NTSTATUS status = STATUS_SUCCESS;
PMOUNTMGR_MOUNT_POINT mountMgrKey = NULL;
ULONG mountMgrKeyLength = 0;
PIRP irp = NULL;
UNICODE_STRING mountMgrName;
PFILE_OBJECT mountMgrFileObject = NULL;
PDEVICE_OBJECT mountMgrDeviceObject = NULL;
IO_STATUS_BLOCK ioStatus;
KEVENT ioEvent;
PMOUNTMGR_MOUNT_POINTS mountMgrMountPoints = NULL;
ULONG mountMgrMountPointsLength = 0;
PAGED_CODE();
__try{
KeInitializeEvent( &ioEvent, NotificationEvent, FALSE);
//
// first try to set up the buffer for the name..
//
mountMgrKeyLength = sizeof(MOUNTMGR_MOUNT_POINT);
mountMgrKeyLength += BufferLength;
mountMgrKey = ExAllocatePoolWithTag( PagedPool,
mountMgrKeyLength,
MY_MOUNTMGR_MOUNT_POINT_TAG );
if (mountMgrKey == NULL) {
status = STATUS_INSUFFICIENT_RESOURCES;
__leave;
}
//
// populate the structure..
//
RtlZeroMemory( mountMgrKey, mountMgrKeyLength);
switch(BufferType) {
case MY_MOUNTMGR_DEVICE:
mountMgrKey->DeviceNameLength = BufferLength;
mountMgrKey->DeviceNameOffset = sizeof(MOUNTMGR_MOUNT_POINT);
break;
case MY_MOUNTMGR_UNIQUE_ID:
mountMgrKey->UniqueIdLength= BufferLength;
mountMgrKey->UniqueIdOffset = sizeof(MOUNTMGR_MOUNT_POINT);
break;
case MY_MOUNTMGR_SYMLINK:
mountMgrKey->SymbolicLinkNameLength= BufferLength;
mountMgrKey->SymbolicLinkNameOffset= sizeof(MOUNTMGR_MOUNT_POINT);
break;
default:
status = STATUS_INVALID_PARAMETER;
__leave;
break;
}
RtlCopyMemory( Add2Ptr(mountMgrKey, sizeof(MOUNTMGR_MOUNT_POINT)),
Buffer,
BufferLength );
//
// now we need a reference to MountMgr
//
RtlInitUnicodeString(&mountMgrName, MOUNTMGR_DEVICE_NAME);
status = IoGetDeviceObjectPointer( &mountMgrName,
FILE_READ_ATTRIBUTES,
&mountMgrFileObject,
&mountMgrDeviceObject);
if (!NT_SUCCESS(status)) {
__leave;
}
mountMgrMountPointsLength = sizeof(MOUNTMGR_MOUNT_POINTS);
status = STATUS_BUFFER_OVERFLOW;
while(status == STATUS_BUFFER_OVERFLOW) {
NT_ASSERT(mountMgrMountPoints == NULL);
mountMgrMountPoints = ExAllocatePoolWithTag( PagedPool,
mountMgrMountPointsLength,
MY_MOUNTMGR_MOUNT_POINT_TAG );
if (mountMgrMountPoints == NULL) {
status = STATUS_INSUFFICIENT_RESOURCES;
__leave;
}
irp = IoBuildDeviceIoControlRequest( IOCTL_MOUNTMGR_QUERY_POINTS,
mountMgrDeviceObject,
mountMgrKey,
mountMgrKeyLength,
mountMgrMountPoints,
mountMgrMountPointsLength,
FALSE,
&ioEvent,
&ioStatus);
if (irp == NULL) {
status = STATUS_INSUFFICIENT_RESOURCES;
__leave;
}
status = IoCallDriver( mountMgrDeviceObject, irp );
if (status == STATUS_PENDING) {
status = KeWaitForSingleObject( &ioEvent,
Executive,
KernelMode,
FALSE,
NULL );
status = ioStatus.Status;
}
switch (status) {
case STATUS_BUFFER_OVERFLOW:
//
// we need a bigger buffer, the Size should tell us how big.
// assert that it's more that we previously had...
//
NT_ASSERT(mountMgrMountPointsLength < mountMgrMountPoints->Size);
mountMgrMountPointsLength = mountMgrMountPoints->Size;
ExFreePoolWithTag( mountMgrMountPoints, MY_MOUNTMGR_MOUNT_POINT_TAG );
mountMgrMountPoints = NULL;
break;
case STATUS_OBJECT_NAME_NOT_FOUND:
//
// it is possible that the IOCTL doesn't find anything, this
// is not a problem...
//
break;
case STATUS_SUCCESS:
//
// we got the links back, all is good... for the delete case
// it's possible we'll get called multiple times..
//
NT_ASSERT((mountMgrMountPoints->NumberOfMountPoints != 0) ||
(MountMgrIoctl == IOCTL_MOUNTMGR_DELETE_POINTS));
break;
default:
NT_ASSERT(!"why are we here ? investigate...");
break;
}
}
}
__finally{
if (mountMgrKey != NULL) {
ExFreePoolWithTag( mountMgrKey, MY_MOUNTMGR_MOUNT_POINT_TAG );
}
if (mountMgrFileObject != NULL) {
ObDereferenceObject( mountMgrFileObject );
}
}
//
// if we have some mount points and we were successful and the caller
// gave us a pointer, then set it in that pointer. Otherwise, free it..
//
if (NT_SUCCESS(status) &&
MountPoints != NULL) {
*MountPoints = mountMgrMountPoints;
}
return status;
}
Subscribe to:
Posts (Atom)
Posts
