Files vs File descriptors
First of all we'll start with the difference between a file structure and a file descriptor. Imagine you have an array of files like this:
files[] -> file1 | file2 | file3
file1, file2 and file3 are file data structures. A file structure is an opaque data structure. Opaque means that we don't really know how that data structure looks like and we don't either bother about it. All we need to know is that a file structure represents a file on a hard disk, a USB stick or whatever other storage device.
Going back to our example, the file descriptors are the indices(plural for index) of the array. So the indices in the above example would look like this:
| index | 0 | 1 | 2 |
| file structure | file1 | file2 | file3 |
Why do we need file descriptors when we have file stuctures?
The truth is that a process keeps track only of file descriptors. The file structures are on the side of the kernel. So the reason is the same reason that we use pointers and not actual data structures when we program in general: to save space and time, or otherwise efficiency. (There is also the element of security but we will not go into that.)
In C code you will often see something like this:
[sourcecode language="cpp" light="true"]
FILE* myfile;
[/sourcecode]
This is something totally different from the a file structure in the kernel that we saw earlier. FILE in C is just a wrapper, or simply said; a structure holding an other structure. In fact FILE in C is just a file descriptor with some extra bells and whistles, nothing more. So why use FILE in C when we can use the file descriptors? Well file descriptors are just numbers. Imagine if we had to open and close numbers all the time. It would be hard to keep track of what we are doing. Everything would be a mess! Except the much friendlier name of a FILE, the FILE structure in C lets us also use more advanced functions which can take a FILE as argument but not a naked file descriptors.
When you start a new process, three file descriptors are created by default. These three file descriptors are called the standard file descriptors and are given the numbers 0, 1, 2. If you remember the Unix mantra, it says that everything in a Unix system is considered a file. That is even true for hardware devices like your monitor and keyboard. In fact there are file structures in the kernel that are corresponding to just those. The file descriptors 0, 1, 2 are indices to these special files. To be more exact, 0 is the index corresponding to the "keyboard file" and 1 and 2 are indices corresponding to the "monitor file".
Streams
Earlier we said that there are three file descriptors created by default: 0, 1 and 2. We said that 0 corresponds to the "keyboard file" in the kernel and 1 and 2 correspond to the "monitor file". If we were to sketch all this on paper, it would look a bit like below.
Process1 has the three default file descriptors we talked about. Notice how we hide the "keyboard file" and "monitor file" in the kernel box. That is mostly for simplicity as there is a lot of things going on in the kernel. Other than that the user(in this case the programmer) is not meant to know the inner workings of the kernel. The only thing the programmer can see is the C-like FILE structures and the file descriptors so we play that way.
Now there is something more than file descriptors in this figure. It's the arrows that show the flow of the data. These arrows, call them channels, call them buses, call them rivers, or whatever you like, have a special name: streams. Stream is just an abstract name to make it easier for the programmer to visualize what is happening with the data. It's merely easier to talk about stream of data than to talk about indices and the file structures in the kernel that those indices correspond to.
Now the three default file descriptors we talked about earlier, have in fact been baptized with their own special names: stdin, stdout and stderr.
These names are just abstract words that we use to talk about three specific channels of data(in most cases characters). Stdin is the data that we get from the user. Stdout is the flow of normal output to the user and stderr is the flow of output when errors happen in the program. Hopefully now it makes sense why file descriptor 0 is corresponding to the keyboard and file descriptors 1 and 2 are corresponding to the monitor. That is the program gets input from the keyboard while output goes to the monitor.
Now in C and many other languages there are three specific macros that are called stdin, stdout and stderr. These are not streams although they have the same name. That wouldn't be possible anyway as streams is just an abstract idea(as said earlier) so that the programmer has it easier to visualize what is happening(even if it makes it a living hell for some). The stdin, stdout and stderr macros in C are just pointers to FILE structures(the C-like ones). You can in fact use these functions as you would use any FILE when programming in C. For example see the code below.
[sourcecode language="cpp" light="true"]
fprintf(stdout, "linux");
fprintf(stderr, "meerkat");
[/sourcecode]
Will print "linuxmeerkat" on the screen. A question arises: Why do we have two macros that direct data to the same place? This is the same question as: Why do we have two streams to the monitor? The answer is rather simple. Many times when we develop a program we need to output errors on a different channel than the normal output. It's just more neat to keep different things separated than mix them. Think for example how easier it is now if we want to hide all error messages in our program or just redirect them to a different place than the monitor.
Finally here is a table with the standard file descriptors(fd) and the corresponding standard streams:
| fd | stream |
|---|---|
| 0 | stdin |
| 1 | stdout |
| 2 | stderr |
File descriptor tables
It's an important detail to understand that file descriptors are the only file-relevant thing that a process can keep track of. As said earlier FILE structures in C are just a wrapper for a file descriptor so they can also be thought as file descriptors to justify the fact that a process has knowledge of only file descriptors. Now, each process keeps its unique file descriptor table. Say that we have two processes: process 1 and process 2.
| process1 |
|---|
| 0 |
| 1 |
| 2 |
| process2 |
|---|
| 0 |
| 1 |
| 2 |
When a new process gets created, file descriptors 0, 1 and 2 are created automatically and mapped to stdin, stdout and stderr.
To the eye the above two file descriptor tables look the same. That is, they have the same numbers. However the file descriptor 0 in process1 can be pointing to a totally different thing than the file descriptor 0 in process2.
(You may ask yourself: "How can file descriptor 0 in the two processes point to different things, if as we assumed, file descriptors are indices?". Well that is a very logical question and the explanation is rather simple. A file descriptor is bundled with a pointer(in the abstract meaning), or if you like, an index to the global Open File Table. That is however nothing we should be worried about, so it's not shown in the diagrams of this article.)
Just because we say that file descriptors 0, 1, 2 are standard it doesn't mean that they are always going to correspond to stdin, stdout and stderr. Standard streams are as we think until the programmer decides it's time to change things around. That is something that I will demonstrate. Bellow you see a visual representation of file descriptors and their connection with the kernel.
This is how the processes look like if we assume that file descriptors are not altered in any way. The arrows show the flow of data. The keyboard and screen icons in the kernel are file structures of the special files. The truth is that things are more complicated in the kernel. However now we focus on keeping track of the file descriptors. That's also the only thing we can alter directly from inside the process. However don't assume that because there are two icons that there are only two file structures.
Now let's change the file descriptors in the first process a bit and see what happens:
[sourcecode language="cpp" light="true"]
close(1); //we close the stdout stream
FILE* f=fopen("myarticle.txt", "w"); //open a file for writing
fprintf(stderr, "file has fd: %d\n", fileno(f));
[/sourcecode]
Pay attention that we fprintf() to the stderr as stdout is closed. From the code above this is what we get printed when we run the program:
file has fd: 1
From this output it's crystal clear that file descriptor 1 is not pointing to stdout anymore. In fact it's pointing to the file myarticle.txt. How do I know? The reason I know is that the usual behavior of the kernel is whenever we try to create a new file descriptor to give it the lowest number possible. That keeps things clean. Following this paradigm, after we close the stdout stream in the example, we immediately create a file for writing. This file needs a file descriptor. The kernel sees that the lowest number that can be used is 1 so the file gets that index number.
What about the file in the kernel that already has index 1? you might ask. As we said, the kernel is a bit more complicated than shown in the above figure. The kernel doesn't mess files from different processes, even if they are the same files. So file descriptor 1 from the first process corresponds to a totally different file structure in the kernel than the file descriptor 1 in the second process. That's something that a programmer shouldn't worry about. As we said.. care only about the file descriptors. The kernel is a magician that you shouldn't be aware of how his tricks work.
Here follows a figure with the file descriptors and the kernel after we ran the above code. The file description tables of both processes look like before. However look how the streams on the first process are now differentiated!
We see a new arrow there pointing to a new icon in the kernel. That icon is a new file data structure created in the kernel. The new arrow is a stream like stdin, stdout, stderr. The only difference is that we don't have a standard name for it. However we can clearly see from the figure that the data is flowing to a file and particularly the file myarticle.txt. Notice that stderr is still flowing to the monitor. That's also the reason we can still print text on fprintf() to the screen. If we closed this channel then we wouldn't be able to output anything on the screen anymore.
TIP: When you want to sketch the file descriptors in paper I find it convenient to write the file descriptor in the first column and in the second column the stream-name or filename in case the stream is nameless. In this example I would write it like bellow.
| process1 | |
|---|---|
| 0 | stdin |
| 1 | myarticle.txt |
| 2 | stderr |
Why clones are helpful (duplicating file descriptors)
When we say that a file descriptor is removed or closed, it means that the file descriptor is destryied! Deallocated. No returning back. Nada! It's gone forever and ever! We have lost it and with it also lost the stream it was connected to.
Now I want to remind you of the popular so called memory-leak in C. Say we allocate some memory in a function for a pointer. If we have only one pointer to the allocated memory space and somehow we lose it, we automatically get a memory leak as there is no way we can reach the memory where the pointer was pointing at. In C a solution would be to have a backup pointer and that's also a solution used with file descriptors.
When we duplicate a file descriptor with dup(), what we actually do is making a second index for a file structure in the kernel. Let's take an example:
[sourcecode language="cpp" light="true"]
int newfd; //we declare a new file descriptor
newfd=dup(1); //we make it a clone of file descriptor 1
printf("newfd: %d\n", newfd);
[/sourcecode]
Output:
newfd: 3The new file descriptor newfd is a clone of file descriptor 1 (stdout). Otherwise said, file descriptor 1 and 3 point to the same file structure in the kernel. That means that destroying file descriptor 1 is not going to have any effect on newfd. See the bellow code where we continue on the same example.
[sourcecode language="cpp" light="true"]
close(1); //destroying file descriptor 1 (stdout)
close(2); //destroying file descriptor 2 (stderr)
dprintf(newfd, "test"); //sending some data to the cloned file descriptor
[/sourcecode]
Output:
testAs you see we destroyied all file descriptors to the monitor. However as we had made a backup file descriptor of 1, we can still print on the screen. By the way dprintf() is the same function as fprintf() with only difference that it takes a file descriptor as parameter instead of a FILE.
Here is also a visualisation of what we did:
I hope you can clearly see now how dup() works. The function merely duplicates a file descriptor. You might wonder about something however. If file descriptors are numbers, then why not just copy the file descriptor number to a new integer variable? Something like this:
[sourcecode language="cpp" light="true"]
FILE *f=fopen("test.txt", "r"); //open a file
int filefd=fileno(f); //get file's fd
int filefd2; //a secondary fd to the file
filefd2=filefd; //this is where we "duplicate"
[/sourcecode]
This is wrong and will not work. You see we don't make a new file description in this case. filefd2 and filefd are going to have the exact same value and thus being the same exact index. The idea of duplication is to make a new file descriptor, a new index. In this example if we delete filefd, there is no way to access the file structure in the kernel through filefd2 as filefd2 and filefd are the exact same thing and thus deleting one is like deleting the other.
The below code is the correct way to do it.
[sourcecode language="cpp" light="true"]
FILE *f=fopen("test.txt", "r"); //open a file
int filefd=fileno(f); //get file's fd
int filefd2; //a secondary fd to the file
filefd2=dup(filefd); //this is where we "duplicate"
[/sourcecode]
If filefd gets deleted now, we can still access it through filefd2.
A few words on pipes
I was concidering leaving pipes outside of this article. However I want people to have a full grasp of the tight relation between pipes and file descriptors so I will mention the basics on pipes here.
Pipe is essentially a pair of file descriptors. One file descriptor is used for input while the other file descriptor is used for output. Now what we feed to the one file descriptor will magically pop out the other file descriptor. Binary, characters, integers, everything is welcome and works. You can think of a pipe as a physical pipe that whatever you drop on the top end, will get out from the end at the bottom.
We declare a pipe just as an array of two integers like this:
[sourcecode language="cpp" light="true"]
int mypipe[2];
[/sourcecode]
Now this alone doesn't do anything. We have to tell the kernel to setup the pipe for us so that we can use it:
[sourcecode language="cpp" light="true"]
pipe(mypipe); //initializing the pipe
[/sourcecode]
Now we have our fully working pipe.
As we said data goes through one file descriptor and comes out through the other one. The file descriptor that takes input is mypipe[1] and the one used for output is mypipe[0]. Notice that the numbers 1 and 0 are not file descriptors. They are the indices of the pipe. You should be very super extra careful on which end of the pipe is supposed to get data and which end is supposed to give data.Programmers are probably some very egoistic bastards. I refer to the people that implemented the pipes in the kernel and you will see why. When it comes to pipes we refer to the write end of the pipe or the read end of the pipe from the view of the programmer and not the pipe itself. So for example the write end of the pipe is mypipe[1] while the read end is mypipe[0]. However if you are a plummer or an electronics person then you are familiar with reading the input/output from the view of the pipe itself. So be extra careful on that detail! You might spend endless hours, days or weeks debugging code just because you mixed the read end with the write end.
Now let's test our pipe:
[sourcecode language="cpp" light="true"]
char buffer[5]="";
write(mypipe[1], "test", 5); //writing to pipe
read(mypipe[0], buffer, 5); //reading from pipe
puts(buffer);
[/sourcecode]
This should output the text "test" on the terminal.
I will not go any deeper into pipes as that needs its own article in my humble opinion. I hope I made it a bit clearer what file descriptors really are and their relation to pipes and files.