In the previous two sections we looked at a collection of filters that would manipulate data for us. In this section we will see how we may join them together to do more powerful data manipulation.There is a bit of reading involved in this section. Even though the mechanisms and their use are quite simple, it is important to understand various characteristics about their behaviour if you wish to use them effectively. ls. barry.txt bob example.png firstfile foo1 video.mpeg.
How To Configure Logging And Log Rotation In Apache On An Ubuntu VPS. The 'LogFormat' command defines a custom format for logs that can be called using the 'CustomLog' directive as we saw in the virtual host definition. Logging Using Pipes. Using a pipe instead of a file is an easy way to allow a separate logging program to handle the.
ls myoutput. ls. barry.txt bob example.png firstfile foo1 myoutput video.mpeg. cat myoutput.
barry.txt. bob. example.png. firstfile. foo1. myoutput.
video.mpeg.Let's break it down:. Line 1 Let's start off by seeing what's in our current directory.
Line 3 Now we'll run the same command but this time we use the to tell the terminal to save the output into the file myoutput. You'll notice that we don't need to create the file before saving to it. The terminal will create it automatically if it does not exist.
Line 4 As you can see, our new file has been created. Line 6 Let's have a look at what was saved in there.Some ObservationsYou'll notice that in the above example, the output saved in the file was one file per line instead of all across one line when printed to the screen. The reason for this is that the screen is a known width and the program can format its output to suit that. When we are redirecting, it may be to a file, or it could be somewhere else, so the safest option is to format it as one entry per line. This also allows us to easier manipulate that data later on as we'll see further down the page.
When piping and redirecting, the actual data will always be the same, but the formatting of that data may be slightly different to what is normally printed to the screen. Keep this in mind.You'll also notice that the file we created to save the data into is also in our listing. The way the mechanism works, the file is created first (if it does not exist already) and then the program is run and output saved into the file. Saving to an Existing FileIf we redirect to a file which does not exist, it will be created automatically for us. If we save into a file which already exists, however, then it's contents will be cleared, then the new output saved to it. wc -l myoutput. 8 myoutput.
wc -l. ls -l video.mpg blah.foo. ls: cannot access blah.foo: No such file or directory.rwxr-r- 1 ryan users 6 May 16 09:14 video.mpg. ls -l video.mpg blah.foo 2 errors.txt.rwxr-r- 1 ryan users 6 May 16 09:14 video.mpg. cat errors.txt. ls: cannot access blah.foo: No such file or directory.Maybe we wish to save both normal output and error messages into a single file.
This can be done by redirecting the STDERR stream to the STDOUT stream and redirecting STDOUT to a file. We redirect to a file first then redirect the error stream. We identify the redirection to a stream by placing an & in front of the stream number (otherwise it would redirect to a file called 1). I often find people try and write their pipes all out in one go and make a mistake somewhere along the line. They then think it is in one point but in fact it is another point. They waste a lot of time trying to fix a problem that is not there while not seeing the problem that is there.
If you build your pipes up incrementally then you won't fall into this trap. Run the first program and make sure it provides the output you were expecting. Then add the second program and check again before adding the third and so on. This will save you a lot of frustration.You may combine pipes and redirection too.
ls head -3 tail -1 myoutput. cat myoutput. example.png.More ExamplesBelow are some more examples to give an idea of the sorts of things you can do with piping. There are many things you can achieve with piping and these are just a few of them. With experience and a little creative thinking I'm sure you'll find many more ways to use piping to make your life easier.All the programs used in the examples are programs we have seen before. I have used some command line arguments that we haven't covered yet however.
Look up the relevant man pages to find out what they do. Also you can try the commands yourself, building up incrementally to see exactly what each step is doing.In this example we are sorting the listing of a directory so that all the directories are listed first.
Almost everything in Linux can be considered a file, but the main difference between a regular file and a named pipe is that a named pipe is a special instance of a file that has no contents on the filesystem.Here is quote from man fifo:A FIFO special file (a named pipe) is similar to a pipe, except that it is accessed as part of the filesystem. It can be opened by multiple processes for reading or writing. When processes are exchanging data via the FIFO, the kernel passes all data internally without writing it to the filesystem. Thus, the FIFO special file has no contents on the filesystem; the filesystem entry merely serves as a reference point so that processes can access the pipe using a name in the filesystem.The kernel maintains exactly one pipe object for each FIFO special file that is opened by at least one process. The FIFO must be opened on both ends (reading and writing) before data can be passed. Normally, opening the FIFO blocks until the other end is opened also.So actually a named pipe does nothing until some process reads and writes to it. It does not take any space on the hard disk (except a bit of meta information), it does not use the CPU.You can check it by doing this:Create a named pipe $ mkfifo /tmp/testpipeGo to some directory, for example /home/user/Documents, and gzip everything inside it, using named pipe.
$ cd /home/user/Documents$ tar cvf -. gzip /tmp/testpipe &1 28584Here you should see the PID of the gzip process. In our example it was 28584.Now check what this PID is doing $ ps u -P 28584USER PID%CPU%MEM VSZ RSS TTY STAT START TIME COMMANDc0rp 28584 0.0 0.0 pts/8 S 00:08 0:00 bashYou will see that it is using no resources. 0% CPU usage, 0% memory usage.Verify hunch regarding file space usage $ du -h /tmp/testpipe0 testpipeAnd again 0, nothing.
The testpipe could be used again if needed.Don't forget to kill gzip, using kill -15 28584. And remove our named pipe using rm /tmp/testpipeExample UsagesYou can redirect almost everything using named pipe. As example you can see this.Also nice explanation of named pipe usage. You can configure two processes on one server to communicate using a named pipe instead of TCP/IP stack. It is much faster, and does not load network resources. For example your Web Server can communicate with the database directly using a named pipe, instead of using localhost address or listening to some port.
You can let a program lie still and listen to a named pipe for some outside event. As soon as the outside event occurs (f.ex. Arrival of some new data)this could be detected by some other program which in turn opens the pipe for write, writing the relevant event data to the pipe. When the close statement is issued, the listening program will receive the stream of data through the pipe via a read statement, and is ready to process what it has got.
Don't forget tor close the pipe after reading the content.The listening program could also return results of its processing via the same, or via another named pipe. Such inter-program communications is veryconvenient at times.