Optimizing tail with Rx

Chris was concerned that the initial Rx implementation was inefficiently reading an entire 5 GB text file throwing everything away except for the last 10 lines. To optimize I/O, I've modified the code to read the file in reverse. Here's a new version that tails a 5 GB text file in less than 150 ms:

const int bufferSize = 2 << 15;
using (var stream = new FileStream (
                        args[0], 
                        FileMode.Open, 
                        FileAccess.Read, 
                        FileShare.Read, 
                        bufferSize, 
                        FileOptions.Asynchronous | FileOptions.RandomAccess)) {
    // seek to (EOF - bufferSize) and read stream backwards, returning IObservable<byte[]>
    stream.ReverseAsyncRead (bufferSize)
        // reverse bytes inside byte[]
        .Select(bytes => bytes.Reverse().ToArray())
        // split byte[] on \r and \n
        .Split (new byte[] { 13 }, new byte[] { 10 })
        // take 10 of the byte[] results
        .Take (10)
        // reverse order of byte[]s, and the bytes inside the byte[]s
        .ToEnumerable ().Reverse ().Select (bytes => bytes.Reverse ().ToArray ()).ToObservable ()
        // using ASCII encoding, convert bytes to string
        .Decode (Encoding.ASCII)
        // output the IObservable<string> to the standard output
        .Run (Console.WriteLine);
}

Starting from the end of the file, reading chunks backwards, reversing the byte[] chunks, delimiting by \r and \n (creating smaller byte[] results), take the first 10, reversing them back, converting to string (via ASCII encoding), and then sending to standard output. In this version we are reading from disk only the information required to tail the file - no unnecessary disk IOs. Unfortunately some of these extension methods didn't exist, so I had to create three to build this:

• ReverseAsyncRead (this Stream): just like AsyncRead (), but in reverse
• Split<T> (this T[]): just like string.Split (), but generic, for any array, and for observable input arrays
• Decode (this byte[]): using Encoding and a Decoder, converts a IObservable<byte[]> to IObservable<string>

Comments

[louis vuitton taiga leather]

of course like your web site but you need to check the spelling on quite a few of your posts. A number of them are rife with spelling issues and I find it very bothersome to tell the truth nevertheless I’ll certainly come back again.louis vuitton taiga leather

[patek philippe replica watches]

hi!,I like your writing so much! share we communicate more about your article on AOL? I need an expert on this area to solve my problem. May be that's you! Looking forward to see you. patek philippe replica watches

[basketball shoes sale]

LA Lakers 92-89 Denver Nuggets, won three straight. LA Lakers (3-2), Andrew Bynum wear adizero ghost suspended four games and come back with 29 points and 13 rebounds, Kobe Bryant wear nike kobe vii 17 points, 10 rebounds and nine assists, Pau Gasol wear nike zoom hyperfuse 17 points and five rebounds. Denver Nuggets (2-2), Nene Hilario wear nike rookie lwp 13 points and eight rebounds, Arron Afflalo wear adizero rose 13 points, Ty Lawson wear adizero rose 2.5 critical moment missed two free throws, 9 points and 8 assists. Off the bench Al Harrington wear adizero rose 2 21 points, Andre Miller wear nike lebron 9 13 points. LA Lakers in the first half after leading 16-10, but only after four minutes the Lakers hit a ball, finally, Denver Nuggets 23-22 lead with 1 minute. LA Lakers outside Section II failed to score, Chris Andersen wear nike kobe vi Section II will come on a dunk, Denver Nuggets25-22 lead. Steve Blake wear nike lebron of two free throws, Pau Gasol wear nike zoom kobe jumper scored 2 points, LA Lakers will lead into 25 levels. Andrew Bynum wear adizero rose dominate immediately scored the basket, LA Lakers re-overtake. Denver Nuggets in this section only the final 1 minute 55 seconds 2 minutes, the first half to 45-46 behind. LA Lakers in the third quarter, shortly after, and finally by Matt Barnes wear nike lebron 8 the first two-thirds vote. LA Lakers lead to 63-56, but the main end, the counterattack was the Denver Nuggets. Denver Nuggets scored six points, to 69-65 exceeded. After three sessions, L. A. Lakers to 69-71 at a disadvantage. Competition also 3 minutes and 40 seconds, Denver Nuggets dunk, Denver Nuggets to 89-84 lead. Kobe Bryant wear nike zoom kobe sent pass, Andrew Bynum alley-oop dunk and cause foul, Nene Hilario wear nike lebron 8 v2 then do not dunk, while another two free throws Kobe Bryant, Andrew Bynum wear adizero crazy light after a layup in the cap, LA Lakers scored six points in the game 1 minute 52 seconds to 91-89 exceeded. And Ty Lawson wear nike lebron 8 ps in the game 1 minute 13 seconds even missed two free throws, missed opportunity to tie. Kobe Bryant wear nike hyperdunk 2011 pointer with 1.1 seconds of two free throws, LA Lakers 3 points ahead. Denver Nuggets desperate not in the third. The last 3 minutes and 40 seconds without a point Denver Nuggets.

[soccercleats]

English Premier League 2011/12 season, the 19th round of a competition game at the Emirates Stadium, Arsenal at home 1-0 Queens Park Rangers, Robin Van Persie scored and broke Henry wear adidas f50 adizero trx annual league scoring record. Theo Walcott wear nike mercurial vapor superfly iii, Aaron Ramsey wear adipower predator sl and Andrey Arshavin wear adidas f50 adizero micoach with the playing rotation to provide support to Robin Van Persie.
Queens Park Rangers Lineup (4-2-3-1): 24-Radek Cerny weart nike total 90 laser iv; 18-Luke Young, 16-Matthew Connolly wear adidas adipure iv sl, 6-Daniel Gabbidon, 13-Armand Traore wear nike ctr360 maestri ii elite (78', 2-Bradley Orr); 11-Alejandro Faurlin wear mercurial vapor superfly iii, 17-Joey Barton; 12-Jamie Mackie wear nike mercurial vapor superfly iii cr (74', 21-Tommy Smith), 7-Adel Taarabt, 32-Shaun Wright-Phillips wear nike total 90 laser iv; 10-Jay Bothroyd (64', 9-DJ Campbell wear nike mercurial vapor superfly iii).
Arsenal Lineup (4-2-3-1): 13-Wojciech Szczęsny wear wayne rooney cleats; 20-Johan Djourou, 6-Laurent Koscielny wear adipure iv, 4-Per Mertesacker, 5-Thomas Vermaelen wear cristiano ronaldo cleats (54', 39-Francis Coquelin); 8-Mikel Arteta wear nike tiempo legend iv elite, 17-Alexandre Song; 14-Theo Walcott wear messi soccer cleats (75', 27-Gervinho), 16-Aaron Ramsey wear adidas adipower predator, 23-Andrey Arshavin (67', 7-Tomas Rosicky wear tiempo legend iv); 10-Robin Van Persie.