Exetools  

Go Back   Exetools > General > General Discussion

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 07-09-2011, 23:31
Kerlingen Kerlingen is offline
VIP
 
Join Date: Feb 2011
Posts: 303
Rept. Given: 0
Rept. Rcvd 274 Times in 97 Posts
Thanks Given: 0
Thanks Rcvd at 261 Times in 79 Posts
Kerlingen Reputation: 200-299 Kerlingen Reputation: 200-299 Kerlingen Reputation: 200-299
Flushing input stream

I've a little programming related problem and I don't know how to solve it.

A console program (lets call it "prog1.exe") has the following pseudo-code:
Code:
while not (Ctrl_Break)
{
  x = fetch_stats();     // Read some info
  writeconsole(x);       // Output the info to the console
  wait(60);              // Wait 60 seconds and repeat the whole process
}
I want to log the output of the program both to screen and to a file. So I wrote a little program ("tee.exe") which does:
Code:
while not (EndOfFile(StdIn))
{
  c = read(StdIn);     // Read one char from StdIn
  write(StdOut, c);    // Write one char to StdOut
  write(FILE, c);      // Write the same char to a file
}
In theory, executing "prog1.exe | tee.exe" would redirect all output from "prog1.exe" as input to "tee.exe", while "tee.exe" would write the input it gets to both the console and a file.

The problem is the output buffer. For performance reasons, the OS doesn't write the output of "prog1.exe" to "tee.exe" char-by-char, but waits until the output buffer is full and then writes the whole buffer to "tee.exe". Since "prog1.exe" does only output a few bytes every minute, it takes about 40 minutes before the output buffer is full and transfered to "tee.exe".

Running "prog1.exe >> output.txt" has a similar result: There is no output to the console any more und the file is also only updated when the output buffer is full.

I don't have access to "prog1.exe" or it's source code, so I can't flush the output buffer. Flushing the input buffer is not possible for some reason. The only workarounds I found for the problem describe how to flush the keyboard buffer.

Does anybody have an idea how to solve this problem?

My goal is simple:
1. See the console output in real time
2. Write the console output to a file
Reply With Quote
  #2  
Old 07-10-2011, 00:41
Git's Avatar
Git Git is offline
Old Git
 
Join Date: Mar 2002
Location: Torino
Posts: 1,095
Rept. Given: 220
Rept. Rcvd 265 Times in 157 Posts
Thanks Given: 91
Thanks Rcvd at 207 Times in 121 Posts
Git Reputation: 200-299 Git Reputation: 200-299 Git Reputation: 200-299
There is an ANSI C _flushall() command that causes all open buffers to be flushed to file. There is also a stream.flush() function if you are using C++ stream i/o. They may help, but I am not quite sure what you mean by flushing an input buffer, but if you don't have the source code to prog1 I don't see how you can affect it's behaviour. Unless.. if you can redirect stdout to a pipe then i/o can be made non-buffered, but I am still unsure if it can work.

Git

Last edited by Git; 07-10-2011 at 00:54.
Reply With Quote
The Following User Gave Reputation+1 to Git For This Useful Post:
chessgod101 (07-10-2011)
  #3  
Old 07-10-2011, 01:39
Kerlingen Kerlingen is offline
VIP
 
Join Date: Feb 2011
Posts: 303
Rept. Given: 0
Rept. Rcvd 274 Times in 97 Posts
Thanks Given: 0
Thanks Rcvd at 261 Times in 79 Posts
Kerlingen Reputation: 200-299 Kerlingen Reputation: 200-299 Kerlingen Reputation: 200-299
"prog1.exe" writes to the console. The default behavior ist unbuffered. The output will be written to the console every 60 seconds.

As soon as the output is redirected (to file ">>" or to pipeline "|"), the OS (both Windows and Linux) switch to buffered output. This means the output will be only written when the output buffer is full, which takes about 40 minutes each time. Somebody could "flush" the output buffer, which would write the contents of the buffer at once, even if it isn't full yet. Or somebody could enable unbuffered output again. But both changes would need to change "prog1.exe", something which is not possible in this situation.

The other end of the pipeline belongs to "tee.exe" (full source available). It reads all data from the standard input. The problem is that "tee.exe" only gets something to read when "prog1.exe" sends something, which will only happen when its output buffer is full.

Flushing the input buffer would mean something like "Dear OS, please send me all data from prog1.exe, even if the buffer is not full yet". Unfortunately this kind of command is not supported for some reason.

I also tried to set "stdin" to unbuffered without success. Using a very small buffer for "stdin" didn't help either, data was only available for read when the output buffer of "prog1.exe" was full, not earlier.
Reply With Quote
  #4  
Old 02-13-2012, 21:06
redbull redbull is offline
Friend
 
Join Date: Mar 2004
Posts: 160
Rept. Given: 17
Rept. Rcvd 5 Times in 4 Posts
Thanks Given: 3
Thanks Rcvd at 6 Times in 6 Posts
redbull Reputation: 5
would a standard close(FILE) not work with stderr and stdout ?
Reply With Quote
  #5  
Old 02-14-2012, 05:23
*RemedY* *RemedY* is offline
Family
 
Join Date: Sep 2003
Posts: 115
Rept. Given: 18
Rept. Rcvd 72 Times in 30 Posts
Thanks Given: 0
Thanks Rcvd at 3 Times in 3 Posts
*RemedY* Reputation: 72
Here is something which I hope it helps:

First, the program that writes to stdout. Since you've written that you don't have access to the source, this one is exchangeable:

Code:
#include <windows.h>

int main()
{
    char c[] = "";
    unsigned long count;
    HANDLE stdout = GetStdHandle(STD_OUTPUT_HANDLE);
    for(int i = 0; i < 10; i++)
    {
        c[0] = i + 0x30;
        WriteFile(stdout, c, 1, &count, NULL);
        Sleep(1000);
    }
    c[0] = 0x0;
    WriteFile(stdout, c, 1, &count, NULL);
    CloseHandle(stdout);
    return 0;
}
Now the one that reads from stdin and writes through at once:

Code:
#include <windows.h>

int main()
{
    char c[] = " ";
    unsigned long incount, outcount, foutcount;
    HANDLE stdin = GetStdHandle(STD_INPUT_HANDLE);
    HANDLE stdout = GetStdHandle(STD_OUTPUT_HANDLE);
    HANDLE File;
    
    File = CreateFile("test.txt", 
                       GENERIC_WRITE, 
                       0, 
                       NULL, 
                       CREATE_NEW, 
                       FILE_ATTRIBUTE_NORMAL|FILE_FLAG_WRITE_THROUGH, 
                       NULL);
    while(c[0])
    {
        ReadFile(stdin, &c[0], 1, &incount, NULL);
        FlushFileBuffers(stdin);
        WriteConsole(stdout, c, 1, &outcount, NULL);
        WriteFile(File, c, 1, &foutcount, NULL);
        FlushFileBuffers(File);
    }
    CloseHandle(File);
    CloseHandle(stdout);
    CloseHandle(stdin);
    return 0;
}
I'm pretty sure you can adjust it to your needs.
If that doesn't help you, I'm wrong and I didn't get you right.
I'm sorry then.

Best regards
*RemedY*
Reply With Quote
  #6  
Old 02-14-2012, 06:55
*RemedY* *RemedY* is offline
Family
 
Join Date: Sep 2003
Posts: 115
Rept. Given: 18
Rept. Rcvd 72 Times in 30 Posts
Thanks Given: 0
Thanks Rcvd at 3 Times in 3 Posts
*RemedY* Reputation: 72
I'm so sorry.
I just saw that this thread is actually seven month old
I was fooled by redbull's reply and didn't recognise the real age of this thread.
Shame on me.

*RemedY*

Anyways, maybe my reply above is useful for someone.
Reply With Quote
  #7  
Old 02-21-2012, 18:08
Kerlingen Kerlingen is offline
VIP
 
Join Date: Feb 2011
Posts: 303
Rept. Given: 0
Rept. Rcvd 274 Times in 97 Posts
Thanks Given: 0
Thanks Rcvd at 261 Times in 79 Posts
Kerlingen Reputation: 200-299 Kerlingen Reputation: 200-299 Kerlingen Reputation: 200-299
The thread is not old to me, the problem still exists.

Thank you for the code you posted, but it doesn't work either.
"ReadFile(stdin)" will not return anything until the output buffer is full.
"FlushFileBuffers(stdin)" will not change anything about this behaviour.

And your code contains a small error: "while(c[0])" will generate an endless loop, since c[0] will contain the last char which was read, which won't be "0" (false condition) when reaching EndOfFile or closing the handle.

@redbull:
While CloseHandle(stdout) might work, it would close the handle, making any further output impossible. "prog1.exe" is also running at higher privileges than "tee.exe", so even if I wanted to close the handle, I wouldn't have permission to do so.
Reply With Quote
  #8  
Old 02-22-2012, 22:31
2late 2late is offline
Friend
 
Join Date: Nov 2003
Posts: 48
Rept. Given: 5
Rept. Rcvd 6 Times in 3 Posts
Thanks Given: 12
Thanks Rcvd at 11 Times in 9 Posts
2late Reputation: 6
have you tried mtee?

Code:
http://www.commandline.co.uk/mtee/index.html
Reply With Quote
  #9  
Old 02-23-2012, 04:00
*RemedY* *RemedY* is offline
Family
 
Join Date: Sep 2003
Posts: 115
Rept. Given: 18
Rept. Rcvd 72 Times in 30 Posts
Thanks Given: 0
Thanks Rcvd at 3 Times in 3 Posts
*RemedY* Reputation: 72
@Kerlingen

no, there is no error.
Did you try the code? I guess you didn't.
In the first program the last char written is 0x0, thus there is no endless loop. Also, it writes directly to stdout and to the file, not only when the buffer is full. At least, it works with the two small programs above under Win7. As I've written, I'm pretty sure you can adjust it to your needs. If you can't get it to work, I'd be interessted in your input file.

Best regards
*RemedY*
Reply With Quote
  #10  
Old 03-06-2012, 09:31
Kerlingen Kerlingen is offline
VIP
 
Join Date: Feb 2011
Posts: 303
Rept. Given: 0
Rept. Rcvd 274 Times in 97 Posts
Thanks Given: 0
Thanks Rcvd at 261 Times in 79 Posts
Kerlingen Reputation: 200-299 Kerlingen Reputation: 200-299 Kerlingen Reputation: 200-299
There is no such thing like an "input file", there is just a service daemon which outputs his log messages to the console.

I want to see the messages in real time on the console and additionally log them to a file.
Reply With Quote
  #11  
Old 03-06-2012, 18:56
*RemedY* *RemedY* is offline
Family
 
Join Date: Sep 2003
Posts: 115
Rept. Given: 18
Rept. Rcvd 72 Times in 30 Posts
Thanks Given: 0
Thanks Rcvd at 3 Times in 3 Posts
*RemedY* Reputation: 72
Ah, OK.

Win or *nix?

I want to know that, because I will set up a small daemon then and try it with that. I hope I can help you.

Best regards
*RemedY*
Reply With Quote
  #12  
Old 03-06-2012, 19:23
Git's Avatar
Git Git is offline
Old Git
 
Join Date: Mar 2002
Location: Torino
Posts: 1,095
Rept. Given: 220
Rept. Rcvd 265 Times in 157 Posts
Thanks Given: 91
Thanks Rcvd at 207 Times in 121 Posts
Git Reputation: 200-299 Git Reputation: 200-299 Git Reputation: 200-299
That really doesn't sound too much to ask for, does it?. it must be possible!

Git
Reply With Quote
  #13  
Old 03-29-2012, 20:34
Sergey Nameless
 
Posts: n/a
It should be possible to change file handler to be non-blocking. Not sure of windows calls, but on *nix this can be done by
flags = fcntl(fd, F_GETFL, 0);
fcntl(fd, F_SETFL, flags | O_NONBLOCK);
Reply With Quote
The Following User Gave Reputation+1 to For This Useful Post:
|roe (03-30-2012)
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[ARMADILLO] 1 serial & 1 key - need input Maltese General Discussion 2 02-18-2005 00:58


All times are GMT +8. The time now is 10:33.


Always Your Best Friend: Aaron, JMI, ahmadmansoor, ZeNiX
( 1998 - 2020 )