Node.js Streams: What They Are and How to Use Them

Step 1 – Understand the Stream Types

• Readable: emits data you can .pipe() out (e.g., file reads, HTTP responses).
• Writable: accepts data you .write() in (e.g., file writes, HTTP requests).
• Duplex: both readable and writable (e.g., TCP sockets).
• Transform: duplex that modifies data along the way (e.g., zlib.createGzip()).

Step 2 – Reading a File as a Stream

Instead of fs.readFile(), use fs.createReadStream() to handle large files without exhausting memory:

const fs = require('fs');

const readStream = fs.createReadStream('large.txt', { encoding: 'utf8' });

readStream.on('data', chunk => {
  console.log('Received', chunk.length, 'bytes');
});

readStream.on('end', () => console.log('File fully read'));

chunk is a Buffer (or string if encoding is specified). Node reads the file in 64 KB blocks by default.

Step 3 – Piping a Readable into a Writable

The .pipe() method connects streams and manages flow automatically:

const writeStream = fs.createWriteStream('copy.txt');
readStream.pipe(writeStream);

Node handles backpressure—pausing the source when the destination’s internal buffer is full and resuming when drained.

Step 4 – Transform Streams (Gzip Example)

Transform streams modify data mid-flight. Compress a file with Gzip:

const zlib = require('zlib');
const gzip = zlib.createGzip();

fs.createReadStream('large.txt')
  .pipe(gzip)
  .pipe(fs.createWriteStream('large.txt.gz'));

Step 5 – Handling Backpressure Manually

Sometimes you need explicit control. Check the return value of write() and wait for the drain event:

const writable = fs.createWriteStream('output.txt');

function writeLots() {
  let ok = true;
  let i = 0;
  while (i < 1e6 && ok) {
    ok = writable.write(`Line ${i++}\n`);
  }
  if (i < 1e6) {
    writable.once('drain', writeLots);
  }
}
writeLots();

drain signals that the buffer has flushed and writing can continue without data loss.

Step 6 – Robust Error Handling

Always attach error listeners to prevent uncaught exceptions:

readStream.on('error', console.error);
writeStream.on('error', console.error);

You can also use pipeline() from stream/promises to automatically handle cleanup on error.