File System & Streams

File Operations with fs

The fs module provides callback, sync, and promise APIs. Prefer fs/promises with async/await for readable control flow without blocking the event loop with sync variants.

Use path.join to build cross-platform paths instead of string concatenation. Check file existence with access or stat before assuming paths are valid.

For large files, read incrementally with streams rather than readFile, which loads entire contents into memory.

• Always specify encoding utf8 for text files
• Handle ENOENT with try/catch for optional config files
• Use mkdir with recursive: true before writing nested paths

import { readFile, writeFile } from 'node:fs/promises';
import path from 'node:path';

const configPath = path.join(process.cwd(), 'config.json');
const config = JSON.parse(await readFile(configPath, 'utf8'));

Readable and Writable Streams

Streams process data in chunks, keeping memory bounded. Readable streams produce data; writable streams consume it. Transform streams modify data in flight.

Listen for data, end, and error events on readables. Writable streams emit drain when it is safe to write more after backpressure.

Node core streams include fs.createReadStream, HTTP request/response bodies, and zlib compression streams.

• Prefer highWaterMark tuning only after profiling
• Destroy streams on early exit to free file descriptors
• Use pipeline() for automatic error forwarding and cleanup

import { createReadStream } from 'node:fs';

const stream = createReadStream('large.log', { encoding: 'utf8' });
stream.on('data', chunk => process.stdout.write(chunk.length + ' chars\n'));
stream.on('error', err => console.error(err));

Piping and pipeline

stream.pipe connects readable output to writable input, handling backpressure automatically. For robust error handling, use pipeline from node:stream/promises or callbacks.

Common pattern: createReadStream -> gzip.createGzip() -> createWriteStream for compressed archives. Each stage processes chunks without loading the full file.

HTTP servers stream response bodies by piping file readables to res with appropriate Content-Type headers.

• Set Content-Length only when size is known ahead of time
• Handle client aborts on HTTP streams to avoid leaks
• Test stream errors at each pipeline stage

import { pipeline } from 'node:stream/promises';
import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';

await pipeline(
  createReadStream('access.log'),
  createGzip(),
  createWriteStream('access.log.gz'),
);

Error Handling and Cleanup

Stream errors must be handled on every participant in a pipeline. Uncaught error events crash Node processes in many configurations.

Use finally blocks or pipeline completion to delete temporary files and close database cursors tied to streaming exports.

For user uploads, enforce size limits with stream counters before writing to disk or cloud storage.

• Wrap stream logic in reusable utility functions
• Log stream abort reasons separately from read failures
• Use async iterators (for await) for simpler consumption in modern code