22 — Reading and Writing Files

Level: Intermediate · Time: ~35 min · Prerequisites: Lesson 21 — Concurrency — Parallel Work, Timeouts and Cancellation

---

Why this matters

Everything the tracker knows lives in memory, and memory dies with the process. The day's yields have to survive a crash, a redeploy and the Ctrl+C that Lesson 21 taught you to handle. A file on disk is the least complicated durable store there is — and the easiest place to lose data: a write interrupted halfway leaves a truncated file where a valid one used to be, a path glued together with string concatenation works on your laptop and breaks on a Windows server, and reading a 400 MB export with readFile costs 400 MB of memory before a single row is processed. This lesson is the small set of habits — one API, real paths, atomic writes, streams — that make file handling the boring part of a program.

One filesystem API, eight calls

Node exposes the same operations twice: the callback interface in node:fs, and a promise interface in node:fs/promises. Learn the promise one — it composes with await, with try/catch and with the Promise.allSettled of Lesson 21 instead of forcing you to unwind callbacks by hand.

import { appendFile, mkdir, readFile, readdir, writeFile } from "node:fs/promises";

await mkdir("data/daily", { recursive: true });                       // no error if it already exists
await writeFile("data/daily/2026-09-14.json", JSON.stringify({ date: "2026-09-14", yieldKwh: 4.2 }));
await appendFile("data/log/runs.log", `${new Date().toISOString()} fetched\n`);
const raw = await readFile("data/daily/2026-09-14.json", "utf8");     // a string, not bytes
const names = await readdir("data/daily");                            // the entries in a directory

Without an encoding readFile resolves to a Buffer, with "utf8" to a string, and every call rejects with an error carrying a code — ENOENT, EACCES, ENOSPC — which Lesson 20's error boundary should wrap into a DomainError.

Call	What it does	The trap
readFile(path, "utf8")	Reads the whole file into a string	Costs memory equal to the file size
writeFile(path, data)	Replaces the content	Truncates first: a crash leaves a half-file
mkdir(path, { recursive: true })	Creates every missing parent	Without recursive, fails when it exists
readdir(path, { withFileTypes: true })	Lists entries with their type	Names only unless you pass the option
rename(from, to)	Moves a file	Atomic within one filesystem, not across two
rm(path, { recursive: true, force: true })	Deletes a tree	recursive is mandatory for a directory

Characters are not bytes

Text on disk is bytes, and "utf8" is a decoder that turns one into the other. That distinction matters the moment a file is not text, and for accents too:

import { readFile, writeFile } from "node:fs/promises";

const pdf = await readFile("data/pvgis-export.pdf");        // Buffer, a Uint8Array subclass
await writeFile("data/copy.pdf", pdf);                      // byte-for-byte copy, no decoding

const encoded = new TextEncoder().encode("Sénégal");        // Uint8Array
console.log(encoded.byteLength, "Sénégal".length);          // 9 7 — bytes are not characters

Reading a PDF as "utf8" does not throw; it hands you a string full of replacement characters, and writing that string back destroys the file. So if a file ends in .pdf, .png, .jpg, .gz or .zip, never let an encoding near it — carry the Buffer from read to write. A Buffer is a Uint8Array, so anything that accepts one (TextDecoder, a hash, writeFile) accepts it directly.

Paths done right

A relative path in readFile is resolved against process.cwd(), not against the file doing the reading. node dist/index.js from another directory then fails with ENOENT while the same command from the project root works — a bug that appears in cron and in production only.

import { basename, dirname, extname, join, resolve, sep } from "node:path";
import { fileURLToPath } from "node:url";

const moduleDir = dirname(fileURLToPath(import.meta.url));   // directory of the running file
const dataDir = resolve(moduleDir, "..", "data");            // absolute; independent of the cwd
const file = join(dataDir, "daily", "2026-09-14.json");

console.log(basename(file), extname(file));                  // 2026-09-14.json .json
console.log(sep);                                            // "/" on Linux, "\\" on Windows

resolve returns an absolute path and collapses ..; join only glues segments together; sep is the separator you should never hard-code. One further rule, and it is a security rule: never join(dataDir, userSupplied) without checking, because a name such as ../../etc/passwd walks out of your directory. Resolve it and confirm the result still starts with dataDir + sep before opening anything.

Atomic writes, and directories you can trust

writeFile truncates the target before it writes the new bytes. If the process dies in between — out of memory, container eviction, a pulled plug — the only copy of the day's data is half a file. The fix is to write a complete file somewhere else and swap it in with one rename, which the operating system performs atomically:

import { mkdir, open, rename } from "node:fs/promises";
import { dirname, join } from "node:path";

export async function writeFileAtomic(filePath: string, contents: string): Promise<void> {
  const directory = dirname(filePath);
  await mkdir(directory, { recursive: true });
  const tempPath = join(directory, `.${Date.now().toString(36)}.tmp`);
  const handle = await open(tempPath, "wx");          // "wx" fails if the temp name already exists
  try {
    await handle.writeFile(contents, "utf8");
    await handle.sync();                              // push the bytes to disk before the swap
  } finally {
    await handle.close();
  }
  await rename(tempPath, filePath);                   // atomic: readers see old or new, never half
}

mkdir(..., { recursive: true }) is idempotent, so create directories on demand rather than assuming they exist. When you walk a tree, skip .git and node_modules explicitly, and remove temp directories with rm(dir, { recursive: true, force: true }) instead of leaving them for the next run to trip over.

export async function findCsvFiles(dir: string): Promise<string[]> {
  const found: string[] = [];
  for (const entry of await readdir(dir, { withFileTypes: true })) {
    if (entry.name === ".git" || entry.name === "node_modules") continue;
    const full = join(dir, entry.name);
    if (entry.isDirectory()) found.push(...(await findCsvFiles(full)));
    else if (entry.name.endsWith(".csv")) found.push(full);
  }
  return found;
}

Streams, backpressure and lines

readFile on a 400 MB CSV puts 400 MB in memory, plus the string copy, plus your parsed rows. Streams move data in chunks, and pipeline connects them so a slow writer applies backpressure to a fast reader instead of letting a buffer grow without limit:

import { createReadStream, createWriteStream } from "node:fs";
import { createInterface } from "node:readline";
import { pipeline } from "node:stream/promises";

await pipeline(createReadStream("data/yields-2024.csv"), createWriteStream("data/yields-copy.csv"));

const lines = createInterface({ input: createReadStream("data/yields-2024.csv"), crlfDelay: Infinity });
let rows = 0;
for await (const line of lines) {
  if (line.trim() === "" || line.startsWith("date,")) continue;
  rows += 1;
}
console.log(`${rows} data rows`);

crlfDelay: Infinity makes a Windows line ending behave like a Unix one, and the for await loop pulls the next line only when the previous iteration has finished, so memory stays flat whatever the file size. Use pipeline rather than wiring "data" events by hand: it propagates a failure to every stream and closes them all.

Watching for changes, and polling

watch is a notification system, not a change log: editors save by writing a temp file and renaming it, so one save can produce several events, and the filename argument can be null. { recursive: true } works on macOS and Windows; on Linux support arrived only in recent Node 20 releases, so treat it as unavailable there.

import { watch } from "node:fs";

const watcher = watch("data/incoming", (eventType: string, filename: string | null) => {
  console.log(`${eventType} ${filename ?? "(unknown)"}`);
});
process.once("SIGINT", () => watcher.close());        // always release the handle

When correctness matters more than latency, poll instead: compare stat(...).mtimeMs every few seconds and act on the change. One stat per interval, identical behaviour on every platform. Watch for interactivity, poll for ingestion.

[!WARNING] writeFile over the only copy of a file is the most common way to lose data in a small program. Write to a temp path in the same directory and rename it into place — that is what git and your editor do as well.

---

Common mistakes

Mistake	What happens	The fix
writeFile straight over the only copy	A crash mid-write leaves a truncated file and the previous data is gone	Write a temp file, then rename it into place
Building paths with + "/" +	Works locally, breaks on Windows and on any path containing ..	join and resolve from node:path
Relative paths in a script	Works from the project root, ENOENT from cron or systemd	Anchor paths on import.meta.url
Reading images or PDFs with "utf8"	Silent corruption: the bytes are re-encoded when written back	Read binary files without an encoding
A user-supplied path joined into data/	../ escapes the directory: arbitrary file read	Resolve, then check the prefix before opening

---

Try it

1. Create src/atomicStore.ts with writeFileAtomic exactly as printed above, exported.

2. Create src/index.ts that writes thirty days of records through it:

import { readdir } from "node:fs/promises";
import { dirname, join, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import { writeFileAtomic } from "./atomicStore.js";

const dailyDir = resolve(dirname(fileURLToPath(import.meta.url)), "..", "data", "daily");
for (let day = 1; day <= 30; day += 1) {
  const date = `2026-09-${String(day).padStart(2, "0")}`;
  await writeFileAtomic(join(dailyDir, `${date}.json`), JSON.stringify({ date, yieldKwh: 4.2, source: "pvgis" }));
}
console.log(`${(await readdir(dailyDir)).length} files`);

Run npx tsc, then start it twice: once as node dist/index.js and once from elsewhere, as cd /tmp && node ~/dev/ts-zero-to-hero/dist/index.js.

Expected output: 30 files both times — the path anchor means the working directory cannot change the outcome.

1. Break one on purpose. Add import { createWriteStream } from "node:fs"; to the top of src/index.ts, this at the end, and run it twice:

const target = join(dailyDir, "2026-09-01.json");
const stream = createWriteStream(target);     // truncates the file immediately
stream.write('{"date":"2026-09-01","yield');
stream.destroy();                             // this is what a crash looks like

Expected output: the file is now a fragment of a couple of dozen bytes and JSON.parse on it throws — the day's record is gone. Replace the stream with await writeFileAtomic(target, newBody) and interrupt the process as often as you like: the file always holds either the previous record or the new one, and it always parses.

---

Key takeaways

• node:fs/promises is the interface to learn: it composes with await, with try/catch and with the concurrency tools of Lesson 21.
• Bytes and characters are different things. Pass "utf8" for text, pass nothing for binary, and never write a decoded binary file back.
• Anchor paths with import.meta.url and build them with node:path; a concatenated path is a bug waiting for another operating system.
• Never truncate the only copy: write a temp file, sync it, and rename it into place.
• Stream large files through pipeline so memory stays flat, and prefer polling over watch when a change must not be missed.

---

Check yourself

1. Why is rename safer than writeFile for replacing a file another process may be reading?
2. Your ingestion script works from the project root but not from a cron job. Which single line is wrong, and what does the fix look like?
3. A colleague reads a 2 MB JPEG with readFile(path, "utf8") and writes the string to a new file. The copy is unreadable — what happened?
4. You must detect new files in a directory on a Linux container. Why is polling mtimeMs defensible, and what does it cost?

---

Next

Lesson 23 — JSON, Validation and Environment Configuration