26 — HTTP Without Magic

Level: Intermediate · Time: ~30 min · Prerequisites: Lesson 25 — Building a Real Console Application

Why this matters

Every API call you will ever make is a text conversation: a request line, some headers, an optional body, then a status line, more headers and a body back. Libraries such as the one you built in Lesson 25 hide that conversation, which is fine until something goes wrong — a 403 where you expected 200, a body you cannot parse, a retry that creates two rows. The people who fix those bugs quickly are the ones who can read the raw exchange, and Lesson 27 puts fetch back on top of it.

One exchange, read line by line

Here is the actual exchange with NASA POWER, captured with curl -v from a terminal:

> GET /api/temporal/daily/point?parameters=ALLSKY_SFC_SW_DWN&community=RE&latitude=14.69 HTTP/2
> Host: power.larc.nasa.gov
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/2 200
< content-type: application/json
< content-length: 542
< content-disposition: inline; filename=POWER_Point_Daily_20260301_20260301_014d69N_017d44W_LST.json
< x-cache: Miss from cloudfront
<
{"type":"Feature","geometry":{"type":"Point","coordinates":[-17.44,14.69,0.31]},
 "properties":{"parameter":{"ALLSKY_SFC_SW_DWN":{"20260301":6.3007,...

Lines beginning > are what your machine sent, < what came back. Under HTTP/1.1 these are text lines ending in CRLF; under HTTP/2 they are binary frames with the same meaning, printed the same way.

Part	What it is	Why you care
Request line (`GET /api/... HTTP/2`)	Method, path and query — no scheme, no host	The path is where routing and versioning live; the query carries parameters
`Host`	Which site on the server you want	One IP usually serves many sites; virtual hosts depend on it
Response status line (`HTTP/2 200`)	Protocol, status code	The first thing to check, before parsing anything
Headers	`key: value` pairs, case-insensitive	They carry content type, length, caching, cookies, pagination links
Blank line	End of headers	Everything after it is the body, and the body can be long
Body	The bytes, in whatever the content type says	`content-length: 542` means exactly 542 bytes — the body is a stream, not a string you already have

Methods are contracts

The method is not decoration: it tells servers, caches and proxies what you mean and whether the request may be repeated safely.

Method	Used for	Safe (no side effects)	Idempotent (repeatable)
`GET`	Read a resource	yes	yes
`HEAD`	Read only the headers	yes	yes
`POST`	Create something, or run a search with a body	no	no
`PUT`	Replace a resource at a known URL	no	yes
`PATCH`	Apply a partial update	no	no
`DELETE`	Remove a resource	no	yes

The practical consequence is retries: a failed GET can be resent without thinking, a failed PUT or DELETE may be repeated because the end state is the same, and a failed POST may create a second record — which is why Lesson 29 introduces idempotency keys. A safe, idempotent GET may also be cached and replayed, so it must never be used to change data.

The status codes worth memorising

Code	Meaning	Whose fault is it?
`200`	Success, with a body	—
`201` / `204`	Created (often with a `location` header) / success with no body	—
`301` / `302`	Moved permanently / temporarily; follow the `location` header	neither
`400`	Malformed request — the server could not even parse it	caller
`401`	Not authenticated: missing or invalid credentials	caller
`403`	Authenticated, but not allowed	caller, usually
`404`	No such resource (sometimes used to hide a forbidden one)	caller
`409`	Conflict: duplicate key, version clash	caller
`422`	Well-formed but semantically invalid — a validation failure	caller
`429`	Rate limited; read `Retry-After` and slow down	caller
`500` / `502` / `503`	Server bug / bad gateway / temporarily unavailable	server

Two rules follow. 4xx means fix the request before sending it again; 5xx and 429 mean the same request may succeed later, so retry with backoff. And 200 never means the operation worked: many APIs answer 200 with an error object inside the body. Here is a real POWER validation failure, a 422 with its message in the body:

{"detail":[{"type":"enum","loc":["query","community"],
  "msg":"Input should be 'ag', 'hy', 'sb' or 're'","input":"xx"}]}

Content types, encodings and the headers that matter

content-type tells you how to read the body: application/json (text, always UTF-8), application/x-www-form-urlencoded (a browser form), multipart/form-data (forms with files, separated by a boundary), text/csv, application/octet-stream (opaque bytes).
charset matters for text that is not JSON or HTML; JSON is always UTF-8, which is why the response above carries no charset at all.
accept says what you can read; accept-encoding: gzip asks for compression and content-encoding reports what was applied. These are transport details: curl --compressed and fetch handle them for you.
authorization: Bearer <token> carries credentials; user-agent identifies your client and some APIs block requests without one; cache-control and etag drive caching, and if-none-match can earn you a 304 Not Modified and an empty body.
Cookies are two headers doing one job: the server sends set-cookie, the client returns cookie. For API clients an authorization header is cleaner — Lesson 28 covers secrets.
CORS is a browser rule, not a server rule: the browser refuses to hand a cross-origin response to your page unless the server sends access-control-allow-origin, while curl and Node are unaffected. Lesson 39 spends a whole lesson on it.

REST, RPC, versions and pagination

REST-style endpoints name resources and use methods as verbs: GET /v1/stations/dakar/readings?from=2026-03-01. RPC-style endpoints name the action: POST /api/getReadings with a JSON body. Both are in production everywhere — read the documentation rather than guessing from the shape of the URL.

Versioning protects you from breaking changes: in the path (/v1/...) it is impossible to miss, in a header (accept: application/vnd.solar.v2+json) it keeps URLs clean. Pin what you tested.

Pagination comes in two shapes. Offset and limit (?offset=200&limit=100) is easy to reason about, but rows inserted while you page shift the window and records can be skipped or duplicated. Cursor pagination (?cursor=eyJpZCI6NDE...) hands you an opaque pointer to the next page and stays stable under inserts. Either way, read the envelope — next, has_more, total — and keep asking until it says there is nothing left: treating the first page as all of it is the commonest silent data bug in this domain.

[!WARNING] The first page is not the data set. If an import never grows past the page size, you are ignoring a next link or a cursor, and the symptom looks exactly like "the API has no data".

Inspecting real traffic without fear

In the browser, open the developer tools before the request you want to see: Network, then the Fetch/XHR filter, then click a request for Headers, Payload and Response — the same four parts as the transcript above, with cookies and timing for free. "Preserve log" keeps them across redirects.

From a terminal, three commands cover most of the work:

URL="https://power.larc.nasa.gov/api/temporal/daily/point?parameters=ALLSKY_SFC_SW_DWN&community=RE&latitude=14.69&longitude=-17.44&start=20260301&end=20260303&format=JSON"
curl -sS -i "$URL" | head -20                                          # status line, headers, body
curl -sS -o /dev/null -w '%{http_code} %{content_type}\n' "$URL"       # status and type only
curl -sS -v -o /dev/null "$URL" 2>&1 | grep -E "^(>|<) "               # request and response

-i prints response headers, -v both directions, -w a format after the transfer.

Common mistakes

Mistake	What happens	The fix
Treating `200` as success	An error envelope parses "fine" and an empty import is reported as done	Check the status, then inspect the body
Retrying a `POST` on failure	The row is created twice and you spend an afternoon deduplicating	Retry only safe, idempotent requests; use an idempotency key (Lesson 29)
Treating `401` and `403` as the same thing	You rotate a perfectly good key when the account lacks permission	`401` means authenticate, `403` means authorise
Building URLs by concatenation	A space or `&` in a value silently truncates the query	Use `URL` and `URLSearchParams` (Lesson 27)
Assuming the first page is everything	The import quietly covers 100 of 4 000 days	Follow `next`/`cursor` until exhausted
Blaming CORS for a `curl`-visible failure	Time is lost in the browser while the server is at fault	CORS is browser-only; compare with `curl` first
Sending credentials in the query string	Keys end up in server logs, proxies and shell history	`authorization` header, or keep the secret server-side

Try it

Work in a scratch directory; every command below reaches a public NASA API and needs no key.

Make a request and read it in both directions:

curl -sS -v -o /tmp/power.json \
  "https://power.larc.nasa.gov/api/temporal/daily/point?parameters=ALLSKY_SFC_SW_DWN&community=RE&latitude=14.69&longitude=-17.44&start=20260301&end=20260303&format=JSON"
head -c 120 /tmp/power.json; echo

Expected: an HTTP/2 200 status line, content-type: application/json, content-length: 542 and a body starting with {"type":"Feature",.... Count how many request headers you sent: three.

Now ask only for the status and content type, without downloading the body:

curl -sS -o /dev/null -w '%{http_code} %{content_type}\n' \
  "https://power.larc.nasa.gov/api/temporal/daily/point?parameters=ALLSKY_SFC_SW_DWN&community=RE&latitude=14.69&longitude=-17.44&start=20260301&end=20260303&format=JSON"

Expected: 200 application/json.

Break the request deliberately. Send community=XX instead of community=RE:

curl -sS -w '\nstatus=%{http_code}\n' \
  "https://power.larc.nasa.gov/api/temporal/daily/point?parameters=ALLSKY_SFC_SW_DWN&community=XX&latitude=14.69&longitude=-17.44&start=20260301&end=20260303&format=JSON"

Expected: status=422 and a body whose detail array says the input should be ag, hy, sb or re. That is the caller's fault, and repeating the identical request will never help.

Run the curl -sS -o /dev/null -w '%{http_code}\n' form against community=XX and against a misspelt parameter name. Both answer 422; note that one returns a detail array and the other a messages array — the same code, two different envelopes.

Key takeaways

The method is a contract about side effects and repeatability; only safe, idempotent requests can be retried blindly, which is why POST needs an idempotency key.
4xx is your fault and must be fixed before retrying; 5xx and 429 are worth retrying with backoff. 200 never guarantees the operation worked.
CORS is enforced by the browser, so a curl request is the honest test of whether the server is at fault.
Paginate until the envelope says there is no next page; the first page is not the data set.

Check yourself

A colleague's importer reports "imported 100 readings" every night, and the database never grows past 100 rows per station. Which HTTP concept did they miss, and how would you confirm it?
Why is a repeated PUT acceptable after a network timeout, while a repeated POST may be a data corruption incident?
Your front end fails with a CORS error while curl to the same URL returns 200. Where is the bug, and what would you check first?
An API answers 429 with Retry-After: 30. What should your code do, and which earlier lesson's tooling would you use for the wait?