Node 23 was released this week, and the hot ticket item probably is the fact that you can now require() files that use ESM (import/export).

This is helpful because ESM and CommonJS (require/module.exports) are kind of different worlds and before this change if you wanted to use a “module” from your CommonJS file, you would need to do something like:

const theThing = await import('some/module/file.mjs');

This is called a dynamic import and it’s extra annoying, because you can’t just put this at the top of your file. It returns a Promise, so you can only import these modules ‘asynchronously’ or in functions. The reason is that you can only await inside functions in CommonJS files. So this syntax prevents importing a “module” and immediately using it. To use a module in CommonJS, you’ll need some kind of initialization logic.

Many Javascript packages resort to shipping both a ‘CommonJS’ and a ‘ESM’ version of their code to reduce this kind of annoyance, but not everyone does.

The Node 23 change

Node 23 makes it possible to load ESM modules transparently via require(). This means that the above example can be rewritten to:

const theThing = require('some/module/file.mjs');

The important part here is not the usage of require, but the absense of await. This allows ESM modules to be loaded without this kind of initialization logic.

But there’s a big caveat:

This only works if the module you loaded in is not using top-level await. “Top level await” means awaiting things outside of (async) functions, which is possible in modules (but not CommonJS).

If a top-level await was used, a ERR_REQUIRE_ASYNC_MODULE error will be thrown. But the important thing to note is that this doesn’t just apply to the file you are directly importing/requiring. It also applies to any files loaded by that file, at any level in either your project or any dependency or sub-dependencies.

Using top-level await is now a BC break

Typically when we think of backwards compatibility breaks that require a major new version in semver, you might think of functions changing, or removing or arguments no longer being supported.

Before this change in Node, if your project was fully switched to ESM you might not think of placing a top-level await anywhere in your code as a backwards compatibility break, but if it’s the first await you might now inadvertently break Node.js users, if they used require() to bring in your module.

This means that the first top-level await in your project (or any of your dependencies) might now constitute a new major version if you follow semver.

If you don’t want to do this, here are some other things you could do:

1. Tell your users you don’t support require()

You could effectively tell them if they use require() they’re on their own. Chances are that your users won’t read this and do it anyway, but arguably

Truncated by Planet PHP, read more at the original (another 4607 bytes)

Say you have an API, and you want to communicate what sort of things a user can do on a specific endpoint. You can use external description formats like OpenAPI or JSON Schema, but sometimes it’s nice to also dynamically communicate this on the API itself.

OPTIONS is the method used for that. You may know this HTTP method from CORS, but it’s general purpose is for clients to passively find out ‘What can I do here?’.

All HTTP clients typically support making OPTIONS request. For example with fetch():

const response = await fetch(
  'https://example.org',
  {method: 'OPTIONS'}
);

A basic OPTIONS response might might look like this:

HTTP/1.1 204 No Content
Date: Mon, 23 Sep 2024 02:57:38 GMT
Server: KKachel/1.2
Allow: GET, PUT, POST, DELETE, OPTIONS

Based on the Allow header you can quickly tell which HTTP methods are available at a given endpoint. Many web frameworks emit this automatically and generate the list of methods dynamically per route, so chances are that you get this one for free.

To find out if your server does, try running the command below (with your URL!):

curl -X OPTIONS http://localhost:3000/some/endpoint/

One nice thing you could do with the Allow header, is that you could also communicate access-control information on a very basic level. For example, you could only include DELETE and PUT if a user has write access to a resource.

Accept and Accept-Encoding

There’s server other standard headers for discovery. Here’s an example showing a few at once:

HTTP/1.1 204 No Content
Date: Mon, 23 Sep 2024 02:57:38 GMT
Server: KKachel/1.2
Allow: GET, PUT, POST, DELETE, OPTIONS
Accept: application/vnd.my-company-api+json, application/json, text/html
Accept-Encoding: gzip,brotli,identity

You may already be familiar with Accept and Accept-Encoding from HTTP requests, but they can also appear in responses. Accept in a response lets you tell the client which kind of mimetypes are available at an endpoint. I like adding text/html to every JSON api endpoint and making sure that API urls

Truncated by Planet PHP, read more at the original (another 11304 bytes)

We had been having an issue with losing cookies in Varnish. Because of how Varnish works, the way to work around cookies is to store them in headers inbetween states of the request process. We were finally able to get some data out of varnishlog that looked like this.

      - ReqHeader DN-Varnish-Offer-Group: 
- LostHeader DN-Varnish-Offer-Sort: price 
- Error out of workspace (req) 
- LostHeader DN-Varnish-Sbtab-Sorts: 
- Error out of workspace (req) 
- LostHeader DN-Varnish-Use-View: 
- Error out of workspace (req) 
- LostHeader DN-Varnish-Ux-Variant: classic_site 
- Error out of workspace (req)

When you first read these errors, you will likely find the settings workspace_client and workspace_backend. Those seem like very logical settings to tweak. However, no matter how big we set them nothing helped. We graph stats coming out of Varnish using the prometheus exporter. We found the metric varnish_main_ws_backend_overflow. That made us believe even more that this was a workspace_backend limit we were hitting. It turns out, there is more to the workspace settings than just this. I read through an old issue on Github and found some folks trying to set other settings related to header size and header limits. In the end, that was our issue. We increased these settings and our overflows disappeared. 

http_req_hdr_len = 64k (default is 8k)
http_req_size = 128k (default is 32k)
http_max_hdr=256 (default is 64)

Hopefully this will help someone else that runs up against this.

A new RFC was released for Structured Fields: RFC9651.

What is it?

HTTP headers have been a bit of a free-for all in terms of how complex values are encoded, with many headers requiring their own mini-parser.

A while back an effort was started to fix this for headers going forward, named ‘Structured Fields’. They’re called Fields and not ‘Headers’ because HTTP has both Headers and Trailers!

Structured fields let you encode things like lists, dictionaries, strings, numbers, booleans and binary data. The original RFC from 2021 is pretty successful and although many existing headers can’t be retrofitted to this format, a lot of new standards are taking advantage.

Some examples:

# Parsed an ASCII string
Header: "foo"

# A simple string, called a 'Token' in the spec
Header: foo

# Parsed as number
Header: 5
Header: -10
Header: 5.01415

# Parsed into boolean
Header: ?1
Header: ?0

# Binaries are base64 encoded
Header: :RE0gbWUgZm9yIGEgZnJlZSBjb29raWU=:

# Items can have parameters
Header: "Hello world"; a="5"

# A simple list
Header: 5, "foo", bar, ?1

# Each element can have parameters
Header: sometoken; param1; param2=hi, 42

# A list can also contain lists itself. These are called 'inner lists' and
# use parenthesis
Header: sometoken, (innerlistitem1 innerlistitem2), (anotherlist)

# A simple dictionary
Header: fn="evert", ln="pot", coffee=?1

# Each item may have parameters too
Header: foo=123; q=1, bar=123, q=0.5

# A dictionary value may be an inner list again
Header: foo=(1 2 3)

The new RFC published last week adds 2 new data types: Dates and ‘Display strings’, which is a Unicode serialization that fits in the HTTP header (and trailer) format.

# Parsed into a Date object
Header: @1686634251

# A Unicode string, called a 'Display String' in the spec. They use
# percent encoding, but encode a different set of characters than
# URLs.
Header %"Frysl%C3%A2n"

Why should you care?

If you encounter these headers in the wild, it’s a really good idea to use a standard parser. One of the reasons is that with using structured-fields, there’s a built-in extension mechanism. You’ll want to make sure that when a new parameter appears your application doesn’t suddenly break.

You may also want to define and use your own HTTP headers. The structured fields format is a very good ‘default choice’ that removes decisions such as ‘How should I encode a key value object’ or ‘how do I encode a UTF-8 string’.

With parsers popping up for every language, you don’t have to worry about writing your own one-off formats.

Javascript package

I’m the maintainer of a Javascript library for Structured Fields, called “structured headers”, which I’ve also updated for this new RFC. I wish I picked the name “structured-fields”, but I picked the name before the original standard changed it’s name.

I’ve just released v2 of this library supporting these new types, and also added ES Modules support.

Comments?

Reply to one of these:

• Mastodon post
• Bluesky post

One week ago on September 24th my son Kian was born, after a 5 year fertility journey with my wife Roxy. Roxy and Kian are well and I’m really excited for everything that comes next!

When Guzzle throws BadResponseException, the message includes information about the method, URL response code and then a truncated part of the body.

For example:

"Client error: `GET https://dev.clientproject.com:4444/oauth2/authorize?client_id=983e98d2fab8756a&scope=scope&response_type=code&redirect_uri=%2Fhome&code_challenge=some_code_challenge_here` resulted in a `400 Bad Request` response:
{"error":"invalid_request","error_description":"The request is missing a required parameter, includes an invalid parame (truncated...)

To retrieve the full text of the body, you grab it from the response property of the exception:

$body = $e->getResponse()->getBody()->getContents();

Changing the truncation limit

If you want more text in the exception message you need to pass a handler stack to the Client's constructor with an httpErrors middleware that has a BodySummarizer with a larger truncation limit.

The easiest way to do this is to create the default stack and then modify it:

use GuzzleHttp\BodySummarizer;
use GuzzleHttp\HandlerStack;

// ...

$stack = HandlerStack::create();
$stack->remove('http_errors');
$stack->unshift(Middleware::httpErrors(new BodySummarizer(1500)), 'http_errors');

Set the number of characters to truncate at in the BodySummarizer's constructor. I picked 1500; the default is 120 (as of this time of writing).

After creating the stack, we remove the current httpErrors middleware and add a new one. Note that it needs to be first in the stack as we want it to handle any error from any subsequent middleware too.

We then create our client with our custom stack:

use GuzzleHttp\Client;

// ...

$this->client = new Client([
    'handler'  => $stack,
    'base_uri' => $this->baseUri,
]);

That's it. There's now more text in the BadResponseException message.