Oof, this one feels like it's been a while coming, but after a whole bunch of work, and a whole bunch of breaking changes I've just released v5.0.0 of Betterer! 🎉

What is Betterer?​

Betterer is a test runner that helps make incremental improvements to your code!

The first time Betterer it runs a test, it will take a snapshot of the current state. From that point on, whenever it runs it will compare against that snapshot. It will either throw an error (if the test got worse ❌), or update the snapshot (if the test got better ✅). That's pretty much it!

You can check out the (newly updated!) documentation at https://phenomnomnominal.github.io/betterer/

What happened to v2, v3, v4...?​

"But Craig", I hear you say, "The last time you posted about Betterer, it was at v1.0.0!? What's been going on?!". 🔥🔥🔥

That's very astute of you dear reader, and let's just put it this way - I sure do love breaking APIs! One of the interesting things about Betterer is that it is a tool designed for problems that emerge in large and old codebases. That means it has to be able to handle large and old codebases from the get go! So I've had a lot of fun as I've tried to figure out the best workflows and APIs for using Betterer.

Between v1.0.0 and now, I've released a bunch of features, consolidated and simplified APIs, and just generally made Betterer more usable and flexible. I'm pretty happy with where it is at now, so I figured it was about time for an update. I've even been talking about it at a few conferences now that they're coming back! What a world 🌍!

What's in v5.0.0?​

Parallel tests:​

Performance is hard. Prior to v5, the default Betterer reporter would struggle pretty badly, especially when lots of tests were running and producing lots of issues. That was because the main thread was responsible for updating the reporter output and running all the tests.

To fix this, Betterer will now execute all your tests using Node.js Worker Threads! That frees up the main thread to focus on rendering and also means that multiple tests can run at the same time. Getting this to work required breaking some APIs, so your test definition file needs to change:

Before:​

// .betterer.ts
import { BettererTest } from '@betterer/betterer';

export default {
  'my test': new BettererTest({
    // ... test config
  }),
  'my other test': new BettererTest({
    // ... test config
  })
};

After:​

// .betterer.ts
import { BettererTest } from '@betterer/betterer';

export default {
  'my test': () =>
    new BettererTest({
      // ... test config
    }),
  'my other test': () =>
    new BettererTest({
      // ... test config
    })
};

But never fear, you can use the betterer upgrade command to do this migration for you! Just running betterer upgrade will show you what the migration will look like, and betterer upgrade --save will actually update your files. Easy ✨. The betterer upgrade command will be used in the future when I (most probably) break more stuff.

Check out the beast of a PR here (and yes, it took me three branches to get it right 😅)

Betterer ❤️ Angular:​

I've published a new Betterer test for incrementally adding Angular compiler configuration to a project! I'm pretty excited by this, as there are a lot of big Angular codebases out there that don't utilise the full power of the Angular compiler. In particular, I think Betterer could be a good way to introduce the strictTemplates option. You can now do that with the following:

//.betterer.ts
import { angular } from '@betterer/angular';

export default {
  'strict templates': () =>
    angular('./tsconfig.json', {
      strictTemplates: true
    }).include('./src/**/*.ts', './src/**/*.html')
};

Expect to see a full post detailing this in the near future!

Simpler BettererFileTest:​

The old BettererFileTest API was a bit clunky and confusing due to the BettererFileResolver thing. I've hidden that away in the internals, so now the public API is less clunky and confusing:

Before:​

import { BettererFileResolver, BettererFileTest } from '@betterer/betterer';

function myFileTest() {
  const resolver = new BettererFileResolver();
  return new BettererFileTest(resolver, async (filePaths, fileTestResult) => {
    // test implementation...
  });
}

After:​

import { BettererFileTest } from '@betterer/betterer';

function myFileTest() {
  return new BettererFileTest(async (filePaths, fileTestResult, resolver) => {
    // test implementation...
  });
}

Smaller public API, less magic, and you only have to use it if you know why, choice! 👍

Improved workflow:​

I'm still working on figuring out the ideal Betterer workflow. For now, I recommend running Betterer in Pre-commit mode as a pre-commit hook (perhaps using husky and lint-staged) and in CI mode on your build server.

But one thing about chonky codebases is that they often have lots of contributors! Lots of contributors making changes (and making things better) means that 👻 merge conflicts 👻 in the results file are quite common!

To try to help with resolving merge conflicts, I've introduced the betterer merge command. You can still fix merge conflicts manually, but betterer merge will do it for you! If you're as lazy as me, you can even enable automerge and you'll never have to think about merging the results file ever again (I hope, this could still be buggy 🐛😅.

To enable automerge run:

betterer init --automerge

Improved caching:​

Betterer got some cool (ish) caching implemented in v4, but turns out caching is a hard problem (😅), so it's taken a little bit to get right.

It works by passing the --cache flag when running Betterer:

betterer --cache

That will create a file something like this:

{
  "version": 2,
  "testCache": {
    "no hack comments": {
      "packages/angular/src/angular.ts": "b66de728222febdecb3cf11d3aa510b3a8a6ae0e37c0539e37787964573a56ad1b7eb6ee378a9087",
      "packages/angular/src/index.ts": "b66de728222febdecb3cf11d3aa510b3a8a6ae0eb9494122f82a750085fc20d2c3b0f14b34897431",
      "packages/betterer/src/betterer.ts": "b66de728222febdecb3cf11d3aa510b3a8a6ae0e94efcd2f99a4cf14222c400693335ac1b94696bb"
      // ...
    }
  }
}

Betterer will use this cache to only re-test files that have actually changes, so it can be much faster (useful for running on pre-commit!) I suspect there are still issues here, so please try it out and create issues. 🙌

Bug fixes and improvements:​

• betterer results command to get a quick summary:

• Negative filters. I already suspect I'll regret this, but you can now use "!" at the start of a filter to negate it. Now --filter myTest will just run "myTest", and --filter !myTest will run every other test.

• Rewrote most of the public API docs. These are now generated from the code, so should hopefully be easier to keep up to date. 🤞

• Removed a bunch of stuff from the public API. This means more consistency, and I'll be less likely to accidentally break stuff in the future. 😇

Thanks ❤️​

Huge thanks to everyone who has helped me with this stuff, if you've read my rambling, cryptic tweets, opened issues on Github, chatted to me about Betterer at conferences, it's all meant a lot! Maybe I'll print some stickers or something? ☀️

Love 🥰 this? Hate 🤬 this? Go off in the comments, DM me on Twitter, or be the third person to join the Betterer Discord. Catch you on the line 💻!

I just released v4.0.0 of Betterer 🎉 (now with sweet new docs!) and it has a bunch of simplified APIs for writing tests. And just before I shipped it, I got an issue asking how to write a Stylelint test, so let's do it here and explain it line by line:

TL;DR;​

Here's the full test:

// stylelint.ts
import { BettererFileTest } from '@betterer/betterer';
import { promises as fs } from 'node:fs';
import { Configuration, lint } from 'stylelint';

export function stylelint(configOverrides: Partial<Configuration> = {}) {
  return new BettererFileTest(async (filePaths, fileTestResult) => {
    const result = await lint({
      files: [...filePaths],
      configOverrides
    });

    await Promise.all(
      result.results.map(async (result) => {
        const contents = await fs.readFile(result.source, 'utf8');
        const file = fileTestResult.addFile(result.source, contents);
        result.warnings.forEach((warning) => {
          const { line, column, text } = warning;
          file.addIssue(line - 1, column - 1, line - 1, column - 1, text, text);
        });
      })
    );
  });
}

And then using the test:

// .betterer.ts
import { stylelint } from './stylelint';

export default {
  'no stylelint issues': stylelint({
    rules: {
      'unit-no-unknown': true
    }
  }).include('./**/*.css')
};

NTL;PR (not that long, please read 😂)​

Stylelint​

So how does it all work? Let's start with the actual Stylelint part.

Stylelint is pretty easy to set-up. You need a stylelintrc.json file with configuration:

{
  "extends": "stylelint-config-standard"
}

And then run it on your CSS files:

stylelint "**/*.css"

Running that does the following:

1. searches for the stylelintrc.json configuration file
2. reads the configuration
3. finds the valid files
4. runs the rules
5. returns the results

Stylelint also has a JS API which we're going to use:

import { lint } from 'stylelint';

const result = await lint({
  // ...
});

We could just run the above and it will test the current state of the files with the current configuration in stylelintrc.json. And that's great ✨!

Augmenting the configuration:​

For the Betterer test we want to augment the stylelintrc.json configuration with some extra rules... and Stylelint has a really easy way to do that:

import { Configuration, lint } from 'stylelint';

function stylelint(configOverrides: Partial<Configuration> = {}) {
  const result = await lint({
    configOverrides
  });
}

Passing the list of files:​

Stylelint also allows us to pass a specific set of files to test:

import { Configuration, lint } from 'stylelint';

function stylelint(configOverrides: Partial<Configuration> = {}, files: Array<string>) {
  const result = await lint({
    files,
    configOverrides
  });
}

So we could call the stylelint function like:

stylelint(
  {
    rules: {
      'unit-no-unknown': true
    }
  },
  './**/*.css'
);

And that will run the Stylelint from the stylelinerc.json file, plus the unit-no-unknown rule, on all .css files! Thats most of the tricky stuff sorted ⭐️!

Hooking into Betterer:​

This test needs to take advantage of all the snapshotting and diffing magic of Betterer, so we need to wrap it in a test. We want to be able to target individual files, so it specifically needs to be a BettererFileTest. The function argument is the actual test, which is an async function that runs the linter.

import { BettererFileTest } from '@betterer/betterer';
import { Configuration, lint } from 'stylelint';

function stylelint(configOverrides: Partial<Configuration> = {}) {
  return new BettererFileTest(async (filePaths) => {
    // ...
  });
}

Each time it runs Betterer will call that function with the relevant set of files, which we will pass along to Stylelint:

import { BettererFileTest } from '@betterer/betterer';
import { Configuration, lint } from 'stylelint';

function stylelint(configOverrides: Partial<Configuration> = {}) {
  return new BettererFileTest(async (filePaths) => {
    const result = await lint({
      files: [...filePaths],
      configOverrides
    });
  });
}

Adding files:​

Next thing is telling Betterer about all the files with issues reported by Stylelint. To do this we can use the BettererFileTestResult object, which is the second parameter of the test function:

new BettererFileTest(resolver, async (filePaths, fileTestResult) => {
  // ...
});

The result object from Stylelint contains a list of results. For each item in that list, we need to read the file with Node's fs module, and then call addFile() with the file path (result.source), and the contents of the file. That returns a BettererFile object:

import { promises as fs } from 'node:fs';

await Promise.all(
  result.results.map(async (result) => {
    const contents = await fs.readFile(result.source, 'utf8');
    const file = fileTestResult.addFile(result.source, contents);
  })
);

Adding issues:​

The last thing to do is convert from Stylelint warnings to Betterer issues. To do that we use the addIssue() function! In this case we will use the following overload:

addIssue(startLine: number, startCol: number, endLine: number, endCol: number, message: string, hash?: string):

Stylelint only gives us the line and column for the start of the issue, so we use that as both the start position and the end position. Betterer expects them to be zero-indexed so we subtract 1 from both. This also means that the VS Code extension will add a diagnostic to the whole token with the issue, which is pretty handy! We also pass the text of the issue twice, once as the message, and a second time as the hash. The hash is used by Betterer to track issues as they move around within a file. Stylelint adds specific details to the message so that makes it a good enough hash for our purposes. All up, converting an issue looks like this:

result.warnings.forEach((warning) => {
  const { line, column, text } = warning;
  file.addIssue(line - 1, column - 1, line - 1, column - 1, text, text);
});

The whole test:​

Putting that all together and you get this:

// stylelint.ts
import { BettererFileTest } from '@betterer/betterer';
import { promises as fs } from 'node:fs';
import { Configuration, lint } from 'stylelint';

export function stylelint(configOverrides: Partial<Configuration> = {}) {
  return new BettererFileTest(async (filePaths, fileTestResult) => {
    const result = await lint({
      files: [...filePaths],
      configOverrides
    });

    await Promise.all(
      result.results.map(async (result) => {
        const contents = await fs.readFile(result.source, 'utf8');
        const file = fileTestResult.addFile(result.source, contents);
        result.warnings.forEach((warning) => {
          const { line, column, text } = warning;
          file.addIssue(line - 1, column - 1, line - 1, column - 1, text, text);
        });
      })
    );
  });
}

And then we can use the test like this:

// .betterer.ts
import { stylelint } from './stylelint';

export default {
  'no stylelint issues': stylelint({
    rules: {
      'unit-no-unknown': true
    }
  }).include('./**/*.css')
};

And that's about it! The Stylelint API is the real MVP here, nice job to their team! 🔥🔥🔥

Hopefully that makes sense! I'm still pretty excited by Betterer, so hit me up on Twitter if you have thoughts/feelings/ideas 🦄

So, yesterday I announced the new release of ☀️ Betterer, thanks if you've checked it out already!

I wanted to write another post describing a different example, this time with a custom test instead of a built-in test! Let's take a look at how we can prevent accessibility regressions (and hopefully encourage improvements!) 👀

Betterer TL;DR​

Betterer is a test runner that helps make incremental improvements to your code! It is based upon Jest's snapshot testing, but with a twist...

Betterer works in two stages. The first time it runs a test, it will take a snapshot of the current state. From that point on, whenever it runs it will compare against that snapshot. It will either throw an error (if the test got worse ❌), or update the snapshot (if the test got better ✅). That's pretty much it!

Our first Betterer test​

Writing a test with Betterer involves implementing two functions! More formally, we need to implement the BettererTest type:

type BettererTest<ResultType> = {
  test: () => ResultType | Promise<ResultType>;
  constraint: (result: ResultType, expected: ResultType) => ResultType | Promise<ResultType>;
};

So we need to write two functions:

• test - the action that needs to happen to get a result,
• constraint - the rule to apply to check if the result is better, worse or the same

We can implement these in their own file, or straight in the .betterer.ts file. To keep it simple, we'll do the latter:

// .betterer.ts
export default {
  'improve accessibility': {
    test: ...?,
    constraint: ...?
  }
};

Writing the test​

To implement our test, we're going to use Puppeteer and Axe. Puppeteer is a tool that will run a browser and load a page. Axe is a set of accessibility audits that we can run over a web page. We're also going to use Axe Puppeteer which makes it a bit easier to use Axe with Puppeteer.

Lucky for us, we can take the example straight from the Axe Puppeteer documentation! 😍

We launch Puppeteer, get the page that it creates for us and navigate to a website. Then we execute Axe and get the results. Next, we close the page and the browser, before finally returning the number of violations! 🤓

import { AxePuppeteer } from 'axe-puppeteer';
import * as puppeteer from 'puppeteer';

async function improveAccessibility() {
  const browser = await puppeteer.launch();
  const [page] = await browser.pages();

  await page.goto('https://phenomnomnominal.github.io/betterer');
  const results = await new AxePuppeteer(page).analyze();

  await page.close();
  await browser.close();

  return results.violations.length;
}

That's our test sorted!

Writing the constraint​

Now what about the constraint? Since our test returns a number, we just need to compare the two results. We want our test to improve when the result is smaller, so the constraint should look something like this:

import { ConstraintResult } from '@betterer/constraint';

function constraint(result: number, expected: number): ConstraintResult {
  if (current === previous) {
    return ConstraintResult.same;
  }
  if (current < previous) {
    return ConstraintResult.better;
  }
  return ConstraintResult.worse;
}

Comparing numbers is fairly common, so we can use the pre-defined smaller or bigger constraints:

import { smaller } from '@betterer/constraint';

So I kind of lied, you can write a test with just one function! 😅

The whole thing​

Putting it all together, we have our test:

// .betterer.ts
import { smaller } from '@betterer/constraint';
import { AxePuppeteer } from 'axe-puppeteer';
import * as puppeteer from 'puppeteer';

export default {
  'improve accessibility': {
    async test() {
      const browser = await puppeteer.launch();
      const [page] = await browser.pages();

      await page.goto('https://phenomnomnominal.github.io/betterer');
      const results = await new AxePuppeteer(page).analyze();

      await page.close();
      await browser.close();

      return results.violations.length;
    },
    constraint: smaller
  }
};

How's that look? Not bad I reckon! Betterer will run the test for us and update the test snapshot whenever the results get better. That will help make sure that our audit score only goes in the right direction.

This test isn't perfect - you may noticed that it doesn't matter what the violations are, but how many there are! That's something that we could improve later! For now it will stop us introducing more audit violations, which is a good start ⭐️⭐️⭐️

We could improve this test by keeping track of the specific violations that occurred, so we can have a clearer definition of what better or worse really is, but let's leave that for another day!

That's it!​

That's all I got for now. Please let me know what you think on Twitter! 🦄

I'm stoked to announce v1.0.0 of Betterer!

I've been locked down in New Zealand for the last little while, and I've used some of that time to smash out what I think is a pretty compelling v1 release of a tool that I'm really excited about!

What is Betterer?​

Betterer is a test runner that helps make incremental improvements to your code! It is based upon Jest's snapshot testing, but with a twist...

I'm sure many of us have been in situations where we've seen big changes we'd like to make, or new standards or design decisions that we'd like to encourage, but we just don't have the time to do it.

Usually one of two things happen:

1. You start a long-lived branch that is awful to maintain and often impossible to merge. It ends up being a time sink ⏱

2. You and your team make an agreement to make the improvement over time. It gets forgotten about and nothing gets better (in fact usually it gets worse!) 😕

I've seen this happen time and time and again! Sometimes it's introducing a new style rule to a codebase. Other times it's enabling stricter compilation, or decreasing the number of accessibility failures!

Betterer works in two stages. The first time it runs a test, it will take a snapshot of the current state. From that point on, whenever it runs it will compare against that snapshot. It will either throw an error (if the test got worse ❌), or update the snapshot (if the test got better ✅). That's pretty much it!

How does it work?​

To get started, you can run the following from the root of your project:

npx @betterer/cli init

That will give you a brand new .betterer.ts config file which looks something like this:

// .betterer.ts

export default {
  // Add tests here ☀️
};

From here, it's up to you to add some tests!

Let's imagine you're working with a codebase that uses Moment.js. You'd like to migrate away from it for performance reasons.

// src/subtract.js
import * as moment from 'moment';

const now = moment();

console.log(now.subtract(4, 'years'));

Let's also imagine that you're using ESLint in this codebase. One approach to remove Moment.js might be to use the no-restricted-imports ESLint rule, add the eslint-disable-next-line comment all over the place, and cross your fingers that people don't just add more... 🤔

Betterer gives us a better option! We can create a test for that specific rule:

// .betterer.ts
import { eslintBetterer } from '@betterer/eslint';

export default {
  'no import from moment': eslintBetterer('./src/**/*.js', [
    'no-restricted-imports',
    [
      'error',
      {
        name: 'moment',
        message: 'Please use "date-fns" instead.'
      }
    ]
  ])
};

The first time we run the test with Betterer, it will look something like this:

Betterer has now created a snapshot of the current state, stored by default in a .betterer.results file:

// BETTERER RESULTS V1.
exports[`no import from moment`] = {
  timestamp: 1589459511808,
  value: `{
    "src/subtract.js:566118541": [
      [0, 0, 33, "\'moment\' import is restricted from being used. Please use \\"date-fns\\" instead.", "4035178381"]
    ]
  }`
};

The snapshot contains information about the current issues in the code.

The next time we run the test, it will look like this:

Now, someone else on the team comes along and doesn't know about the new rule, and they add a new file that uses Moment.js:

// src/add.js
import * as moment from 'moment';

const now = moment();

console.log(now.add(4, 'years'));

When Betterer runs on their code, they get a nice big error:

Even though a new issue has been introduced, the .betterer.results file doesn't change!

Our teammate reads the helpful error message from ESLint and they update their code to use date-fns...

// src/add.js
import { addYears } from 'date-fns';

console.log(addYears(Date.now(), 4));

... and once again Betterer tells them that the result is the same:

Our teammate has a bit of time on their hands, so they decide to fix up our usage of Moment.js as well! 🎉

This time when they run Betterer, everything is good:

There are now no remaining issues, so this test has met its goal. Since the existing issue has been resolved, it is removed from the snapshot in the .betterer.results file. This means we can move the rule from Betterer over to the normal ESLint configuration, so we don't reintroduce the issues again.

Pretty neat eh! That's an example of the built-in @betterer/eslint test, but there are other built-in tests too. And you can of course write your own tests! Check out the documentation for more details (still a WIP 🚧)!

What's in v1.0.0?​

Everything I've mentioned so far has been working for a while! Over the last few months I've really solidified the implementation (basically a whole rewrite to be honest!):

• Better error handling and error messages
• Better issue comparison. It now understands file renames and issues that around within the same file
• The ability to run tests on a single file via the JS API, with betterer.single
• A whole bunch more tests!

But I've also added a few key features that are worthy of a 1.0.0 release! 🔥🔥🔥

Force Update (!)​

First things first, you can now run Betterer with the --update option, and the snapshot will be updated even if it got worse! This is handy for when you need to ship something, even if it makes it temporarily worse:

betterer --update

And because this is shamelessly stolen from Jest, you can also use -u.

Watch mode (!!!)​

This one is huge! All the changes that I made were building up to this. You can now run Betterer in watch mode and get feedback as you fix up issues:

Same rules apply here, the snapshot will update whenever the test gets better!

There's a bunch of other cool things that could happen with watch mode (gamification much?), so I'm pumped that it's working! 🤩

VS Code extension (!!!!!)​

Way to bury the lede! This is probably the coolest bit! Let's just say it was a build up.

Betterer runs entirely in its own world, so the usual ESLint or TypeScript extensions can't report the issues. But everyone loves seeing red squiggly lines in the code editor, so Betterer now has its very own VS Code Extension 🤯!

Initialise in a new project​

You can run the betterer.init command in a project! It will generate config files and update your package.json with scripts and dependencies:

See all issues in a file​

And when you've got some tests setup, it will show you all the existing issues in a file, and when they were first created. And it will show you any new issues as you make them:

I reckon that's pretty sweet!

So, what's next?​

Well, you're going to try out Betterer and open lots of issues and help me make it better(er)! If you'd like to contribute...

• There are definitely bugs to fix
• The test coverage is pretty good, but there aren't any E2E tests for the VS Code extension yet
• The logging and reporting implementation could use some work
• There is so much documentation to be written

What a time to be alive! 🤓

In case you can't tell, I'm unreasonably excited about this and I really think this will help with large and legacy codebases. Please let me know what you think on the Twitters!