- Continuous code generation - Automated Utopia
- Continuous code generation - Versioning
- Continuous code generation - TypeScript libraries
- Continuous code generation - .NET libraries
- Continuous code generation - Automatically set up new libraries and APIs
First time putting a video together since, forever, so don't judge too harshly. No idea why it's so matrix-like at times, and no idea how to fix it, so it is what it is
The video shows from start to finish the current state of the continuous code generation for TypeScript libraries. So what is happening in the video?
First, an API change happens, where we add a new channel for the public Rust API, through a PR (read more about the setup here). The API version is then bumped to 0.8.0 as it was a minor version change.
Once the version changes, it's time to remotely trigger the code generation process, in this example for the rust-ts-public-api library. The code generation process is what the previous blog post outlined. The CI system then provides a PR with the desired change based on the version change of the API and code template.
Once merged the library is then released with the appropriate new version and then afterward bumps the package version to 0.5.0.
As the video shows, it's not quite fully automated, as I still need to manually merge the PRs, it is, however, an "easy fix", as you can get the CI to auto-merge and even approve PRs if it's a specific bot that creates it (we have a similar setup at AsyncAPI). I just have not come around to include it yet even though it's the idea.
Generate new code
Because I choose to host (or at least that's how it came to be, as initially it was hosted on Gitlab) it on GitHub, many of the automation steps are for GitHub actions. However, it should be "rather easy" to translate it to another CI system as the building blocks and processes are there.
Find the current version of the library
As part of the generation script, one step was to find the current version of the library to know where we are left off. For the TypeScript library, it is straightforward, as we can read the version with jq from the package.json file.
library_last_version=$(cat ./package.json | jq -r '.version')Auto-generate the library
Next is to trigger the code generation when the API receives a version change.
To achieve this I choose to add a generation workflow in the API library and one where the AsyncAPI documents are located, to remotely trigger the generation workflow. This is not the only way to achieve it, but it was the one I found most suiting in this case.
For the generation workflow, it's rather simple as all it does is execute the generate.sh script, read the results of the generation process - and then create appropriate PR with the desired conventional commit. The workflow is then set to be triggered on workflow dispatch which can be done remotely.
1name: Auto update generated client
2on:
3 workflow_dispatch:
4
5jobs:
6 update-code:
7 runs-on: ubuntu-latest
8 steps:
9 - uses: actions/checkout@v3
10
11 - uses: actions/setup-node@v3
12 with:
13 node-version: '14'
14
15 - name: Generate code
16 run: chmod +x ./generate.sh && ./generate.sh
17
18 - name: Set Environment Variables from generator file
19 uses: ./.github/actions/setvars
20 with:
21 varFilePath: ./.github/variables/generator.env
22
23 # Follow conventional commit for PR's to ensure accurate updates
24 - name: Create pull request for major version if needed
25 if: ${{env.major_version_change == 'true'}}
26 uses: peter-evans/create-pull-request@v4
27 with:
28 title: "feat!: major version change"
29 body: ${{env.commit_message}}
30
31 - name: Create pull request for minor version if needed
32 if: ${{env.minor_version_change == 'true'}}
33 uses: peter-evans/create-pull-request@v4
34 with:
35 title: "feat: minor version change"
36 body: ${{env.commit_message}}
37
38 - name: Create pull request for patch version if needed
39 if: ${{env.patch_version_change == 'true'}}
40 uses: peter-evans/create-pull-request@v4
41 with:
42 title: "fix: patch version change"
43 body: ${{env.commit_message}}To remotely trigger the code generation from where the AsyncAPI documents are located, we can then use the GitHub action benc-uk/workflow-dispatch to trigger the workflow remotely. This is run whenever the bundled AsyncAPI document change. This is my lazy way of detecting version changes, as the AsyncAPI documents are bundled on each release.
1name: Trigger remote code generation for Rust public API
2on:
3 workflow_dispatch:
4 push:
5 branches:
6 - main
7 paths:
8 - 'bundled/rust_public.asyncapi.json'
9jobs:
10 generate:
11 runs-on: ubuntu-latest
12 steps:
13 - name: Update TS public game API library
14 uses: benc-uk/workflow-dispatch@v1
15 with:
16 workflow: Auto update generated library
17 token: ${{ secrets.GH_TOKEN }}
18 repo: GamingAPI/rust-ts-public-api
19 - # Update ... public game API libraryThe only thing we cannot trigger upon is when the code template receives a change. You could fall back to a cronjob which could trigger the generation workflow at some specific time, so it's semi-automatic.
Release the new library
This setup is a page out of the AsyncAPI setup, where I reuse the exact setup that Lukasz authored. If you want a more in-depth description of the setup, you can watch the Let's talk about contributing - CI/CD at AsyncAPI, otherwise, I will make sure to recap the major parts below.
It has two core concepts, the package.json file has specific release scripts and release configurations which is being triggered by a release workflow (for the library it's this workflow) that ensures everything is released based on the conventional commits it finds since the last release. Similar to what the setup is for the AsyncAPI files. Once released we bump the version of the library through this workflow.
Turning towards our code generation, the first dilemma is that the code template is generating a full project, meaning it already generate a package.json file. This means it does not contain our needed release scripts or dependencies for supporting the continuous release.
Luckily it's JSON, so we can jq once again to make it contain what we need.
We can therefore create a custom_package.json file to contain all our customized properties (more or less copy-pasted from the AsyncAPI setup):
1{
2 "name": "@gamingapi/rust-ts-public-api",
3 "description": "TypeScript public API wrapper for rust",
4 "publishConfig": {
5 "access": "public"
6 },
7 "keywords": [],
8 "author": "Jonas Lagoni ([email protected])",
9 "license": "Apache-2.0",
10 "bugs": {
11 "url": "https://github.com/GamingAPI/rust-ts-public-api/issues"
12 },
13 "files": [
14 "/lib",
15 "./README.md",
16 "./LICENSE"
17 ],
18 "homepage": "https://github.com/GamingAPI/rust-ts-public-api#readme",
19 "scripts": {
20 "generate:assets": "npm run build && npm run docs",
21 "bump:version": "npm --no-git-tag-version --allow-same-version version $VERSION",
22 "release": "semantic-release",
23 "prepublishOnly": "npm run generate:assets"
24 },
25 "devDependencies": {
26 "@semantic-release/commit-analyzer": "^8.0.1",
27 "@semantic-release/github": "^7.0.4",
28 "@semantic-release/npm": "^7.0.3",
29 "@semantic-release/release-notes-generator": "^9.0.1",
30 "conventional-changelog-conventionalcommits": "^4.2.3",
31 "semantic-release": "^17.0.4"
32 },
33 "release": {
34 "branches": [
35 "main"
36 ],
37 "plugins": [
38 [
39 "@semantic-release/commit-analyzer",
40 {
41 "preset": "conventionalcommits"
42 }
43 ],
44 [
45 "@semantic-release/release-notes-generator",
46 {
47 "preset": "conventionalcommits"
48 }
49 ],
50 "@semantic-release/npm",
51 "@semantic-release/github"
52 ]
53 }
54}The generator script can then be adapted so after the code is generated, we can merge the package.json file from the code template without custom_package.json.
1# Merge custom package file with template generated
2jq -s '.[0] * .[1]' ./package.json ./custom_package.json > ./package_tmp.json
3rm ./package.json
4mv ./package_tmp.json ./package.jsonSome reflection
While that finishes the setup for TypeScript libraries, it's far from over - cause this is quite a complex setup and quite a few steps to take to achieve it. I still think how all of this can be reduced to a single step... Maybe it's not necessary to have the library in a repository, but rather just releasing it through a library. But I need some more experience in how different languages do things to put something together.
Photo by Iswanto Arif on Unsplash