Improve documentation (#9)

* Add JSDoc comments to common configuration options

* Clean up PR comment docs

* Move Discord notification docs to dedicated file

* Overhaul README

README.md

@@ -1,63 +1,123 @@
 # astro-ecosystem-ci
 
-This repository is used to run integration tests for astro ecosystem projects
+This repository is used to run integration tests for Astro ecosystem projects.
 
-## via github workflow
+## Running tests
 
-### scheduled
+### GitHub workflows
 
-Workflows are scheduled to run automatically every Monday, Wednesday and Friday
+#### Scheduled runs
 
-### manually
+Workflows are scheduled to run automatically every Monday, Wednesday and Friday.
 
-- open [workflow](../../actions/workflows/ecosystem-ci-selected.yml)
-- click 'Run workflow' button on top right of the list
-- select suite to run in dropdown
-- start workflow
+#### Manual runs
 
-## via shell script
+If you are an Astro maintainer, follow these steps to trigger a manual run for one of the ecosystem test suites:
 
-- clone this repo
-- run `pnpm i`
-- run `pnpm test` to run all suites
-- or `pnpm test <suitename>` to select a suite
-- or `tsx ecosystem-ci.ts`
+1. Open the [ecosystem CI workflow](https://github.com/withastro/astro-ecosystem-ci/actions/workflows/ecosystem-ci-selected.yml) on GitHub.
+2. Click the <kbd>Run workflow</kbd> button at the top right of the list.
+3. Select the test suite to run from the dropdown menu.
+4. Confirm by clicking <kbd>Run workflow</kbd>.
 
-You can pass `--tag v2.8.0-beta.1`, `--branch somebranch` or `--commit abcd1234` option to select a specific astro version to build.
-If you pass `--release 2.7.13`, astro build will be skipped and astro is fetched from the registry instead
+#### Trigger a workflow run with a PR comment
 
-The repositories are checked out into `workspace` subdirectory as shallow clones
+Astro maintainers can trigger a workflow run with a comment in an Astro PR (requires triage permissions for the `withastro/astro` repository).
 
-## via comment on PR
+To trigger a run, comment `/ecosystem-ci run` on a PR.
 
-- comment `/ecosystem-ci run` on a PR
-- or `/ecosystem-ci run <suitename>` to select a suite
+You can optionally specify a test suite in the comment: `/ecosystem-ci run <suitename>`
 
-Users with triage permission to withastro/astro repository can only use this.
+See the [“Setting up PR comment triggers”](./docs/pr-comment-setup.md) guide for how to configure this feature.
 
-See [docs/pr-comment-setup.md](./docs/pr-comment-setup.md) for how to setup this feature.
+### Run tests locally
 
-# how to add a new integration test
+1. Clone this repo
 
-- check out the existing [tests](./tests) and add one yourself. Thanks to some utilities it is really easy
-- once you are confident the suite works, add it to the lists of suites in the [workflows](../../actions/)
+2. Install dependencies:
 
-# reporting results
+   ```sh
+    pnpm install
+   ```
 
-## Discord
+3. Run all tests or a single suite as required:
 
-Results are posted automatically to `#ecosystem-ci` on [astro discord](https://astro.build/chat)
+   ```sh
+   # run all tests:
+   pnpm test
+   # run a single test suite, e.g. pnpm test adapters
+   pnpm test <suitename>
+   ```
 
-### on your own server
+   **Optional flags:**
 
-- Go to `Server settings > Integrations > Webhooks` and click `New Webhook`
-- Give it a name, icon and a channel to post to
-- copy the webhook url
-- get in touch with admins of this repo so they can add the webhook
+   You can pass options to the CLI command to select a specific Astro version to use with tests:
 
-#### how to add a discord webhook here
+   - `--tag v2.8.0-beta.1` — use a specific Git tag in the Astro repo
+   - `--branch somebranch` — use a specific Git branch in the Astro repo
+   - `--commit abcd1234` — use a specific Git hash in the Astro repo
+   - `--release 2.7.13` — fetch the specified release from the NPM registry (this will skip building Astro from source)
 
-- Go to `<github repo>/settings/secrets/actions` and click on `New repository secret`
-- set `Name` as `DISCORD_WEBHOOK_URL`
-- paste the discord webhook url you copied from above into `Value`
-- Click `Add secret`
+The repositories for each test suite are checked out into the `workspace/` subdirectory as shallow clones.
+
+## How to add a new test suite
+
+1. If you are not an Astro maintainer, fork and clone this repository.
+
+2. Create a new file in the [tests/](./tests/) directory.
+
+3. Export a `test()` function, which calls the `runInRepo()` utility to configure and run tests for your project.
+
+   For example, the following test would clone a fictional `nasa/apollo` repo from GitHub, set up overrides for internal Astro packages, and run the `test` script specified in that repo’s `package.json`:
+
+   ```ts
+   // tests/nasa-apollo.ts
+   import type { RunOptions } from "../types.d.ts";
+   import { runInRepo } from "../utils.ts";
+
+   export async function test(options: RunOptions) {
+   	await runInRepo({
+   		...options,
+   		repo: "nasa/apollo",
+   		overrides: {
+   			"@astrojs/internal-helpers": true,
+   			"@astrojs/markdown-remark": true,
+   			"@astrojs/prism": true,
+   			"@astrojs/telemetry": true,
+   		},
+   		test: "test",
+   	});
+   }
+   ```
+
+   Use the existing tests as a guide for common patterns in test files.
+
+4. Test your changes locally by running:
+
+   ```sh
+   pnpm test your-file-name-here
+   ```
+
+5. Once you are confident your new test suite works, add its name to the lists of suites in the [GitHub workflow configuration files](./.github/workflows/).
+
+   1. In [`ecosystem-ci-from-pr.yml`](./.github/workflows/ecosystem-ci-from-pr.yml), add your suite name to:
+
+      - `on.workflow_dispatch.inputs.suite.options`
+      - `jobs.execute-all.strategy.matrix.suite`
+
+   2. In [`ecosystem-ci-selected.yml`](./.github/workflows/ecosystem-ci-selected.yml), add your suite name to:
+
+      - `on.workflow_dispatch.inputs.suite.options`
+
+   3. In [`ecosystem-ci.yml`](./.github/workflows/ecosystem-ci.yml), add your suite name to`:
+
+      - `jobs.test-ecosystem.strategy.matrix.suite`
+
+6. Open a PR adding your new files.
+
+## Reporting results
+
+### On Discord
+
+Results are posted automatically to the `#ecosystem-ci` channel in the [Astro Discord server](https://astro.build/chat).
+
+See the [“Setting up a Discord webhook notification”](./docs/discord-webhook-setup.md) guide for how to configure this initially.

docs/discord-webhook-setup.md

@@ -0,0 +1,19 @@
+# Setting up a Discord webhook notification
+
+If you want to configure notifications in the Astro Discord server, follow these steps:
+
+1. On the Astro Discord server, navigate to `Server settings > Integrations > Webhooks` and click <kbd>New Webhook</kbd>.
+
+2. Give it a name, icon, and a channel to post to.
+
+3. Copy the webhook URL.
+
+4. Open the [secrets settings for this repository](https://github.com/withastro/astro-ecosystem-ci/settings/secrets/actions) (requires admin privileges).
+
+5. click on <kbd>New repository secret</kbd>.
+
+6. Set the `Name` for the new secret to `DISCORD_WEBHOOK_URL`.
+
+7. Paste the Discord webhook URL you copied into `Value`.
+
+8. Click <kbd>Add secret</kbd>.

docs/pr-comment-setup.md

@@ -1,27 +1,39 @@
-# Setting up "PR comment trigger" feature
+# Setting up PR comment triggers
 
-## (1) Create a GitHub App
+## 1. Create a GitHub App
+
+1. [Create a GitHub App](https://docs.github.com/en/apps/creating-github-apps/registering-a-github-app/registering-a-github-app). Webhook is not needed.
+
+   The following permissions are required:
+
+   - **Metadata:** Read only
+   - **Actions:** Read and Write
+   - **Issues:** Read and Write
+   - **Pull requests:** Read and Write
+
+2. Install the app to the `withastro` org and give it access to `withastro/astro` and `withastro/astro-ecosystem-ci`.
+
+3. Get the **App ID** for your app.
+
+   You can find this at `https://github.com/settings/apps/<github-app-name-slug>`, for example:
 
-1. [Create a GitHub App](https://docs.github.com/en/apps/creating-github-apps/registering-a-github-app/registering-a-github-app). Webhook is not needed. The following permissions are required:
-   - Metadata: Read only
-   - Actions: Read and Write
-   - Issues: Read and Write
-   - Pull requests: Read and Write
-1. Install that App to the organization/user. Give that App access to vitejs/vite and vitejs/vite-ecosystem-ci.
-1. Check the App ID. It's written on `https://github.com/settings/apps/<github-app-name-slug>`. This is used later.
    ![GitHub App ID](github_app_id.png)
-1. Generate a private key. It can be generated on the same page with the App ID. The key will be downloaded when you generate it.
+
+4. Generate a private key.
+
+   This can be generated on the same page as the App ID. The key will be downloaded when you generate it.
+
    ![GitHub App private key](github_app_private_key.png)
 
-## (2) Adding secrets to vitejs/vite and vitejs/vite-ecosystem-ci
+## 2. Add secrets to `withastro/astro` and `withastro/astro-ecosystem-ci`
 
-- vitejs/vite
+- `withastro/astro`:
   - `ECOSYSTEM_CI_GITHUB_APP_ID`: ID of the created GitHub App
   - `ECOSYSTEM_CI_GITHUB_APP_PRIVATE_KEY`: the content of the private key of the created GitHub App
-- vitejs/vite-ecosystem-ci
+- `withastro/astro-ecosystem-ci`:
   - `PR_GITHUB_APP_ID`: ID of the created GitHub App
   - `PR_GITHUB_APP_PRIVATE_KEY`: the content of the private key of the created GitHub App
 
-## (3) Adding workflows to vitejs/vite
+## 3. Add workflows to `withastro/astro`
 
 Add [this workflow](https://github.com/vitejs/vite/blob/main/.github/workflows/ecosystem-ci-trigger.yml).

types.d.ts

@@ -1,4 +1,4 @@
-import type { Agent } from '@antfu/ni';
+import type { Agent } from "@antfu/ni";
 export interface EnvironmentData {
 	root: string;
 	workspace: string;
@@ -11,15 +11,48 @@ export interface RunOptions {
 	workspace: string;
 	root: string;
 	astroPath: string;
-	// viteMajor: number
+	/**
+	 * @default true
+	 */
 	verify?: boolean;
+	/**
+	 * Whether to skip cloning and setting up the remote Git repository.
+	 * @default false
+	 */
 	skipGit?: boolean;
+	/**
+	 * Equivalent to setting a custom value for `astro` in the `overrides` option.
+	 * Used by the `--release` CLI flag.
+	 */
 	release?: string;
+	/** The package manager to use to run commands. If not set, tries to detect this automatically. */
 	agent?: Agent;
+	/** One or more scripts to run in the repository to build/prepare the project. */
 	build?: Task | Task[];
+	/**
+	 * One or more scripts to run in the repository to execute tests.
+	 *
+	 * These can range from very simple:
+	 * ```js
+	 * // runs the `test` script defined in `package.json` with the current package manager
+	 * test: "test",
+	 * ```
+	 *
+	 * To a more complex series steps:
+	 * ```js
+	 * test: [
+	 *   "node ./setup-script",
+	 *   "pnpm --filter package-A test",
+	 *   "pnpm --filter package-B type-check",
+	 * ],
+	 * ```
+	 */
 	test?: Task | Task[];
+	/** One or more scripts to run before installing dependencies. */
 	beforeInstall?: Task | Task[];
+	/** One or more scripts to run before building the project. */
 	beforeBuild?: Task | Task[];
+	/** One or more scripts to run before running tests. */
 	beforeTest?: Task | Task[];
 }
 
@@ -37,12 +70,56 @@ export interface CommandOptions {
 }
 
 export interface RepoOptions {
+	/**
+	 * A `username/repo` identifier or Git URL for the repository to test. Examples:
+	 *
+	 * - `"withastro/starlight"`
+	 * - `"https://github.com/withastro/starlight.git"`
+	 *
+	 * The shorthand style is assumed to be hosted on GitHub. The URL could point to other Git hosts.
+	 */
 	repo: string;
+	/**
+	 * A subdirectory of the cloned repository to `cd` into before running tests.
+	 * @default undefined
+	 */
 	dir?: string;
+	/**
+	 * The branch of the repository to test.
+	 * @default "main"
+	 */
 	branch?: string;
+	/**
+	 * A Git tag to check out for tests. Overrides the `branch` and `commit` options if set.
+	 * @default undefined
+	 */
 	tag?: string;
+	/**
+	 * A Git commit hash to check out for tests. Overrides the `branch` option if set.
+	 * @default undefined
+	 */
 	commit?: string;
+	/**
+	 * Whether the repository should be shallow or deep cloned.
+	 * @default true
+	 */
 	shallow?: boolean;
+	/**
+	 * A map of dependencies to override in the repository being tested.
+	 *
+	 * `astro` will be overridden by default and you can specify additional `@astrojs/*`
+	 * packages to override here. For example:
+	 *
+	 * ```js
+	 * {
+	 *   "@astrojs/internal-helpers": true,
+	 *   "@astrojs/markdown-remark": true,
+	 * }
+	 * ```
+	 *
+	 * An override can also be set to a custom path to the override to use, but usually the
+	 * automatic behavior provided by setting `true` is sufficient.
+	 */
 	overrides?: Overrides;
 }