diff --git a/.agents/skills/e2e-cucumber-playwright/SKILL.md b/.agents/skills/e2e-cucumber-playwright/SKILL.md new file mode 100644 index 0000000000..de6b58f26d --- /dev/null +++ b/.agents/skills/e2e-cucumber-playwright/SKILL.md @@ -0,0 +1,79 @@ +--- +name: e2e-cucumber-playwright +description: Write, update, or review Dify end-to-end tests under `e2e/` that use Cucumber, Gherkin, and Playwright. Use when the task involves `.feature` files, `features/step-definitions/`, `features/support/`, `DifyWorld`, scenario tags, locator/assertion choices, or E2E testing best practices for this repository. +--- + +# Dify E2E Cucumber + Playwright + +Use this skill for Dify's repository-level E2E suite in `e2e/`. Use [`e2e/AGENTS.md`](../../../e2e/AGENTS.md) as the canonical guide for local architecture and conventions, then apply Playwright/Cucumber best practices only where they fit the current suite. + +## Scope + +- Use this skill for `.feature` files, Cucumber step definitions, `DifyWorld`, hooks, tags, and E2E review work under `e2e/`. +- Do not use this skill for Vitest or React Testing Library work under `web/`; use `frontend-testing` instead. +- Do not use this skill for backend test or API review tasks under `api/`. + +## Read Order + +1. Read [`e2e/AGENTS.md`](../../../e2e/AGENTS.md) first. +2. Read only the files directly involved in the task: + - target `.feature` files under `e2e/features/` + - related step files under `e2e/features/step-definitions/` + - `e2e/features/support/hooks.ts` and `e2e/features/support/world.ts` when session lifecycle or shared state matters + - `e2e/scripts/run-cucumber.ts` and `e2e/cucumber.config.ts` when tags or execution flow matter +3. Read [`references/playwright-best-practices.md`](references/playwright-best-practices.md) only when locator, assertion, isolation, or waiting choices are involved. +4. Read [`references/cucumber-best-practices.md`](references/cucumber-best-practices.md) only when scenario wording, step granularity, tags, or expression design are involved. +5. Re-check official docs with Context7 before introducing a new Playwright or Cucumber pattern. + +## Local Rules + +- `e2e/` uses Cucumber for scenarios and Playwright as the browser layer. +- `DifyWorld` is the per-scenario context object. Type `this` as `DifyWorld` and use `async function`, not arrow functions. +- Keep glue organized by capability under `e2e/features/step-definitions/`; use `common/` only for broadly reusable steps. +- Browser session behavior comes from `features/support/hooks.ts`: + - default: authenticated session with shared storage state + - `@unauthenticated`: clean browser context + - `@authenticated`: readability/selective-run tag only unless implementation changes + - `@fresh`: only for `e2e:full*` flows +- Do not import Playwright Test runner patterns that bypass the current Cucumber + `DifyWorld` architecture unless the task is explicitly about changing that architecture. + +## Workflow + +1. Rebuild local context. + - Inspect the target feature area. + - Reuse an existing step when wording and behavior already match. + - Add a new step only for a genuinely new user action or assertion. + - Keep edits close to the current capability folder unless the step is broadly reusable. +2. Write behavior-first scenarios. + - Describe user-observable behavior, not DOM mechanics. + - Keep each scenario focused on one workflow or outcome. + - Keep scenarios independent and re-runnable. +3. Write step definitions in the local style. + - Keep one step to one user-visible action or one assertion. + - Prefer Cucumber Expressions such as `{string}` and `{int}`. + - Scope locators to stable containers when the page has repeated elements. + - Avoid page-object layers or extra helper abstractions unless repeated complexity clearly justifies them. +4. Use Playwright in the local style. + - Prefer user-facing locators: `getByRole`, `getByLabel`, `getByPlaceholder`, `getByText`, then `getByTestId` for explicit contracts. + - Use web-first `expect(...)` assertions. + - Do not use `waitForTimeout`, manual polling, or raw visibility checks when a locator action or retrying assertion already expresses the behavior. +5. Validate narrowly. + - Run the narrowest tagged scenario or flow that exercises the change. + - Run `pnpm -C e2e check`. + - Broaden verification only when the change affects hooks, tags, setup, or shared step semantics. + +## Review Checklist + +- Does the scenario describe behavior rather than implementation? +- Does it fit the current session model, tags, and `DifyWorld` usage? +- Should an existing step be reused instead of adding a new one? +- Are locators user-facing and assertions web-first? +- Does the change introduce hidden coupling across scenarios, tags, or instance state? +- Does it document or implement behavior that differs from the real hooks or configuration? + +Lead findings with correctness, flake risk, and architecture drift. + +## References + +- [`references/playwright-best-practices.md`](references/playwright-best-practices.md) +- [`references/cucumber-best-practices.md`](references/cucumber-best-practices.md) diff --git a/.agents/skills/e2e-cucumber-playwright/agents/openai.yaml b/.agents/skills/e2e-cucumber-playwright/agents/openai.yaml new file mode 100644 index 0000000000..605cce041d --- /dev/null +++ b/.agents/skills/e2e-cucumber-playwright/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "E2E Cucumber + Playwright" + short_description: "Write and review Dify E2E scenarios." + default_prompt: "Use $e2e-cucumber-playwright to write or review a Dify E2E scenario under e2e/." diff --git a/.agents/skills/e2e-cucumber-playwright/references/cucumber-best-practices.md b/.agents/skills/e2e-cucumber-playwright/references/cucumber-best-practices.md new file mode 100644 index 0000000000..d7a1a52852 --- /dev/null +++ b/.agents/skills/e2e-cucumber-playwright/references/cucumber-best-practices.md @@ -0,0 +1,93 @@ +# Cucumber Best Practices For Dify E2E + +Use this reference when writing or reviewing Gherkin scenarios, step definitions, parameter expressions, and step reuse in Dify's `e2e/` suite. + +Official sources: + +- https://cucumber.io/docs/guides/10-minute-tutorial/ +- https://cucumber.io/docs/cucumber/step-definitions/ +- https://cucumber.io/docs/cucumber/cucumber-expressions/ + +## What Matters Most + +### 1. Treat scenarios as executable specifications + +Cucumber scenarios should describe examples of behavior, not test implementation recipes. + +Apply it like this: + +- write what the user does and what should happen +- avoid UI-internal wording such as selector details, DOM structure, or component names +- keep language concrete enough that the scenario reads like living documentation + +### 2. Keep scenarios focused + +A scenario should usually prove one workflow or business outcome. If a scenario wanders across several unrelated behaviors, split it. + +In Dify's suite, this means: + +- one capability-focused scenario per feature path +- no long setup chains when existing bootstrap or reusable steps already cover them +- no hidden dependency on another scenario's side effects + +### 3. Reuse steps, but only when behavior really matches + +Good reuse reduces duplication. Bad reuse hides meaning. + +Prefer reuse when: + +- the user action is genuinely the same +- the expected outcome is genuinely the same +- the wording stays natural across features + +Write a new step when: + +- the behavior is materially different +- reusing the old wording would make the scenario misleading +- a supposedly generic step would become an implementation-detail wrapper + +### 4. Prefer Cucumber Expressions + +Use Cucumber Expressions for parameters unless regex is clearly necessary. + +Common examples: + +- `{string}` for labels, names, and visible text +- `{int}` for counts +- `{float}` for decimal values +- `{word}` only when the value is truly a single token + +Keep expressions readable. If a step needs complicated parsing logic, first ask whether the scenario wording should be simpler. + +### 5. Keep step definitions thin and meaningful + +Step definitions are glue between Gherkin and automation, not a second abstraction language. + +For Dify: + +- type `this` as `DifyWorld` +- use `async function` +- keep each step to one user-visible action or assertion +- rely on `DifyWorld` and existing support code for shared context +- avoid leaking cross-scenario state + +### 6. Use tags intentionally + +Tags should communicate run scope or session semantics, not become ad hoc metadata. + +In Dify's current suite: + +- capability tags group related scenarios +- `@unauthenticated` changes session behavior +- `@authenticated` is descriptive/selective, not a behavior switch by itself +- `@fresh` belongs to reset/full-install flows only + +If a proposed tag implies behavior, verify that hooks or runner configuration actually implement it. + +## Review Questions + +- Does the scenario read like a real example of product behavior? +- Are the steps behavior-oriented instead of implementation-oriented? +- Is a reused step still truthful in this feature? +- Is a new tag documenting real behavior, or inventing semantics that the suite does not implement? +- Would a new reader understand the outcome without opening the step-definition file? diff --git a/.agents/skills/e2e-cucumber-playwright/references/playwright-best-practices.md b/.agents/skills/e2e-cucumber-playwright/references/playwright-best-practices.md new file mode 100644 index 0000000000..02e763d46b --- /dev/null +++ b/.agents/skills/e2e-cucumber-playwright/references/playwright-best-practices.md @@ -0,0 +1,96 @@ +# Playwright Best Practices For Dify E2E + +Use this reference when writing or reviewing locator, assertion, isolation, or synchronization logic for Dify's Cucumber-based E2E suite. + +Official sources: + +- https://playwright.dev/docs/best-practices +- https://playwright.dev/docs/locators +- https://playwright.dev/docs/test-assertions +- https://playwright.dev/docs/browser-contexts + +## What Matters Most + +### 1. Keep scenarios isolated + +Playwright's model is built around clean browser contexts so one test does not leak into another. In Dify's suite, that principle maps to per-scenario session setup in `features/support/hooks.ts` and `DifyWorld`. + +Apply it like this: + +- do not depend on another scenario having run first +- do not persist ad hoc scenario state outside `DifyWorld` +- do not couple ordinary scenarios to `@fresh` behavior +- when a flow needs special auth/session semantics, express that through the existing tag model or explicit hook changes + +### 2. Prefer user-facing locators + +Playwright recommends built-in locators that reflect what users perceive on the page. + +Preferred order in this repository: + +1. `getByRole` +2. `getByLabel` +3. `getByPlaceholder` +4. `getByText` +5. `getByTestId` when an explicit test contract is the most stable option + +Avoid raw CSS/XPath selectors unless no stable user-facing contract exists and adding one is not practical. + +Also remember: + +- repeated content usually needs scoping to a stable container +- exact text matching is often too brittle when role/name or label already exists +- `getByTestId` is acceptable when semantics are weak but the contract is intentional + +### 3. Use web-first assertions + +Playwright assertions auto-wait and retry. Prefer them over manual state inspection. + +Prefer: + +- `await expect(page).toHaveURL(...)` +- `await expect(locator).toBeVisible()` +- `await expect(locator).toBeHidden()` +- `await expect(locator).toBeEnabled()` +- `await expect(locator).toHaveText(...)` + +Avoid: + +- `expect(await locator.isVisible()).toBe(true)` +- custom polling loops for DOM state +- `waitForTimeout` as synchronization + +If a condition genuinely needs custom retry logic, use Playwright's polling/assertion tools deliberately and keep that choice local and explicit. + +### 4. Let actions wait for actionability + +Locator actions already wait for the element to be actionable. Do not preface every click/fill with extra timing logic unless the action needs a specific visible/ready assertion for clarity. + +Good pattern: + +- assert a meaningful visible state when that is part of the behavior +- then click/fill/select via locator APIs + +Bad pattern: + +- stack arbitrary waits before every action +- wait on unstable implementation details instead of the visible state the user cares about + +### 5. Match debugging to the current suite + +Playwright's wider ecosystem supports traces and rich debugging tools. Dify's current suite already captures: + +- full-page screenshots +- page HTML +- console errors +- page errors + +Use the existing artifact flow by default. If a task is specifically about improving diagnostics, confirm the change fits the current Cucumber architecture before importing broader Playwright tooling. + +## Review Questions + +- Would this locator survive DOM refactors that do not change user-visible behavior? +- Is this assertion using Playwright's retrying semantics? +- Is any explicit wait masking a real readiness problem? +- Does this code preserve per-scenario isolation? +- Is a new abstraction really needed, or does it bypass the existing `DifyWorld` + step-definition model? diff --git a/.claude/skills/e2e-cucumber-playwright b/.claude/skills/e2e-cucumber-playwright new file mode 120000 index 0000000000..71b0eae34f --- /dev/null +++ b/.claude/skills/e2e-cucumber-playwright @@ -0,0 +1 @@ +../../.agents/skills/e2e-cucumber-playwright \ No newline at end of file diff --git a/e2e/AGENTS.md b/e2e/AGENTS.md index ae642768f5..da3d76210d 100644 --- a/e2e/AGENTS.md +++ b/e2e/AGENTS.md @@ -165,3 +165,132 @@ Open the HTML report locally with: ```bash open cucumber-report/report.html ``` + +## Writing new scenarios + +### Workflow + +1. Create a `.feature` file under `features//` +2. Add step definitions under `features/step-definitions//` +3. Reuse existing steps from `common/` and other definition files before writing new ones +4. Run with `pnpm -C e2e e2e -- --tags @your-tag` to verify +5. Run `pnpm -C e2e check` before committing + +### Feature file conventions + +Tag every feature or scenario with a capability tag. Add auth tags only when they clarify intent or change the browser session behavior: + +```gherkin +@datasets @authenticated +Feature: Create dataset + Scenario: Create a new empty dataset + Given I am signed in as the default E2E admin + When I open the datasets page + ... +``` + +- Capability tags (`@apps`, `@auth`, `@datasets`, …) group related scenarios for selective runs +- Auth/session tags: + - default behavior — scenarios run with the shared authenticated storageState unless marked otherwise + - `@unauthenticated` — uses a clean BrowserContext with no cookies or storage + - `@authenticated` — optional intent tag for readability or selective runs; it does not currently change hook behavior on its own +- `@fresh` — only runs in `e2e:full` mode (requires uninitialized instance) +- `@skip` — excluded from all runs + +Keep scenarios short and declarative. Each step should describe **what** the user does, not **how** the UI works. + +### Step definition conventions + +```typescript +import { When, Then } from '@cucumber/cucumber' +import { expect } from '@playwright/test' +import type { DifyWorld } from '../../support/world' + +When('I open the datasets page', async function (this: DifyWorld) { + await this.getPage().goto('/datasets') +}) +``` + +Rules: + +- Always type `this` as `DifyWorld` for proper context access +- Use `async function` (not arrow functions — Cucumber binds `this`) +- One step = one user-visible action or one assertion +- Keep steps stateless across scenarios; use `DifyWorld` properties for in-scenario state + +### Locator priority + +Follow the Playwright recommended locator strategy, in order of preference: + +| Priority | Locator | Example | When to use | +| -------- | ------------------ | ----------------------------------------- | ----------------------------------------- | +| 1 | `getByRole` | `getByRole('button', { name: 'Create' })` | Default choice — accessible and resilient | +| 2 | `getByLabel` | `getByLabel('App name')` | Form inputs with visible labels | +| 3 | `getByPlaceholder` | `getByPlaceholder('Enter name')` | Inputs without visible labels | +| 4 | `getByText` | `getByText('Welcome')` | Static text content | +| 5 | `getByTestId` | `getByTestId('workflow-canvas')` | Only when no semantic locator works | + +Avoid raw CSS/XPath selectors. They break when the DOM structure changes. + +### Assertions + +Use `@playwright/test` `expect` — it auto-waits and retries until the condition is met or the timeout expires: + +```typescript +// URL assertion +await expect(page).toHaveURL(/\/datasets\/[a-f0-9-]+\/documents/) + +// Element visibility +await expect(page.getByRole('button', { name: 'Save' })).toBeVisible() + +// Element state +await expect(page.getByRole('button', { name: 'Submit' })).toBeEnabled() + +// Negation +await expect(page.getByText('Loading')).not.toBeVisible() +``` + +Do not use manual `waitForTimeout` or polling loops. If you need a longer wait for a specific assertion, pass `{ timeout: 30_000 }` to the assertion. + +### Cucumber expressions + +Use Cucumber expression parameter types to extract values from Gherkin steps: + +| Type | Pattern | Example step | +| ---------- | ------------- | ---------------------------------- | +| `{string}` | Quoted string | `I select the "Workflow" app type` | +| `{int}` | Integer | `I should see {int} items` | +| `{float}` | Decimal | `the progress is {float} percent` | +| `{word}` | Single word | `I click the {word} tab` | + +Prefer `{string}` for UI labels, names, and text content — it maps naturally to Gherkin's quoted values. + +### Scoping locators + +When the page has multiple similar elements, scope locators to a container: + +```typescript +When('I fill in the app name in the dialog', async function (this: DifyWorld) { + const dialog = this.getPage().getByRole('dialog') + await dialog.getByPlaceholder('Give your app a name').fill('My App') +}) +``` + +### Failure diagnostics + +The `After` hook automatically captures on failure: + +- Full-page screenshot (PNG) +- Page HTML dump +- Console errors and page errors + +Artifacts are saved to `cucumber-report/artifacts/` and attached to the HTML report. No extra code needed in step definitions. + +## Reusing existing steps + +Before writing a new step definition, inspect the existing step definition files first. Reuse a matching step when the wording and behavior already fit, and only add a new step when the scenario needs a genuinely new user action or assertion. Steps in `common/` are designed for broad reuse across all features. + +Or browse the step definition files directly: + +- `features/step-definitions/common/` — auth guards and navigation assertions shared by all features +- `features/step-definitions//` — domain-specific steps scoped to a single feature area diff --git a/e2e/features/auth/sign-out.feature b/e2e/features/auth/sign-out.feature index 0f377ea133..9112f1220a 100644 --- a/e2e/features/auth/sign-out.feature +++ b/e2e/features/auth/sign-out.feature @@ -6,3 +6,13 @@ Feature: Sign out And I open the account menu And I sign out Then I should be on the sign-in page + + Scenario: Redirect back to sign-in when reopening the apps console after signing out + Given I am signed in as the default E2E admin + When I open the apps console + And I open the account menu + And I sign out + Then I should be on the sign-in page + When I open the apps console + Then I should be redirected to the signin page + And I should see the "Sign in" button diff --git a/e2e/features/support/hooks.ts b/e2e/features/support/hooks.ts index 9e8c025ef8..7a8319463b 100644 --- a/e2e/features/support/hooks.ts +++ b/e2e/features/support/hooks.ts @@ -3,7 +3,7 @@ import { chromium, type Browser } from '@playwright/test' import { mkdir, writeFile } from 'node:fs/promises' import path from 'node:path' import { fileURLToPath } from 'node:url' -import { ensureAuthenticatedState } from '../../fixtures/auth' +import { AUTH_BOOTSTRAP_TIMEOUT_MS, ensureAuthenticatedState } from '../../fixtures/auth' import { baseURL, cucumberHeadless, cucumberSlowMo } from '../../test-env' import type { DifyWorld } from './world' @@ -31,7 +31,7 @@ const writeArtifact = async ( return artifactPath } -BeforeAll(async () => { +BeforeAll({ timeout: AUTH_BOOTSTRAP_TIMEOUT_MS }, async () => { await mkdir(artifactsDir, { recursive: true }) browser = await chromium.launch({ diff --git a/e2e/fixtures/auth.ts b/e2e/fixtures/auth.ts index 853bfff5ed..14aee52634 100644 --- a/e2e/fixtures/auth.ts +++ b/e2e/fixtures/auth.ts @@ -12,7 +12,7 @@ export type AuthSessionMetadata = { usedInitPassword: boolean } -const WAIT_TIMEOUT_MS = 120_000 +export const AUTH_BOOTSTRAP_TIMEOUT_MS = 120_000 const e2eRoot = fileURLToPath(new URL('..', import.meta.url)) export const authDir = path.join(e2eRoot, '.auth') @@ -39,40 +39,54 @@ const escapeRegex = (value: string) => value.replaceAll(/[.*+?^${}()|[\]\\]/g, ' const appURL = (baseURL: string, pathname: string) => new URL(pathname, baseURL).toString() -const waitForPageState = async (page: Page) => { +type AuthPageState = 'install' | 'login' | 'init' + +const getRemainingTimeout = (deadline: number) => Math.max(deadline - Date.now(), 1) + +const waitForPageState = async (page: Page, deadline: number): Promise => { const installHeading = page.getByRole('heading', { name: 'Setting up an admin account' }) const signInButton = page.getByRole('button', { name: 'Sign in' }) const initPasswordField = page.getByLabel('Admin initialization password') - const deadline = Date.now() + WAIT_TIMEOUT_MS - - while (Date.now() < deadline) { - if (await installHeading.isVisible().catch(() => false)) return 'install' as const - if (await signInButton.isVisible().catch(() => false)) return 'login' as const - if (await initPasswordField.isVisible().catch(() => false)) return 'init' as const - - await page.waitForTimeout(1_000) + try { + return await Promise.any([ + installHeading + .waitFor({ state: 'visible', timeout: getRemainingTimeout(deadline) }) + .then(() => 'install'), + signInButton + .waitFor({ state: 'visible', timeout: getRemainingTimeout(deadline) }) + .then(() => 'login'), + initPasswordField + .waitFor({ state: 'visible', timeout: getRemainingTimeout(deadline) }) + .then(() => 'init'), + ]) + } catch { + throw new Error(`Unable to determine auth page state for ${page.url()}`) } - - throw new Error(`Unable to determine auth page state for ${page.url()}`) } -const completeInitPasswordIfNeeded = async (page: Page) => { +const completeInitPasswordIfNeeded = async (page: Page, deadline: number) => { const initPasswordField = page.getByLabel('Admin initialization password') - if (!(await initPasswordField.isVisible({ timeout: 3_000 }).catch(() => false))) return false + + const needsInitPassword = await initPasswordField + .waitFor({ state: 'visible', timeout: Math.min(getRemainingTimeout(deadline), 3_000) }) + .then(() => true) + .catch(() => false) + + if (!needsInitPassword) return false await initPasswordField.fill(initPassword) await page.getByRole('button', { name: 'Validate' }).click() await expect(page.getByRole('heading', { name: 'Setting up an admin account' })).toBeVisible({ - timeout: WAIT_TIMEOUT_MS, + timeout: getRemainingTimeout(deadline), }) return true } -const completeInstall = async (page: Page, baseURL: string) => { +const completeInstall = async (page: Page, baseURL: string, deadline: number) => { await expect(page.getByRole('heading', { name: 'Setting up an admin account' })).toBeVisible({ - timeout: WAIT_TIMEOUT_MS, + timeout: getRemainingTimeout(deadline), }) await page.getByLabel('Email address').fill(adminCredentials.email) @@ -81,13 +95,13 @@ const completeInstall = async (page: Page, baseURL: string) => { await page.getByRole('button', { name: 'Set up' }).click() await expect(page).toHaveURL(new RegExp(`^${escapeRegex(baseURL)}/apps(?:\\?.*)?$`), { - timeout: WAIT_TIMEOUT_MS, + timeout: getRemainingTimeout(deadline), }) } -const completeLogin = async (page: Page, baseURL: string) => { +const completeLogin = async (page: Page, baseURL: string, deadline: number) => { await expect(page.getByRole('button', { name: 'Sign in' })).toBeVisible({ - timeout: WAIT_TIMEOUT_MS, + timeout: getRemainingTimeout(deadline), }) await page.getByLabel('Email address').fill(adminCredentials.email) @@ -95,12 +109,13 @@ const completeLogin = async (page: Page, baseURL: string) => { await page.getByRole('button', { name: 'Sign in' }).click() await expect(page).toHaveURL(new RegExp(`^${escapeRegex(baseURL)}/apps(?:\\?.*)?$`), { - timeout: WAIT_TIMEOUT_MS, + timeout: getRemainingTimeout(deadline), }) } export const ensureAuthenticatedState = async (browser: Browser, configuredBaseURL?: string) => { const baseURL = resolveBaseURL(configuredBaseURL) + const deadline = Date.now() + AUTH_BOOTSTRAP_TIMEOUT_MS await mkdir(authDir, { recursive: true }) @@ -111,25 +126,28 @@ export const ensureAuthenticatedState = async (browser: Browser, configuredBaseU const page = await context.newPage() try { - await page.goto(appURL(baseURL, '/install'), { waitUntil: 'networkidle' }) + await page.goto(appURL(baseURL, '/install'), { + timeout: getRemainingTimeout(deadline), + waitUntil: 'domcontentloaded', + }) - let usedInitPassword = await completeInitPasswordIfNeeded(page) - let pageState = await waitForPageState(page) + let usedInitPassword = await completeInitPasswordIfNeeded(page, deadline) + let pageState = await waitForPageState(page, deadline) while (pageState === 'init') { - const completedInitPassword = await completeInitPasswordIfNeeded(page) + const completedInitPassword = await completeInitPasswordIfNeeded(page, deadline) if (!completedInitPassword) throw new Error(`Unable to validate initialization password for ${page.url()}`) usedInitPassword = true - pageState = await waitForPageState(page) + pageState = await waitForPageState(page, deadline) } - if (pageState === 'install') await completeInstall(page, baseURL) - else await completeLogin(page, baseURL) + if (pageState === 'install') await completeInstall(page, baseURL, deadline) + else await completeLogin(page, baseURL, deadline) await expect(page.getByRole('button', { name: 'Create from Blank' })).toBeVisible({ - timeout: WAIT_TIMEOUT_MS, + timeout: getRemainingTimeout(deadline), }) await context.storageState({ path: authStatePath }) diff --git a/e2e/scripts/common.ts b/e2e/scripts/common.ts index bb82121079..5bca7fb4c9 100644 --- a/e2e/scripts/common.ts +++ b/e2e/scripts/common.ts @@ -1,4 +1,5 @@ import { spawn, type ChildProcess } from 'node:child_process' +import { createHash } from 'node:crypto' import { access, copyFile, readFile, writeFile } from 'node:fs/promises' import net from 'node:net' import path from 'node:path' @@ -38,6 +39,10 @@ export const middlewareEnvExampleFile = path.join(dockerDir, 'middleware.env.exa export const webEnvLocalFile = path.join(webDir, '.env.local') export const webEnvExampleFile = path.join(webDir, '.env.example') export const apiEnvExampleFile = path.join(apiDir, 'tests', 'integration_tests', '.env.example') +export const e2eWebEnvOverrides = { + NEXT_PUBLIC_API_PREFIX: 'http://127.0.0.1:5001/console/api', + NEXT_PUBLIC_PUBLIC_API_PREFIX: 'http://127.0.0.1:5001/api', +} satisfies Record const formatCommand = (command: string, args: string[]) => [command, ...args].join(' ') @@ -166,13 +171,16 @@ export const ensureLineInFile = async (filePath: string, line: string) => { await writeFile(filePath, `${normalizedContent}${line}\n`, 'utf8') } -export const ensureWebEnvLocal = async () => { - await ensureFileExists(webEnvLocalFile, webEnvExampleFile) - - const fileContent = await readFile(webEnvLocalFile, 'utf8') - const nextContent = fileContent.replaceAll('http://localhost:5001', 'http://127.0.0.1:5001') - - if (nextContent !== fileContent) await writeFile(webEnvLocalFile, nextContent, 'utf8') +export const getWebEnvLocalHash = async () => { + const fileContent = await readFile(webEnvLocalFile, 'utf8').catch(() => '') + return createHash('sha256') + .update( + JSON.stringify({ + envLocal: fileContent, + overrides: e2eWebEnvOverrides, + }), + ) + .digest('hex') } export const readSimpleDotenv = async (filePath: string) => { diff --git a/e2e/scripts/setup.ts b/e2e/scripts/setup.ts index 6f38598df4..4bd9de09d2 100644 --- a/e2e/scripts/setup.ts +++ b/e2e/scripts/setup.ts @@ -1,14 +1,15 @@ -import { access, mkdir, rm } from 'node:fs/promises' +import { access, mkdir, readFile, rm, writeFile } from 'node:fs/promises' import path from 'node:path' import { waitForUrl } from '../support/process' import { apiDir, apiEnvExampleFile, dockerDir, + e2eWebEnvOverrides, e2eDir, ensureFileExists, ensureLineInFile, - ensureWebEnvLocal, + getWebEnvLocalHash, isMainModule, isTcpPortReachable, middlewareComposeFile, @@ -23,6 +24,7 @@ import { } from './common' const buildIdPath = path.join(webDir, '.next', 'BUILD_ID') +const webBuildEnvStampPath = path.join(webDir, '.next', 'e2e-web-env.sha256') const middlewareDataPaths = [ path.join(dockerDir, 'volumes', 'db', 'data'), @@ -110,27 +112,47 @@ const waitForDependency = async ({ } export const ensureWebBuild = async () => { - await ensureWebEnvLocal() + const envHash = await getWebEnvLocalHash() + const buildEnv = { + ...e2eWebEnvOverrides, + } if (process.env.E2E_FORCE_WEB_BUILD === '1') { await runCommandOrThrow({ command: 'pnpm', args: ['run', 'build'], cwd: webDir, + env: buildEnv, }) + await writeFile(webBuildEnvStampPath, `${envHash}\n`, 'utf8') return } try { - await access(buildIdPath) - console.log('Reusing existing web build artifact.') + const [buildExists, previousEnvHash] = await Promise.all([ + access(buildIdPath) + .then(() => true) + .catch(() => false), + readFile(webBuildEnvStampPath, 'utf8') + .then((value) => value.trim()) + .catch(() => ''), + ]) + + if (buildExists && previousEnvHash === envHash) { + console.log('Reusing existing web build artifact.') + return + } } catch { - await runCommandOrThrow({ - command: 'pnpm', - args: ['run', 'build'], - cwd: webDir, - }) + // Fall through to rebuild when the existing build cannot be verified. } + + await runCommandOrThrow({ + command: 'pnpm', + args: ['run', 'build'], + cwd: webDir, + env: buildEnv, + }) + await writeFile(webBuildEnvStampPath, `${envHash}\n`, 'utf8') } export const startWeb = async () => { @@ -141,6 +163,7 @@ export const startWeb = async () => { args: ['run', 'start'], cwd: webDir, env: { + ...e2eWebEnvOverrides, HOSTNAME: '127.0.0.1', PORT: '3000', }, @@ -152,14 +175,25 @@ export const startApi = async () => { await runCommandOrThrow({ command: 'uv', - args: ['run', '--project', '.', 'flask', 'upgrade-db'], + args: ['run', '--project', '.', '--no-sync', 'flask', 'upgrade-db'], cwd: apiDir, env, }) await runForegroundProcess({ command: 'uv', - args: ['run', '--project', '.', 'flask', 'run', '--host', '127.0.0.1', '--port', '5001'], + args: [ + 'run', + '--project', + '.', + '--no-sync', + 'flask', + 'run', + '--host', + '127.0.0.1', + '--port', + '5001', + ], cwd: apiDir, env, })