Remove `endpoint.write_to_disk()` in favor of standardized `endpoint.output()` #75059

wbinnssmith · 2025-01-18T01:17:32Z

This:

Makes the Endpoint trait not manage any file writing on its own, instead returning a structure containing OutputAssets, the paths where the OutputAssets can be found, and the associated Project.
Allows for OutputAssets to be written at a future point in Turbopack
Implements endpoint_write_to_disk for compatibility in dev. This continues to be used for prod builds, but in the future it won’t.

In a future PR, production builds will dedupe and unify all output assets, writing them to disk at once, rather than exposing individual endpoint writes to JS to occur sequentially.

Test Plan: CI

wbinnssmith · 2025-01-18T01:17:44Z

Remove endpoint.write_to_disk() in favor of standardized endpoint.output() #75059 👈 (View in Graphite)
refactor(turbopack): Only perform strongly consistent reads/resolves on VcOperation #75016 : 1 other dependent PR (#75051 )
perf(turbo-tasks): Optionally schedule ResolveNative/ResolveTrait tasks as local tasks #69126
refactor(turbopack/next-api): Make VcArc use OperationVc #74479
fix(turbopack-swc-utils): Don't panic if issues aren't emitted, use #[must_use] annotation #74906
canary

This stack of pull requests is managed by Graphite. Learn more about stacking.

ijjk · 2025-01-18T01:29:00Z

Failing test suites

Commit: 192bdc6

pnpm test-dev-turbo test/development/acceptance/ReactRefreshRequire.test.ts (turbopack)

ReactRefreshRequire > propagates hot update to all inverse dependencies

Expand output

● ReactRefreshRequire › propagates hot update to all inverse dependencies

expect(received).toEqual(expected) // deep equality

Expected: ["init LeafV1", "init MiddleAV1", "init MiddleBV1", "init MiddleCV1", "init RootV1"]
Received: undefined

  243 |     )
  244 |
> 245 |     expect(await session.evaluate(() => (window as any).log)).toEqual([
      |                                                               ^
  246 |       'init LeafV1',
  247 |       'init MiddleAV1',
  248 |       'init MiddleBV1',

  at Object.toEqual (development/acceptance/ReactRefreshRequire.test.ts:245:63)

Read more about building and testing Next.js in contributing.md.

pnpm test-dev-turbo test/development/acceptance-app/ReactRefreshRequire.test.ts (turbopack)

ReactRefreshRequire app > re-runs accepted modules
ReactRefreshRequire app > propagates a hot update to closest accepted module
ReactRefreshRequire app > propagates hot update to all inverse dependencies

Expand output

● ReactRefreshRequire app › re-runs accepted modules

expect(received).toEqual(expected) // deep equality

Expected: ["init FooV1", "init BarV1"]
Received: undefined

  35 |       `require('./foo'); export default function Noop() { return null; };`
  36 |     )
> 37 |     expect(await session.evaluate(() => (window as any).log)).toEqual([
     |                                                               ^
  38 |       'init FooV1',
  39 |       'init BarV1',
  40 |     ])

  at Object.toEqual (development/acceptance-app/ReactRefreshRequire.test.ts:37:63)

● ReactRefreshRequire app › propagates a hot update to closest accepted module

expect(received).toEqual(expected) // deep equality

Expected: ["init FooV1", "init BarV1"]
Received: undefined

   96 |     )
   97 |
>  98 |     expect(await session.evaluate(() => (window as any).log)).toEqual([
      |                                                               ^
   99 |       'init FooV1',
  100 |       'init BarV1',
  101 |     ])

  at Object.toEqual (development/acceptance-app/ReactRefreshRequire.test.ts:98:63)

● ReactRefreshRequire app › propagates hot update to all inverse dependencies

expect(received).toEqual(expected) // deep equality

Expected: ["init LeafV1", "init MiddleAV1", "init MiddleBV1", "init MiddleCV1", "init RootV1"]
Received: undefined

  244 |     )
  245 |
> 246 |     expect(await session.evaluate(() => (window as any).log)).toEqual([
      |                                                               ^
  247 |       'init LeafV1',
  248 |       'init MiddleAV1',
  249 |       'init MiddleBV1',

  at Object.toEqual (development/acceptance-app/ReactRefreshRequire.test.ts:246:63)

Read more about building and testing Next.js in contributing.md.

pnpm test test/integration/app-document-remove-hmr/test/index.test.js (turbopack)

_app removal HMR > should HMR when _app is removed

Expand output

● _app removal HMR › should HMR when _app is removed

TIMED OUT: success

<head><meta charset="utf-8" data-next-head=""><meta name="viewport" content="width=device-width" data-next-head=""><noscript data-n-css=""></noscript><script src="/_next/static/chunks/packages_next_dist_compiled_d4dc90._.js" defer=""></script><script src="/_next/static/chunks/packages_next_dist_shared_lib_cd2c78._.js" defer=""></script><script src="/_next/static/chunks/packages_next_dist_client_0487f6._.js" defer=""></script><script src="/_next/static/chunks/packages_next_dist_04dd8f._.js" defer=""></script><script src="/_next/static/chunks/packages_next_app_1d3324.js" defer=""></script><script src="/_next/static/chunks/test_integration_app-document-remove-hmr_pages__app_6c205e._.js" defer=""></script><script src="/_next/static/chunks/%5Broot%20of%20the%20server%5D__f97c22._.js" defer=""></script><script src="/_next/static/chunks/node_modules__pnpm_9f95c7._.js" defer=""></script><script src="/_next/static/chunks/test_integration_app-document-remove-hmr_pages__app_5771e1._.js" defer=""></script><script src="/_next/static/chunks/test_integration_app-document-remove-hmr_pages__app_a577fe._.js" defer=""></script><script src="/_next/static/chunks/packages_next_dist_29c433._.js" defer=""></script><script src="/_next/static/chunks/test_integration_app-document-remove-hmr_pages_c8bddd._.js" defer=""></script><script src="/_next/static/chunks/%5Broot%20of%20the%20server%5D__7c0a5e._.js" defer=""></script><script src="/_next/static/chunks/node_modules__pnpm_ab5364._.js" defer=""></script><script src="/_next/static/chunks/test_integration_app-document-remove-hmr_pages_index_5771e1._.js" defer=""></script><script src="/_next/static/chunks/test_integration_app-document-remove-hmr_pages_index_89ecb5._.js" defer=""></script><script src="/_next/static/development/_ssgManifest.js" defer=""></script><script src="/_next/static/development/_buildManifest.js" defer=""></script><noscript id="__next_css__DO_NOT_USE__"></noscript></head><body><p>custom _document</p><div id="__next"><p>index page updated</p></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{}},"page":"/","query":{},"buildId":"development","nextExport":true,"autoExport":true,"isFallback":false,"scriptLoader":[]}</script><div id="__next-build-indicator" style="position: fixed; bottom: 10px; right: 20px; width: 0px; height: 0px; z-index: 99999;"></div><next-route-announcer><p aria-live="assertive" id="__next-route-announcer__" role="alert" style="border: 0px; clip: rect(0px, 0px, 0px, 0px); height: 1px; margin: -1px; overflow: hidden; padding: 0px; position: absolute; top: 0px; width: 1px; white-space: nowrap; overflow-wrap: normal;"></p></next-route-announcer></body>

undefined

  731 |
  732 |   if (hardError) {
> 733 |     throw new Error('TIMED OUT: ' + regex + '\n\n' + content + '\n\n' + lastErr)
      |           ^
  734 |   }
  735 |   return false
  736 | }

  at check (lib/next-test-utils.ts:733:11)
  at Object.<anonymous> (integration/app-document-remove-hmr/test/index.test.js:59:7)

Read more about building and testing Next.js in contributing.md.

pnpm test-dev-turbo test/development/basic/hmr/error-recovery.test.ts (turbopack)

HMR - Error Recovery, nextConfig: { basePath: '', assetPrefix: '' } > should recover after webpack parse error in an imported file
HMR - Error Recovery, nextConfig: { basePath: '', assetPrefix: '/asset-prefix' } > should recover after webpack parse error in an imported file
HMR - Error Recovery, nextConfig: { basePath: '/docs', assetPrefix: '' } > should recover after webpack parse error in an imported file
HMR - Error Recovery, nextConfig: { basePath: '/docs', assetPrefix: '/asset-prefix' } > should recover after webpack parse error in an imported file

Expand output

● HMR - Error Recovery, nextConfig: { basePath: '', assetPrefix: '' } › should recover after webpack parse error in an imported file

Expected Redbox but found none

  519 |         )
  520 |
> 521 |         await assertHasRedbox(browser)
      |         ^
  522 |         expect(await getRedboxHeader(browser)).toMatch('Failed to compile')
  523 |
  524 |         if (process.env.TURBOPACK) {

  at Object.<anonymous> (development/basic/hmr/error-recovery.test.ts:521:9)

● HMR - Error Recovery, nextConfig: { basePath: '', assetPrefix: '/asset-prefix' } › should recover after webpack parse error in an imported file

Expected Redbox but found none

  519 |         )
  520 |
> 521 |         await assertHasRedbox(browser)
      |         ^
  522 |         expect(await getRedboxHeader(browser)).toMatch('Failed to compile')
  523 |
  524 |         if (process.env.TURBOPACK) {

  at Object.<anonymous> (development/basic/hmr/error-recovery.test.ts:521:9)

● HMR - Error Recovery, nextConfig: { basePath: '/docs', assetPrefix: '' } › should recover after webpack parse error in an imported file

Expected Redbox but found none

  519 |         )
  520 |
> 521 |         await assertHasRedbox(browser)
      |         ^
  522 |         expect(await getRedboxHeader(browser)).toMatch('Failed to compile')
  523 |
  524 |         if (process.env.TURBOPACK) {

  at Object.<anonymous> (development/basic/hmr/error-recovery.test.ts:521:9)

● HMR - Error Recovery, nextConfig: { basePath: '/docs', assetPrefix: '/asset-prefix' } › should recover after webpack parse error in an imported file

Expected Redbox but found none

  519 |         )
  520 |
> 521 |         await assertHasRedbox(browser)
      |         ^
  522 |         expect(await getRedboxHeader(browser)).toMatch('Failed to compile')
  523 |
  524 |         if (process.env.TURBOPACK) {

  at Object.<anonymous> (development/basic/hmr/error-recovery.test.ts:521:9)

Read more about building and testing Next.js in contributing.md.

ijjk · 2025-01-18T01:33:21Z

Stats from current PR

Default Build (Increase detected ⚠️)

General

	vercel/next.js canary	vercel/next.js wbinnssmith/endpoint-output-assets	Change
buildDuration	20.8s	17.9s	N/A
buildDurationCached	16.9s	14.3s	N/A
nodeModulesSize	418 MB	418 MB	✓
nextStartRea..uration (ms)	456ms	434ms	N/A

Client Bundles (main, webpack)

	vercel/next.js canary	vercel/next.js wbinnssmith/endpoint-output-assets	Change
5306-HASH.js gzip	54 kB	54 kB	N/A
8276.HASH.js gzip	169 B	168 B	N/A
8377-HASH.js gzip	5.46 kB	5.46 kB	N/A
bccd1874-HASH.js gzip	52.9 kB	52.9 kB	✓
framework-HASH.js gzip	57.5 kB	57.5 kB	N/A
main-app-HASH.js gzip	241 B	242 B	N/A
main-HASH.js gzip	34.4 kB	34.4 kB	N/A
webpack-HASH.js gzip	1.71 kB	1.71 kB	N/A
Overall change	52.9 kB	52.9 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	vercel/next.js wbinnssmith/endpoint-output-assets	Change
polyfills-HASH.js gzip	39.4 kB	39.4 kB	✓
Overall change	39.4 kB	39.4 kB	✓

Client Pages

	vercel/next.js canary	vercel/next.js wbinnssmith/endpoint-output-assets	Change
_app-HASH.js gzip	193 B	193 B	✓
_error-HASH.js gzip	193 B	193 B	✓
amp-HASH.js gzip	512 B	510 B	N/A
css-HASH.js gzip	343 B	342 B	N/A
dynamic-HASH.js gzip	1.84 kB	1.84 kB	✓
edge-ssr-HASH.js gzip	265 B	265 B	✓
head-HASH.js gzip	363 B	362 B	N/A
hooks-HASH.js gzip	393 B	392 B	N/A
image-HASH.js gzip	4.59 kB	4.58 kB	N/A
index-HASH.js gzip	268 B	268 B	✓
link-HASH.js gzip	2.35 kB	2.35 kB	N/A
routerDirect..HASH.js gzip	328 B	328 B	✓
script-HASH.js gzip	397 B	397 B	✓
withRouter-HASH.js gzip	323 B	326 B	N/A
1afbb74e6ecf..834.css gzip	106 B	106 B	✓
Overall change	3.59 kB	3.59 kB	✓

Client Build Manifests

	vercel/next.js canary	vercel/next.js wbinnssmith/endpoint-output-assets	Change
_buildManifest.js gzip	748 B	747 B	N/A
Overall change	0 B	0 B	✓

Rendered Page Sizes

	vercel/next.js canary	vercel/next.js wbinnssmith/endpoint-output-assets	Change
index.html gzip	523 B	523 B	✓
link.html gzip	538 B	538 B	✓
withRouter.html gzip	519 B	520 B	N/A
Overall change	1.06 kB	1.06 kB	✓

Edge SSR bundle Size

	vercel/next.js canary	vercel/next.js wbinnssmith/endpoint-output-assets	Change
edge-ssr.js gzip	129 kB	129 kB	N/A
page.js gzip	208 kB	208 kB	N/A
Overall change	0 B	0 B	✓

Middleware size

	vercel/next.js canary	vercel/next.js wbinnssmith/endpoint-output-assets	Change
middleware-b..fest.js gzip	671 B	664 B	N/A
middleware-r..fest.js gzip	155 B	156 B	N/A
middleware.js gzip	31.3 kB	31.3 kB	N/A
edge-runtime..pack.js gzip	844 B	844 B	✓
Overall change	844 B	844 B	✓

Next Runtimes

	vercel/next.js canary	vercel/next.js wbinnssmith/endpoint-output-assets	Change
274-experime...dev.js gzip	322 B	322 B	✓
274.runtime.dev.js gzip	314 B	314 B	✓
app-page-exp...dev.js gzip	375 kB	375 kB	✓
app-page-exp..prod.js gzip	130 kB	130 kB	✓
app-page-tur..prod.js gzip	143 kB	143 kB	✓
app-page-tur..prod.js gzip	139 kB	139 kB	✓
app-page.run...dev.js gzip	363 kB	363 kB	✓
app-page.run..prod.js gzip	126 kB	126 kB	✓
app-route-ex...dev.js gzip	37.6 kB	37.6 kB	✓
app-route-ex..prod.js gzip	25.6 kB	25.6 kB	✓
app-route-tu..prod.js gzip	25.6 kB	25.6 kB	✓
app-route-tu..prod.js gzip	25.4 kB	25.4 kB	✓
app-route.ru...dev.js gzip	39.2 kB	39.2 kB	✓
app-route.ru..prod.js gzip	25.4 kB	25.4 kB	✓
pages-api-tu..prod.js gzip	9.69 kB	9.69 kB	✓
pages-api.ru...dev.js gzip	11.6 kB	11.6 kB	✓
pages-api.ru..prod.js gzip	9.68 kB	9.68 kB	✓
pages-turbo...prod.js gzip	21.9 kB	21.9 kB	✓
pages.runtim...dev.js gzip	27.7 kB	27.7 kB	✓
pages.runtim..prod.js gzip	21.9 kB	21.9 kB	✓
server.runti..prod.js gzip	916 kB	916 kB	✓
Overall change	2.47 MB	2.47 MB	✓

build cache Overall increase ⚠️

	vercel/next.js canary	vercel/next.js wbinnssmith/endpoint-output-assets	Change
0.pack gzip	2.1 MB	2.1 MB	⚠️ +3.75 kB
index.pack gzip	75.5 kB	75.8 kB	⚠️ +326 B
Overall change	2.17 MB	2.18 MB	⚠️ +4.08 kB

Diff details

Diff for main-HASH.js

Diff too large to display

Commit: 192bdc6

…output()` This: - Makes the Endpoint trait not manage any file writing on its own, instead returning a structure containing OutputAssets, the paths where the OutputAssets can be found, and the associated Project. - Allows for OutputAssets to be written at a future point in Turbopack - Implements `endpoint_write_to_disk` for compatibility in dev. This continues to be used for prod builds, but in the future it won’t. In a future PR, production builds will dedupe and unify all output assets, writing them to disk at once, rather than exposing individual endpoint writes to JS to occur sequentially. Test Plan: CI

ijjk added created-by: Turbopack team PRs by the Turbopack team. Turbopack Related to Turbopack with Next.js. labels Jan 18, 2025

wbinnssmith force-pushed the wbinnssmith/endpoint-output-assets branch from efad310 to b7a60c2 Compare January 20, 2025 19:11

wbinnssmith marked this pull request as ready for review January 20, 2025 19:11

wbinnssmith requested review from sokra and mischnic January 20, 2025 19:11

bgw force-pushed the bgw/operation-strong-resolve branch from 63bd765 to 89fa28f Compare January 20, 2025 22:22

bgw mentioned this pull request Jan 20, 2025

fix(turbopack/napi): Flush optional task cache hit statistics upon completion of build #75122

Merged

wbinnssmith force-pushed the wbinnssmith/endpoint-output-assets branch from b7a60c2 to f36636b Compare January 21, 2025 01:03

bgw force-pushed the bgw/operation-strong-resolve branch from 89fa28f to 57af58e Compare January 21, 2025 01:04

bgw changed the base branch from bgw/operation-strong-resolve to graphite-base/75059 January 21, 2025 16:28

bgw force-pushed the graphite-base/75059 branch from 57af58e to 029a819 Compare January 21, 2025 16:30

bgw force-pushed the wbinnssmith/endpoint-output-assets branch from f36636b to af331d2 Compare January 21, 2025 16:30

bgw changed the base branch from graphite-base/75059 to canary January 21, 2025 16:31

bgw force-pushed the wbinnssmith/endpoint-output-assets branch from af331d2 to 192bdc6 Compare January 21, 2025 16:31

mischnic approved these changes Jan 21, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove `endpoint.write_to_disk()` in favor of standardized `endpoint.output()` #75059

Remove `endpoint.write_to_disk()` in favor of standardized `endpoint.output()` #75059

wbinnssmith commented Jan 18, 2025 •

edited

Loading

wbinnssmith commented Jan 18, 2025 •

edited

Loading

ijjk commented Jan 18, 2025 •

edited

Loading

ijjk commented Jan 18, 2025 •

edited

Loading

Remove endpoint.write_to_disk() in favor of standardized endpoint.output() #75059

Are you sure you want to change the base?

Remove endpoint.write_to_disk() in favor of standardized endpoint.output() #75059

Conversation

wbinnssmith commented Jan 18, 2025 • edited Loading

wbinnssmith commented Jan 18, 2025 • edited Loading

ijjk commented Jan 18, 2025 • edited Loading

Failing test suites

ijjk commented Jan 18, 2025 • edited Loading

Stats from current PR

Remove `endpoint.write_to_disk()` in favor of standardized `endpoint.output()` #75059

Remove `endpoint.write_to_disk()` in favor of standardized `endpoint.output()` #75059

wbinnssmith commented Jan 18, 2025 •

edited

Loading

wbinnssmith commented Jan 18, 2025 •

edited

Loading

ijjk commented Jan 18, 2025 •

edited

Loading

ijjk commented Jan 18, 2025 •

edited

Loading