Repository navigation
Content mapper docs - #65582
Content mapper docs#65582Gabriela Araujo Britto (gabritto) wants to merge 1 commit into
Conversation
Andrew Branch (andrewbranch)
left a comment
There was a problem hiding this comment.
Thank you so much for doing this! I feel so much better about my odds of getting everything done by our release 😄
One additional short section in the implementation doc should be about compatibility:
- The protocol will only change in backward-compatible ways after 7.1 is officially released. New features may be added with an LSP-like capabilities negotiation in
initialize. - For forward compatibility, tuples should be considered open-ended. A mapper should not assert an exact tuple length or read elements beyond ones it knows how to handle.
|
|
||
| ### `transform` | ||
|
|
||
| When TypeScript finds a non-TS file in a project supported by a content mapper, it will send a `transform` request to the content mapper. |
There was a problem hiding this comment.
Structure: I think by the time I've gotten here, I should have a more thorough understanding of how a non-TS file gets into the program. I think this could even go in the intro doc, where you touch on this. By default, the registered extensions get considered in include, and they can be imported. Relative imports require the file extension to be included. (Package.json entrypoints and paths targets can also point to non-TS files, making it incorrect to say that the module specifier must always include the file extension, but I don't think this needs to be spelled out.)
|
|
||
| TypeScript will spawn the content mapper process using the command specified in the `"exec"` option of the content mapper's `package.json`. | ||
| It will then send an `initialize` request containing a list of position encodings that TypeScript supports, and optionally a locale that the mapper should use for its diagnostics. | ||
| The content mapper must respond with one of the supported position encodings that it will use throughout, and also a diagnostic source string that will be used to identify diagnostics emitted by the content mapper. |
There was a problem hiding this comment.
I think it would be helpful to include the TypeScript types describing the request params and result types for each of these messages after the prose.
| Span mappings map between spans (start position + length) in the original file and virtual file. | ||
| TypeScript needs these mappings to emit diagnostics and to power other language server features. | ||
| For instance, in the CLI, these mappings are used to show TypeScript-generated diagnostics in the original, non-TypeScript content. | ||
| A mapping can be of kind `verbatim`, meaning the original and virtual content are the same and have the same length; it can be of kind `atom`, meaning the original and virtual content may have different lengths and contents, and it can be an `alias`, meaning the original and virtual contents may have different lengths and content, but that the content of both is a *name* for the same entity. |
There was a problem hiding this comment.
Formatting: I think this would read well as a bulleted list
| ``` | ||
|
|
||
| Gaps in the span map are treated as fully synthesized content and cannot be mapped to a location in the original text. | ||
| Diagnostics in unmappable regions are not discarded. |
There was a problem hiding this comment.
Content: somewhere, we should clarify that a few kinds of diagnostics are filtered out—currently unused-identifier diagnostics, but I'm proposing to expand the list slightly: #65573
|
|
||
| ``` | ||
|
|
||
| In this example, the span for identifier `active` inside the template, i.e. in `<template><p :class="{ active }">Hello</p></template>`, is mapped to three virtual spans: |
There was a problem hiding this comment.
Structure: I'm wondering if this level of detail should be moved into a later section about language server features. I almost forgot we're not done describing the protocol.
| ```ts | ||
| import { ref } from 'vue' | ||
| const count = ref<number>(0); | ||
| // ... vue-specific definitions ... |
There was a problem hiding this comment.
On one hand, I don't think we should assault the reader with a full code block of realistic Vue virtual content yet, but I do think this first example should show some non-verbatim content and the default export, since you're about to show an example of importing this file.
| Note that if any content mapper you're using in your project specifies `outputExtension`, you'll need to enable `rewriteRelativeImportExtensions` in your project's compiler options. | ||
| For more details, see [the implementation PR](https://github.com/microsoft/TypeScript/pull/64581). |
There was a problem hiding this comment.
| Note that if any content mapper you're using in your project specifies `outputExtension`, you'll need to enable `rewriteRelativeImportExtensions` in your project's compiler options. | |
| For more details, see [the implementation PR](https://github.com/microsoft/TypeScript/pull/64581). | |
| Note that if any content mapper you're using in your project specifies `outputExtensions`, you'll need to enable `rewriteRelativeImportExtensions` in your project's compiler options. | |
| For more details, see [the implementation PR](https://github.com/microsoft/TypeScript/pull/64581). |
But I think this can probably be skipped from the intro content, because you're going to get the same info immediately upon configuration if you need to worry about it, and most users probably won't (I expect outputExtensions to be uncommon).
| @@ -0,0 +1,436 @@ | |||
| # Implementing content mappers | |||
|
|
|||
| A content mapper can be implemented as any kind of program that satisfies the content mapping protocol that TypeScript expects, but it must be provided as a package with a `package.json` specifying how to invoke it: | |||
There was a problem hiding this comment.
I would probably lead with the child process / STDIO nature here.
| A content mapper can be implemented as any kind of program that satisfies the content mapping protocol that TypeScript expects, but it must be provided as a package with a `package.json` specifying how to invoke it: | |
| A content mapper is a child process spawned by the TypeScript compiler that receives and responds to JSON-RPC messages over STDIO. It can be implemented in any language, but must be provided as a Node.js package with a `package.json` specifying how to launch it: |
| Besides activating TypeScript on content mapped files, and providing content mapper support for inferred projects, a companion extension can also provide customized language server features beyond what content mappers enable. | ||
| For instance, consider the following Svelte file: | ||
|
|
||
| ```svelte | ||
| <script lang="ts"> | ||
| let { visible = true }: { visible?: boolean } = $props(); | ||
| </script> | ||
|
|
||
| {#if visible} | ||
| <p>Hello</p> | ||
| {:else} | ||
| <p>Goodbye</p> | ||
| {/if} | ||
| ``` | ||
|
|
||
| While TypeScript can provide completions inside the script blocks, which are content-mapped into TypeScript content, it will not provide completions outside of those, e.g. completions for the closing keyword in `{/if}`. Those have to be provided by the Svelte extension. | ||
|
|
||
| Extensions can also intercept TypeScript language server responses and process them. | ||
| As an example, consider the following Svelte file: | ||
|
|
||
| `HoverChild.svelte`: | ||
| ```svelte | ||
| <p>Hello</p> | ||
| ``` | ||
|
|
||
| When content mapped, the virtual content will look something like this: | ||
|
|
||
| `HoverChild.ts`: | ||
| ```ts | ||
| ... | ||
| class HoverChild__Svelte_Component_ { ... } | ||
| ... | ||
| ``` | ||
|
|
||
| The special Svelte suffix is necessary to avoid name clashes, but now if that class is used somewhere, and the user hovers over it, they would see the generated name instead of the original name. | ||
| To fix this, the Svelte extension processes TypeScript hover responses, and maps those generated names back to user-facing ones (e.g. `HoverChild__Svelte_Component_` -> `HoverChild`). |
There was a problem hiding this comment.
This section invests a good bit into explaining the motivation for doing additional things in a VS Code extension, but doesn't go into how whatsoever. Since those things aren't directly content mapper related, and fleshing out the "how" would extend it quite a bit more, I think this could be shortened to a pretty brief mention of why you might want to stand up an additional LSP server and the existence of middleware registration.
|
|
||
| <!-- TODO: I also have a prototype of a VS Code extension that lets you see the virtual content and span mappings, which I’ll share after this PR is merged. --> | ||
|
|
||
| ## Editor extension |
There was a problem hiding this comment.
One thing we should add in here is contributing a JSON schema for mapper options that will augment the tsconfig.json schema. Example here: andrewbranch/language-tools@76d07d9
| } | ||
| ``` | ||
|
|
||
| The package.json of the content mapper package must specify a `typescript` top-level field with a nested contentMapper field describing how to spawn the content mapper process and what compiler options its transform requires. |
There was a problem hiding this comment.
| The package.json of the content mapper package must specify a `typescript` top-level field with a nested contentMapper field describing how to spawn the content mapper process and what compiler options its transform requires. | |
| The `package.json` of the content mapper package must specify a `typescript` top-level field with a nested `contentMapper` field describing how to spawn the content mapper process and what compiler options its transform requires. |
| A mapper that reads additional project-specific configuration from external sources beyond those compiler options should additionally declare `dynamicConfig: true`. | ||
|
|
||
| For example, if the Vue content mapper's `tsconfig.json` options set `target: "auto"`, it reads the nearest `node_modules/vue/package.json` to determine the Vue version. | ||
| Changing its `version` from `3.5.0` to `3.6.0` can affect the virtual content produced by the content mapper for a Vue file, so the Vue content mapper's manifesto needs to specify `dynamicConfig: true`. |
There was a problem hiding this comment.
| Changing its `version` from `3.5.0` to `3.6.0` can affect the virtual content produced by the content mapper for a Vue file, so the Vue content mapper's manifesto needs to specify `dynamicConfig: true`. | |
| Changing its `version` from `3.5.0` to `3.6.0` can affect the virtual content produced by the content mapper for a Vue file, so the Vue content mapper's manifest needs to specify `dynamicConfig: true`. |
There was a problem hiding this comment.
I just find this section a little bit confusing - where are we seeing 3.5.0 and 3.6.0?
| We'll call that configuration the content mapper's **manifest**. | ||
| A mapper that reads additional project-specific configuration from external sources beyond those compiler options should additionally declare `dynamicConfig: true`. | ||
|
|
||
| For example, if the Vue content mapper's `tsconfig.json` options set `target: "auto"`, it reads the nearest `node_modules/vue/package.json` to determine the Vue version. |
There was a problem hiding this comment.
What do you mean by target: auto? I don't think that exists? Do you mean target: null?
| In addition to the options above, a content mapper manifest can optionally specify a mapping of output extensions. | ||
| This is useful when the original, content mapped files supported by the content mapper will be emited into different files by another tool. | ||
| For instance, when writing Ember components as `.gts` files, we expect an Ember build tool to emit `.js` files for `.gts` files, so an Ember content mapper's manifest would look like this: | ||
|
|
||
| ```jsonc | ||
| { | ||
| "name": "ember-content-mapper", | ||
| "typescript": { | ||
| "contentMapper": { | ||
| "exec": ["node", "server.js"], | ||
| "outputExtensions": { | ||
| ".gts": ".js" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
| This allows TypeScript to produce the right declaration files for a `.gts` file (i.e. `Component.gts` -> `Component.d.ts` instead of `Component.d.gts.ts`). |
There was a problem hiding this comment.
It's not critical, but I personally think this might read better with a flow like
- You can also have explicit output extensions
- What it looks like (example
jsoncfile withoutputExtensions) - Rationale of why this is important.
| Changing its `version` from `3.5.0` to `3.6.0` can affect the virtual content produced by the content mapper for a Vue file, so the Vue content mapper's manifesto needs to specify `dynamicConfig: true`. | ||
|
|
||
| In addition to the options above, a content mapper manifest can optionally specify a mapping of output extensions. | ||
| This is useful when the original, content mapped files supported by the content mapper will be emited into different files by another tool. |
There was a problem hiding this comment.
| This is useful when the original, content mapped files supported by the content mapper will be emited into different files by another tool. | |
| This is useful when the original content mapped files will be emitted to a different format, rather than left as-is. |
| } | ||
| } | ||
| ``` | ||
| This allows TypeScript to produce the right declaration files for a `.gts` file (i.e. `Component.gts` -> `Component.d.ts` instead of `Component.d.gts.ts`). |
There was a problem hiding this comment.
| This allows TypeScript to produce the right declaration files for a `.gts` file (i.e. `Component.gts` -> `Component.d.ts` instead of `Component.d.gts.ts`). | |
| In the example above, the manifest indicates that `.gts` files will be emitted to `.js` files. | |
| With this information, TypeScript emits declaration files with the correct file extensions (i.e. `Component.gts` -> `Component.d.ts` instead of `Component.d.gts.ts`). |
I'm not sure if this is true, but might be worth adding:
| This allows TypeScript to produce the right declaration files for a `.gts` file (i.e. `Component.gts` -> `Component.d.ts` instead of `Component.d.gts.ts`). | |
| In the example above, the manifest indicates that `.gts` files will be emitted to `.js` files. | |
| With this information, TypeScript emits declaration files with the correct file extensions (i.e. `Component.gts` -> `Component.d.ts` instead of `Component.d.gts.ts`), and also permits `.js` import paths to resolve to `.gts` files as well. |
|
|
||
| A content mapper communicates with TypeScript using JSON-RPC over STDIO. | ||
| A JSON schema of the content mapper protocol types can be found in the TypeScript npm package, under `/schemas/contentmapper.schema.json`. | ||
| TypeScript will send requests to the content mapper; the content mapper should not send requests or notifications to TypeScript, only responses. |
There was a problem hiding this comment.
| TypeScript will send requests to the content mapper; the content mapper should not send requests or notifications to TypeScript, only responses. | |
| TypeScript will spawn content mappers and send requests as needed. | |
| Content mapper must not send requests or notifications to TypeScript; only responses. |
| TypeScript will spawn the content mapper process using the command specified in the `"exec"` option of the content mapper's `package.json`. | ||
| It will then send an `initialize` request containing a list of position encodings that TypeScript supports, and optionally a locale that the mapper should use for its diagnostics. | ||
| The content mapper must respond with one of the supported position encodings that it will use throughout, and also a diagnostic source string that will be used to identify diagnostics emitted by the content mapper. | ||
| The diagnostic source must not be `typescript`, `tsc`, or any other extension supported by TypeScript (`ts`, `tsx`, `json, `js`, etc), so that the content mapper diagnostics are not confused with regular TypeScript diagnostics. |
There was a problem hiding this comment.
| The diagnostic source must not be `typescript`, `tsc`, or any other extension supported by TypeScript (`ts`, `tsx`, `json, `js`, etc), so that the content mapper diagnostics are not confused with regular TypeScript diagnostics. | |
| The diagnostic source must not be `typescript`, `tsc`, or any other extension supported by TypeScript (`ts`, `tsx`, `json`, `js`, etc), so that the content mapper diagnostics are not confused with regular TypeScript diagnostics. |
| The content mapper must respond with one of the supported position encodings that it will use throughout, and also a diagnostic source string that will be used to identify diagnostics emitted by the content mapper. | ||
| The diagnostic source must not be `typescript`, `tsc`, or any other extension supported by TypeScript (`ts`, `tsx`, `json, `js`, etc), so that the content mapper diagnostics are not confused with regular TypeScript diagnostics. | ||
|
|
||
| After initialization, the content mapper should be ready to receive the other kinds of requests. |
There was a problem hiding this comment.
| After initialization, the content mapper should be ready to receive the other kinds of requests. | |
| After initialization, the content mapper must be ready to receive the other kinds of requests. |
https://www.rfc-editor.org/info/rfc2119/ - while we don't need to use the UPPERCASE versions of these, I think when describing a protocol we can SHOULD be more intentional here.
|
|
||
| The first time TypeScript needs to transform a file in a project, it will send an `openProject` request containing a project handle that uniquely identifies the project, the project's relevant compiler options (as specified in the content mapper's manifest), the content mapper options specified in the project's `tsconfig.json`, and optionally a path to a tsconfig for that project, if any exists. | ||
|
|
||
| The content mapper should respond with diagnostics for the project's content mapper options, if any exist. |
There was a problem hiding this comment.
Is a response required? Or only if there are diagnostics? I would have guessed that the content mapper should always respond, and should respond with any diagnostics if they exist.
| Additionally, if the content mapper specified `dynamicConfig: true` in its manifest, the content mapper's response must also contain a `configIdentity`: a stable fingerprint of all *dynamically discovered configuration* that can affect transforms for the project. | ||
| If `dynamicConfig: true`, the content mapper's response may also contain a list of `watchedFiles`, which are the additional files that, if modified, may change the project's `configIdentity` and affect the content mapper's transform. | ||
| In that case, whenever a file in the list of `watchedFiles` changes, TypeScript will send a request to close the project (see [`closeProject`](#closeproject)). | ||
| TypeScript may then call `openProject` for the same project again when needed; if indeed the watched file changes could affect the content mapper's transform behavior, then that mapper should return a different `configIdentity` for this new `openProject` request. |
There was a problem hiding this comment.
| TypeScript may then call `openProject` for the same project again when needed; if indeed the watched file changes could affect the content mapper's transform behavior, then that mapper should return a different `configIdentity` for this new `openProject` request. | |
| TypeScript may then call `openProject` for the same project again when needed. | |
| If a change to a watched file could affect the content mapper's transform behavior, then that mapper should return a different `configIdentity` for this new `openProject` request. |
| In that case, whenever a file in the list of `watchedFiles` changes, TypeScript will send a request to close the project (see [`closeProject`](#closeproject)). | ||
| TypeScript may then call `openProject` for the same project again when needed; if indeed the watched file changes could affect the content mapper's transform behavior, then that mapper should return a different `configIdentity` for this new `openProject` request. | ||
|
|
||
| Note that the compiler options will only be sent by TypeScript as part of the `openProject` request, so the content mapper should store that information associated with that particular project, and use it during transforms. |
There was a problem hiding this comment.
| Note that the compiler options will only be sent by TypeScript as part of the `openProject` request, so the content mapper should store that information associated with that particular project, and use it during transforms. | |
| Note that the compiler options will only be sent by TypeScript as part of the `openProject` request. | |
| Content mappers should store information associated with each opened project using the project handle if they require the use of compiler options later on. |
| TypeScript may then call `openProject` for the same project again when needed; if indeed the watched file changes could affect the content mapper's transform behavior, then that mapper should return a different `configIdentity` for this new `openProject` request. | ||
|
|
||
| Note that the compiler options will only be sent by TypeScript as part of the `openProject` request, so the content mapper should store that information associated with that particular project, and use it during transforms. | ||
| Also note that a single content mapper process may handle multiple projects, because TypeScript tries to reuse a content mapper process for any project that matches that content mapper. |
There was a problem hiding this comment.
Maybe a better way to rephrase this paragraph is
Note that TypeScript may try to reuse a single content mapper process across multiple projects.
This means that content mapper processes must be able to handle multiple projects.
While this might mean very little for simple content mappers, this is important in cases where TypeScript compiler options affect the result of [`transform`](#transform) responses.
Because compiler options are only sent during the `openProject` request, content mappers should store this information for later, and may do so using this request's project handle.
No description provided.