Merge #7536

7536: Make architecture more informative r=matklad a=matklad

bors r+
🤖

Co-authored-by: Aleksey Kladov <[email protected]>

2 files changed, 317 insertions, 201 deletions

diff --git a/docs/dev/README.md b/docs/dev/README.md
index 4cc608b07..9c0af68e3 100644
--- a/docs/dev/README.md
+++ b/docs/dev/README.md
@@ -9,8 +9,9 @@ $ cargo test
 
 should be enough to get you started!
 
-To learn more about how rust-analyzer works, see
-[./architecture.md](./architecture.md) document.
+To learn more about how rust-analyzer works, see [./architecture.md](./architecture.md) document.
+It also explains the high-level layout of the source code.
+Do skim through that document.
 
 We also publish rustdoc docs to pages:
 
@@ -99,25 +100,6 @@ I don't have a specific workflow for this case.
 Additionally, I use `cargo run --release -p rust-analyzer -- analysis-stats path/to/some/rust/crate` to run a batch analysis.
 This is primarily useful for performance optimizations, or for bug minimization.
 
-## Parser Tests
-
-Tests for the parser (`parser`) live in the `syntax` crate (see `test_data` directory).
-There are two kinds of tests:
-
-* Manually written test cases in `parser/ok` and `parser/err`
-* "Inline" tests in `parser/inline` (these are generated) from comments in `parser` crate.
-
-The purpose of inline tests is not to achieve full coverage by test cases, but to explain to the reader of the code what each particular `if` and `match` is responsible for.
-If you are tempted to add a large inline test, it might be a good idea to leave only the simplest example in place, and move the test to a manual `parser/ok` test.
-
-To update test data, run with `UPDATE_EXPECT` variable:
-
-```bash
-env UPDATE_EXPECT=1 cargo qt
-```
-
-After adding a new inline test you need to run `cargo xtest codegen` and also update the test data as described above.
-
 ## TypeScript Tests
 
 If you change files under `editors/code` and would like to run the tests and linter, install npm and run:
@@ -128,73 +110,6 @@ npm ci
 npm run lint
 ```
 
-# Code organization
-
-All Rust code lives in the `crates` top-level directory, and is organized as a single Cargo workspace.
-The `editors` top-level directory contains code for integrating with editors.
-Currently, it contains the plugin for VS Code (in TypeScript).
-The `docs` top-level directory contains both developer and user documentation.
-
-We have some automation infra in Rust in the `xtask` package.
-It contains stuff like formatting checking, code generation and powers `cargo xtask install`.
-The latter syntax is achieved with the help of cargo aliases (see `.cargo` directory).
-
-# Architecture Invariants
-
-This section tries to document high-level design constraints, which are not
-always obvious from the low-level code.
-
-## Incomplete syntax trees
-
-Syntax trees are by design incomplete and do not enforce well-formedness.
-If an AST method returns an `Option`, it *can* be `None` at runtime, even if this is forbidden by the grammar.
-
-## LSP independence
-
-rust-analyzer is independent from LSP.
-It provides features for a hypothetical perfect Rust-specific IDE client.
-Internal representations are lowered to LSP in the `rust-analyzer` crate (the only crate which is allowed to use LSP types).
-
-## IDE/Compiler split
-
-There's a semi-hard split between "compiler" and "IDE", at the `hir` crate.
-Compiler derives new facts about source code.
-It explicitly acknowledges that not all info is available (i.e. you can't look at types during name resolution).
-
-IDE assumes that all information is available at all times.
-
-IDE should use only types from `hir`, and should not depend on the underling compiler types.
-`hir` is a facade.
-
-## IDE API
-
-The main IDE crate (`ide`) uses "Plain Old Data" for the API.
-Rather than talking in definitions and references, it talks in Strings and textual offsets.
-In general, API is centered around UI concerns -- the result of the call is what the user sees in the editor, and not what the compiler sees underneath.
-The results are 100% Rust specific though.
-Shout outs to LSP developers for popularizing the idea that "UI" is a good place to draw a boundary at.
-
-## LSP is stateless
-
-The protocol is implemented in the mostly stateless way.
-A good mental model is HTTP, which doesn't store per-client state, and instead relies on devices like cookies to maintain an illusion of state.
-If some action requires multi-step protocol, each step should be self-contained.
-
-A good example here is code action resolving process.
-TO display the lightbulb, we compute the list of code actions without computing edits.
-Figuring out the edit is done in a separate `codeAction/resolve` call.
-Rather than storing some `lazy_edit: Box<dyn FnOnce() -> Edit>` somewhere, we use a string ID of action to re-compute the list of actions during the resolve process.
-(See [this post](https://rust-analyzer.github.io/blog/2020/09/28/how-to-make-a-light-bulb.html) for more details.)
-The benefit here is that, generally speaking, the state of the world might change between `codeAction` and `codeAction` resolve requests, so any closure we store might become invalid.
-
-While we don't currently implement any complicated refactors with complex GUI, I imagine we'd use the same techniques for refactors.
-After clicking each "Next" button during refactor, the client would send all the info which server needs to re-recreate the context from scratch.
-
-## CI
-
-CI does not test rust-analyzer, CI is a core part of rust-analyzer, and is maintained with above average standard of quality.
-CI is reproducible -- it can only be broken by changes to files in this repository, any dependence on externalities is a bug.
-
 # Code Style & Review Process
 
 Do see [./style.md](./style.md).
diff --git a/docs/dev/architecture.md b/docs/dev/architecture.md
index b5831f47c..feda20dd7 100644
--- a/docs/dev/architecture.md
+++ b/docs/dev/architecture.md
@@ -1,174 +1,375 @@
 # Architecture
 
 This document describes the high-level architecture of rust-analyzer.
-If you want to familiarize yourself with the code base, you are just
-in the right place!
+If you want to familiarize yourself with the code base, you are just in the right place!
 
-See also the [guide](./guide.md), which walks through a particular snapshot of
-rust-analyzer code base.
+See also the [guide](./guide.md), which walks through a particular snapshot of rust-analyzer code base.
 
-Yet another resource is this playlist with videos about various parts of the
-analyzer:
+Yet another resource is this playlist with videos about various parts of the analyzer:
 
 https://www.youtube.com/playlist?list=PL85XCvVPmGQho7MZkdW-wtPtuJcFpzycE
 
-Note that the guide and videos are pretty dated, this document should be in
-generally fresher.
+Note that the guide and videos are pretty dated, this document should be in generally fresher.
 
-## The Big Picture
+See also this implementation-oriented blog posts:
+
+* https://rust-analyzer.github.io/blog/2019/11/13/find-usages.html
+* https://rust-analyzer.github.io/blog/2020/07/20/three-architectures-for-responsive-ide.html
+* https://rust-analyzer.github.io/blog/2020/09/16/challeging-LR-parsing.html
+* https://rust-analyzer.github.io/blog/2020/09/28/how-to-make-a-light-bulb.html
+* https://rust-analyzer.github.io/blog/2020/10/24/introducing-ungrammar.html
+
+## Bird's Eye View
 
 ![](https://user-images.githubusercontent.com/1711539/50114578-e8a34280-0255-11e9-902c-7cfc70747966.png)
 
-On the highest level, rust-analyzer is a thing which accepts input source code
-from the client and produces a structured semantic model of the code.
+On the highest level, rust-analyzer is a thing which accepts input source code from the client and produces a structured semantic model of the code.
+
+More specifically, input data consists of a set of test files (`(PathBuf, String)` pairs) and information about project structure, captured in the so called `CrateGraph`.
+The crate graph specifies which files are crate roots, which cfg flags are specified for each crate and what dependencies exist between the crates.
+This the input (ground) state.
+The analyzer keeps all this input data in memory and never does any IO.
+Because the input data are source code, which typically measures in tens of megabytes at most, keeping everything in memory is OK.
+
+A "structured semantic model" is basically an object-oriented representation of modules, functions and types which appear in the source code.
+This representation is fully "resolved": all expressions have types, all references are bound to declarations, etc.
+This is derived state.
+
+The client can submit a small delta of input data (typically, a change to a single file) and get a fresh code model which accounts for changes.
 
-More specifically, input data consists of a set of test files (`(PathBuf,
-String)` pairs) and information about project structure, captured in the so
-called `CrateGraph`. The crate graph specifies which files are crate roots,
-which cfg flags are specified for each crate and what dependencies exist between
-the crates. The analyzer keeps all this input data in memory and never does any
-IO. Because the input data are source code, which typically measures in tens of
-megabytes at most, keeping everything in memory is OK.
+The underlying engine makes sure that model is computed lazily (on-demand) and can be quickly updated for small modifications.
 
-A "structured semantic model" is basically an object-oriented representation of
-modules, functions and types which appear in the source code. This representation
-is fully "resolved": all expressions have types, all references are bound to
-declarations, etc.
 
-The client can submit a small delta of input data (typically, a change to a
-single file) and get a fresh code model which accounts for changes.
+## Code Map
 
-The underlying engine makes sure that model is computed lazily (on-demand) and
-can be quickly updated for small modifications.
+This section talks briefly about various important directories an data structures.
+Pay attention to the **Architecture Invariant** sections.
+They often talk about things which are deliberately absent in the source code.
 
+Note also which crates are **API Boundaries**.
+Remember, [rules at the boundary are different](https://www.tedinski.com/2018/02/06/system-boundaries.html).
 
-## Code generation
+### `xtask`
 
-Some of the components of this repository are generated through automatic
-processes. `cargo xtask codegen` runs all generation tasks. Generated code is
-committed to the git repository.
+This is rust-analyzer's "build system".
+We use cargo to compile rust code, but there are also various other tasks, like release management or local installation.
+They are handled by Rust code in the xtask directory.
 
-In particular, `cargo xtask codegen` generates:
+### `editors/code`
 
-1. [`syntax_kind/generated`](https://github.com/rust-analyzer/rust-analyzer/blob/a0be39296d2925972cacd9fbf8b5fb258fad6947/crates/ra_parser/src/syntax_kind/generated.rs)
-  -- the set of terminals and non-terminals of rust grammar.
+VS Code plugin.
 
-2. [`ast/generated`](https://github.com/rust-analyzer/rust-analyzer/blob/a0be39296d2925972cacd9fbf8b5fb258fad6947/crates/ra_syntax/src/ast/generated.rs)
-  -- AST data structure.
+### `libs/`
 
-3. [`doc_tests/generated`](https://github.com/rust-analyzer/rust-analyzer/blob/a0be39296d2925972cacd9fbf8b5fb258fad6947/crates/assists/src/doc_tests/generated.rs),
-  [`test_data/parser/inline`](https://github.com/rust-analyzer/rust-analyzer/tree/a0be39296d2925972cacd9fbf8b5fb258fad6947/crates/ra_syntax/test_data/parser/inline)
-  -- tests for assists and the parser.
+rust-analyzer independent libraries which we publish to crates.io.
+It not heavily utilized at the moment.
 
-The source for 1 and 2 is in [`ast_src.rs`](https://github.com/rust-analyzer/rust-analyzer/blob/a0be39296d2925972cacd9fbf8b5fb258fad6947/xtask/src/ast_src.rs).
+### `crates/parser`
 
-## Code Walk-Through
+It is a hand-written recursive descent parser, which produces a sequence of events like "start node X", "finish node Y".
+It works similarly to
+[kotlin's parser](https://github.com/JetBrains/kotlin/blob/4d951de616b20feca92f3e9cc9679b2de9e65195/compiler/frontend/src/org/jetbrains/kotlin/parsing/KotlinParsing.java),
+which is a good source of inspiration for dealing with syntax errors and incomplete input.
+Original [libsyntax parser](https://github.com/rust-lang/rust/blob/6b99adeb11313197f409b4f7c4083c2ceca8a4fe/src/libsyntax/parse/parser.rs) is what we use for the definition of the Rust language.
+`TreeSink` and `TokenSource` traits bridge the tree-agnostic parser from `grammar` with `rowan` trees.
 
-### `crates/ra_syntax`, `crates/parser`
+**Architecture Invariant:** the parser is independent of the particular tree structure and particular representation of the tokens.
+It transforms one flat stream of events into another flat stream of events.
+Token independence allows us to pares out both text-based source code and `tt`-based macro input.
+Tree independence allows us to more easily vary the syntax tree implementation.
+It should also unlock efficient light-parsing approaches.
+For example, you can extract the set of names defined in a file (for typo correction) without building a syntax tree.
 
-Rust syntax tree structure and parser. See
-[RFC](https://github.com/rust-lang/rfcs/pull/2256) and [./syntax.md](./syntax.md) for some design notes.
+**Architecture Invariant:** parsing never fails, the parser produces `(T, Vec<Error>)` rather than `Result<T, Error>`.
+
+### `crates/syntax`
+
+Rust syntax tree structure and parser.
+See [RFC](https://github.com/rust-lang/rfcs/pull/2256) and [./syntax.md](./syntax.md) for some design notes.
 
 - [rowan](https://github.com/rust-analyzer/rowan) library is used for constructing syntax trees.
-- `grammar` module is the actual parser. It is a hand-written recursive descent parser, which
-  produces a sequence of events like "start node X", "finish node Y". It works similarly to [kotlin's parser](https://github.com/JetBrains/kotlin/blob/4d951de616b20feca92f3e9cc9679b2de9e65195/compiler/frontend/src/org/jetbrains/kotlin/parsing/KotlinParsing.java),
-  which is a good source of inspiration for dealing with syntax errors and incomplete input. Original [libsyntax parser](https://github.com/rust-lang/rust/blob/6b99adeb11313197f409b4f7c4083c2ceca8a4fe/src/libsyntax/parse/parser.rs)
-  is what we use for the definition of the Rust language.
-- `TreeSink` and `TokenSource` traits bridge the tree-agnostic parser from `grammar` with `rowan` trees.
 - `ast` provides a type safe API on top of the raw `rowan` tree.
-- `ast_src` description of the grammar, which is used to generate `syntax_kinds`
-  and `ast` modules, using `cargo xtask codegen` command.
+- `ungrammar` description of the grammar, which is used to generate `syntax_kinds` and `ast` modules, using `cargo xtask codegen` command.
+
+Tests for ra_syntax are mostly data-driven.
+`test_data/parser` contains subdirectories with a bunch of `.rs` (test vectors) and `.txt` files with corresponding syntax trees.
+During testing, we check `.rs` against `.txt`.
+If the `.txt` file is missing, it is created (this is how you update tests).
+Additionally, running `cargo xtask codegen` will walk the grammar module and collect all `// test test_name` comments into files inside `test_data/parser/inline` directory.
+
+To update test data, run with `UPDATE_EXPECT` variable:
 
-Tests for ra_syntax are mostly data-driven: `test_data/parser` contains subdirectories with a bunch of `.rs`
-(test vectors) and `.txt` files with corresponding syntax trees. During testing, we check
-`.rs` against `.txt`. If the `.txt` file is missing, it is created (this is how you update
-tests). Additionally, running `cargo xtask codegen` will walk the grammar module and collect
-all `// test test_name` comments into files inside `test_data/parser/inline` directory.
+```bash
+env UPDATE_EXPECT=1 cargo qt
+```
 
-Note
-[`api_walkthrough`](https://github.com/rust-analyzer/rust-analyzer/blob/2fb6af89eb794f775de60b82afe56b6f986c2a40/crates/ra_syntax/src/lib.rs#L190-L348)
+After adding a new inline test you need to run `cargo xtest codegen` and also update the test data as described above.
+
+Note  [`api_walkthrough`](https://github.com/rust-analyzer/rust-analyzer/blob/2fb6af89eb794f775de60b82afe56b6f986c2a40/crates/ra_syntax/src/lib.rs#L190-L348)
 in particular: it shows off various methods of working with syntax tree.
 
-See [#93](https://github.com/rust-analyzer/rust-analyzer/pull/93) for an example PR which
-fixes a bug in the grammar.
+See [#93](https://github.com/rust-analyzer/rust-analyzer/pull/93) for an example PR which fixes a bug in the grammar.
+
+**Architecture Invariant:** `syntax` crate is completely independent from the rest of rust-analyzer, it knows nothing about salsa or LSP.
+This is important because it is possible to useful tooling using only syntax tree.
+Without semantic information, you don't need to be able to _build_ code, which makes the tooling more robust.
+See also https://web.stanford.edu/~mlfbrown/paper.pdf.
+You can view the `syntax` crate as an entry point to rust-analyzer.
+`sytax` crate is an **API Boundary**.
+
+**Architecture Invariant:** syntax tree is a value type.
+The tree is fully determined by the contents of its syntax nodes, it doesn't need global context (like an interner) and doesn't store semantic info.
+Using the tree as a store for semantic info is convenient in traditional compilers, but doesn't work nicely in the IDE.
+Specifically, assists and refactors require transforming syntax trees, and that becomes awkward if you need to do something with the semantic info.
+
+**Architecture Invariant:** syntax tree is build for a single file.
+This is to enable parallel parsing of all files.
+
+**Architecture Invariant:**  Syntax trees are by design incomplete and do not enforce well-formedness.
+If an AST method returns an `Option`, it *can* be `None` at runtime, even if this is forbidden by the grammar.
 
 ### `crates/base_db`
 
-We use the [salsa](https://github.com/salsa-rs/salsa) crate for incremental and
-on-demand computation. Roughly, you can think of salsa as a key-value store, but
-it also can compute derived values using specified functions. The `base_db` crate
-provides basic infrastructure for interacting with salsa. Crucially, it
-defines most of the "input" queries: facts supplied by the client of the
-analyzer. Reading the docs of the `base_db::input` module should be useful:
-everything else is strictly derived from those inputs.
+We use the [salsa](https://github.com/salsa-rs/salsa) crate for incremental and on-demand computation.
+Roughly, you can think of salsa as a key-value store, but it also can compute derived values using specified functions. The `base_db` crate provides basic infrastructure for interacting with salsa.
+Crucially, it defines most of the "input" queries: facts supplied by the client of the analyzer.
+Reading the docs of the `base_db::input` module should be useful: everything else is strictly derived from those inputs.
+
+**Architecture Invariant:** particularities of the build system are *not* the part of the ground state.
+In particular, `base_db` knows nothing about cargo.
+The `CrateGraph` structure is used to represent the dependencies between the crates abstractly.
+
+**Architecture Invariant:** `base_db` doesn't know about file system and file paths.
+Files are represented with opaque `FileId`, there's no operation to get an `std::path::Path` out of the `FileId`.
+
+### `crates/hir_expand`, `crates/hir_def`, `crates/hir_ty`
+
+These crates are the *brain* of rust-analyzer.
+This is the compiler part of the IDE.
 
-### `crates/hir*` crates
+`hir_xxx` crates have a strong ECS flavor, in that they work with raw ids and directly query the database.
+There's little abstraction here.
+These crates integrate deeply with salsa and chalk.
 
-HIR provides high-level "object oriented" access to Rust code.
+Name resolution, macro expansion and type inference all happen here.
+These crates also define various intermediate representations of the core.
 
-The principal difference between HIR and syntax trees is that HIR is bound to a
-particular crate instance. That is, it has cfg flags and features applied. So,
-the relation between syntax and HIR is many-to-one. The `source_binder` module
-is responsible for guessing a HIR for a particular source position.
+`ItemTree` condenses a single `SyntaxTree` into a "summary" data structure, which is stable over modifications to function bodies.
 
-Underneath, HIR works on top of salsa, using a `HirDatabase` trait.
+`DefMap` contains the module tree of a crate and stores module scopes.
 
-`hir_xxx` crates have a strong ECS flavor, in that they work with raw ids and
-directly query the database.
+`Body` stores information about expressions.
 
-The top-level `hir` façade crate wraps ids into a more OO-flavored API.
+**Architecture Invariant:** this crates are not, and will never be, an api boundary.
+
+**Architecture Invariant:** these creates explicitly care about being incremental.
+The core invariant we maintain is "typing inside a function's body never invalidates global derived data".
+Ie, if you change body of `foo`, all facts about `bar` should remain intact.
+
+**Architecture Invariant:** hir exists only in context of particular crate instance with specific CFG flags.
+The same syntax may produce several instances of HIR if the crate participates in the crate graph more than once.
+
+### `crates/hir`
+
+The top-level `hir` crate is an **API Boundary**.
+If you think about "using rust-analyzer as a library", `hir` crate is most likely the façade you'll be talking to.
+
+It wraps ECS-style internal API into a more OO-flavored API (with an extra `db` argument for each call).
+
+**Architecture Invariant:** `hir` provides a static, fully resolved view of the code.
+While internal `hir_*` crates _compute_ things, `hir`, from the outside, looks like an inert data structure.
+
+`hir` also handles the delicate task of going from syntax to the corresponding `hir`.
+Remember that the mapping here is one-to-many.
+See `Semantics` type and `source_to_def` module.
+
+Note in particular a curious recursive structure in `source_to_def`.
+We first resolve the parent _syntax_ node to the parent _hir_ element.
+Then we ask the _hir_ parent what _syntax_ children does it have.
+Then we look for our node in the set of children.
+
+This is the heart of many IDE features, like goto definition, which start with figuring out a hir node at the cursor.
+This is some kind of (yet unnamed) uber-IDE pattern, as it is present in Roslyn and Kotlin as well.
 
 ### `crates/ide`
 
-A stateful library for analyzing many Rust files as they change. `AnalysisHost`
-is a mutable entity (clojure's atom) which holds the current state, incorporates
-changes and hands out `Analysis` --- an immutable and consistent snapshot of
-the world state at a point in time, which actually powers analysis.
+The `ide` crate build's on top of `hir` semantic model to provide high-level IDE features like completion or goto definition.
+It is an **API Boundary**.
+If you want to use IDE parts of rust-analyzer via LSP, custom flatbuffers-based protocol or just as a library in your text editor, this is the right API.
 
-One interesting aspect of analysis is its support for cancellation. When a
-change is applied to `AnalysisHost`, first all currently active snapshots are
-canceled. Only after all snapshots are dropped the change actually affects the
-database.
+**Architecture Invariant:** `ide` crate's API is build out of POD types with public fields.
+The API uses editor's terminology, it talks about offsets and string labels rathe than in terms of definitions or types.
+It is effectively the view in MVC and viewmodel in [MVVM](https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93viewmodel).
+All arguments and return types are conceptually serializable.
+In particular, syntax tress and and hir types are generally absent from the API (but are used heavily in the implementation).
+Shout outs to LSP developers for popularizing the idea that "UI" is a good place to draw a boundary at.
 
-APIs in this crate are IDE centric: they take text offsets as input and produce
-offsets and strings as output. This works on top of rich code model powered by
-`hir`.
+`ide` is also the first crate which has the notion of change over time.
+`AnalysisHost` is a state to which you can transactonally `apply_change`.
+`Analysis` is an immutable snapshot of the state.
+
+Internally, `ide` is split across several crates. `ide_assists`, `ide_completion` and `ide_ssr` implement large isolated features.
+`ide_db` implements common IDE functionality (notably, reference search is implemented here).
+The `ide` contains a public API/façade, as well as implementation for a plethora of smaller features.
+
+**Architecture Invariant:** `ide` crate strives to provide a _perfect_ API.
+Although at the moment it has only one consumer, the LSP server, LSP *does not* influence it's API design.
+Instead, we keep in mind a hypothetical _ideal_ client -- an IDE tailored specifically for rust, every nook and cranny of which is packed with Rust-specific goodies.
 
 ### `crates/rust-analyzer`
 
-An LSP implementation which wraps `ide` into a language server protocol.
+This crate defines the `rust-analyzer` binary, so it is the **entry point**.
+It implements the language server.
+
+**Architecture Invariant:** `rust-analyzer` is the only crate that knows about LSP and JSON serialization.
+If you want to expose a datastructure `X` from ide to LSP, don't make it serializable.
+Instead, create a serializable counterpart in `rust-analyzer` crate and manually convert between the two.
+
+`GlobalState` is the state of the server.
+The `main_loop` defines the server event loop which accepts requests and sends responses.
+Requests that modify the state or might block user's typing are handled on the main thread.
+All other requests are processed in background.
+
+**Architecture Invariant:** the server is stateless, a-la HTTP.
+Sometimes state needs to be preserved between requests.
+For example, "what is the `edit` for the fifth's completion item of the last completion edit?".
+For this, the second request should include enough info to re-create the context from scratch.
+This generally means including all the parameters of the original request.
+
+`reload` module contains the code that handles configuration and Cargo.toml changes.
+This is a tricky business.
+
+**Architecture Invariant:** `rust-analyzer` should be partially available even when the build is broken.
+Reloading process should not prevent IDE features from working.
+
+### `crates/toolchain`, `crates/project_model`, `crates/flycheck`
 
-### `crates/vfs`
+These crates deal with invoking `cargo` to learn about project structure and get compiler errors for the "check on save" feature.
 
-Although `hir` and `ide` don't do any IO, we need to be able to read
-files from disk at the end of the day. This is what `vfs` does. It also
-manages overlays: "dirty" files in the editor, whose "true" contents is
-different from data on disk. 
+They use `crates/path` heavily instead of `std::path`.
+A single `rust-analyzer` process can serve many projects, so it is important that server's current directory does not leak.
 
-## Testing Infrastructure
+### `crates/mbe`, `crated/tt`, `crates/proc_macro_api`, `crates/proc_macro_srv`
 
-Rust Analyzer has three interesting [systems
-boundaries](https://www.tedinski.com/2018/04/10/making-tests-a-positive-influence-on-design.html)
-to concentrate tests on.
+These crates implement macros as token tree -> token tree transforms.
+They are independent from the rest of the code.
 
-The outermost boundary is the `rust-analyzer` crate, which defines an LSP
-interface in terms of stdio. We do integration testing of this component, by
-feeding it with a stream of LSP requests and checking responses. These tests are
-known as "heavy", because they interact with Cargo and read real files from
-disk. For this reason, we try to avoid writing too many tests on this boundary:
-in a statically typed language, it's hard to make an error in the protocol
-itself if messages are themselves typed.
+### `crates/vfs`, `crates/vfs-notify`
 
-The middle, and most important, boundary is `ide`. Unlike
-`rust-analyzer`, which exposes API, `ide` uses Rust API and is intended to
-use by various tools. Typical test creates an `AnalysisHost`, calls some
-`Analysis` functions and compares the results against expectation.
+These crates implement a virtual fils system.
+They provide consistent snapshots of the underlying file system and insulate messy OS paths.
 
-The innermost and most elaborate boundary is `hir`. It has a much richer
-vocabulary of types than `ide`, but the basic testing setup is the same: we
-create a database, run some queries, assert result.
+**Architecture Invariant:** vfs doesn't assume a single unified file system.
+IE, a single rust-analyzer process can act as a remote server for two different machines, where the same `/tmp/foo.rs` path points to different files.
+For this reason, all path APIs generally take some existing path as a "file system witness".
+
+### `crates/stdx`
+
+This crate contains various non-rust-analyzer specific utils, which could have been in std.
+
+### `crates/profile`
+
+This crate contains utilities for CPU and memory profiling.
+
+
+## Cross-Cutting Concerns
+
+This sections talks about the things which are everywhere and nowhere in particular.
+
+### Code generation
+
+Some of the components of this repository are generated through automatic processes.
+`cargo xtask codegen` runs all generation tasks.
+Generated code is generally committed to the git repository.
+There are tests to check that the generated code is fresh.
+
+In particular, we generate:
+
+* API for working with syntax trees (`syntax::ast`, the `ungrammar` crate).
+* Various sections of the manual:
+
+    * features
+    * assists
+    * config
+
+* Documentation tests for assists
+
+**Architecture Invariant:** we avoid bootstrapping.
+For codegen we need to parse Rust code.
+Using rust-analyzer for that would work and would be fun, but it would also complicate the build process a lot.
+For that reason, we use syn and manual string parsing.
+
+### Cancellation
+
+Let's say that the IDE is in the process of computing syntax highlighting, when the user types `foo`.
+What should happen?
+`rust-analyzer`s answer is that the highlighting process should be cancelled -- its results are now stale, and it also blocks modification of the inputs.
+
+The salsa database maintains a global revision counter.
+When applying a change, salsa bumps this counter and waits until all other threads using salsa finish.
+If a thread does salsa-based computation and notices that the counter is incremented, it panics with a special value (see `Canceled::throw`).
+That is, rust-analyzer requires unwinding.
+
+`ide` is the boundary where the panic is caught and transformed into a `Result<T, Cancelled>`.
+
+### Testing
+
+Rust Analyzer has three interesting [systems boundaries](https://www.tedinski.com/2018/04/10/making-tests-a-positive-influence-on-design.html) to concentrate tests on.
+
+The outermost boundary is the `rust-analyzer` crate, which defines an LSP interface in terms of stdio.
+We do integration testing of this component, by feeding it with a stream of LSP requests and checking responses.
+These tests are known as "heavy", because they interact with Cargo and read real files from disk.
+For this reason, we try to avoid writing too many tests on this boundary: in a statically typed language, it's hard to make an error in the protocol itself if messages are themselves typed.
+Heavy tests are only run when `RUN_SLOW_TESTS` env var is set.
+
+The middle, and most important, boundary is `ide`.
+Unlike `rust-analyzer`, which exposes API, `ide` uses Rust API and is intended to use by various tools.
+Typical test creates an `AnalysisHost`, calls some `Analysis` functions and compares the results against expectation.
+
+The innermost and most elaborate boundary is `hir`.
+It has a much richer vocabulary of types than `ide`, but the basic testing setup is the same: we create a database, run some queries, assert result.
 
 For comparisons, we use the `expect` crate for snapshot testing.
 
-To test various analysis corner cases and avoid forgetting about old tests, we
-use so-called marks. See the `marks` module in the `test_utils` crate for more.
+To test various analysis corner cases and avoid forgetting about old tests, we use so-called marks.
+See the `marks` module in the `test_utils` crate for more.
+
+**Architecture Invariant:** rust-analyzer tests do not use libcore or libstd.
+All required library code must be a part of the tests.
+This ensures fast test execution.
+
+**Architecture Invariant:** tests are data driven and do not test API.
+Tests which directly call various API functions are a liability, because they make refactoring the API significantly more complicated.
+So most of the tests look like this:
+
+```rust
+fn check(input: &str, expect: expect_test::Expect) {
+    // The single place that actually exercises a particular API
+}
+
+
+#[test]
+fn foo() {
+    check("foo", expect![["bar"]]);
+}
+
+#[test]
+fn spam() {
+    check("spam", expect![["eggs"]]);
+}
+// ...and a hundred more tests that don't care about the specific API at all.
+```
+
+To specify input data, we use a single string literal in a special format, which can describe a set of rust files.
+See the `Fixture` type.
+
+**Architecture Invariant:** all code invariants are tested by `#[test]` tests.
+There's no additional checks in CI, formatting and tidy tests are run with `cargo test`.
+
+**Architecture Invariant:** tests do not depend on any kind of external resources, they are perfectly reproducible.
+
+### Observability
+
+I've run out of steam here :)
+rust-analyzer is a long-running process, so its important to understand what's going on inside.
+We have hierarchical profiler (`RA_PROFILER=1`) and object counting (`RA_COUNT=1`).