aboutsummaryrefslogtreecommitdiff
path: root/guide.md
diff options
context:
space:
mode:
Diffstat (limited to 'guide.md')
-rw-r--r--guide.md575
1 files changed, 0 insertions, 575 deletions
diff --git a/guide.md b/guide.md
deleted file mode 100644
index abbe4c154..000000000
--- a/guide.md
+++ /dev/null
@@ -1,575 +0,0 @@
1# Guide to rust-analyzer
2
3## About the guide
4
5This guide describes the current state of rust-analyzer as of 2019-01-20 (git
6tag [guide-2019-01]). Its purpose is to document various problems and
7architectural solutions related to the problem of building IDE-first compiler
8for Rust. There is a video version of this guide as well:
9https://youtu.be/ANKBNiSWyfc.
10
11[guide-2019-01]: https://github.com/rust-analyzer/rust-analyzer/tree/guide-2019-01
12
13## The big picture
14
15On the highest possible level, rust-analyzer is a stateful component. A client may
16apply changes to the analyzer (new contents of `foo.rs` file is "fn main() {}")
17and it may ask semantic questions about the current state (what is the
18definition of the identifier with offset 92 in file `bar.rs`?). Two important
19properties hold:
20
21* Analyzer does not do any I/O. It starts in an empty state and all input data is
22 provided via `apply_change` API.
23
24* Only queries about the current state are supported. One can, of course,
25 simulate undo and redo by keeping a log of changes and inverse changes respectively.
26
27## IDE API
28
29To see the bigger picture of how the IDE features works, let's take a look at the [`AnalysisHost`] and
30[`Analysis`] pair of types. `AnalysisHost` has three methods:
31
32* `default()` for creating an empty analysis instance
33* `apply_change(&mut self)` to make changes (this is how you get from an empty
34 state to something interesting)
35* `analysis(&self)` to get an instance of `Analysis`
36
37`Analysis` has a ton of methods for IDEs, like `goto_definition`, or
38`completions`. Both inputs and outputs of `Analysis`' methods are formulated in
39terms of files and offsets, and **not** in terms of Rust concepts like structs,
40traits, etc. The "typed" API with Rust specific types is slightly lower in the
41stack, we'll talk about it later.
42
43[`AnalysisHost`]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_ide_api/src/lib.rs#L265-L284
44[`Analysis`]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_ide_api/src/lib.rs#L291-L478
45
46The reason for this separation of `Analysis` and `AnalysisHost` is that we want to apply
47changes "uniquely", but we might also want to fork an `Analysis` and send it to
48another thread for background processing. That is, there is only a single
49`AnalysisHost`, but there may be several (equivalent) `Analysis`.
50
51Note that all of the `Analysis` API return `Cancelable<T>`. This is required to
52be responsive in an IDE setting. Sometimes a long-running query is being computed
53and the user types something in the editor and asks for completion. In this
54case, we cancel the long-running computation (so it returns `Err(Canceled)`),
55apply the change and execute request for completion. We never use stale data to
56answer requests. Under the cover, `AnalysisHost` "remembers" all outstanding
57`Analysis` instances. The `AnalysisHost::apply_change` method cancels all
58`Analysis`es, blocks until all of them are `Dropped` and then applies changes
59in-place. This may be familiar to Rustaceans who use read-write locks for interior
60mutability.
61
62Next, let's talk about what the inputs to the `Analysis` are, precisely.
63
64## Inputs
65
66Rust Analyzer never does any I/O itself, all inputs get passed explicitly via
67the `AnalysisHost::apply_change` method, which accepts a single argument, a
68`AnalysisChange`. [`AnalysisChange`] is a builder for a single change
69"transaction", so it suffices to study its methods to understand all of the
70input data.
71
72[`AnalysisChange`]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_ide_api/src/lib.rs#L119-L167
73
74The `(add|change|remove)_file` methods control the set of the input files, where
75each file has an integer id (`FileId`, picked by the client), text (`String`)
76and a filesystem path. Paths are tricky; they'll be explained below, in source roots
77section, together with the `add_root` method. The `add_library` method allows us to add a
78group of files which are assumed to rarely change. It's mostly an optimization
79and does not change the fundamental picture.
80
81The `set_crate_graph` method allows us to control how the input files are partitioned
82into compilation unites -- crates. It also controls (in theory, not implemented
83yet) `cfg` flags. `CrateGraph` is a directed acyclic graph of crates. Each crate
84has a root `FileId`, a set of active `cfg` flags and a set of dependencies. Each
85dependency is a pair of a crate and a name. It is possible to have two crates
86with the same root `FileId` but different `cfg`-flags/dependencies. This model
87is lower than Cargo's model of packages: each Cargo package consists of several
88targets, each of which is a separate crate (or several crates, if you try
89different feature combinations).
90
91Procedural macros should become inputs as well, but currently they are not
92supported. Procedural macro will be a black box `Box<dyn Fn(TokenStream) -> TokenStream>`
93function, and will be inserted into the crate graph just like dependencies.
94
95Soon we'll talk how we build an LSP server on top of `Analysis`, but first,
96let's deal with that paths issue.
97
98
99## Source roots (a.k.a. "Filesystems are horrible")
100
101This is a non-essential section, feel free to skip.
102
103The previous section said that the filesystem path is an attribute of a file,
104but this is not the whole truth. Making it an absolute `PathBuf` will be bad for
105several reasons. First, filesystems are full of (platform-dependent) edge cases:
106
107* it's hard (requires a syscall) to decide if two paths are equivalent
108* some filesystems are case-sensitive (e.g. on macOS)
109* paths are not necessary UTF-8
110* symlinks can form cycles
111
112Second, this might hurt reproducibility and hermeticity of builds. In theory,
113moving a project from `/foo/bar/my-project` to `/spam/eggs/my-project` should
114not change a bit in the output. However, if the absolute path is a part of the
115input, it is at least in theory observable, and *could* affect the output.
116
117Yet another problem is that we really *really* want to avoid doing I/O, but with
118Rust the set of "input" files is not necessary known up-front. In theory, you
119can have `#[path="/dev/random"] mod foo;`.
120
121To solve (or explicitly refuse to solve) these problems rust-analyzer uses the
122concept of a "source root". Roughly speaking, source roots are the contents of a
123directory on a file systems, like `/home/matklad/projects/rustraytracer/**.rs`.
124
125More precisely, all files (`FileId`s) are partitioned into disjoint
126`SourceRoot`s. Each file has a relative UTF-8 path within the `SourceRoot`.
127`SourceRoot` has an identity (integer ID). Crucially, the root path of the
128source root itself is unknown to the analyzer: A client is supposed to maintain a
129mapping between `SourceRoot` IDs (which are assigned by the client) and actual
130`PathBuf`s. `SourceRoot`s give a sane tree model of the file system to the
131analyzer.
132
133Note that `mod`, `#[path]` and `include!()` can only reference files from the
134same source root. It is of course is possible to explicitly add extra files to
135the source root, even `/dev/random`.
136
137## Language Server Protocol
138
139Now let's see how the `Analysis` API is exposed via the JSON RPC based language server protocol. The
140hard part here is managing changes (which can come either from the file system
141or from the editor) and concurrency (we want to spawn background jobs for things
142like syntax highlighting). We use the event loop pattern to manage the zoo, and
143the loop is the [`main_loop_inner`] function. The [`main_loop`] does a one-time
144initialization and tearing down of the resources.
145
146[`main_loop`]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_lsp_server/src/main_loop.rs#L51-L110
147[`main_loop_inner`]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_lsp_server/src/main_loop.rs#L156-L258
148
149
150Let's walk through a typical analyzer session!
151
152First, we need to figure out what to analyze. To do this, we run `cargo
153metadata` to learn about Cargo packages for current workspace and dependencies,
154and we run `rustc --print sysroot` and scan the "sysroot" (the directory containing the current Rust toolchain's files) to learn about crates like
155`std`. Currently we load this configuration once at the start of the server, but
156it should be possible to dynamically reconfigure it later without restart.
157
158[main_loop.rs#L62-L70](https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_lsp_server/src/main_loop.rs#L62-L70)
159
160The [`ProjectModel`] we get after this step is very Cargo and sysroot specific,
161it needs to be lowered to get the input in the form of `AnalysisChange`. This
162happens in [`ServerWorldState::new`] method. Specifically
163
164* Create a `SourceRoot` for each Cargo package and sysroot.
165* Schedule a filesystem scan of the roots.
166* Create an analyzer's `Crate` for each Cargo **target** and sysroot crate.
167* Setup dependencies between the crates.
168
169[`ProjectModel`]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_lsp_server/src/project_model.rs#L16-L20
170[`ServerWorldState::new`]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_lsp_server/src/server_world.rs#L38-L160
171
172The results of the scan (which may take a while) will be processed in the body
173of the main loop, just like any other change. Here's where we handle:
174
175* [File system changes](https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_lsp_server/src/main_loop.rs#L194)
176* [Changes from the editor](https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_lsp_server/src/main_loop.rs#L377)
177
178After a single loop's turn, we group the changes into one `AnalysisChange` and
179[apply] it. This always happens on the main thread and blocks the loop.
180
181[apply]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_lsp_server/src/server_world.rs#L216
182
183To handle requests, like ["goto definition"], we create an instance of the
184`Analysis` and [`schedule`] the task (which consumes `Analysis`) on the
185threadpool. [The task] calls the corresponding `Analysis` method, while
186massaging the types into the LSP representation. Keep in mind that if we are
187executing "goto definition" on the threadpool and a new change comes in, the
188task will be canceled as soon as the main loop calls `apply_change` on the
189`AnalysisHost`.
190
191["goto definition"]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_lsp_server/src/server_world.rs#L216
192[`schedule`]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_lsp_server/src/main_loop.rs#L426-L455
193[The task]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_lsp_server/src/main_loop/handlers.rs#L205-L223
194
195This concludes the overview of the analyzer's programing *interface*. Next, lets
196dig into the implementation!
197
198## Salsa
199
200The most straightforward way to implement an "apply change, get analysis, repeat"
201API would be to maintain the input state and to compute all possible analysis
202information from scratch after every change. This works, but scales poorly with
203the size of the project. To make this fast, we need to take advantage of the
204fact that most of the changes are small, and that analysis results are unlikely
205to change significantly between invocations.
206
207To do this we use [salsa]: a framework for incremental on-demand computation.
208You can skip the rest of the section if you are familiar with rustc's red-green
209algorithm (which is used for incremental compilation).
210
211[salsa]: https://github.com/salsa-rs/salsa
212
213It's better to refer to salsa's docs to learn about it. Here's a small excerpt:
214
215The key idea of salsa is that you define your program as a set of queries. Every
216query is used like a function `K -> V` that maps from some key of type `K` to a value
217of type `V`. Queries come in two basic varieties:
218
219* **Inputs**: the base inputs to your system. You can change these whenever you
220 like.
221
222* **Functions**: pure functions (no side effects) that transform your inputs
223 into other values. The results of queries is memoized to avoid recomputing
224 them a lot. When you make changes to the inputs, we'll figure out (fairly
225 intelligently) when we can re-use these memoized values and when we have to
226 recompute them.
227
228
229For further discussion, its important to understand one bit of "fairly
230intelligently". Suppose we have two functions, `f1` and `f2`, and one input,
231`z`. We call `f1(X)` which in turn calls `f2(Y)` which inspects `i(Z)`. `i(Z)`
232returns some value `V1`, `f2` uses that and returns `R1`, `f1` uses that and
233returns `O`. Now, let's change `i` at `Z` to `V2` from `V1` and try to compute
234`f1(X)` again. Because `f1(X)` (transitively) depends on `i(Z)`, we can't just
235reuse its value as is. However, if `f2(Y)` is *still* equal to `R1` (despite
236`i`'s change), we, in fact, *can* reuse `O` as result of `f1(X)`. And that's how
237salsa works: it recomputes results in *reverse* order, starting from inputs and
238progressing towards outputs, stopping as soon as it sees an intermediate value
239that hasn't changed. If this sounds confusing to you, don't worry: it is
240confusing. This illustration by @killercup might help:
241
242<img alt="step 1" src="https://user-images.githubusercontent.com/1711539/51460907-c5484780-1d6d-11e9-9cd2-d6f62bd746e0.png" width="50%">
243
244<img alt="step 2" src="https://user-images.githubusercontent.com/1711539/51460915-c9746500-1d6d-11e9-9a77-27d33a0c51b5.png" width="50%">
245
246<img alt="step 3" src="https://user-images.githubusercontent.com/1711539/51460920-cda08280-1d6d-11e9-8d96-a782aa57a4d4.png" width="50%">
247
248<img alt="step 4" src="https://user-images.githubusercontent.com/1711539/51460927-d1340980-1d6d-11e9-851e-13c149d5c406.png" width="50%">
249
250## Salsa Input Queries
251
252All analyzer information is stored in a salsa database. `Analysis` and
253`AnalysisHost` types are newtype wrappers for [`RootDatabase`] -- a salsa
254database.
255
256[`RootDatabase`]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_ide_api/src/db.rs#L88-L134
257
258Salsa input queries are defined in [`FilesDatabase`] (which is a part of
259`RootDatabase`). They closely mirror the familiar `AnalysisChange` structure:
260indeed, what `apply_change` does is it sets the values of input queries.
261
262[`FilesDatabase`]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_db/src/input.rs#L150-L174
263
264## From text to semantic model
265
266The bulk of the rust-analyzer is transforming input text into a semantic model of
267Rust code: a web of entities like modules, structs, functions and traits.
268
269An important fact to realize is that (unlike most other languages like C# or
270Java) there isn't a one-to-one mapping between source code and the semantic model. A
271single function definition in the source code might result in several semantic
272functions: for example, the same source file might be included as a module into
273several crate, or a single "crate" might be present in the compilation DAG
274several times, with different sets of `cfg`s enabled. The IDE-specific task of
275mapping source code position into a semantic model is inherently imprecise for
276this reason, and is handled by the [`source_binder`].
277
278[`source_binder`]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_hir/src/source_binder.rs
279
280The semantic interface is declared in the [`code_model_api`] module. Each entity is
281identified by an integer ID and has a bunch of methods which take a salsa database
282as an argument and returns other entities (which are also IDs). Internally, these
283methods invoke various queries on the database to build the model on demand.
284Here's [the list of queries].
285
286[`code_model_api`]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_hir/src/code_model_api.rs
287[the list of queries]: https://github.com/rust-analyzer/rust-analyzer/blob/7e84440e25e19529e4ff8a66e521d1b06349c6ec/crates/ra_hir/src/db.rs#L20-L106
288
289The first step of building the model is parsing the source code.
290
291## Syntax trees
292
293An important property of the Rust language is that each file can be parsed in
294isolation. Unlike, say, `C++`, an `include` can't change the meaning of the
295syntax. For this reason, rust-analyzer can build a syntax tree for each "source
296file", which could then be reused by several semantic models if this file
297happens to be a part of several crates.
298
299The representation of syntax trees that rust-analyzer uses is similar to that of `Roslyn`
300and Swift's new [libsyntax]. Swift's docs give an excellent overview of the
301approach, so I skip this part here and instead outline the main characteristics
302of the syntax trees:
303
304* Syntax trees are fully lossless. Converting **any** text to a syntax tree and
305 back is a total identity function. All whitespace and comments are explicitly
306 represented in the tree.
307
308* Syntax nodes have generic `(next|previous)_sibling`, `parent`,
309 `(first|last)_child` functions. You can get from any one node to any other
310 node in the file using only these functions.
311
312* Syntax nodes know their range (start offset and length) in the file.
313
314* Syntax nodes share the ownership of their syntax tree: if you keep a reference
315 to a single function, the whole enclosing file is alive.
316
317* Syntax trees are immutable and the cost of replacing the subtree is
318 proportional to the depth of the subtree. Read Swift's docs to learn how
319 immutable + parent pointers + cheap modification is possible.
320
321* Syntax trees are build on best-effort basis. All accessor methods return
322 `Option`s. The tree for `fn foo` will contain a function declaration with
323 `None` for parameter list and body.
324
325* Syntax trees do not know the file they are built from, they only know about
326 the text.
327
328The implementation is based on the generic [rowan] crate on top of which a
329[rust-specific] AST is generated.
330
331[libsyntax]: https://github.com/apple/swift/tree/5e2c815edfd758f9b1309ce07bfc01c4bc20ec23/lib/Syntax
332[rowan]: https://github.com/rust-analyzer/rowan/tree/100a36dc820eb393b74abe0d20ddf99077b61f88
333[rust-specific]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_syntax/src/ast/generated.rs
334
335The next step in constructing the semantic model is ...
336
337## Building a Module Tree
338
339The algorithm for building a tree of modules is to start with a crate root
340(remember, each `Crate` from a `CrateGraph` has a `FileId`), collect all `mod`
341declarations and recursively process child modules. This is handled by the
342[`module_tree_query`], with two slight variations.
343
344[`module_tree_query`]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_hir/src/module_tree.rs#L116-L123
345
346First, rust-analyzer builds a module tree for all crates in a source root
347simultaneously. The main reason for this is historical (`module_tree` predates
348`CrateGraph`), but this approach also enables accounting for files which are not
349part of any crate. That is, if you create a file but do not include it as a
350submodule anywhere, you still get semantic completion, and you get a warning
351about a free-floating module (the actual warning is not implemented yet).
352
353The second difference is that `module_tree_query` does not *directly* depend on
354the "parse" query (which is confusingly called `source_file`). Why would calling
355the parse directly be bad? Suppose the user changes the file slightly, by adding
356an insignificant whitespace. Adding whitespace changes the parse tree (because
357it includes whitespace), and that means recomputing the whole module tree.
358
359We deal with this problem by introducing an intermediate [`submodules_query`].
360This query processes the syntax tree and extracts a set of declared submodule
361names. Now, changing the whitespace results in `submodules_query` being
362re-executed for a *single* module, but because the result of this query stays
363the same, we don't have to re-execute [`module_tree_query`]. In fact, we only
364need to re-execute it when we add/remove new files or when we change mod
365declarations.
366
367[`submodules_query`]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_hir/src/module_tree.rs#L41
368
369We store the resulting modules in a `Vec`-based indexed arena. The indices in
370the arena becomes module IDs. And this brings us to the next topic:
371assigning IDs in the general case.
372
373## Location Interner pattern
374
375One way to assign IDs is how we've dealt with modules: Collect all items into a
376single array in some specific order and use the index in the array as an ID. The
377main drawback of this approach is that these IDs are not stable: Adding a new item can
378shift the IDs of all other items. This works for modules, because adding a module is
379a comparatively rare operation, but would be less convenient for, for example,
380functions.
381
382Another solution here is positional IDs: We can identify a function as "the
383function with name `foo` in a ModuleId(92) module". Such locations are stable:
384adding a new function to the module (unless it is also named `foo`) does not
385change the location. However, such "ID" types ceases to be a `Copy`able integer and in
386general can become pretty large if we account for nesting (for example: "third parameter of
387the `foo` function of the `bar` `impl` in the `baz` module").
388
389[`LocationInterner`] allows us to combine the benefits of positional and numeric
390IDs. It is a bidirectional append-only map between locations and consecutive
391integers which can "intern" a location and return an integer ID back. The salsa
392database we use includes a couple of [interners]. How to "garbage collect"
393unused locations is an open question.
394
395[`LocationInterner`]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_db/src/loc2id.rs#L65-L71
396[interners]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_hir/src/db.rs#L22-L23
397
398For example, we use `LocationInterner` to assign IDs to definitions of functions,
399structs, enums, etc. The location, [`DefLoc`] contains two bits of information:
400
401* the ID of the module which contains the definition,
402* the ID of the specific item in the modules source code.
403
404We "could" use a text offset for the location of a particular item, but that would play
405badly with salsa: offsets change after edits. So, as a rule of thumb, we avoid
406using offsets, text ranges or syntax trees as keys and values for queries. What
407we do instead is we store "index" of the item among all of the items of a file
408(so, a positional based ID, but localized to a single file).
409
410[`DefLoc`]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_hir/src/ids.rs#L127-L139
411
412One thing we've glossed over for the time being is support for macros. We have
413only proof of concept handling of macros at the moment, but they are extremely
414interesting from an "assigning IDs" perspective.
415
416## Macros and recursive locations
417
418The tricky bit about macros is that they effectively create new source files.
419While we can use `FileId`s to refer to original files, we can't just assign them
420willy-nilly to the pseudo files of macro expansion. Instead, we use a special
421ID, [`HirFileId`] to refer to either a usual file or a macro-generated file:
422
423```rust
424enum HirFileId {
425 FileId(FileId),
426 Macro(MacroCallId),
427}
428```
429
430`MacroCallId` is an interned ID that specifies a particular macro invocation.
431Its `MacroCallLoc` contains:
432
433* `ModuleId` of the containing module
434* `HirFileId` of the containing file or pseudo file
435* an index of this particular macro invocation in this file (positional id
436 again).
437
438Note how `HirFileId` is defined in terms of `MacroCallLoc` which is defined in
439terms of `HirFileId`! This does not recur infinitely though: any chain of
440`HirFileId`s bottoms out in `HirFileId::FileId`, that is, some source file
441actually written by the user.
442
443[`HirFileId`]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_hir/src/ids.rs#L18-L125
444
445Now that we understand how to identify a definition, in a source or in a
446macro-generated file, we can discuss name resolution a bit.
447
448## Name resolution
449
450Name resolution faces the same problem as the module tree: if we look at the
451syntax tree directly, we'll have to recompute name resolution after every
452modification. The solution to the problem is the same: We [lower] the source code of
453each module into a position-independent representation which does not change if
454we modify bodies of the items. After that we [loop] resolving all imports until
455we've reached a fixed point.
456
457[lower]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_hir/src/nameres/lower.rs#L113-L117
458[loop]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_hir/src/nameres.rs#L186-L196
459
460And, given all our preparation with IDs and a position-independent representation,
461it is satisfying to [test] that typing inside function body does not invalidate
462name resolution results.
463
464[test]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_hir/src/nameres/tests.rs#L376
465
466An interesting fact about name resolution is that it "erases" all of the
467intermediate paths from the imports: in the end, we know which items are defined
468and which items are imported in each module, but, if the import was `use
469foo::bar::baz`, we deliberately forget what modules `foo` and `bar` resolve to.
470
471To serve "goto definition" requests on intermediate segments we need this info
472in the IDE, however. Luckily, we need it only for a tiny fraction of imports, so we just ask
473the module explicitly, "What does the path `foo::bar` resolve to?". This is a
474general pattern: we try to compute the minimal possible amount of information
475during analysis while allowing IDE to ask for additional specific bits.
476
477Name resolution is also a good place to introduce another salsa pattern used
478throughout the analyzer:
479
480## Source Map pattern
481
482Due to an obscure edge case in completion, IDE needs to know the syntax node of
483an use statement which imported the given completion candidate. We can't just
484store the syntax node as a part of name resolution: this will break
485incrementality, due to the fact that syntax changes after every file
486modification.
487
488We solve this problem during the lowering step of name resolution. The lowering
489query actually produces a *pair* of outputs: `LoweredModule` and [`SourceMap`].
490The `LoweredModule` module contains [imports], but in a position-independent form.
491The `SourceMap` contains a mapping from position-independent imports to
492(position-dependent) syntax nodes.
493
494The result of this basic lowering query changes after every modification. But
495there's an intermediate [projection query] which returns only the first
496position-independent part of the lowering. The result of this query is stable.
497Naturally, name resolution [uses] this stable projection query.
498
499[imports]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_hir/src/nameres/lower.rs#L52-L59
500[`SourceMap`]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_hir/src/nameres/lower.rs#L52-L59
501[projection query]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_hir/src/nameres/lower.rs#L97-L103
502[uses]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_hir/src/query_definitions.rs#L49
503
504## Type inference
505
506First of all, implementation of type inference in rust-analyzer was spearheaded
507by [@flodiebold]. [#327] was an awesome Christmas present, thank you, Florian!
508
509Type inference runs on per-function granularity and uses the patterns we've
510discussed previously.
511
512First, we [lower the AST] of a function body into a position-independent
513representation. In this representation, each expression is assigned a
514[positional ID]. Alongside the lowered expression, [a source map] is produced,
515which maps between expression ids and original syntax. This lowering step also
516deals with "incomplete" source trees by replacing missing expressions by an
517explicit `Missing` expression.
518
519Given the lowered body of the function, we can now run [type inference] and
520construct a mapping from `ExprId`s to types.
521
522[@flodiebold]: https://github.com/flodiebold
523[#327]: https://github.com/rust-analyzer/rust-analyzer/pull/327
524[lower the AST]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_hir/src/expr.rs
525[positional ID]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_hir/src/expr.rs#L13-L15
526[a source map]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_hir/src/expr.rs#L41-L44
527[type inference]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_hir/src/ty.rs#L1208-L1223
528
529## Tying it all together: completion
530
531To conclude the overview of the rust-analyzer, let's trace the request for
532(type-inference powered!) code completion!
533
534We start by [receiving a message] from the language client. We decode the
535message as a request for completion and [schedule it on the threadpool]. This is
536the also place where we [catch] canceled errors if, immediately after completion, the
537client sends some modification.
538
539In [the handler] we a deserialize LSP request into the rust-analyzer specific data
540types (by converting a file url into a numeric `FileId`), [ask analysis for
541completion] and serializer results to LSP.
542
543The [completion implementation] is finally the place where we start doing the actual
544work. The first step is to collect the `CompletionContext` -- a struct which
545describes the cursor position in terms of Rust syntax and semantics. For
546example, `function_syntax: Option<&'a ast::FnDef>` stores a reference to
547enclosing function *syntax*, while `function: Option<hir::Function>` is the
548`Def` for this function.
549
550To construct the context, we first do an ["IntelliJ Trick"]: we insert a dummy
551identifier at the cursor's position and parse this modified file, to get a
552reasonably looking syntax tree. Then we do a bunch of "classification" routines
553to figure out the context. For example, we [find an ancestor `fn` node] and we get a
554[semantic model] for it (using the lossy `source_binder` infrastructure).
555
556The second step is to run a [series of independent completion routines]. Let's
557take a closer look at [`complete_dot`], which completes fields and methods in
558`foo.bar|`. First we extract a semantic function and a syntactic receiver
559expression out of the `Context`. Then we run type-inference for this single
560function and map our syntactic expression to `ExprId`. Using the ID, we figure
561out the type of the receiver expression. Then we add all fields & methods from
562the type to completion.
563
564[receiving a message]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_lsp_server/src/main_loop.rs#L203
565[schedule it on the threadpool]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_lsp_server/src/main_loop.rs#L428
566[catch]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_lsp_server/src/main_loop.rs#L436-L442
567[the handler]: https://salsa.zulipchat.com/#narrow/stream/181542-rfcs.2Fsalsa-query-group/topic/design.20next.20steps
568[ask analysis for completion]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_ide_api/src/lib.rs#L439-L444
569[completion implementation]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_ide_api/src/completion.rs#L46-L62
570[`CompletionContext`]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_ide_api/src/completion/completion_context.rs#L14-L37
571["IntelliJ Trick"]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_ide_api/src/completion/completion_context.rs#L72-L75
572[find an ancestor `fn` node]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_ide_api/src/completion/completion_context.rs#L116-L120
573[semantic model]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_ide_api/src/completion/completion_context.rs#L123
574[series of independent completion routines]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_ide_api/src/completion.rs#L52-L59
575[`complete_dot`]: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/crates/ra_ide_api/src/completion/complete_dot.rs#L6-L22