From 1065c2bf1db2aaf78286b1f9f3c13237baac155b Mon Sep 17 00:00:00 2001 From: Aleksey Kladov Date: Wed, 29 Jan 2020 14:45:32 +0100 Subject: Freshen dev docs a tiny bits --- docs/dev/README.md | 84 ++++++++++++++++++++++----------------------------- docs/dev/debugging.md | 2 ++ 2 files changed, 38 insertions(+), 48 deletions(-) (limited to 'docs/dev') diff --git a/docs/dev/README.md b/docs/dev/README.md index 2f6215d6b..a2be99858 100644 --- a/docs/dev/README.md +++ b/docs/dev/README.md @@ -26,15 +26,6 @@ Discussion happens in this Zulip stream: https://rust-lang.zulipchat.com/#narrow/stream/185405-t-compiler.2Fwg-rls-2.2E0 -# Work List - -We have this "work list" paper document: - -https://paper.dropbox.com/doc/RLS-2.0-work-list--AZ3BgHKKCtqszbsi3gi6sjchAQ-42vbnxzuKq2lKwW0mkn8Y - -It shows what everyone is working on right now. If you want to (this is not -mandatory), add yourself to the list! - # Issue Labels * [good-first-issue](https://github.com/rust-analyzer/rust-analyzer/labels/good%20first%20issue) @@ -50,10 +41,12 @@ mandatory), add yourself to the list! # CI -We use Travis for CI. Most of the things, including formatting, are checked by +We use GitHub Actions for CI. Most of the things, including formatting, are checked by `cargo test` so, if `cargo test` passes locally, that's a good sign that CI will -be green as well. We use bors-ng to enforce the [not rocket -science](https://graydon2.dreamwidth.org/1597.html) rule. +be green as well. The only exception is that long-running by default a skipped locally. +Use `env RUN_SLOW_TESTS=1 cargo test` to run the full suite. + +We use bors-ng to enforce the [not rocket science](https://graydon2.dreamwidth.org/1597.html) rule. You can run `cargo xtask install-pre-commit-hook` to install git-hook to run rustfmt on commit. @@ -81,42 +74,37 @@ relevant test and execute it (VS Code includes an action for running a single test). However, launching a VS Code instance with locally build language server is -possible. There's even a VS Code task for this, so just F5 should -work (thanks, [@andrew-w-ross](https://github.com/andrew-w-ross)!). - -I often just install development version with `cargo xtask install --server --jemalloc` and -restart the host VS Code. - -See [./debugging.md](./debugging.md) for how to attach to rust-analyzer with -debugger, and don't forget that rust-analyzer has useful `pd` snippet and `dbg` -postfix completion for printf debugging :-) - -# Working With VS Code Extension - -To work on the VS Code extension, launch code inside `editors/code` and use `F5` -to launch/debug. To automatically apply formatter and linter suggestions, use -`npm run fix`. - -Tests are located inside `src/test` and are named `*.test.ts`. They use the -[Mocha](https://mochajs.org) test framework and the builtin Node -[assert](https://nodejs.org/api/assert.html) module. Unlike normal Node tests -they must be hosted inside a VS Code instance. This can be done in one of two -ways: - -1. When `F5` debugging in VS Code select the `Extension Tests` configuration - from the drop-down at the top of the Debug View. This will launch a temporary - instance of VS Code. The test results will appear in the "Debug Console" tab - of the primary VS Code instance. - -2. Run `npm test` from the command line. Although this is initiated from the - command line it is not headless; it will also launch a temporary instance of - VS Code. - -Due to the requirements of running the tests inside VS Code they are **not run -on CI**. When making changes to the extension please ensure the tests are not -broken locally before opening a Pull Request. - -To install **only** the VS Code extension, use `cargo xtask install --client-code`. +possible. There's "Run Extension (Dev Server)" launch configuration for this. + +In general, I use one of the following workflows for fixing bugs and +implementing features. + +If the problem concerns only internal parts of rust-analyzer (ie, I don't need +to touch `ra_lsp_server` crate or typescript code), there is a unit-test for it. +So, I use **Rust Analyzer: Run** action in VS Code to run this single test, and +then just do printf-driven development/debugging. As a sanity check after I'm +done, I use `cargo xtask install --server` and **Reload Window** action in VS +Code to sanity check that the thing works as I expect. + +If the problem concerns only the VS Code extension, I use **Run Extension** +launch configuration from `launch.json`. Notably, this uses the usual +`ra_lsp_server` binary from `PATH`. After I am done with the fix, I use `cargo +xtask install --client-code` to try the new extension for real. + +If I need to fix something in the `ra_lsp_server` crate, I feel sad because it's +on the boundary between the two processes, and working there is slow. I usually +just `cargo xtask install --server` and poke changes from my live environment. +Note that this uses `--release`, which is usually faster overall, because +loading stdlib into debug version of rust-analyzer takes a lot of time. To speed +things up, sometimes I open a temporary hello-world project which has +`"rust-analyzer.withSysroot": false` in `.code/settings.json`. This flag causes +rust-analyzer to skip loading the sysroot, which greatly reduces the amount of +things rust-analyzer needs to do, and makes printf's more useful. Note that you +should only use `eprint!` family of macros for debugging: stdout is used for LSP +communication, and `print!` would break it. + +If I need to fix something simultaneously in the server and in the client, I +feel even more sad. I don't have a specific workflow for this case. # Logging diff --git a/docs/dev/debugging.md b/docs/dev/debugging.md index f868e6998..1ccf4dca2 100644 --- a/docs/dev/debugging.md +++ b/docs/dev/debugging.md @@ -1,5 +1,7 @@ # Debugging vs Code plugin and the Language Server +**NOTE:** the information here is mostly obsolete + Install [LLDB](https://lldb.llvm.org/) and the [LLDB Extension](https://marketplace.visualstudio.com/items?itemName=vadimcn.vscode-lldb). Checkout rust rust-analyzer and open it in vscode. -- cgit v1.2.3 From 84dfbfbd1d72c276a93518fea41196f75069d17e Mon Sep 17 00:00:00 2001 From: Aleksey Kladov Date: Wed, 29 Jan 2020 15:08:31 +0100 Subject: Freshen Architecture.md document --- docs/dev/README.md | 4 +++ docs/dev/architecture.md | 79 +++++++++++++++++++++++++----------------------- 2 files changed, 45 insertions(+), 38 deletions(-) (limited to 'docs/dev') diff --git a/docs/dev/README.md b/docs/dev/README.md index a2be99858..d30727786 100644 --- a/docs/dev/README.md +++ b/docs/dev/README.md @@ -106,6 +106,10 @@ communication, and `print!` would break it. If I need to fix something simultaneously in the server and in the client, I feel even more sad. I don't have a specific workflow for this case. +Additionally, I use `cargo run --release -p ra_cli -- analysis-stats +path/to/some/rust/crate` to run a batch analysis. This is primaraly useful for +performance optimiations, or for bug minimization. + # Logging Logging is done by both rust-analyzer and VS Code, so it might be tricky to diff --git a/docs/dev/architecture.md b/docs/dev/architecture.md index 629645757..9675ed0b6 100644 --- a/docs/dev/architecture.md +++ b/docs/dev/architecture.md @@ -12,6 +12,9 @@ analyzer: https://www.youtube.com/playlist?list=PL85XCvVPmGQho7MZkdW-wtPtuJcFpzycE +Note that the guide and videos are pretty dated, this document should be in +generally fresher. + ## The Big Picture ![](https://user-images.githubusercontent.com/1711539/50114578-e8a34280-0255-11e9-902c-7cfc70747966.png) @@ -20,13 +23,12 @@ On the highest level, rust-analyzer is a thing which accepts input source code from the client and produces a structured semantic model of the code. More specifically, input data consists of a set of test files (`(PathBuf, -String)` pairs) and information about project structure, captured in the so called -`CrateGraph`. The crate graph specifies which files are crate roots, which cfg -flags are specified for each crate (TODO: actually implement this) and what -dependencies exist between the crates. The analyzer keeps all this input data in -memory and never does any IO. Because the input data is source code, which -typically measures in tens of megabytes at most, keeping all input data in -memory is OK. +String)` pairs) and information about project structure, captured in the so +called `CrateGraph`. The crate graph specifies which files are crate roots, +which cfg flags are specified for each crate and what dependencies exist between +the crates. The analyzer keeps all this input data in memory and never does any +IO. Because the input data are source code, which typically measures in tens of +megabytes at most, keeping everything in memory is OK. A "structured semantic model" is basically an object-oriented representation of modules, functions and types which appear in the source code. This representation @@ -43,37 +45,39 @@ can be quickly updated for small modifications. ## Code generation Some of the components of this repository are generated through automatic -processes. These are outlined below: +processes. `cargo xtask codegen` runs all generation tasks. Generated code is +commited to the git repository. + +In particular, `cargo xtask codegen` generates: + +1. [`syntax_kind/generated`](https://github.com/rust-analyzer/rust-analyzer/blob/a0be39296d2925972cacd9fbf8b5fb258fad6947/crates/ra_parser/src/syntax_kind/generated.rs) + -- the set of terminals and non-terminals of rust grammar. -- `cargo xtask codegen`: The kinds of tokens that are reused in several places, so a generator - is used. We use `quote!` macro to generate the files listed below, based on - the grammar described in [grammar.ron]: - - [ast/generated.rs][ast generated] - - [syntax_kind/generated.rs][syntax_kind generated] +2. [`ast/generated`](https://github.com/rust-analyzer/rust-analyzer/blob/a0be39296d2925972cacd9fbf8b5fb258fad6947/crates/ra_syntax/src/ast/generated.rs) + -- AST data structure. -[grammar.ron]: ../../crates/ra_syntax/src/grammar.ron -[ast generated]: ../../crates/ra_syntax/src/ast/generated.rs -[syntax_kind generated]: ../../crates/ra_parser/src/syntax_kind/generated.rs +.3 [`doc_tests/generated`](https://github.com/rust-analyzer/rust-analyzer/blob/a0be39296d2925972cacd9fbf8b5fb258fad6947/crates/ra_assists/src/doc_tests/generated.rs), + [`test_data/parser/inline`](https://github.com/rust-analyzer/rust-analyzer/tree/a0be39296d2925972cacd9fbf8b5fb258fad6947/crates/ra_syntax/test_data/parser/inline) + -- tests for assists and the parser. + +The source for 1 and 2 is in [`ast_src.rs`](https://github.com/rust-analyzer/rust-analyzer/blob/a0be39296d2925972cacd9fbf8b5fb258fad6947/xtask/src/ast_src.rs). ## Code Walk-Through ### `crates/ra_syntax`, `crates/ra_parser` Rust syntax tree structure and parser. See -[RFC](https://github.com/rust-lang/rfcs/pull/2256) for some design notes. +[RFC](https://github.com/rust-lang/rfcs/pull/2256) and [./syntax.md](./syntax.md) for some design notes. - [rowan](https://github.com/rust-analyzer/rowan) library is used for constructing syntax trees. - `grammar` module is the actual parser. It is a hand-written recursive descent parser, which produces a sequence of events like "start node X", "finish node Y". It works similarly to [kotlin's parser](https://github.com/JetBrains/kotlin/blob/4d951de616b20feca92f3e9cc9679b2de9e65195/compiler/frontend/src/org/jetbrains/kotlin/parsing/KotlinParsing.java), which is a good source of inspiration for dealing with syntax errors and incomplete input. Original [libsyntax parser](https://github.com/rust-lang/rust/blob/6b99adeb11313197f409b4f7c4083c2ceca8a4fe/src/libsyntax/parse/parser.rs) is what we use for the definition of the Rust language. -- `parser_api/parser_impl` bridges the tree-agnostic parser from `grammar` with `rowan` trees. - This is the thing that turns a flat list of events into a tree (see `EventProcessor`) +- `TreeSink` and `TokenSource` traits bridge the tree-agnostic parser from `grammar` with `rowan` trees. - `ast` provides a type safe API on top of the raw `rowan` tree. -- `grammar.ron` RON description of the grammar, which is used to - generate `syntax_kinds` and `ast` modules, using `cargo xtask codegen` command. -- `algo`: generic tree algorithms, including `walk` for O(1) stack - space tree traversal (this is cool). +- `ast_src` description of the grammar, which is used to generate `syntax_kinds` + and `ast` modules, using `cargo xtask codegen` command. Tests for ra_syntax are mostly data-driven: `test_data/parser` contains subdirectories with a bunch of `.rs` (test vectors) and `.txt` files with corresponding syntax trees. During testing, we check @@ -81,6 +85,10 @@ Tests for ra_syntax are mostly data-driven: `test_data/parser` contains subdirec tests). Additionally, running `cargo xtask codegen` will walk the grammar module and collect all `// test test_name` comments into files inside `test_data/parser/inline` directory. +Note +[`api_walkthrough`](https://github.com/rust-analyzer/rust-analyzer/blob/2fb6af89eb794f775de60b82afe56b6f986c2a40/crates/ra_syntax/src/lib.rs#L190-L348) +in particular: it shows off various methods of working with syntax tree. + See [#93](https://github.com/rust-analyzer/rust-analyzer/pull/93) for an example PR which fixes a bug in the grammar. @@ -94,18 +102,22 @@ defines most of the "input" queries: facts supplied by the client of the analyzer. Reading the docs of the `ra_db::input` module should be useful: everything else is strictly derived from those inputs. -### `crates/ra_hir` +### `crates/ra_hir*` crates HIR provides high-level "object oriented" access to Rust code. The principal difference between HIR and syntax trees is that HIR is bound to a -particular crate instance. That is, it has cfg flags and features applied (in -theory, in practice this is to be implemented). So, the relation between -syntax and HIR is many-to-one. The `source_binder` module is responsible for -guessing a HIR for a particular source position. +particular crate instance. That is, it has cfg flags and features applied. So, +the relation between syntax and HIR is many-to-one. The `source_binder` module +is responsible for guessing a HIR for a particular source position. Underneath, HIR works on top of salsa, using a `HirDatabase` trait. +`ra_hir_xxx` crates have a strong ECS flavor, in that they work with raw ids and +directly query the databse. + +The top-level `ra_hir` façade crate wraps ids into a more OO-flavored API. + ### `crates/ra_ide` A stateful library for analyzing many Rust files as they change. `AnalysisHost` @@ -135,18 +147,9 @@ different from data on disk. This is more or less the single really platform-dependent component, so it lives in a separate repository and has an extensive cross-platform CI testing. -### `crates/gen_lsp_server` - -A language server scaffold, exposing a synchronous crossbeam-channel based API. -This crate handles protocol handshaking and parsing messages, while you -control the message dispatch loop yourself. - -Run with `RUST_LOG=sync_lsp_server=debug` to see all the messages. - ### `crates/ra_cli` -A CLI interface to rust-analyzer. - +A CLI interface to rust-analyzer, mainly for testing. ## Testing Infrastructure -- cgit v1.2.3 From 134cc70097f2386bf3af6a15e3c29de9c317ed24 Mon Sep 17 00:00:00 2001 From: Florian Diebold Date: Sat, 8 Feb 2020 16:02:37 +0100 Subject: Docs cleanups --- docs/dev/README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) (limited to 'docs/dev') diff --git a/docs/dev/README.md b/docs/dev/README.md index d30727786..732e4bdd3 100644 --- a/docs/dev/README.md +++ b/docs/dev/README.md @@ -43,7 +43,7 @@ https://rust-lang.zulipchat.com/#narrow/stream/185405-t-compiler.2Fwg-rls-2.2E0 We use GitHub Actions for CI. Most of the things, including formatting, are checked by `cargo test` so, if `cargo test` passes locally, that's a good sign that CI will -be green as well. The only exception is that long-running by default a skipped locally. +be green as well. The only exception is that some long-running tests are skipped locally by default. Use `env RUN_SLOW_TESTS=1 cargo test` to run the full suite. We use bors-ng to enforce the [not rocket science](https://graydon2.dreamwidth.org/1597.html) rule. @@ -54,9 +54,9 @@ You can run `cargo xtask install-pre-commit-hook` to install git-hook to run rus All Rust code lives in the `crates` top-level directory, and is organized as a single Cargo workspace. The `editors` top-level directory contains code for -integrating with editors. Currently, it contains plugins for VS Code (in -typescript) and Emacs (in elisp). The `docs` top-level directory contains both -developer and user documentation. +integrating with editors. Currently, it contains the plugin for VS Code (in +typescript). The `docs` top-level directory contains both developer and user +documentation. We have some automation infra in Rust in the `xtask` package. It contains stuff like formatting checking, code generation and powers `cargo xtask install`. @@ -107,8 +107,8 @@ If I need to fix something simultaneously in the server and in the client, I feel even more sad. I don't have a specific workflow for this case. Additionally, I use `cargo run --release -p ra_cli -- analysis-stats -path/to/some/rust/crate` to run a batch analysis. This is primaraly useful for -performance optimiations, or for bug minimization. +path/to/some/rust/crate` to run a batch analysis. This is primarily useful for +performance optimizations, or for bug minimization. # Logging -- cgit v1.2.3