Merge #4397

4397: Textmate cooperation r=matklad a=georgewfraser

This PR tweaks the fallback TextMate scopes to make them more consistent with the existing grammar and other languages, and edits the builtin TextMate grammar to align with semantic coloring. Before is on the left, after is on the right:

<img width="855" alt="Screen Shot 2020-05-10 at 1 45 51 PM" src="https://user-images.githubusercontent.com/1369240/81512320-a8be7e80-92d4-11ea-8940-2c03f6769015.png">

**Use keyword.other for regular keywords instead of keyword**. This is a really peculiar quirk of TextMate conventions, but virtually *all* TextMate grammars use `keyword.other` (colored blue in VSCode Dark+) for regular keywords and `keyword.control` (colored purple in VSCode Dark+) for control keywords. The TextMate scope `keyword` is colored like control keywords, not regular keywords. It may seem strange that the `keyword` scope is not the right fallback for the `keyword` semantic token, but TextMate has a long and weird history. Note how keywords change from purple back to blue (what they were before semantic coloring was added):

**(1) Use punctuation.section.embedded for format specifiers**. This aligns with how Typescript colors formatting directives:

<img width="238" alt="Screen Shot 2020-05-09 at 10 54 01 AM" src="https://user-images.githubusercontent.com/1369240/81481258-93b5f280-91e3-11ea-99c2-c6d258c5bcad.png">

**(2) Consistently use `entity.name.type.*` scopes for type names**. Avoid using `entity.name.*` which gets colored like a keyword.

**(3) Use Property instead of Member for fields**. Property and Member are very similar, but if you look at the TextMate fallback scopes, it's clear that Member is intended for function-like-things (methods?) and Property is intended for variable-like-things.

**(4) Color `for` as a regular keyword when it's part of `impl Trait for Struct`**. 

**(5) Use `variable.other.constant` for constants instead of `entity.name.constant`**. In the latest VSCode insiders, variable.other.constant has a subtly different color that differentiates constants from ordinary variables. It looks close to the green of types but it's not the same---it's a new color recently added to take advantage of semantic coloring.

I also made some minor changes that make the TextMate scopes better match the semantic scopes. The effect of this for the user is you observe less of a change when semantic coloring "activates". You can see the changes I made relative to the built-in TextMate grammar here:

https://github.com/rust-analyzer/rust-analyzer/pull/4397/files/a91d15c80c337dd1afb0eddd5eb048010d098ac7..97428b6d52d25f810dbd7d7a8d787740c58bfbd2#diff-6966c729b862f79f79bf7258eb3e0885


Co-authored-by: George Fraser <[email protected]>

3 files changed, 695 insertions, 8 deletions

diff --git a/editors/code/.vscodeignore b/editors/code/.vscodeignore
index ac411f8e2..257b744bf 100644
--- a/editors/code/.vscodeignore
+++ b/editors/code/.vscodeignore
@@ -3,5 +3,6 @@
 !package.json
 !package-lock.json
 !ra_syntax_tree.tmGrammar.json
+!rust.tmGrammar.json
 !icon.png
 !README.md
diff --git a/editors/code/package.json b/editors/code/package.json
index c6fc13519..f46684c76 100644
--- a/editors/code/package.json
+++ b/editors/code/package.json
@@ -479,6 +479,11 @@
         ],
         "grammars": [
             {
+                "language": "rust",
+                "scopeName": "source.rust",
+                "path": "rust.tmGrammar.json"
+            },
+            {
                 "language": "ra_syntax_tree",
                 "scopeName": "source.ra_syntax_tree",
                 "path": "ra_syntax_tree.tmGrammar.json"
@@ -596,28 +601,28 @@
                         "support.type.primitive"
                     ],
                     "lifetime": [
-                        "entity.name.lifetime.rust"
+                        "storage.modifier.lifetime.rust"
                     ],
                     "typeAlias": [
-                        "entity.name.typeAlias"
+                        "entity.name.type.typeAlias"
                     ],
                     "union": [
-                        "entity.name.union"
+                        "entity.name.type.union"
                     ],
                     "struct": [
                         "entity.name.type.struct"
                     ],
-                    "keyword.unsafe": [
-                        "keyword.other.unsafe"
-                    ],
                     "keyword": [
-                        "keyword"
+                        "keyword.other"
                     ],
                     "keyword.controlFlow": [
                         "keyword.control"
                     ],
                     "variable.constant": [
-                        "entity.name.constant"
+                        "variable.other.constant"
+                    ],
+                    "formatSpecifier": [
+                        "punctuation.section.embedded"
                     ]
                 }
             }
diff --git a/editors/code/rust.tmGrammar.json b/editors/code/rust.tmGrammar.json
new file mode 100644
index 000000000..27982c13a
--- /dev/null
+++ b/editors/code/rust.tmGrammar.json
@@ -0,0 +1,681 @@
+{
+        "name": "Rust",
+        "scopeName": "source.rust",
+        "patterns": [
+                {
+                        "comment": "Implementation",
+                        "begin": "\\b(impl)\\b",
+                        "end": "\\{",
+                        "beginCaptures": {
+                                "1": {
+                                        "name": "storage.type.rust"
+                                }
+                        },
+                        "patterns": [
+                                {
+                                        "include": "#block_comment"
+                                },
+                                {
+                                        "include": "#line_comment"
+                                },
+                                {
+                                        "include": "#sigils"
+                                },
+                                {
+                                        "include": "#mut"
+                                },
+                                {
+                                        "include": "#dyn"
+                                },
+                                {
+                                        "include": "#ref_lifetime"
+                                },
+                                {
+                                        "include": "#core_types"
+                                },
+                                {
+                                        "include": "#core_marker"
+                                },
+                                {
+                                        "include": "#core_traits"
+                                },
+                                {
+                                        "include": "#std_types"
+                                },
+                                {
+                                        "include": "#std_traits"
+                                },
+                                {
+                                        "include": "#type_params"
+                                },
+                                {
+                                        "include": "#where"
+                                },
+                                {
+                                        "name": "storage.type.rust",
+                                        "match": "\\bfor\\b"
+                                },
+                                {
+                                        "include": "#type"
+                                }
+                        ]
+                },
+                {
+                        "include": "#block_doc_comment"
+                },
+                {
+                        "include": "#block_comment"
+                },
+                {
+                        "include": "#line_doc_comment"
+                },
+                {
+                        "include": "#line_comment"
+                },
+                {
+                        "comment": "Attribute",
+                        "name": "meta.attribute.rust",
+                        "begin": "#\\!?\\[",
+                        "end": "\\]",
+                        "patterns": [
+                                {
+                                        "include": "#string_literal"
+                                },
+                                {
+                                        "include": "#block_doc_comment"
+                                },
+                                {
+                                        "include": "#block_comment"
+                                },
+                                {
+                                        "include": "#line_doc_comment"
+                                },
+                                {
+                                        "include": "#line_comment"
+                                }
+                        ]
+                },
+                {
+                        "comment": "Single-quote string literal (character)",
+                        "name": "string.quoted.single.rust",
+                        "match": "b?'([^'\\\\]|\\\\(x[0-9A-Fa-f]{2}|[0-2][0-7]{0,2}|3[0-6][0-7]?|37[0-7]?|[4-7][0-7]?|.))'"
+                },
+                {
+                        "include": "#string_literal"
+                },
+                {
+                        "include": "#raw_string_literal"
+                },
+                {
+                        "comment": "Floating point literal (fraction)",
+                        "name": "constant.numeric.float.rust",
+                        "match": "\\b[0-9][0-9_]*\\.[0-9][0-9_]*([eE][+-]?[0-9_]+)?(f32|f64)?\\b"
+                },
+                {
+                        "comment": "Floating point literal (exponent)",
+                        "name": "constant.numeric.float.rust",
+                        "match": "\\b[0-9][0-9_]*(\\.[0-9][0-9_]*)?[eE][+-]?[0-9_]+(f32|f64)?\\b"
+                },
+                {
+                        "comment": "Floating point literal (typed)",
+                        "name": "constant.numeric.float.rust",
+                        "match": "\\b[0-9][0-9_]*(\\.[0-9][0-9_]*)?([eE][+-]?[0-9_]+)?(f32|f64)\\b"
+                },
+                {
+                        "comment": "Integer literal (decimal)",
+                        "name": "constant.numeric.integer.decimal.rust",
+                        "match": "\\b[0-9][0-9_]*([ui](8|16|32|64|128|s|size))?\\b"
+                },
+                {
+                        "comment": "Integer literal (hexadecimal)",
+                        "name": "constant.numeric.integer.hexadecimal.rust",
+                        "match": "\\b0x[a-fA-F0-9_]+([ui](8|16|32|64|128|s|size))?\\b"
+                },
+                {
+                        "comment": "Integer literal (octal)",
+                        "name": "constant.numeric.integer.octal.rust",
+                        "match": "\\b0o[0-7_]+([ui](8|16|32|64|128|s|size))?\\b"
+                },
+                {
+                        "comment": "Integer literal (binary)",
+                        "name": "constant.numeric.integer.binary.rust",
+                        "match": "\\b0b[01_]+([ui](8|16|32|64|128|s|size))?\\b"
+                },
+                {
+                        "comment": "Static storage modifier",
+                        "name": "storage.modifier.static.rust",
+                        "match": "\\bstatic\\b"
+                },
+                {
+                        "comment": "Boolean constant",
+                        "name": "constant.language.boolean.rust",
+                        "match": "\\b(true|false)\\b"
+                },
+                {
+                        "comment": "Control keyword",
+                        "name": "keyword.control.rust",
+                        "match": "\\b(async|await|break|continue|else|if|in|for|loop|match|return|try|while)\\b"
+                },
+                {
+                        "comment": "Keyword",
+                        "name": "keyword.other.rust",
+                        "match": "\\b(crate|extern|mod|let|ref|use|super|move|as)\\b"
+                },
+                {
+                        "comment": "Reserved keyword",
+                        "name": "invalid.deprecated.rust",
+                        "match": "\\b(abstract|alignof|become|do|final|macro|offsetof|override|priv|proc|pure|sizeof|typeof|virtual|yield)\\b"
+                },
+                {
+                        "include": "#unsafe"
+                },
+                {
+                        "include": "#sigils"
+                },
+                {
+                        "include": "#self"
+                },
+                {
+                        "include": "#mut"
+                },
+                {
+                        "include": "#dyn"
+                },
+                {
+                        "include": "#impl"
+                },
+                {
+                        "include": "#box"
+                },
+                {
+                        "include": "#lifetime"
+                },
+                {
+                        "include": "#ref_lifetime"
+                },
+                {
+                        "include": "#const"
+                },
+                {
+                        "include": "#pub"
+                },
+                {
+                        "comment": "Miscellaneous operator",
+                        "name": "keyword.operator.misc.rust",
+                        "match": "(=>|::)"
+                },
+                {
+                        "comment": "Comparison operator",
+                        "name": "keyword.operator.comparison.rust",
+                        "match": "(&&|\\|\\||==|!=)"
+                },
+                {
+                        "comment": "Assignment operator",
+                        "name": "keyword.operator.assignment.rust",
+                        "match": "(\\+=|-=|/=|\\*=|%=|\\^=|&=|\\|=|<<=|>>=|=)"
+                },
+                {
+                        "comment": "Arithmetic operator",
+                        "name": "keyword.operator.arithmetic.rust",
+                        "match": "(!|\\+|-|/|\\*|%|\\^|&|\\||<<|>>)"
+                },
+                {
+                        "comment": "Comparison operator (second group because of regex precedence)",
+                        "name": "keyword.operator.comparison.rust",
+                        "match": "(<=|>=|<|>)"
+                },
+                {
+                        "include": "#core_types"
+                },
+                {
+                        "include": "#core_vars"
+                },
+                {
+                        "include": "#core_marker"
+                },
+                {
+                        "include": "#core_traits"
+                },
+                {
+                        "include": "#std_types"
+                },
+                {
+                        "include": "#std_traits"
+                },
+                {
+                        "comment": "Built-in macro",
+                        "name": "support.function.builtin.rust",
+                        "match": "\\b(macro_rules|compile_error|format_args|env|option_env|concat_idents|concat|line|column|file|stringify|include|include_str|include_bytes|module_path|cfg)!"
+                },
+                {
+                        "comment": "Core macro",
+                        "name": "support.function.core.rust",
+                        "match": "\\b(panic|assert|assert_eq|assert_ne|debug_assert|debug_assert_eq|debug_assert_ne|try|write|writeln|unreachable|unimplemented)!"
+                },
+                {
+                        "comment": "Standard library macro",
+                        "name": "support.function.std.rust",
+                        "match": "\\b(format|print|println|eprint|eprintln|select|vec)!"
+                },
+                {
+                        "comment": "Logging macro",
+                        "name": "support.function.log.rust",
+                        "match": "\\b(log|error|warn|info|debug|trace|log_enabled)!"
+                },
+                {
+                        "comment": "Invokation of a macro",
+                        "match": "\\b([a-zA-Z_][a-zA-Z0-9_]*\\!)\\s*[({\\[]",
+                        "captures": {
+                                "1": {
+                                        "name": "entity.name.function.macro.rust"
+                                }
+                        }
+                },
+                {
+                        "comment": "Function call",
+                        "match": "\\b([A-Za-z][A-Za-z0-9_]*|_[A-Za-z0-9_]+)\\s*\\(",
+                        "captures": {
+                                "1": {
+                                        "name": "entity.name.function.rust"
+                                }
+                        }
+                },
+                {
+                        "comment": "Function call with type parameters",
+                        "begin": "\\b([A-Za-z][A-Za-z0-9_]*|_[A-Za-z0-9_]+)\\s*(::)(?=\\s*<.*>\\s*\\()",
+                        "end": "\\(",
+                        "captures": {
+                                "1": {
+                                        "name": "entity.name.function.rust"
+                                },
+                                "2": {
+                                        "name": "keyword.operator.misc.rust"
+                                }
+                        },
+                        "patterns": [
+                                {
+                                        "include": "#type_params"
+                                }
+                        ]
+                },
+                {
+                        "comment": "Function definition",
+                        "begin": "\\b(fn)\\s+([A-Za-z][A-Za-z0-9_]*|_[A-Za-z0-9_]+)",
+                        "end": "[\\{;]",
+                        "beginCaptures": {
+                                "1": {
+                                        "name": "keyword.other.fn.rust"
+                                },
+                                "2": {
+                                        "name": "entity.name.function.rust"
+                                }
+                        },
+                        "patterns": [
+                                {
+                                        "include": "#block_comment"
+                                },
+                                {
+                                        "include": "#line_comment"
+                                },
+                                {
+                                        "include": "#sigils"
+                                },
+                                {
+                                        "include": "#self"
+                                },
+                                {
+                                        "include": "#mut"
+                                },
+                                {
+                                        "include": "#dyn"
+                                },
+                                {
+                                        "include": "#impl"
+                                },
+                                {
+                                        "include": "#ref_lifetime"
+                                },
+                                {
+                                        "include": "#core_types"
+                                },
+                                {
+                                        "include": "#core_marker"
+                                },
+                                {
+                                        "include": "#core_traits"
+                                },
+                                {
+                                        "include": "#std_types"
+                                },
+                                {
+                                        "include": "#std_traits"
+                                },
+                                {
+                                        "include": "#type_params"
+                                },
+                                {
+                                        "include": "#const"
+                                },
+                                {
+                                        "include": "#where"
+                                },
+                                {
+                                        "include": "#unsafe"
+                                },
+                                {
+                                        "comment": "Function arguments",
+                                        "match": "\bfn\b",
+                                        "name": "keyword.other.fn.rust"
+                                }
+                        ]
+                },
+                {
+                        "comment": "Type declaration",
+                        "begin": "\\b(enum|struct|trait|union)\\s+([a-zA-Z_][a-zA-Z0-9_]*)",
+                        "end": "[\\{\\(;]",
+                        "beginCaptures": {
+                                "1": {
+                                        "name": "storage.type.rust"
+                                },
+                                "2": {
+                                        "name": "entity.name.type.rust"
+                                }
+                        },
+                        "patterns": [
+                                {
+                                        "include": "#block_comment"
+                                },
+                                {
+                                        "include": "#line_comment"
+                                },
+                                {
+                                        "include": "#core_traits"
+                                },
+                                {
+                                        "include": "#std_traits"
+                                },
+                                {
+                                        "include": "#type_params"
+                                },
+                                {
+                                        "include": "#core_types"
+                                },
+                                {
+                                        "include": "#pub"
+                                },
+                                {
+                                        "include": "#where"
+                                }
+                        ]
+                },
+                {
+                        "comment": "Type alias",
+                        "begin": "\\b(type)\\s+([a-zA-Z_][a-zA-Z0-9_]*)",
+                        "end": ";",
+                        "beginCaptures": {
+                                "1": {
+                                        "name": "storage.type.rust"
+                                },
+                                "2": {
+                                        "name": "entity.name.type.rust"
+                                }
+                        },
+                        "patterns": [
+                                {
+                                        "include": "#block_comment"
+                                },
+                                {
+                                        "include": "#line_comment"
+                                },
+                                {
+                                        "include": "#sigils"
+                                },
+                                {
+                                        "include": "#mut"
+                                },
+                                {
+                                        "include": "#dyn"
+                                },
+                                {
+                                        "include": "#impl"
+                                },
+                                {
+                                        "include": "#lifetime"
+                                },
+                                {
+                                        "include": "#ref_lifetime"
+                                },
+                                {
+                                        "include": "#core_types"
+                                },
+                                {
+                                        "include": "#core_marker"
+                                },
+                                {
+                                        "include": "#core_traits"
+                                },
+                                {
+                                        "include": "#std_types"
+                                },
+                                {
+                                        "include": "#std_traits"
+                                },
+                                {
+                                        "include": "#type_params"
+                                }
+                        ]
+                }
+        ],
+        "repository": {
+                "block_doc_comment": {
+                        "comment": "Block documentation comment",
+                        "name": "comment.block.documentation.rust",
+                        "begin": "/\\*[\\*!](?![\\*/])",
+                        "end": "\\*/",
+                        "patterns": [
+                                {
+                                        "include": "#block_doc_comment"
+                                },
+                                {
+                                        "include": "#block_comment"
+                                }
+                        ]
+                },
+                "block_comment": {
+                        "comment": "Block comment",
+                        "name": "comment.block.rust",
+                        "begin": "/\\*",
+                        "end": "\\*/",
+                        "patterns": [
+                                {
+                                        "include": "#block_doc_comment"
+                                },
+                                {
+                                        "include": "#block_comment"
+                                }
+                        ]
+                },
+                "line_doc_comment": {
+                        "comment": "Single-line documentation comment",
+                        "name": "comment.line.documentation.rust",
+                        "begin": "//[!/](?=[^/])",
+                        "end": "$"
+                },
+                "line_comment": {
+                        "comment": "Single-line comment",
+                        "name": "comment.line.double-slash.rust",
+                        "begin": "//",
+                        "end": "$"
+                },
+                "escaped_character": {
+                        "name": "constant.character.escape.rust",
+                        "match": "\\\\(x[0-9A-Fa-f]{2}|[0-2][0-7]{0,2}|3[0-6][0-7]?|37[0-7]?|[4-7][0-7]?|.)"
+                },
+                "string_literal": {
+                        "comment": "Double-quote string literal",
+                        "name": "string.quoted.double.rust",
+                        "begin": "b?\"",
+                        "end": "\"",
+                        "patterns": [
+                                {
+                                        "include": "#escaped_character"
+                                }
+                        ]
+                },
+                "raw_string_literal": {
+                        "comment": "Raw double-quote string literal",
+                        "name": "string.quoted.double.raw.rust",
+                        "begin": "b?r(#*)\"",
+                        "end": "\"\\1"
+                },
+                "sigils": {
+                        "comment": "Sigil",
+                        "name": "keyword.operator.sigil.rust",
+                        "match": "[&*](?=[a-zA-Z0-9_\\(\\[\\|\\\"]+)"
+                },
+                "self": {
+                        "comment": "Self variable",
+                        "name": "variable.language.rust",
+                        "match": "\\bself\\b"
+                },
+                "mut": {
+                        "comment": "Mutable storage modifier",
+                        "name": "storage.modifier.mut.rust",
+                        "match": "\\bmut\\b"
+                },
+                "dyn": {
+                        "comment": "Dynamic modifier",
+                        "name": "storage.modifier.dyn.rust",
+                        "match": "\\bdyn\\b"
+                },
+                "impl": {
+                        "comment": "Existential type modifier",
+                        "name": "storage.modifier.impl.rust",
+                        "match": "\\bimpl\\b"
+                },
+                "box": {
+                        "comment": "Box storage modifier",
+                        "name": "storage.modifier.box.rust",
+                        "match": "\\bbox\\b"
+                },
+                "const": {
+                        "comment": "Const storage modifier",
+                        "name": "storage.modifier.const.rust",
+                        "match": "\\bconst\\b"
+                },
+                "pub": {
+                        "comment": "Visibility modifier",
+                        "name": "storage.modifier.visibility.rust",
+                        "match": "\\bpub\\b"
+                },
+                "unsafe": {
+                        "comment": "Unsafe code keyword",
+                        "name": "keyword.other.unsafe.rust",
+                        "match": "\\bunsafe\\b"
+                },
+                "where": {
+                        "comment": "Generic where clause",
+                        "name": "keyword.other.where.rust",
+                        "match": "\\bwhere\\b"
+                },
+                "lifetime": {
+                        "comment": "Named lifetime",
+                        "name": "storage.modifier.lifetime.rust",
+                        "match": "'([a-zA-Z_][a-zA-Z0-9_]*)\\b"
+                },
+                "ref_lifetime": {
+                        "comment": "Reference with named lifetime",
+                        "match": "(&)('[a-zA-Z_][a-zA-Z0-9_]*)\\b",
+                        "captures": {
+                                "1": {
+                                        "name": "keyword.operator.sigil.rust"
+                                },
+                                "2": {
+                                        "name": "storage.modifier.lifetime.rust"
+                                }
+                        }
+                },
+                "core_types": {
+                        "comment": "Built-in/core type",
+                        "name": "support.type.primitive",
+                        "match": "\\b(bool|char|usize|isize|u8|u16|u32|u64|u128|i8|i16|i32|i64|i128|f32|f64|str|Self)\\b"
+                },
+                "core_vars": {
+                        "comment": "Core type variant",
+                        "name": "support.constant.core.rust",
+                        "match": "\\b(Some|None|Ok|Err)\\b"
+                },
+                "core_marker": {
+                        "comment": "Core trait (marker)",
+                        "name": "entity.name.type.marker.rust",
+                        "match": "\\b(Copy|Send|Sized|Sync)\\b"
+                },
+                "core_traits": {
+                        "comment": "Core trait",
+                        "name": "entity.name.type.core.rust",
+                        "match": "\\b(Drop|Fn|FnMut|FnOnce|Clone|PartialEq|PartialOrd|Eq|Ord|AsRef|AsMut|Into|From|Default|Iterator|Extend|IntoIterator|DoubleEndedIterator|ExactSizeIterator)\\b"
+                },
+                "std_types": {
+                        "comment": "Standard library type",
+                        "name": "entity.name.type.class.std.rust",
+                        "match": "\\b(Box|String|Vec|Path|PathBuf|Option|Result)\\b"
+                },
+                "std_traits": {
+                        "comment": "Standard library trait",
+                        "name": "entity.name.type.std.rust",
+                        "match": "\\b(ToOwned|ToString)\\b"
+                },
+                "type": {
+                        "comment": "A type",
+                        "name": "entity.name.type.rust",
+                        "match": "\\b([A-Za-z][_A-Za-z0-9]*|_[_A-Za-z0-9]+)\\b"
+                },
+                "type_params": {
+                        "comment": "Type parameters",
+                        "name": "meta.type_params.rust",
+                        "begin": "<(?![=<])",
+                        "end": "(?<![-])>",
+                        "patterns": [
+                                {
+                                        "include": "#block_comment"
+                                },
+                                {
+                                        "include": "#line_comment"
+                                },
+                                {
+                                        "include": "#sigils"
+                                },
+                                {
+                                        "include": "#mut"
+                                },
+                                {
+                                        "include": "#dyn"
+                                },
+                                {
+                                        "include": "#impl"
+                                },
+                                {
+                                        "include": "#lifetime"
+                                },
+                                {
+                                        "include": "#core_types"
+                                },
+                                {
+                                        "include": "#core_marker"
+                                },
+                                {
+                                        "include": "#core_traits"
+                                },
+                                {
+                                        "include": "#std_types"
+                                },
+                                {
+                                        "include": "#std_traits"
+                                },
+                                {
+                                        "include": "#type_params"
+                                }
+                        ]
+                }
+        }
+}
\ No newline at end of file