Merge #2173

2173: MBE: Add TokenId shift in macro_rules r=matklad a=edwin0cheng

As discussed in #2169 , for fixing duplication TokenId during expansion :

> What we can do here is to re-number the tokens during expansion. Specifically:
> * when we create macro_rules, we note the highest id of the token we have as shift>
> * when we expand macro rules, if we need to output a token from definition, we just re-use its id
> * if we need to output a token from the argument, we increase its id by shift (so it's guaranteed to not to collide with anything from the definition)
> * finally, when we have a HirFileId of the expansion, we can look up the original value of shift and classify node to the arg/def by comparing it's id with shift.
> 

This PR implement first 3 points of above solution. 

Co-authored-by: Edwin Cheng <[email protected]>

1 files changed, 27 insertions, 0 deletions

diff --git a/crates/ra_mbe/src/tests.rs b/crates/ra_mbe/src/tests.rs
index f34ab52e1..a23e3afe3 100644
--- a/crates/ra_mbe/src/tests.rs
+++ b/crates/ra_mbe/src/tests.rs
@@ -59,6 +59,33 @@ mod rule_parsing {
 // * Make it pass :-)
 
 #[test]
+fn test_token_id_shift() {
+    let macro_definition = r#"
+macro_rules! foobar {
+    ($e:ident) => { foo bar $e }
+}
+"#;
+    let rules = create_rules(macro_definition);
+    let expansion = expand(&rules, "foobar!(baz);");
+
+    fn get_id(t: &tt::TokenTree) -> Option<u32> {
+        if let tt::TokenTree::Leaf(tt::Leaf::Ident(ident)) = t {
+            return Some(ident.id.0);
+        }
+        None
+    }
+
+    assert_eq!(expansion.token_trees.len(), 3);
+    // ($e:ident) => { foo bar $e }
+    //   0 1            2   3   4
+    assert_eq!(get_id(&expansion.token_trees[0]), Some(2));
+    assert_eq!(get_id(&expansion.token_trees[1]), Some(3));
+
+    // So baz should be 5
+    assert_eq!(get_id(&expansion.token_trees[2]), Some(5));
+}
+
+#[test]
 fn test_convert_tt() {
     let macro_definition = r#"
 macro_rules! impl_froms {