Age | Commit message (Collapse) | Author |
|
This can get triggered even if the list of statements only contains
a single statement. This is necessary to properly support compiling
```ruby
defined? (;a)
defined? (a;)
```
as "expression". Previously these were parsed as statements lists
with single statements in them.
https://github.com/ruby/prism/commit/b63b5d67a9
|
|
```
(a,), = []
PARSER====================
s(:masgn,
s(:mlhs,
s(:mlhs,
s(:lvasgn, :a))),
s(:array))
PRISM====================
s(:masgn,
s(:mlhs,
s(:lvasgn, :a)),
s(:array))
```
https://github.com/ruby/prism/commit/8aa1f4690e
|
|
In https://github.com/ruby/prism/commit/26370079291a420c6b2b7be5cdbd5c609da62f21 I added tests but didn't modify them correctly
https://github.com/ruby/prism/commit/de021e74de
|
|
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation
https://github.com/ruby/prism/commit/4edfe9d981
|
|
https://github.com/ruby/prism/commit/422d5c4c64
|
|
`Integer#chr` performs some validation that we don't want/need. Octal escapes can go above 255, where it will then raise trying to convert.
`append_as_bytes` actually allows to pass a number, so we can just skip that call.
Although, on older rubies of course we still need to handle this in the polyfill.
I don't really like using `pack` but don't know of another way to do so.
For the utf-8 escapes, this is not an issue. Invalid utf-8 in these is simply a syntax error.
https://github.com/ruby/prism/commit/161c606b1f
|
|
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation
https://github.com/ruby/prism/commit/4edfe9d981
|
|
Also fixes a token incompatibility for the word separator. parser only considers whitespace until the first newline
https://github.com/ruby/prism/commit/bd3dd2b62a
|
|
When the line contains no real newline but contains unescaped ones, then there will be one less entry
https://github.com/ruby/prism/commit/4ef093b600
|
|
translator
This is a followup to #3373, where the implementation
was extracted
https://github.com/ruby/prism/commit/2637007929
|
|
The offset cache contains an entry for each byte so it can't be accessed via the string length.
Adds tests for all variants except for this:
```
"fo
o" "ba
’"
```
For some reason, this still has the wrong offset.
https://github.com/ruby/prism/commit/a651126458
|
|
|
|
1. The string starts out as binary
2. `ち` is appended, forcing it back into utf-8
3. Some invalid byte sequences are tried to append
> incompatible character encodings: UTF-8 and BINARY (ASCII-8BIT)
This makes use of my wish to use `append_as_bytes`. Unfortunatly that method is rather new
so it needs a fallback
https://github.com/ruby/prism/commit/e31e94a775
|
|
Fixes [Bug #21137]
https://github.com/ruby/prism/commit/ca493e6797
|
|
Fixes [Bug #21048]
https://github.com/ruby/prism/commit/07202005cb
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/12574
|
|
https://github.com/ruby/prism/commit/7a93a307ac
|
|
In https://github.com/ruby/prism/pull/3393 I made a mistake.
When there is no previous token, it wraps around to -1. Oops
Additionally, if a comment has no newline then the offset should be kept as is
https://github.com/ruby/prism/commit/3c266f1de4
|
|
Skipping detecting the encoding is almost always right, just for binary it should actually happen.
A symbol containing escapes that are invalid
in utf-8 would fail to parse since symbols must be valid in the script encoding.
Additionally, the parser gem would raise an exception somewhere during string handling
https://github.com/ruby/prism/commit/fa0154d9e4
|
|
|
|
https://github.com/ruby/prism/commit/ab9d80987a
|
|
|
|
Fixes [Bug #20744]
https://github.com/ruby/prism/commit/f1b8b1b2a2
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11624
|
|
https://github.com/ruby/prism/commit/ebd2889bee
|
|
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for left parenthesis.
## Parser gem (Expected)
Returns `tLPAREN2` token:
```console
$ bundle exec ruby -Ilib -rparser/ruby33 \
-ve 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]],
[:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]]
```
## `Prism::Translation::Parser` (Actual)
Previously, the parser returned `tLPAREN` token when parsing the following:
```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN, ["(", #<Parser::Source::Range example.rb 3...4>]],
[:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]]
```
After the update, the parser now returns `tLPAREN2` token for the same input:
```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]],
[:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]]
```
The `PARENTHESIS_LEFT` token in Prism is classified as either `tLPAREN` or `tLPAREN2` in the Parser gem.
The tokens that were previously all classified as `tLPAREN` are now also classified to `tLPAREN2`.
With this change, the following code could be removed from `test/prism/ruby/parser_test.rb`:
```diff
- when :tLPAREN
- actual_token[0] = expected_token[0] if expected_token[0] == :tLPAREN2
```
https://github.com/ruby/prism/commit/04d6f3478d
|
|
https://github.com/ruby/prism/commit/21cb9b78ae
|
|
Rename some fields that do not quite make sense.
* CaseMatchNode#consequent -> CaseMatchNode#else_clause
* CaseNode#consequent -> CaseNode#else_clause
* IfNode#consequent -> IfNode#subsequent
* RescueNode#consequent -> RescueNode#subsequent
* UnlessNode#consequent -> UnlessNode#else_clause
Notes:
Merged: https://github.com/ruby/ruby/pull/11480
|
|
If we have a splat, add a flag for it named `contains_splat`.
https://github.com/ruby/prism/commit/5be97a75c8
|
|
https://github.com/ruby/prism/commit/098b3f08bc
|
|
https://github.com/ruby/prism/commit/18a8597276
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11163
|
|
https://github.com/ruby/prism/commit/f4152c1f50
|
|
https://github.com/ruby/prism/commit/d0143865c2
|
|
https://github.com/ruby/prism/commit/9f12a56fd6
|
|
https://github.com/ruby/prism/commit/4cc0eda4ca
|
|
https://github.com/ruby/prism/commit/f09db18e46
|
|
https://github.com/ruby/prism/commit/aee84cf42c
|
|
https://github.com/ruby/prism/commit/a255653447
|
|
https://github.com/ruby/prism/commit/95e2484903
|
|
This eliminates the subnode on RationalNode and replaces it with two
integer fields, which represent the ratio for the rational. It also
reduces those two integers if they both fit into 32 bits.
Importantly, this PR does not implement bignum reduction. That's something
I'd like to consider for the future, but it's simple enough for now to
leave them unreduced, which makes it more useful than it used to be.
https://github.com/ruby/prism/commit/86e06c7068
|
|
https://github.com/ruby/prism/commit/b5e47f5c42
|
|
https://github.com/ruby/prism/commit/e950dc1e83
|
|
This has been requested for a long time, and I'm finally doing it
now. Unfortunately this is a breaking change for all of the APIs.
I've added in a Ruby method for `#child` that is deprecated so that
existing usage doesn't break, but for everyone else this is going
to be a bit of a pain.
https://github.com/ruby/prism/commit/9cbe74464e
|
|
If a single string that is a static literal is interpolated, it
does not impact whether or not the parent is a static literal. In
this way, if you have something like a regular expression that
interpolates a string literal, it's possible that you will end up
pushing just a single regexp onto the stack as opposed to calling
out to toregexp.
https://github.com/ruby/prism/commit/4f096c2257
|
|
https://github.com/ruby/prism/commit/450541d2c3
|
|
|
|
https://github.com/ruby/prism/commit/0d32af5719
|
|
https://github.com/ruby/prism/commit/da939bbd4e
|
|
https://github.com/ruby/prism/commit/a0b978d25b
|