Skip to content

Commit 2c44e7e

Browse files
committed
Fix token incompatibility for Prism::Translation::Parser::Lexer
This PR fixes token incompatibility for `Prism::Translation::Parser::Lexer` when using escaped backslash in string literal: ```ruby "\\ foo \\ bar" ``` ## Parser gem (Expected) ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Parser::Ruby33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] [s(:str, "\\ foo \\ bar"), [], [[:tSTRING, ["\\ foo \\ bar", #<Parser::Source::Range example.rb 0...15>]], [:tNL, [nil, #<Parser::Source::Range example.rb 15...16>]]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the tokens returned by the Parser gem were different. The escaped backslash does not match in the `tSTRING` token: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] [s(:str, "\\ foo \\ bar"), [], [[:tSTRING, ["\\\\ foo \\\\ bar", #<Parser::Source::Range example.rb 0...15>]], [:tNL, [nil, #<Parser::Source::Range example.rb 15...16>]]]] ``` After this correction, the AST and tokens returned by the Parser gem are the same: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] [s(:str, "\\ foo \\ bar"), [], [[:tSTRING, ["\\ foo \\ bar", #<Parser::Source::Range example.rb 0...15>]], [:tNL, [nil, #<Parser::Source::Range example.rb 15...16>]]]] ``` The reproduction test is based on the following strings.txt and exists: https://github.com/ruby/prism/blob/v0.24.0/test/prism/fixtures/strings.txt#L79 But, the restoration has not yet been performed due to remaining other issues in strings.txt.
1 parent da1d484 commit 2c44e7e

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

lib/prism/translation/parser/lexer.rb

+1-1
Original file line numberDiff line numberDiff line change
@@ -284,7 +284,7 @@ def to_a
284284
elsif ["\"", "'"].include?(value) && (next_token = lexed[index][0]) && next_token.type == :STRING_CONTENT && next_token.value.lines.count <= 1 && (next_next_token = lexed[index + 1][0]) && next_next_token.type == :STRING_END
285285
next_location = token.location.join(next_next_token.location)
286286
type = :tSTRING
287-
value = next_token.value
287+
value = next_token.value.gsub("\\\\", "\\")
288288
location = Range.new(source_buffer, offset_cache[next_location.start_offset], offset_cache[next_location.end_offset])
289289
index += 2
290290
elsif value.start_with?("<<")

0 commit comments

Comments
 (0)