JSR-45: better support for debuging inlined calls

[Kordyjan]

Fixes #9715
This implements the same idea as scala/scala#9121, although due to differences in how inlining is handled in scala 2, the implementation for scala 3 is not using any code from the mentioned PR.

More info about SourceDebugExtension and SMAPs can be found here:

• http://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.7.11
• https://jcp.org/aboutJava/communityprocess/final/jsr045/index.html

How it works:

• For every inlined call it assumes that empty lines are appended to the source file. These lines are not added anywhere in physical form. We only assume that they exist only to be used by LineNumberTable and SourceDebugExtension. The number of these virtual lines is every time equal to the size of the line range of the expansion of inlined call.
• It generates SMAP (as defined by JSR-45) containing two strata. The first stratum (Scala) is describing the mapping from the real source files to the real and virtual lines in our assumed source. The second stratum (ScalaDebug) is mapping from virtual lines to corresponding inlined calls.
• Generated SMAP is written to the bytecode in SourceDebugExtension
• During the generation of the bytecode backed is asking InlinedSourceMap about the position of all trees that have source different from the main source of the given compilation unit. The response to that request is the number of the virtual line that is corresponding to the particular line from the other source.
• Debuggers can use information stored in LineNumberTable and SourceDebugExtension to correctly guess what line of the inlined method is currently executed. They can also construct stack frames for inlined calls.

Example:

Let's assume we have two source files, each containing one function:

def usage(): Unit =
  inlined(5)
  println("x")
  inlined(6)

and

inline def inlined(arg: Int) =
  println("start")
  println(arg)
  println("stop")

The classfile generated for the first sourcefile contains these two pieces:

LineNumberTable:
        line 2: 0
        line 5: 0
        line 6: 8
        line 2: 11
        line 7: 18
        line 3: 26
        line 8: 34
        line 9: 42
        line 4: 45
        line 10: 53

and

SourceDebugExtension:
  SMAP
  Usage.scala
  Scala
  *S Scala
  *F
  1 Usage.scala
  + 2 Inlined.scala
  Inlined$package$
  *L
  1,4:1
  2#2,3:5
  2#2,3:8
  *E
  *S ScalaDebug
  *F
  1 Usage.scala
  *L
  2:5,3
  4:8,3
  *E

The file contains two inlined calls, each having three lines of expansion. We assume then that its source has 10 lines: 4 - corresponding to the original file and 3 for each inline call. Let's analyze the LineNumberTable:

 2     <- start of the `usage` method
 5     <- we are in the inlined call so we jump to the virtual line
 6     <- next line of the first inlined call
 2     <- we jump back to the original method to evaluate the argument of inlined call (`iconst_5` in this example)
 7     <- we continue in the inlined call
 3     <- we execute the next line of the `usage` method
 8     <- second inlined call
 9     <- continue the second inlined call
 4     <- evaluation of the second inlined call argument
10     <- we are finishing the second inlined call

Then let's take a look at the Scala stratum of our SMAP. It contains two files: 1 is Usage.scala which is the main source of this classfile. Number 2 is Inlined.scala that contains the definition of inlined calls expansions. It contains three mappings:

• 1,4:1 is describing lines 1-4 as lines 1-4 from the original file
• 2#2,3:5 is describing lines 5-7 as lines 2-4 from file number 2 (the first inlined call)
• 2#2,3:8 is describing lines 8-10 again as lines 2-4 from file number 2 (the second inlined call)

The ScalaDebug stratum contains mappings necessary for the construction of stack frames for inlined calls in the debugger. It contains two mappings:

• 2:5,3 is mapping line 2 to lines 5-7 which can be interpreted by debugger as lines 5-7 being a body of a function called from line 2
• 4:8,3 is mapping line 4 to lines 8-10 which can be interpreted by debugger as lines 8-10 being a body of a function called from line 4

Notes about current implementation:

1. I removed YCheckPosition from the compiler pipeline as now we can represent in the bytecode positions coming from other sources. Should I repurpose it or should it be completely removed?
2. There is a bug that if we compile code with two exactly the same calls of inline methods one after another without any bytecode generated between them the compiler will output slightly incorrect LineNumberTable. It will look like we are reentering the first call where we should enter the second one. The only solutions I can think about right now involve recursive applying of the same attachment to all trees in inlined call expansion or making BCodeSkelBuilder.lineNumber logic much more complicated by adding some context tracking entered inlined calls.
3. I want to provide some automatic tests for SMAPs and line tables that would take a bunch of files, compile them, then compares pieces of generated bytecode with some expected output. How should I approach this using current test architecture?
4. The implementation of SMAP for Scala 2 was using internal class names as paths. There is no 1-1 mapping between them and source files, but we can assume that debuggers will be using this information only to locate the source of a particular classfile. In this case, we should be ok providing any name of the class that is generated from the target source. For now, I reimplemented it as it was in scala 2 for backward compatibility sake, but I think it may be beneficial to think about any other way of providing paths to the sources in SMAPs. The generation of mappings between source files and internal class names is really cumbersome since we need to attach class symbols in the Erasure phase and then process them only during the generation of the bytecode. Removing this will simplify the code.

[nicolasstucki]

Maybe try

Suggested change

	val topLevelClass = Some(inlined.call.symbol.topLevelClass).filter(_.exists)
	val topLevelClass =
	if inlined.call.isEmpty then None
	else Some(inlined.call.symbol.topLevelClass)

In theory, the symbol would not exist only if the inlined.call is the EmptyTree.

[smarter]

/cc @errikos who might want to have a look

[anatoliykmetyuk]

@Kordyjan what's the status on this one? Are you planning to continue working on it or can it be closed?

[Kordyjan]

Are you planning to continue working on it or can it be closed?

Haven't had time for that recently, but I definitely plan to continue my work on this solution.

[lrytz]

Nice to see this being worked on here 🙏

One idea for testing would be to use the JDI API; from a comment I made on the Scala 2 PR: https://gist.github.com/lrytz/a6a139491e3dad997c2d78d99b5bf504

[smarter]

One idea for testing would be to use the JDI API;

That could be nice yes. We don't have that currently but we do have some tests that just check the behavior of jdb, with infrastructure in https://github.com/lampepfl/dotty/tree/master/compiler/test/debug and actual tests in https://github.com/lampepfl/dotty/tree/master/tests/debug