Handling of Room & Kotlin symbols

△ Picture from Unsplash provided by Marc Reichelt

Jetpack Room library provides an abstraction layer on SQLite, which can provide the ability to verify SQL queries at compile time without any boilerplate code. It achieves the above behavior by processing code annotations and generating Java source code.

Annotation processors are very powerful, but they will increase build time. This is usually acceptable for code written in Java, but for Kotlin, the compile time consumption will be very obvious, because Kotlin does not have a built-in annotation processing pipeline. Instead, it generates stub Java code through Kotlin code to support the annotation processor, and then sends it to the Java compiler for processing.

Since not all the content in the Kotlin source code can be expressed in Java, some information will be lost in this conversion. Similarly, Kotlin is a multi-platform language, but KAPT only works for Java bytecode.

Recognize Kotlin Symbol Processing

With the widespread use of annotation processors on Android, KAPT has become a performance bottleneck at compile time. In order to solve this problem, the Google Kotlin compiler team began to study an alternative to provide first-class annotation processing support for Kotlin. When this project was born, we were very excited because it will help Room to better support Kotlin. Starting from Room 2.4, it has experimental support for KSP. We found that the compilation speed has increased by 2 times, especially in the case of full compilation.

The content of this article does not focus on the processing of annotations, Room or KSP. Instead, it focuses on the challenges and trade-offs we faced when adding KSP support to Room. In order to understand this article, you do not need to know Room or KSP, but you must be familiar with annotation processing.

Note: We started using it before the stable version of KSP was released. Therefore, it is not yet certain whether some of the decisions made before are applicable to the present.

This article aims to allow the authors of annotation processors to fully understand the issues that need attention before adding KSP support to their projects.

Introduction to the working principle of Room

The annotation processing of Room is divided into two steps. There are some "Processor" classes that traverse the user's code, verify and extract the necessary information into the "value object". These value objects are sent to the "Writer" class, which converts them into code. Like many other annotation processors, Room relies heavily on the Auto-Common and packages (Java annotation processing API packages).

In order to support KSP, we have three options:

1. Copy each "Processor" class of JavaAP and KSP, they will have the same value object as output, we can input it into Writer;
2. Create an abstraction layer on top of KSP/Java AP, so that the processor has an implementation based on the abstraction layer;
3. Use KSP instead of JavaAP, and require developers to also use KSP to process Java code.

Option C is actually not feasible because it will cause serious interference to Java users. As the number of rooms used increases, such disruptive changes are impossible. Between "A" and "B", we decided to choose "B" because the processor has a considerable amount of business logic and it is not easy to decompose it.

Know X-Processing

Creating a common abstraction on JavaAP and KSP is not easy. Kotlin and Java can interoperate, but the mode is different, for example, the types of special classes in Kotlin such as Kotlin value classes or static methods in Java. In addition, Java classes have fields and methods, while Kotlin has properties and functions.

We decided to realize " room need 1618b43cbb26cb" instead of trying to pursue perfect abstraction. Literally, find javax.lang.model in Room and move it to the abstraction of X-Processing. In this way, TypeElement becomes XTypeElement , ExecutableElemen becomes XExecutableElemen and so on.

Unfortunately, the javax.lang.model API is very widely used in Room. Creating all these X classes at once will bring a very serious psychological burden to reviewers. Therefore, we need to find a way to iterate this implementation.

On the other hand, we need to prove that this is feasible. So we first made the prototype design, once verified that this is a reasonable choice, we used their own test to re-implement all X class one by one.

There is a good example about the realization of "what is needed for Room" that I said, we can see in the field change about the class. When Room deals with the fields of a class, it is always interested in all its fields, including the fields in the parent class. So when we created the corresponding X-Processing API, only added the ability to get all fields.

interface XTypeElement {
  fun getAllFieldsIncludingPrivateSupers(): List<XVariableElement>
}

If we are designing a common library, it may never pass the API review. But because our goal is only Room, and it already has an TypeElement , copying it can reduce the risk of the project.

Once we have the basic X-Processing APIs and their test methods, the next step is for Room to call this abstraction. This is also the place where "what is needed to realize the Room" is rewarded well. Room already has extended functions/attributes for basic functions on the javax.lang.model API (for example, the method to TypeElement We first updated these extensions to look similar to the X-Processing API, and then migrated Room to X-Processing 1 CL

Improve API usability

Keeping a JavaAP-like API does not mean that we can't improve anything. After migrating Room to X-Processing, we implemented a series of API improvements.

For example, Room calls MoreElement/MoreTypes multiple times to convert between different javax.lang.model types (for example, MoreElements.asType ). The related calls usually look like this:

val element: Element ...
if (MoreElements.isType(element)) {
  val typeElement:TypeElement = MoreElements.asType(element)
}

We put all the calls to Kotlin contracts , so that it can be written as:

val element: XElement ...
if (element.isTypeElement()) {
  // 编译器识别到元素是一个 XTypeElement
}

Another good example is to find a way TypeElement Usually in JavaAP, you need to call ElementFilter class to get the methods in TypeElement. In contrast, we directly set it as an attribute in XTypeElement.

// 前
val methods = ElementFilter.methodsIn(typeElement.enclosedElements)
// 后
val methods = typeElement.declaredMethods

The last example, which may also be one of my favorite examples, is distributable. In JavaAP, if you want to check whether a given TypeMirror can be assigned by another TypeMirror, you need to call Types.isAssignable .

val type1: TypeMirror ...
val type2: TypeMirror ...
if (typeUtils.isAssignable(type1, type2)) {
  ...
}

This code is really hard to read, because you can't even guess whether it verifies that type 1 can be specified by type 2, or the opposite result. We already have an extension function as follows:

fun TypeMirror.isAssignableFrom(
  types: Types,
  otherType: TypeMirror
): Boolean

In X-Processing, we can convert it to a regular function on XType

interface XType {
  fun isAssignableFrom(other: XType): Boolean
}

Implement KSP backend for X-Processing

Each of these X-Processing interfaces has its own test suite. We are not writing them to test AutoCommon or JavaAP, on the contrary, they are written so that when we have their KSP implementation, we can run test cases to verify whether it meets Room's expectations.

Since the original X-Processing API is avax.lang.model , they are not always suitable for KSP, so we have also improved these APIs to provide better support for Kotlin when needed.

This creates a new problem. The existing Room code base was written to handle Java source code. When the application is written in Kotlin, Room can only recognize the appearance of the Kotlin in the Java stub. We decided to maintain similar behavior in the KSP implementation of X-Processing.

For example, the suspend function in Kotlin generates the following signature at compile time:

// kotlin
suspend fun foo(bar:Bar):Baz
// java
Object foo(bar:Bar, Continuation<? extends Baz>)

In order to maintain the same behavior, XMethodElement KSP is implemented as a suspend method to synthesize a new parameter and a new return type. ( KspMethodElement.kt )

Note: This works well, because Room generates Java code, even in KSP. When we add support for Kotlin code generation, some changes may be caused.

Another example is related to attributes. Kotlin attributes may also have a composition getter/setter (accessor) based on their signature. Since Room expects to find these accessors as methods (see: KspTypeElement.kt ), XTypeElement implements these synthesis methods.

Note : We have plans to change the XTypeElement API to provide attributes instead of fields, because this is what Room really wants to get. As you guessed right now, we decided not to do this "for now" to reduce Room modifications. Hopefully, one day we can do this. When we do, the JavaAP implementation of XTypeElement will bundle methods and fields together as properties.

When adding KSP implementations to X-Processing, the last interesting issue is API coupling. These processors API regularly visit each other, so if you do not realize XField / XMethod , can not be achieved in KSP in XTypeElement , and XField / XMethod itself quoted XType and so on. While adding these KSP implementations, we wrote separate test cases for their implementation. When the implementation of KSP becomes more complete, we gradually start all X-Processing tests through the KSP backend.

It should be noted that at this stage we only run tests in the X-Processing project, so even if we know that the content of the test is okay, we cannot guarantee that all Room tests will pass (also called unit tests vs integration tests) ). We need a way to run all Room tests using the KSP backend, and "X-Processing-Testing" came into being.

Know X-Processing-Testing

The preparation of the annotation processor contains 20% of the processor code and 80% of the test code. You need to consider all possible developer errors and make sure to report error messages truthfully. In order to write these tests, Room has provided an auxiliary method as follows:

fun runTest(
  vararg javaFileObjects: JavaFileObject,
  process: (TestInvocation) -> Unit
): CompilationResult

runTest Google Compile Testing library at the bottom, and allows us to simply unit test the processor. It synthesizes a Java annotation processor and calls the process method provided by the processor in it.

val entitySource : JavaFileObject //示例 @Entity 注释类
val result = runTest(entitySource) { invocation ->
  val element = invocation.processingEnv.findElement("Subject")
  val entityValueObject = EntityProcessor(...).process(element)
  // 断言 entityValueObject
}
// 断言结果是否有误，警告等

Unfortunately, Google Compile Testing only supports Java source code. In order to test Kotlin we need another library, fortunately there is Kotlin Compile Testing , which allows us to write tests for Kotlin, and we have contributed KSP support for this library.

Note : We later internal implementation to simplify the Kotlin/KSP update in AndroidX Repo. We also added a better assertion API, which requires us to perform API-incompatible modification operations on KCT.

As the last step to allow KSP to run all tests, we created the following test API:

fun runProcessorTest(
  sources: List<Source>,
  handler: (XTestInvocation) -> Unit
): Unit

The main difference between this and the original version is that it runs tests through both KSP and JavaAP (or KAPT, depending on the source). Because it runs the test multiple times and the judgment results of KSP and JavaAP are different, it cannot return a single result.

Therefore, we thought of a way:

fun XTestInvocation.assertCompilationResult(
  assertion: (XCompilationResultSubject) -> Unit
}

After each compilation, it will call the result assertion (if there is no failure prompt, check whether the compilation was successful). We refactored each Room test as follows:

val entitySource : Source //示例 @Entity 注释类
runProcessorTest(listOf(entitySource)) { invocation ->
  // 该代码块运行两次，一次使用 JavaAP/KAPT，一次使用 KSP
  val element = invocation.processingEnv.findElement("Subject")
  val entityValueObject = EntityProcessor(...).process(element)
  //  断言 entityValueObject
  invocation.assertCompilationResult {
    // 结果被断言为是否有 error，warning 等
    hasWarningContaining("...")
  }
}

The next thing is very simple. Migrate the compilation test of each Room to the new API. Once a new KSP / X-Processing error is found, it will be reported, and then a temporary solution will be implemented; this action is repeated. Since KSP is under vigorous development, we did encounter a lot of bugs. Every time we report a bug, link to it from the Room source, and then move on (or fix it). Whenever the KSP is released, we will search the code base to find the fixed problems, delete the temporary solutions and start the test.

Once the compilation test coverage is good, we will use KSP to run Room's integration test in the next step. These are actual Android test applications, and their behavior will also be tested at runtime. Fortunately, Android supports Gradle variants, so using KSP and KAPT to run our Kotlin integration test is quite easy.

Next

Adding KSP support to the Room is only the first step. Now, we need to update Room to use it. For example, all type checks in Room ignore nullability because javax.lang.model of TypeMirror does not understand nullability . Therefore, when calling your Kotlin code, Room sometimes triggers NullPointerException at runtime. With KSP, these checks can now create new KSP bugs in the Room (for example, b/193437407 ). We have added some temporary solutions, but ideally, we still want improve Room to handle these situations correctly.

Similarly, even if we support KSP, Room still only generates Java code. This restriction prevents us from adding support for certain Kotlin features, such as Value Classes . Hope that in the future, we can also provide some support for generating Kotlin code to provide first-class support for Kotlin in Room. Next, maybe more :).

Can I use X-Processing in my project?

The answer is not yet; at least it is different from the way you use any other Jetpack library. As mentioned earlier, we only implemented the parts required by Room. There is a lot of investment in writing a real Jetpack library, such as documentation, API stability, Codelabs, etc. We can't afford these tasks. Having said that, Dagger and Airbnb ( Paris , DeeplinkDispatch ) both started to use X-Processing to support KSP (and contributed what they needed 🙏). Maybe one day we will break it out of the Room. Technically, you can still Google Maven library , but there is no API guarantee to do so, so you should definitely use the shade technology.

Summary

We added KSP support to Room, which is not easy but it is definitely worth it. If you are maintaining an annotation processor, please add support for KSP to provide a better Kotlin developer experience.

Special thanks to Zac Sweers and Eli Hart the early version of this article. They are also excellent KSP contributors.

More resources

• About Room supported by KSP Issue Tracker
• X-Processing source code
• X-Processing-Testing source code
• KSP source code

You are welcome to click here to submit feedback to us, or share your favorite content or problems found. Your feedback is very important to us, thank you for your support!