Why is Flutter Fast? — Part 3: Rendering Pipeline

In many cases, using the set of widgets included in Flutter SDK would be enough to deliver a good user experience for end-users. However, modern application trends are going in a direction where designers and users demand more motion-rich, highly customizable UI components. Thanks to the pixel-driven architecture rather than relying on the available widgets in the platforms, Flutter targets high UI customizability and consistency across all platforms at the same level [1].

I believe implementing rich and quality UI components efficiently requires a good understanding of how the Flutter framework works. This is why my talk at the Flutter Vikings Conference “From Motion Design Specs To Flutter Code” starts with an explanation of Flutter’s rendering pipeline.

Rendering pipeline is not a Flutter-specific term. In computer graphics, it is used to define a model that consists of sequential steps for rendering objects to produce a frame. The end result of the rendering pipeline is the pixels we see in a window.

Since the rendering pipeline is a very detailed topic, we will mention only a portion in this article. For more detailed information, I would suggest reading this comprehensive blog post from Alibaba Clouder and watching the famous talk from Adam Barth, “Flutter’s Rendering Pipeline”.

• Part 1: Sublinear Building
• Part 2: Layered Architecture
• Part 3: Rendering Pipeline

As Flutter app developers, we have minimal roles in the rendering pipeline:

1. Compose a widget tree and provide it to the framework during the build phase.
2. Let the framework know which widgets need to be marked as “need builds” when we want to update the UI. For example by calling the setState method on the State object of a StatefulWidget with a gesture event, we implicitly mark the widget as needing build and its corresponding element as dirty for the next frame.

Performance of a UI toolkit

In the early days of the mobile application development, preparing a UI to users fast enough was the main concern [7]. As the devices have gotten more computation capabilities, the goals for the performance is shifted to different areas such as:

• Using the device resources efficiently so that the device has enough resources to perform background tasks while during the rendering in the UI thread
• Drawing 60 or120 frame per second performance for smooth animations which gives 16 or 8 milliseconds to draw a frame.
• Optimizing the usage of processors in the device so that the energy spent to draw a frame is decreased and and as a result battery is saved.

“Dirty Region Management has been a staple of UI frameworks for decades, but the state of the art has changed quite a bit from the early days of CPU rendering on chips clocked in MHz to the modern age of GHz GPU rendering on pretty much any device. [7]

Addressing these goals requires efficient UI rendering algorithms and an a good dirty region management. Flutter framework has already great solutions to prevent unnecessary computation in different phases of the rendering pipeline which we will mention throughout this article. However, the team is aware that there is still room for improvement. For example, in this document, you can read the discussion for minimizing the repaint area on each frame.

runApp

While I was working on this series of articles, I noticed this great video “Understanding the runApp function” from the Flutter team by Andrew Fitz Gibbon. It was great to see how he explores the source code as I did while I was trying to figure out how the framework works.

Flutter app developers start working with the widgets layer by calling the runApp method. It is the method that bootstraps the Flutter applications. Two important things are done in this method:

1. The root widget which is provided as a parameter to the runApp method is attached to the top of the widget tree. For the initial build, it is a linear top to down process which is mentioned in the first article as Case 5 for all the updateChild method calls. In this case, as all the widgets are inflated in the widget tree, their corresponding elements and render objects are created.
2. The Flutter framework layer and the engine layer are bound with the help of WidgetsFlutterBinding class. This singleton class is initialized in the runApp method. It contains two important manager classes for the rendering pipeline: BuildOwner and PipelineOwner.

Although the WidgetsFlutterBinding class looks like a God object in the framework, it implements a number of singleton mixins to separate the concerns in the implementation. Each mixin is a glue between the engine and the framework with a specific responsibility.

• GestureBinding is the binding for the gesture recognizers.
• SchedulerBinding is binding for the scheduler frame callbacks such as transient callbacks, persistent callbacks, post-frame callback, and non-rendering tasks in-between frames. It makes sure that the tasks are run only when appropriate.
• ServicesBinding is binding for the platform specific services exposed to the framework layer.
• PaintingBinding is binding for the Flutter painting library that wraps the engine’s painting API.
• SemanticsBinding is the binding for the semantics layer and the engine.
• WidgetsBinding is the binding between the widget tree and the engine. This is the mixin where the BuildOwner starts and finalizes the build phase.
• RendererBinding is the binding between the render tree and the engine. This is the mixin where the PipelineOwner executes the rendering pipeline phases between the start and the finalization of the build phase.

drawFrame

When it is time to produce and layout a frame, the engine calls the drawFrame method on the WidgetsFlutterBinding class which starts the rendering pipeline with the build phase.

Image taken from alibabacloud blog

The output of the rendering pipeline is the layer tree which is consumed in the raster thread that runs in the CPU for rasterizing and compositing. This thread was previously named as the GPU thread by the Flutter team, but then renamed to raster thread because it was misleading and giving impression that it was run in GPU. Skia runs on the raster thread and, the final output is translated into GPU render instructions to draw pixels on the screen.

The BuildOwner is the manager class for the widget tree. The elements that need build are added to the dirty elements list by calling the scheduleBuildFor method on the BuildOwner.

The PipelineOwner is the manager class for the render tree. After the build phase, it stores the dirty states for layout, compositing bits, paint, and semantics stages of the rendering pipeline. For example, if a RenderObject is marked as dirty for the layout phase, then the layout is computed for that RenderObject during the rendering of the frame.

“Anytime anything changes on a render object that would affect the layout of that object, it should call markNeedsLayout. ” [6]

list of dirty render objects in PipelineOwner

Single Pass Layout

In Flutter, the layout phase is unidirectional: from top to down and one-pass. The parent RenderObject calls layout method on the child render objects by providing them their constraints: maxWidth, minWidth, maxHeight, and minHeight values.

“Flutter performs one layout per frame.” [6]

As in the build phase, the layout phase has also linear performance initially, and sub-linear performance subsequently. Each child RenderObject is visited and returns its size only once during the layout pass until the next frame. Children have no right to size themselves beyond the provided constraints.

“Constraints go down. Sizes go up. Parent sets position.” [2]

The parent RenderObject visits and passes the constraints down in the render tree to its children one by one. For example, let’s think about a parent with a Flex layout that has two children: one flexible child and one inflexible child. In this case, the parent first lays out the inflexible child. The laid-out child passes its size up in the render tree to the parent RenderObject . After that, the parent calculates the free space for the flexible child. Finally, the flexible child will receive its constraints according to the remaining space and return its size up in the render tree to the parent RenderObject.

Constraints & Size direction in Flutter

“During layout, the only information that flows from parent to child are the constraints and the only information that flows from child to parent is the geometry.” [6]

In Flutter, positioning a child is controlled by the parent independently from the laying out process, and can’t be completed before laying out all the children. Some render objects do the positioning as late as possible. Even in some cases, the positioning is done in the paint phase.

Sub-linear performance with a single pass algorithm in build and layout phases is a significant optimization considering how it is done in different platforms natively. For example, drawing the layout is a two-pass process for the native Android apps: measure pass and layout pass. In the measure pass, the parent views call the measure method on the child views to find out how big the child views should be within the given constraints. When this method returns from a child view, the child view and all of its descendants measure themselves. Depending on the case, a parent may call the measure method on a child more than once. For example, in the first call, the parent gets the idea of how big or small the children would want to be if there were no constraints. Then, in the second call, the parent provides the actual constraints. The layout pass starts when the measure pass completes. In this pass, the parents position their children according to their desired measurements [3].

Wrap Up With Example

Let’s wrap up what we learned in this series of articles with an example. In the widget tree above, we have the root widget A which has two children: widget B and a stateful widget C. Inside the State object of the widget C , we declare a widget subtree by providing a child widget E on top.

Polluting the Tree

Let’s say that widget E’s child widget F is a GestureDetector widget where we call the setState method on its onTap callback method. In this case, the element C will be marked as needs build since setState method is called inside its State object. As a result, before the new frame starts, the element C will be added to the dirty element list.

Now, let’s go a bit deeper into what we mean by “dirt”. The State object of Widget C has access to its permanently associated element C. When the setState method is called on this state object, it will call the markNeedsBuild method of element C. Inside this method, the BuildOwner class will schedule a build for the element C.

By calling the scheduleBuildFor method for Element C, the element will be added to the list of dirty elements that the BuildOwner class keeps track of. These dirty elements will be cleaned during the next frame.

Drawing the New Frame

When the drawFrame method is called on the WidgetsFlutterBinding , the rendering pipeline for the next frame starts.

The build phase is triggered in the WidgetsBinding mixin of the WidgetsFlutterBinding instance by calling the buildScope method on the BuildOwner . Inside this method, all the dirty elements are rebuilt one by one.

Updating Child Elements

It is now time for the element C to examine its child elements by calling the updateChild method on them.

Case 3 — canUpdate returns true

In the first slot, the element E is examined. Looks like the new widget’s runtime type for that slot is again E , and let’s assume that the old and new widget E have the same key. Hence, canUpdate method will return true , the element E and render object E will be updated with the new configuration described with the new widget E.

• Let’s say that the width or the height of the widget E is different in this build. Then, the Render node E will mark itself as needs layout by calling its markNeedsLayout method. A visual update will be scheduled for the current frame in the layout phase.
• Let’s assume that the opacity of Widget E is different in this build. Then, the render node E will mark itself as needs paint by calling its markNeedsPaint method. A visual update will be scheduled for the current frame in the paint phase.
• Let’s assume that the semantics configuration of Widget E for accessibility purposes is different in this build. Then, the render node E will mark itself as needs semantics update by calling its markNeedsSemanticUpdate method. As a result, a semantics update will be scheduled for the current frame in the semantics phase.

Note that when the RenderObject is updated, calling each of these needs methods on it is optional.

Case 5 — inflate widget

Different than the previous build phase, the next slot of the element C is no longer empty. There is a new widget (Y) as a direct child of the element C. In this case, the parent element C will inflate the widget Y and assign a new slot for the newly created element Y . A new render object Y will also be instantiated.

Case 4 — canUpdate returns false

It is time for the element E to visit its child elements. In the last build phase, Element E had the element F in its first slot. In the current build phase, it has a new widget with a different runtime type (Z) for that slot. As a result, canUpdate method will return false , the element F will be set as inactive, the widget Z will be inflated, element Z and the render object Z will be instantiated.

Case 2 — Cutting of the build walk

In the next slot of element E, we have the widget G with a const constructor. Let’s assume that its constructor parameters are also const variables. Then the build walk for widget G subtree will stop because widget G is a const widget and element G will be reused without an update.

Case 1 — Deactivating the unused

The build walk down continues for the Element Z . It will examine the old child elements for their slots. Unlike the previous build, there are no child widgets for these slots in the current build. Hence, it will be Case 1 where the element H and element I will be set as inactive, and their corresponding element and render nodes will be removed from the trees.

Time to Flush

Now the build phase is done in the WidgetsBinding, and we visited all the subtrees of the dirty widgets. It is time to flush layout, compositing bits, paint, and semantics in the RendererBinding!

After the build phase, the rendering pipeline continues with the layout phase. This phase starts with calling the flushLayout method of the PipelineOwner in the RendererBinding mixin of the WidgetsFlutterBinding instance. In this phase, all the dirty render objects in the render tree will calculate geometry for themselves so that their calculated sizes will be up-to-date for the next phases.

The compositing bits phase follows the layout phase and starts with calling the flushCompositingBits method of the PipelineOwner . This phase is about the layer management in the render objects. I would suggest going deeper into this amazing article about compositing bits in Flutter.

“Within the context of the framework, compositing typically refers to the allocation of render objects to layers with respect to painting.” [4]

The paint phase starts after the compositing bits phase by calling the flushPaint method of the PipelineOwner . In this phase, all the dirty render objects are painted onto a Canvas with methods such as drawLine , clipRect, rotate , transform . The drawings performed in this phase will be composited in the nodes of a layer tree.

“Canvas represents a graphical context supporting a number of drawing operations.” [5]

By the end of the painting phase, all the drawing instructions are stored in the layer tree. Then, it is time to call the compositeFrame on the root object which is also known as the renderView . After the layer tree is submitted in the raster thread, the render objects with dirty semantics will be updated. The semantic events listeners such as the accessibility system of the operating system will be informed about the UI changes.

Finalizing the build

In the first article, we mentioned the inactive child elements as a result of the updateChild method calls in Case 1 and Case 4. The rendering pipeline for the current frame ends with a call to finalizeTree on the BuildOwner class In this method the inactive child elements that are not reused in the tree during the drawing of the same frame are demounted and their resources are cleared. In our case, we didn’t do any tree surgery during the build phase. Hence, element F, H , and I will be unmounted.

Conclusion

In this part, we explained Flutter’s rendering pipeline and went through the steps for drawing a frame on a window. It may sound like a complicated topic, but I believe the most important phases of the rendering pipeline to learn are the build, layout, and paint. Understanding these steps would help us to get the most out of the widgets library. For example, we can utilize the CustomSingleChildLayout and CustomMultiChildLayout to customize the layout logic while staying in the widgets layer. Another example is Flow widget which enables us to add logic that is executed in the paint phase.

This is the final article of this series. I hope that this was helpful to understand the internals of the framework. Let’s appreciate one more time the brilliant engineers behind this technology. Special thanks to Adam Barth, Eric Seidel, Ian Hickson Hixie, and the entire Flutter team for providing great documentation and training materials for everyone.

Spread the word, and stay with Flutter!

References

1. FAQ, docs.flutter.dev
2. Flutter: The Advanced Layout Rule Even Beginners Must Know, Marcelo Glasberg.
3. How Android Draws Views
4. Compositing, flutter.megathink.com
5. Painting, flutter.megathink.com
6. Inside Flutter
7. Dirty Region Management