Problems using ratatui

[arxanas]

I am working on an interactive commit selector for source control tooling. It looks like this:

record-720p.mov

Here are some problems with tui-rs/ratatui that I had to work around in my own code, which might inform future development. Let me know if you have any solutions.

• Relative positioning. If you want to render widget Y below widget X, but you don't know the size of widget X ahead of time, then you don't know where to render widget Y.
  • The constraint solver doesn't help, as the solved sizes are irrespective of the actual size of the components.
  • You could imagine a LinearLayout that renders components in sequence in a certain direction. I haven't implemented this; I simply modified the rendering system to return the actually-drawn Rect, and then do math to position the next widget after it.
• Rendering outside of terminal bounds. To implement scrolling, etc., it would be convenient to render a widget to an arbitrary signed (x, y) coordinate, and have the framework ensure that it only renders in the displayable area. It's very natural to want to render a widget to a negative x- or y- coordinate (for example, a document that has scrolled up), but this is impossible at present because render_widget accepts a Rect with unsigned x-/y-coordinates.
  • It's possible to clamp the Rect into the displayable area, but it's not easy to write widgets that behave well in this paradigm. When being rendered, the widget either has to do math to figure out what part of itself will be in the displayable Rect and only render there (not tenable in my experience), or else render to a virtual canvas internally and then copy the relevant portion to be displayed in the Rect.
  • Likewise, the constraint solver can't directly be used to calculate layouts with signed coordinates.
  • My own implementation renders everything to a virtual canvas and then renders only the displayable portion of the virtual canvas to the terminal.
• "Masking"/"compositing". Related to the last point, when rendering multiple widgets to the screen, it is often desirable to only display portions of that widget, such as when emulating window-like components.
  • Note that the "mask" is not the same as the target Rect to be rendered to. Imagine that you're implementing a scrolling widget: you would want to render the entire widget content to the Rect, and have only a portion of it visible on the screen.
  • My own implementation stores a mask stack and lets the caller push a new mask when rendering a widget. It's clunky as currently written.
• Widget location. After having drawn a widget, if the user clicks it, it's hard to find the widget corresponding to the (x, y) location on screen. This can be managed by keeping track of which widgets render where, but it requires additional bookkeeping.
  • Similarly, if you want to manage widget focus with the keyboard, you may want to look up the widget that is above/to the left/etc. of the currently-focused widget. You can manage this yourself, but it would be nice to have in the framework.
  • My code has each widget declare an associated Id type and stores a map from Id to actually rendered Rect. When clicking, to identify the clicked widget, it looks for the "top-most" (most-recently-rendered) widget that has been rendered to that position. There are subtleties dealing with the fact that only some kinds of renders want to create a logically-new clickable widget; for example, text is usually not its own clickable entity, but part of the parent widget. This approach results in infectious generics, so it may not be suitable for inclusion in the framework.
  • This is also useful to implement scrolling to a certain widget. You need to have rendered the widget/calculated its position to know where to scroll to.
• Event handling and widget focus. Some widgets may be able to respond to only some events. For example, a combo box can handle a few keystroke events (selecting an option), but would want to defer other keyboard shortcuts to the parent widget.
  • My code identifies every kind of renderable widget and tries to match every possible input event with the currently-focused widget to see if that widget should handle it. This results in a combinatorial explosion of state/event pairs as more kinds of widgets are used in the program. It's also not always statically possible to determine if an input event should be handled by a given widget, as it may depend on the widget's runtime state, which makes it hard to use e.g. exhaustive matches to help manage this.
  • Even if ratatui is supposed to be input-agnostic, a good start would be to expose the widget render tree so that the user can walk the widget chain and implement their own event-handling system. But realistically, a "textarea" widget will need to be able to respond to input events, so we may have to standardize on some kind of event system to make these widgets truly reusable and interoperable.

[kdheepak]

Thanks for starting this discussion! There's lots of interesting pieces here to talk about, and it'll be good to have more people weigh in on this and discuss how best to address these issues.

Slightly unrelated, is your code public?

[joshka]

Looks great! I have some ideas about a bunch of those points.

• Crossing all the points, is that Ratatui is an immediate mode gui rather than retained mode library, and so a lot of these things are inherently by design. A widget from that standpoint is instructions on what to draw in a single frame (it could be worth imagining Widget being named "Renderer" and then thinking about some of the points). There's a good blog post about masonry from earlier this year that has a different widget trait that has on_event, layout, paint, children. So a lot of the approaches to addressing those misses will be thinking about what Widget is missing and then writing new tools that provide that (e.g. Component / View / Composite / RenderContext / WidgetLayout / WidgetIdentity / LayoutHierarchy etc.). It should be noted that ratatui is a library, not a framework - your code calls ratatui, not the other way around.

• Relative positioning.

  • We've got some parts of this happening in feat(Rect): add offset method #533 (would love some feedback)
  • We've talked about using Taffy a bit to replace the current layout. No clear direction as yet, but some of the feedback here will help that definitely.
  • Definitely your idea about having a linear layout is good. This could be implemented as an iterator that returns Rects that fit certain sizes or something like that.

• Rendering outside of terminal bounds.

  • I think this might be possible to implement as a Widget (call it something likeViewport) that provides a faux buffer for the widget to render on and then remaps that to the real coordinates. Thinking this through for the paragraph, you want to render the paragraph from 0,0 to width, height, scrolled to column x, row y. The paragraph can skip the right number of lines and start rendering zero based and expect that the Viewport puts it in the right place. A lot of widgets currently always have to do the Rect offset calcs, perhaps they shouldn't.
  • Not sure I see any real benefit of signed coordinates over dealing with an offset into an unsigned coordinate system (e.g. rendering from 0,0 to 100, 40 with an offset of 200, 50 is the same as rendering from -200, -50 to -100, -10.

• "Masking"/"compositing"

  • good points - sounds like that Viewport needs to make it easy to handle noops in regions that don't matter in widgets
  • and handle rendering order issues neatly (again noting the issue of Widget = instructions to render, not some object)

• Widget location.

  • requires widgets to have identity rather than just being rendering instructions (again, immediate vs retained mode here)
  • requires that identity to be tracked on render and the rects to be associated with the order
  • could probably be implemented using a Component and Composite / Container type or similar

• Event handling and widget focus

  • this requires widget identity, visual + non-visual drawing hierarchy and event bubbling. All retained mode concepts, but seriously missing in immediate mode.

All up, I agree with every single point you're making. Ratatui doesn't do any of those. You could build most of that on top of Ratatui though (and many apps have done bits and pieces of all that.

https://poignardazur.github.io/2023/02/02/masonry-01-and-my-vision-for-rust-ui is a particularly good article that I re-read from time to time. It has a whole suite of things that a good rust GUI library should be able to do, and there are many inspirations for things to add to Ratatui or a framework built on top of Ratatui.

• hot reload
• viewing partial renders (for prototyping)
• inspectable render pipeline
• logging
• warnings of developer errors (rendering out of bounds)
• unit testing by rendering headless and driving events
• etc.

All of that said, I suspect that there probably are some things that can be added to Ratatui to make some of the above more able to work well with this sort of application approach. I'm not 100% sure what all those things are, so keep that feedback coming. It's really useful.

[arxanas]

Of course, I should have linked my code, currently at https://github.com/arxanas/git-branchless/tree/master/scm-record; will comment on the other points later.

[arxanas]

Not sure I see any real benefit of signed coordinates over dealing with an offset into an unsigned coordinate system (e.g. rendering from 0,0 to 100, 40 with an offset of 200, 50 is the same as rendering from -200, -50 to -100, -10.

I'm fairly confident that using signed coordinates is the right way to go, ultimately.

• It simplifies a lot of math in places where you would have to check for underflow, etc. You can just subtract and not worry.
• Currently, there is no way to render (or even express the idea of) e.g. a Block such that the first line is off-screen.
  • I wanted to render a Block and scroll it off-screen, but there seems to be no way to do it currently.
  • It could be fixed via a Viewport with signed coordinates, but then most likely everything will want to go through the Viewport, so there ends up being limited use for having unsigned coordinates.
    • There are several largely useless conversions between ratatui::layout::Rect and my own signed Rect in my code.
• It's not always possible to determine the proper starting coordinate since the layout may depend on information not known until render-time.
  • For example, a word-wrapped paragraph scrolled to line 3 will need to start rendering at line 3, won't know what's on line 3 until it tries to do word-wrapping, so it would have to render to an intermediate buffer. It turns out that this is a rather common situation when composing widgets, so the Viewport with support for signed coordinates should really be the default way of rendering.
  • I originally had all my components calculate what "should" be on-screen, but it was incredibly bug-prone and essentially impossible to iterate, so I gave up and used a viewport/virtual canvas instead.

Are there any advantages to using unsigned coordinates that I'm not aware of?

I think this might be possible to implement as a Widget (call it something like Viewport) that provides a faux buffer for the widget to render on and then remaps that to the real coordinates.

Yep, my code defines a Viewport type doing that (although I don't think its interface should be used exactly as-is).

Widget location [...] requires widgets to have identity rather than just being rendering instructions (again, immediate vs retained mode here)

Yes, I think that Ratatui needs to return the rendered widget tree to the caller because there is critical layout information that can only be accessed in that way (unless you write the entire layout stack on top of Ratatui like I did, but then you can't use anyone else's components directly).

We should do something similar to your linked article https://raphlinus.github.io/rust/gui/2022/05/07/ui-architecture.html and have nodes with identities (if not node paths, as they suggest).

• My code defines an associated type (probably similar to how they discussed): https://github.com/arxanas/git-branchless/blob/7520d2251451c4ee6ddd5514c9cf4fe3449d57d9/scm-record/src/render.rs#L565-L570.
• My code only records objects drawn with draw_component and not draw_span, although a more explicit approach to deciding which components get recorded is probably best.
• Generally, I caution against using too much static polymorphism, as it tends to overload the compiler and eventually produce bad developer experiences. The error messages also become inscrutable.
• Rendering "sticky headers" ends up being tricky, because during the render, the widget needs to know which components are visible in the viewport, and then it has to draw the sticky header over that component in the right place.

Event handling and widget focus [...] this requires widget identity, visual + non-visual drawing hierarchy and event bubbling. All retained mode concepts, but seriously missing in immediate mode.

It seems like Druid thinks it's fine to do event bubbling by traverse up the render tree, and in my UI experience so far, it also seems pretty reasonable to me.

[joshka]

I'm fairly confident that using signed coordinates is the right way to go, ultimately.

I'm having a bit of trouble following the abstract example of the viewport enough to grok where the pain point is. I wonder if you could expand on this and talk about the concrete problem you're facing? Try not to hide any "difficult" details or substitute a simpler example, as often these bits are the bits that help explain the issue the best (if you're familiar the https://xyproblem.info/ then you know what I'm talking about here).

Are there any advantages to using unsigned coordinates that I'm not aware of?

Right now that's just that we draw to a buffer that is the size of the screen, and the screen is addressed from 0,0 to width,height, so there's no translation going on and no extra space allocated for stuff outside the screen. Anything rendered outside of those bounds currently panics (which is a bit annoying I'd admit, but something difficult to change without breaking all code I'd guess).

Yes, I think that Ratatui needs to return the rendered widget tree to the caller because there is critical layout information that can only be accessed in that way (unless you write the entire layout stack on top of Ratatui like I did, but then you can't use anyone else's components directly).

Do you have any thoughts about where this would fit in the current Ratatui API? The biggest concern we'd have on this is how to evolve the APIs we have without breaking everything.

For example, a word-wrapped paragraph scrolled to line 3 will need to start rendering at line 3, won't know what's on line 3 until it tries to do word-wrapping, so it would have to render to an intermediate buffer. It turns out that this is a rather common situation when composing widgets, so the Viewport with support for signed coordinates should really be the default way of rendering.

Wrapping combined with scrolling is something I've thought a bunch about See:

• RFC: Text Wrapping Design #293
• RFC: Design of Scrollable Widgets #174
  Both are fairly large issues by themselves, and are somewhat stalled. In brief though scrolling to line in a wrapped paragraph has a fairly obvious implementation (though it has some complex edge cases IIRC)
• instead of keeping a single scroll value, keep two values, the scroll position that indicates the unwrapped line and the number of lines offset past that position. (position: usize and offset: i32)
• Say for example we have 100 lines, each that wrap across 5 lines total if we find that when we render the position and offset that the actual offset moves to the previous or next line (say our offset is -1 or +6, then the paragraph updates the scroll state to correct the issue. The act of rendering fixes the scroll position up for you by rendering enough lines prior to the scroll position to properly fill the paragraph, or afterwards
• Doing this allows us to only call the wrap functions for the text currently in view (i.e. pre-filter the text to the potentially viewable text instead of wrapping all the text), and should be fairly fast except for crazy long lines.
• The List widget takes the update scroll position out of view part of this approach already, though it doesn't render partial rows (it always starts at the first row of a list item)

[arxanas]

I'm having a bit of trouble following the abstract example of the viewport enough to grok where the pain point is. I wonder if you could expand on this and talk about the concrete problem you're facing?

The example this one:

Currently, there is no way to render (or even express the idea of) e.g. a Block such that the first line is off-screen.
I wanted to render a Block and scroll it off-screen, but there seems to be no way to do it currently.

So signed coordinates allow the expression of such a Block.

Do you have any thoughts about where this would fit in the current Ratatui API? The biggest concern we'd have on this is how to evolve the APIs we have without breaking everything.

Changing from using a physical viewport to a virtual viewport seems fundamentally disruptive, so I probably don't have anything to offer that you haven't already thought of. You could build the new Viewport APIs and update all built-in components to support rendering to a Viewport in addition to the current behavior (and also introduce SignedRect or other types as appropriate); then mark the old types as deprecated; then alias the old types to the new ones (in case it works out that there is code that would work in the new model without any changes?); then remove the aliases.

Wrapping combined with scrolling is something I've thought a bunch about [...]

I skimmed the threads and don't think I saw exactly the approach that I am using:

• A request to render a widget accepts logical bounds (on an infinite, signed canvas) and the mask bounds to be used for rendering. If I understand correctly, several of the problems in various approaches are addressed by having two sets of bounds instead of one.
  • The mask is strictly informational, as the Viewport will suppress rendering outside the masked area no matter what the widget does.
  • From the perspective of the widget, the mask is the hint it needs to skip rendering invisible portions. Most user widgets don't need to worry about it at all, and can just continue to render the entire widget, which is likely sufficient for most applications. High-performance widgets can use the mask information to determine which portions are visible and only render those.
  • Scrolling to the nth item in a list is handled checking the previous render tree for the location of the nth item, then applying the delta between the current and desired scroll locations.
  • Tangential: I say "bounds" rather than "position" because I've found neither "a single coordinate" nor "a Rect" are quite descriptive enough in cases where layout and sizing are linked. For example, a LinearLayout may render/mask with bounds on the left/top/right, but not the bottom.
  • Tangential: I would be frustrated with a hypothetical scrolling list component could only start rendering at the top of a list item. In my application, some of the individual list items are larger than one screenful! We have to be able to scroll and render a partial list item for it to be useful.

[joshka]

Tangential: I would be frustrated with a hypothetical scrolling list component could only start rendering at the top of a list item. In my application, some of the individual list items are larger than one screenful! We have to be able to scroll and render a partial list item for it to be useful.

Ignoring the viewport approach for this, it is fairly easy to ensure generically with two fields and is in scope for any scrolling solution (and basically aligns with your note about the nth item above.)

Scrolling needs 2 fields - the usize index of the item to start with (for text this is the unwrapped line), and a signed offset into that item. This allows for page up/page down etc. to be handled without prior knowledge of the rendered state. When the item is rendered, the rendering process should normalize these two values so that the index points at the actual first item rendered. E.g. (assuming all elements have 3 rows / lines):

• index = 10, offset = -4 => index = 8, offset = 2
• index = 10, offset = -3 => index = 9, offset = 0
• index = 10, offset = -1 => index = 0, offset = 2
• index = 10, offset = {0,1,2} => no change
• index = 10, offset = 3 => index = 11, offset = 0

[joshka]

It's also worth reading https://raphlinus.github.io/rust/gui/2022/05/07/ui-architecture.html which covers a similar set of gaps.