Implementing GUIs using Clojure and LWJGL Nuklear bindings
11 May 2024
For game development I have been using LWJGL3 which is a great Java library for cross-platform development.
Among other things it has bindings for OpenGL, GLFW, and STB.
Recently I discovered that it also has Nuklear bindings.
Nuklear is a small C library useful for developing GUIs for games.
It receives control input and commands to populate a GUI and converts those into render instructions.
Nuklear focuses solely on the user interface, while input and graphics backend are handled by the application.
It is therefore very flexible and can be integrated into a 3D game implemented using OpenGL, DirectX, Vulkan, or other.
LWJGL Nuklear bindings come with the GLFWDemo.java example.
In this article I have basically translated the input and graphics backend to Clojure.
I also added examples for several different controls.
I have pushed the source code to Github if you want to look at it straight away.
A big thank you to Ioannis Tsakpinis who developed LWJGL and GLFWDemo.java in particular.
And a big thank you to Micha Mettke who developed the Nuklear library.
The demo is more than 400 lines of code.
This is because it has to implement the graphics backend, input conversion, and truetype font conversion to bitmap font.
If you are rather looking for a Clojure GUI library which does not require you to do this, you might want to look at HumbleUI.
There also is Quil which seems to be more about graphics and animations.
Dependencies
Here is the deps.edn file:
The file contains dependencies to:
LWJGL core library
OpenGL for rendering
Nuklear for the GUI
GLFW for creating a window and handling input
STB for loading images and for converting a truetype font to a texture
If you are not using the natives-linux bindings, there are more native packages for lwjgl-opengl such as natives-windows and natives-macos.
Graphics Setup
GLFW Window
We start with a simple program showing a window which can be closed by the user.
Here is the initial nukleartest.clj file:
You can run the program using clj -M -m nukleartest and it should show a blank window.
The vertex shader passes through texture coordinates and fragment colors.
Furthermore it scales the input position to OpenGL normalized device coordinates (we will set the projection matrix later).
The fragment shader performs a texture lookup and multiplies the result with the fragment color value.
The Clojure code compiles and links the shaders and checks for possible errors.
Vertex Array Object
Next an OpenGL vertex array object is defined.
An array buffer containing the position, texture coordinates, and colors are allocated.
Furthermore an element array buffer is allocated which contains element indices.
A row in the array buffer contains 20 bytes:
2 times 4 bytes for floating point “position”
2 times 4 bytes for floating point texture coordinate “texcoord”
4 bytes for RGBA color value “color”
The Nuklear library needs to be configured with the same layout of the vertex array buffer.
For this purpose a Nuklear vertex layout object is initialised using the NK_VERTEX_ATTRIBUTE_COUNT attribute as a terminator:
Null Texture
For drawing flat colors using the shader program above, Nuklear needs to specify a null texture.
The null texture basically just consists of a single white pixel so that the shader term texture(tex, frag_uv) evaluates to vec4(1, 1, 1, 1).
The Nuklear null texture uses fixed texture coordinates for lookup (here: 0.5, 0.5).
I.e. it is possible to embed the null texture in a bigger multi-purpose texture to save a texture slot.
Nuklear GUI
Nuklear Context, Command Buffer, and Configuration
Finally we can set up a Nuklear context object “context”, a render command buffer “cmds”, and a rendering configuration “config”.
Nuklear even delegates allocating and freeing up memory, so we need to register callbacks for that as well.
We also created an empty font object which we will initialise properly later.
Setup Rendering
OpenGL needs to be configured for rendering the Nuklear GUI.
Blending with existing pixel data is enabled and the blending equation and function are set
Culling of back or front faces is disabled
Depth testing is disabled
Scissor testing is enabled
The first texture slot is enabled
Also the uniform projection matrix for mapping pixel coordinates [0, width] x [0, height] to [-1, 1] x [-1, 1] is defined.
The projection matrix also flips the y-coordinates since the direction of the OpenGL y-axis is reversed in relation to the pixel y-coordinates.
Minimal Test GUI
Now we will add a minimal GUI just using a progress bar for testing rendering without fonts.
First we set up a few values and then in the main loop we start Nuklear input using Nuklear/nk_input_begin, call GLFW to process events, and then end Nuklear input.
We will implement the GLFW callbacks to convert events to Nuklear calls later.
We start populating the GUI by calling Nuklear/nk_begin thereby specifying the window size.
We increase the progress value and store it in a PointerBuffer object.
The call (Nuklear/nk_layout_row_dynamic context 32 1) sets the GUI layout to 32 pixels height and one widget per row.
Then a progress bar is created and the GUI is finalised using Nuklear/nk_end.
Rendering Backend
Now we are ready to add the rendering backend.
The rendering backend sets the viewport and then array buffers for the vertex data and the indices are allocated.
Then the buffers are mapped to memory resulting in the two java.nio.DirectByteBuffer objects “vertices” and “elements”.
The two static buffers are then converted to Nuklear buffer objects using Nuklear/nk_buffer_init_fixed.
Then the core method of the Nuklear library Nuklear/nk_convert is called. It populates the (dynamic) command buffer “cmds” which we initialised earlier as well as the mapped vertex buffer and index buffer.
After the conversion, the two OpenGL memory mappings are undone.
A Clojure loop then is used to get chunks of type NkDrawCommand from the render buffer.
Each draw command requires setting the texture id and the clipping region.
Then a part of the index and vertex buffer is rendered using GL11/glDrawElements.
Finally Nuklear/nk_clear is used to reset the GUI specification for the next frame and Nuklear/nk_buffer_clear is used to empty the command buffer.
GLFW/glfwSwapBuffers is used to publish the new rendered frame.
Now we finally have a widget working!
Mouse Events
Cursor Position and Buttons
The next step one can do is converting GLFW mouse events to Nuklear input.
The first callback is to process mouse cursor movement events.
The second callback converts mouse button press and release events to Nuklear input.
The progress bar is modifyable and you should now be able to change it by clicking on it.
Note that using a case statement instead of cond did not work for some reason.
Scroll Events
The Nuklear library can also be informed about scroll events.
Here is the corresponding GLFW callback to pass scroll events on to the Nuklear library.
The scroll events can later be tested when we implement a combo box.
Fonts
Converting Truetype Font to Bitmap Font
To display other GUI controls, text output is required.
Using the STB library a Truetype font is converted to a bitmap font of the desired size.
Basically the font file is read and converted to a java.nio.DirectByteBuffer (let me know if you find a more straightforward way to do this).
The data is used to initialise an STB font info object.
The next steps I can’t explain in detail but they basically pack the glyphs into a greyscale bitmap.
Finally a white RGBA texture data is created with the greyscale bitmap as the alpha channel.
You can write out the RGBA data to a PNG file and inspect it using GIMP or your favourite image editor.
Font Texture and Nuklear Callbacks
The RGBA bitmap font can now be converted to an OpenGL texture with linear interpolation and the texture id of the NkUserFont object is set.
To get the font working with Nuklear, callbacks for text width, text height, and glyph region in the texture need to be added.
I don’t fully understand yet how the “width” and “query” implementations work.
Hopefully I find a way to do a unit-tested reimplementation to get a better understanding later.
On a positive note though, at this point it is possible to render text.
In the following code we add a button for stopping and a button for starting the progress bar.
The GUI now looks like this:
Trying out Widgets
Menubar
The following code adds a main menu with an exit item to the window.
Here is a screenshot of the result:
Option Labels
One can add option labels to implement a choice between different options.
Here is a screenshot with option labels showing the three options easy, intermediate, and hard.
Check Labels
Check labels are easy to add. They look similar to the option labels, but use squares instead of circles.
Here is a screenshot with two check labels.
Property Widgets
Property widgets let you change a value by either clicking on the arrows or by clicking and dragging across the widget.
Here is a screenshot with an integer and a float property.
Symbol and Image Buttons
Nuklear has several stock symbols for symbol buttons.
Furthermore one can register a texture to be used in an image button.
Each button method returns true if the button was clicked.
Here is a screenshot showing the buttons instantiated in the code above.
Combo Boxes
A combo box lets you choose an option using a drop down menu.
It is even possible to have combo boxes with multiple columns.
The combo box in the following screenshot uses one column.
Drawing Custom Widgets
It is possible to use draw commands to draw a custom widget.
There are also methods for checking if the mouse is hovering over the widget or if the mouse was clicked.
Here we have just drawn a filled rectangle and a filled circle.
Keyboard Input
Finally we need keyboard input.
The GLFWDemo.java example uses two GLFW keyboard callbacks and even implements callbacks for the clipboard.
Character Callback
First a character callback is implemented.
To test it, we also add an edit field for entering text.
The following screenshot shows the edit field with some text entered.
Control Characters
To get control characters working, the second GLFW callback is implemented.
Now it is possible to move the cursor in the text box and also delete characters.
Clipboard and other Control Key Combinations
Finally one can implement some Control key combinations.
Except for undo and redo I managed to get the keyboard combinations from the GLFWDemo.java example to work.
We also implement the clipboard integration.
The following screenshot shows the text edit field with some text selected to copy to the clipboard.
Styling
You can get Nuklear styles from Nuklear/demo/common/style.c.
My favourite is the dark theme.
The style is set by populating a color table and then using nk_style_from_table to overwrite the style.
Here is a fix to get repeat keypress events for control characters working:
Polygon Visualisation
If you enable OpenGL polygon mode and clear the image, you can see the polygons Nuklear is creating.
Grouping Widgets
One can group widgets.
The group even gets a scrollbar if the content is bigger than the allocated region.
The group can also have a title.
The screenshot shows the compression and quality property widgets grouped together.
Slider
Sliders are an alternative to property widgets.
They display the value using a circle instead of using a textual representation.
Here is a screenshot showing a slider.
Nested Layouts
Nuklear does not seem to support nested layouts.
However as shown by Komari Spaghetti one can use groups for nesting layouts in Nuklear.
Basically you just need to set window padding to zero temporarily and disable the scroll bars.
In the following sample there are five buttons using two columns with different button sizes.
The screenshot shows the layout achieved in this case.
Type hints
In order to improve performance, one can use Clojure type hints.
This is especially effective when applied to the implementations of width and query method or the NkUserFont object.
width gets called for each string and query even gets called for each character in the GUI.
One can enable reflection warnings in order to find where type hints are needed to improve performance.
By fixing all reflection warnings (see v1.1), I managed to reduce the CPU load of the GUI prototype from 50.8% down to 9.6%!