The Procedure Authoring Process

Being a visual tool where all content is created within the editor it is of paramount importance that the authoring process is a smooth one.  The tools and UI are being developed to be a low friction and intuitive experience, and I am always thinking about ways to improve them.  Let's take a look at the procedure authoring process.

Managing Procedures

The first step is to add a new procedure to the procedure browser.  Pressing Alt+N creates one called "new procedure".  It will inherit the category of the currently selected procedure.  Single click to select a procedure (highlighted in yellow) and you will see its properties appear in the property panel.  Here you can change its name from the default to something suitable.  If you want, you can change its category too.  As you rename or re-categorise a procedure it will move to the appropriate place in the procedure browser.  Scroll around or use the filter box at the top of the panel to find it.

Creating a new procedure

Procedure Files

A procedure is stored as XML data in a .proc file on disk, along-side this lives a .procedit file containing any data that is only needed in the editor such as visual graph layout and element descriptions.

Procedure files and their editor data companions

Each category has its own folder on disk.  As you rename/recategorise a procedure, these files are renamed and moved around accordingly.

Viewing & Editing

When you are working on a procedure you will probably want to see the results in the 3D view, to make a procedure the subject of this view select it and press the View button at the top of the browser panel.

To edit the procedure content, either select it and click the Edit button or just double-click on it and it will open in the procedure graph which we will look at next.

Editing and Viewing a new procedure

Procedure graph

The main part of the editing window is dedicated to viewing and editing the procedure graph.  A procedure has a perimeter surrounding it's content as well as hosting the input and output connections for it.  The procedures title appears at the top of this area, inputs on the left, and outputs on the right.

The viewing area can be panned around by holding the right mouse button and dragging, and zoomed in and out using the mouse wheel.  Normally the view is fully zoomed in and you will only need to zoom as you work on larger procedures.

Operator Instances

The most important part of creating a procedure is the adding of operator instances into it.  This is done by dragging an operator from the procedure or operator browser panel onto the procedure.

Placing operators by dragging onto procedure

New instances are unconnected and have default values for all of their inputs.  Operators can be removed by selecting them and pressing Delete.

Selection & Manipulation

Operators (and procedures) placed in your procedure can be singly or multiply selected to allow movement, and input value editing.  Shift+Click to add to the selection, and Control+Click to toggle inclusion.  You can also drag a marquee (from an empty part of the window) around operators to select them.  Click on the empty background to deselect.

Selected procedures can be dragged around to be repositioned, both individually and in multiselected groups.  If you move operators near to the edge of the procedure the boundary will be expanded to accommodate it.

Wiring

To use an operator it needs to be connected up with the visual wiring metaphor we use to show where inputs should get their values from.  Each input and output has a name label and a connection point.  Hovering over a connection point and dragging creates a wire attached to that point.  You can now interactively choose the appropriate connection point you want to connect to.  Compatible connection points (same type) are highlighted during this process.  Hover over the target connection point and release the mouse button to make the connection.

Connecting operators together

Existing wires can be moved around easily by grabbing one end and dragging it to somewhere else.  As you hover over a wire it will highlight, both the whole wire (thicker) and one of the ends (white).  This helps you see what a wire is used for in a complex procedure, as well as allowing you to specify an end to be reconnected.  If you drag a wire and drop it away from any connection points the wire will be removed.  Any disconnected inputs will revert to their previously set constant value.

Highlighting and disconnecting wires

Constants

Operator instances with unconnected inputs assume a constant value.  To specify this value simply select the operator and the property panel will list its inputs and values for editing.

Editing an operator instances input values

Connected inputs can't have their value set as they implicitly get their value from another output which is evaluated at synthesis time.

Procedure IO

Procedure inputs (along the left edge) and outputs (along the right edge) are created by starting a new wire on an operator output or input and dragging it outside the procedure boundary.

Creating procedure inputs and outputs

By default this new input/output assumes the name and type of the operator connection point you start from but they can be edited in the property panel by selecting the whole input or output area.

Editing procedure inputs

Editing procedure outputs

Procedure inputs and outputs can also be selected individually for editing, or removal (press Delete).

Once created, procedure inputs and outputs remain present and can be connected/disconnected/reconnected the same way as operator instance inputs and outputs.  In fact; procedure outputs behave exactly like operator instance inputs and procedure inputs behave exactly like operator instance outputs.

Notes

To help document procedures, visual notes can be added.  These are rectangular panels with a title and description text you can adorn a procedure with to explain what is going on.  You can also use them to surround operators to group them such that they can be moved around as a unit.  To include/exclude an operator from a group just drag it into or out of the note boundary.

3D view

The currently viewed procedure is submitted for synthesis and the resulting models displayed in the 3D renderer view-port.

The output of our procedure shown in the 3D view

As edits are made to the procedure structure and constants it is regularly re-submitted so that the models update to reflect these changes providing interactive feedback for your design.

Interactive editing of procedure data

Camera

The 3D view provides a standard range of camera controls including:

• FPS style navigation - right mouse button to look around, WASD to move forward/strafe plus QE to raise/lower.
• Modelling camera - Middle mouse button to pan, with Alt to orbit, with Control to raise/lower (Z axis), with Shift to move around the XY plane.  The mouse wheel can be used to adjust the orbit distance.
• Auto-rotate - left click to toggle a carousel style orbit mode.

Grid

By default, the 3D scene includes a ground-plane grid and an axis indicator at the origin to help visualise the 3D space and scale of objects modelled.  This can be disabled and adjusted if needed by clicking the Grid tab at the top.

Drawing aids and their settings

Expanded View

The normal 3D view is fairly small but fine for a lot of modelling needs.  For cases where more detail is needed though you can toggle it to large size by pressing the Space bar.  In this view you only have the 3D view and the property panel visible.  This is a great mode to tweak values in.

Toggling expanded 3D view

Property Editing

Many elements in the editor can, when selected, have their properties displayed in the property panel for review and editing.  This used for operator input constants, procedure properties, synthesis and rendering statistics, and grid and view-port diagnostics settings.

Types 

Most data types are view-able and editable in the property panel, including; integer, floating point, boolean, colour, string, vector, and even frames have basic editing control.

Controls 

Some types have specialised controls, for example, numerical values have a slider control to aid interactive adjustment.  The minimum and maximum range of the slider is editable too, and stored with the procedure so it is there for convenient editing of the value next time the procedure is opened.
Integer inputs can also be set up in the operator definition to have enumerated values.  This can be presented as a drop-down selection or a series of buttons.  Inputs that represent a set of flags can have independant toggle buttons for each flag as well as some composite value buttons, e.g. All, for convenience.

General

A few of the more general features of the editor are worth mentioning.

Undo

Most editing operations, including property changes are command based and enable full undo/redo support.  The usual Ctrl+Z/Ctrl+Shift+Z keys are used to navigate the command history. 

Save & Load 

Procedures with unsaved changes show in bold font in the procedure browser.

Unsaved procedures appear in bold

Pressing Ctrl+S or the Save button at the top of the browser will save all unsaved work in one go.  The save process is two stage and uses temporary intermediate files to protect your data from problems during the save process.  All procedure and editing files are also kept in a back-up history.  This is stored in a backup folder alongside the procedure files.

All procedures are loaded at startup by default.  This makes managing them much simpler.

Updates

It's worth mentioning how procedure updates are propagated and applied to the models in the 3D view.  Any time you perform an editing operation on a procedure, it can potentially affect the generated output.  By following the dependency graph back from the edited procedure a list of all potentially affected procedures can be built.  This is then used to determine if the procedure you are viewing needs to be re-synthesised.

If you are adjusting a slider and potentially generating lots of edits in a short space of time the engine will try to update as fast as it can without swamping the synthesisers.

Summary

The functionality described here corresponds to the current state of play.  This is the tool set I am currently using to build and test procedures.  There is lots of scope for improvement and I have a large wish-list of features and tweaks to add.  Usually these are implemented when I am building more involved demonstration procedures and find bottlenecks in the process.

Next

I am going to be away for a couple of weeks so the next few blog updates may not be as regular as they have been so far.

A Look Around The Editor

For this post I'll show you round the different parts of the editor application's user interface.  Next time I'll dig into the editing functionality and how to use Apparance to actually build procedures.

An Editor?

Why do we need an editor?  Well, we are trying something very different here, by way of workflow, modelling paradigm, and output.  As essential parts of the Apparance concept, building a custom editing application was the only way to achieve this level of bespoke requirements.  Some of the important features it needs are:

• Creation and management of procedures
• Data-flow based visual graph editing
• Preview of resulting procedure output
• Real-time, interactive authoring and tweaking

Looking through the image gallery you can see how the user interface developed.  Initially as I was proving the procedure data representation and the synthesis process it was just driven as raw XML data.  This was fine for testing, but as you can imagine it was incredibly unwieldy for anything but very simple procedures.  As the project progressed I worked on each of the main interface elements in turn, improving them again and again.  Let's look at them in more detail.

The Apparance Editor

Browser

Good design means factoring out functionality into smaller, re-usable, chunks, and consequently we will need to be able to work with many procedures.  At the moment, procedures are organised in a simple two level hierarchy with a Category and a Name.  This will probably need expanding in the future, for larger projects, but provides a way of grouping procedures together for now.

Procedure/Operator browser and properties of selected procedure

A browsing panel lists all the procedures and as a navigation aid there is a filter box to narrow down those displayed.  As well as procedures, the fundamental operators they are built from are also listed, in their own browsing panel and can be filtered in the same way.

Procedure Editing

Once you create a procedure you need to start specifying the functionality within it and the connections in and out.  This is performed within the main area of the editor in a scrollable, zoom-able window.

Zoomed-out overview of a large procedure in the editing window

Often your operator graph will fit within the window, but for more complicated creations you will need to zoom out or pan around.  Operators are boxes with the name of the operation at the top, inputs on the left, and outputs on the right.  The procedure itself has its inputs on the left and outputs on the right too.  Consequently, the natural visual 'flow of data' is from left to right, most connections and chains of functionality propagating information to the right.  This doesn't mean you can't make connections in any direction and create all manner of spaghetti. Careful factoring out of messy bits into sub-procedures helps here.

The inputs and outputs of the procedure that you specify and name here are what you will see and be able to connect to when you place your procedure down within another procedure.

Procedure IO editing

3D View

There is a rendering window in the corner of the editor where you can view a procedures output.  At the moment all output is 3D model geometry, and as we are targeting 3D worlds this is all you need to see a model in place.

The 3D preview window

By electing to view a procedure, you are specifying the starting point of the geometry synthesis process.  In order to do this with procedures that have inputs, you need to be able to specify their values.  This can be done where you edit the input connections to your procedure (see above) and are effectively the default values your procedure comes with.  This means you can preview any procedure as each come with some starting values.  These are also the values your procedure starts with at its inputs when you place it down.

The 3D view-port has pretty standard camera navigation controls, with orbit, and FPS style movement as well as an auto-rotate mode for showing off a model.

To help with construction and spatial orientation, a ground-plane grid is drawn for you.  This is implemented as another procedure that can be edited just like any other if it needs customising (e.g. turn off, adjust colour/intensity, spacing, scale, etc).

To get a better look at your scene you can expand the 3D view to occupy the whole editing and browser area.  This leaves the property editing panel (which expands to occupy the space where the 3D view was).  This mode is ideal for tweaking values, simply select the operators who's inputs you want to change and switch to expanded mode.

Toggling the large 3D preview window

Property Panel

Most editing environments include some form of properly panel where a list of the individual adjustable elements of an object are shown.  The Apparance editor uses this for editing (and viewing) a number of things, such as: Operator input constant values, procedure IO name and description, new procedure name and description, renderer settings and statistics, view-port visualisation modes (see below) and diagnostics, and grid settings.

Property viewing and editing panel

Most data types are fully editable, some with specific enhancements such as sliders for floating point values and toggle buttons for enumerations.  Sliders have editable min/max values too so you can set them to a sensible range for the value the slider controls.

Development

In line with the live/interactive editing model adopted here, most of the user interface can be updated at run-time.  This has made development of the UI much, much faster and allowed much in the way of polish that would have otherwise been left.  The editor UI is implemented in WPF which supports dynamic loading/parsing of the backing XAML design data.  Custom text editing panels can be expanded to allow live editing of most of the editor interface.

Live editing of the editor UI

The synthesis process can be monitored in a custom panel showing each of the synthesisers, with a timeline of the jobs each works on.  For each job a breakdown of memory use and any issues encountered is displayed.  This is needed to diagnose any technical modelling problems.

Synthesis statistics and diagnostics

Another panel allows exploring of the internal engine structure and any properties exposed by each part.

Engine exploration; here showing view-port modes and settings

There are a few ways to analyse the operation of the engine, the synthesiser, the procedures, and the tools, including: GraphViz dumps of each synthesis run, the scenehierarchy, and procedure capture analysis process, as well as in-editor visualisations of the detail refinement hierarchy, the editor tool stack, and the UI stack.  All helpful in working out why things aren't going as expected and important to understand how best to build procedures that work well with the engine.

Next

Next time I will talk about procedure creation, editing, and viewing.

The Rendering System

Previously we covered how geometry is created.  This time we will look at how it is managed and rendered.

Models

It was mentioned in my last post that geometry is built into fixed size buffers.  This limits the amount of geometry that can be created by one a procedure.  For a small object, or one of low detail (farther away), this may not be a problem, but if we are to build huge, detailed worlds then it most certainly is.  To overcome this, a number of systems and techniques are used.

Refinement

Models are managed within a spacial octree, each node being responsible for any models that fit reasonably within its own bounds.  Smaller models are managed by the small nodes deeper in the octree.

During the synthesis process, the sub-procedures used (and any bounding information that can be obtained from them) are analysed, and in certain cases stored.  The aim here is to capture a set of sub procedures that fully represent the model built, but as smaller component parts.  These parts can then be used to build more detailed versions of parts of the whole model, and which can be managed by the smaller octree nodes that are more suitably sized.  This effectively provides a way of re-synthesising successively smaller parts of any model as we need the extra detail deeper in the octree.  This 'refinement' process is driven by proximity to the viewpoint, using the deeper more detailed model parts in areas that are nearer the camera.

Successive octree levels, and the geometry managed by each

Authoring

Procedures do need to be built with this process in mind somewhat.  There are certainly ways to help or hinder the process and prevent the system from operating at its best, but the tools provide feedback and diagnostics to help you optimise them.  This is another area that I will dig into in more detail in another post.

Rendering

The rendering engine for Apparance has always been fairly basic as most of the work has been in proving out the procedure synthesis and detail refinement techniques.  All that the renderer needed to be able to do was render some coloured triangles with a couple of fixed light sources.  This was implemented in DirectX 9 and based on a fairly simple cube rendering sample.  Even with no materials, no texturing, and simple primitives I have been able to make quite a wide range of examples.

Small sample of results achieved with basic renderer

The renderer itself has been written to be fairly robust and flexible, with support for multiple viewports, cameras, and scenes, it runs on its own thread, and supports window resizing and device loss properly.

Shaders

Current focus

Driven mainly by the need to start blending between meshes of different detail levels, I decided that I needed to add shader support and this is my current focus.

With the flexibility and power shader based rendering brings I will be able to implement an elegant blending system, as well as better lighting, and start experimenting with more realistic surface properties.

I decided that I should certainly allow run-time authoring of shaders as this is an important premise of the Apparance tool philosophy.  To do this I also decided that the shader code should be procedurally constructed by the same systems the models are built.  Not only does this mean I can easily re-use shader functions and constructs, but pieces of code, and even allowing parameterisation of the shader code itself.  This should have all sorts of interesting effect creation potential.

Trouble

During the testing of DirectX 9 shaders I hit some nasty snags to do with background compilation of shaders during rendering, shader lifetime management, and finally with a crash on ending and releasing of shader resources that I couldn't resolve.  Even using my simple training app I couldn't solve the issue and under Windows 10 it turns out that debugging and diagnostics in DirectX 9 isn't supported, so no help there.  My solution was to bite the bullet and upgrade the engine to DirectX 11, which represents a significant improvement in features and support, as well as being fully integrated into the OS and with significant debugging support.  Unfortunately this did mean learning about all the differences and writing another learning app, but it seems like it will be a good move in the long run as I was probably going to need it at some point anyway and DirectX 11 has some nice improvements in the way you handle shaders that it will be good to get used to.

New rendering and shader test app for DirectX 11

Graphics Fu

Eventually I am going to need some fairly fancy rendering features to show off the models properly, such as multi-texturing, advanced light sources, high quality shadows, ambient occlusion, and maybe even global illumination.  I am treating these as 'solved' problems and prioritising many other, more unique, features over them.  I am also likely to need help with the harder graphics tech and should start to involve others in the project more closely, but that will depend on how much interest I can raise in the project and whether I can find funds to build a team around it in the future. We shall see...

Next

I was going to describe my development setup a little here, but I think I'll leave it until a later post.  Next time I'll talk about the editor and how it is used to develop procedures.