The Rendering System

Previously we covered how geometry is created.  This time we will look at how it is managed and rendered.

Models

It was mentioned in my last post that geometry is built into fixed size buffers.  This limits the amount of geometry that can be created by one a procedure.  For a small object, or one of low detail (farther away), this may not be a problem, but if we are to build huge, detailed worlds then it most certainly is.  To overcome this, a number of systems and techniques are used.

Refinement

Models are managed within a spacial octree, each node being responsible for any models that fit reasonably within its own bounds.  Smaller models are managed by the small nodes deeper in the octree.

During the synthesis process, the sub-procedures used (and any bounding information that can be obtained from them) are analysed, and in certain cases stored.  The aim here is to capture a set of sub procedures that fully represent the model built, but as smaller component parts.  These parts can then be used to build more detailed versions of parts of the whole model, and which can be managed by the smaller octree nodes that are more suitably sized.  This effectively provides a way of re-synthesising successively smaller parts of any model as we need the extra detail deeper in the octree.  This 'refinement' process is driven by proximity to the viewpoint, using the deeper more detailed model parts in areas that are nearer the camera.

Successive octree levels, and the geometry managed by each

Authoring

Procedures do need to be built with this process in mind somewhat.  There are certainly ways to help or hinder the process and prevent the system from operating at its best, but the tools provide feedback and diagnostics to help you optimise them.  This is another area that I will dig into in more detail in another post.

Rendering

The rendering engine for Apparance has always been fairly basic as most of the work has been in proving out the procedure synthesis and detail refinement techniques.  All that the renderer needed to be able to do was render some coloured triangles with a couple of fixed light sources.  This was implemented in DirectX 9 and based on a fairly simple cube rendering sample.  Even with no materials, no texturing, and simple primitives I have been able to make quite a wide range of examples.

Small sample of results achieved with basic renderer

The renderer itself has been written to be fairly robust and flexible, with support for multiple viewports, cameras, and scenes, it runs on its own thread, and supports window resizing and device loss properly.

Shaders

Current focus

Driven mainly by the need to start blending between meshes of different detail levels, I decided that I needed to add shader support and this is my current focus.

With the flexibility and power shader based rendering brings I will be able to implement an elegant blending system, as well as better lighting, and start experimenting with more realistic surface properties.

I decided that I should certainly allow run-time authoring of shaders as this is an important premise of the Apparance tool philosophy.  To do this I also decided that the shader code should be procedurally constructed by the same systems the models are built.  Not only does this mean I can easily re-use shader functions and constructs, but pieces of code, and even allowing parameterisation of the shader code itself.  This should have all sorts of interesting effect creation potential.

Trouble

During the testing of DirectX 9 shaders I hit some nasty snags to do with background compilation of shaders during rendering, shader lifetime management, and finally with a crash on ending and releasing of shader resources that I couldn't resolve.  Even using my simple training app I couldn't solve the issue and under Windows 10 it turns out that debugging and diagnostics in DirectX 9 isn't supported, so no help there.  My solution was to bite the bullet and upgrade the engine to DirectX 11, which represents a significant improvement in features and support, as well as being fully integrated into the OS and with significant debugging support.  Unfortunately this did mean learning about all the differences and writing another learning app, but it seems like it will be a good move in the long run as I was probably going to need it at some point anyway and DirectX 11 has some nice improvements in the way you handle shaders that it will be good to get used to.

New rendering and shader test app for DirectX 11

Graphics Fu

Eventually I am going to need some fairly fancy rendering features to show off the models properly, such as multi-texturing, advanced light sources, high quality shadows, ambient occlusion, and maybe even global illumination.  I am treating these as 'solved' problems and prioritising many other, more unique, features over them.  I am also likely to need help with the harder graphics tech and should start to involve others in the project more closely, but that will depend on how much interest I can raise in the project and whether I can find funds to build a team around it in the future. We shall see...

Next

I was going to describe my development setup a little here, but I think I'll leave it until a later post.  Next time I'll talk about the editor and how it is used to develop procedures.

An Introduction to Procedures

As there is a huge amount to cover I'm going to spread it out over multiple posts.  The first few will describe the technology (engine) and after that I'll cover the tooling (editor).  This will by no means cover all the technical detail, but it should give you a good idea of how it works and what (I hope) it will be able to do.

Authoring

Currently the engine allows building of procedural models; objects formed of triangles (and lines) and rendered in a 3D viewport.  These are expressed as 'procedures'; collections of modelling and calculation operations that feed into each other forming a network or graph.  The 'inputs' to this are various constant values within the graph and the output is a 'value' corresponding to the generated geometry.

A simple procedure and the resulting model

Each node is some form of fundamental operator (like add or subtract) implemented in code, or it can be another procedure, itself made from operators and procedures.  The term 'operator' will be used to mean either in the context of a procedures content as once placed down they can be treated in exactly the same way.

A multiplication operator feeding into a luminance procedure

Each operator generally has one or more inputs and one or more outputs, which can be connected up to other operators.  An input can only be connected to one output but an output can connect to multiple inputs.  There are several fundamental data types available for information to be passed between operators.  So far we have: Integer, Float, Bool, Colour, Vector, Matrix, String, Frame, and Model Segment.  These last two are explained more below.  Unconnected inputs are considered constants and the value can be explicitly specified.

Operator inputs of various data types

When creating a procedure, you get to define it's inputs and outputs, and their names and types, these then become available for connecting-to wherever an instance of the procedure is placed down.

Procedures represent blocks of functionality and can easily be used to encapsulate and re-use groups of operators.  For example you might build a colour blend procedure out of mathematical operators if a dedicated operator wasn't available or didn't meet your needs.

Bespoke colour blending procedure

The new blend procedure in use

Operators

There is a small library of built-in operators implemented already to build procedures from, these are roughly divided into:

• Mathematical operations - all the usual maths functions.
• Comparisons and conditional switching - test and flow control.
• Conversion - e.g. changing type or break-out/re-combine (for multi-element types).
• Constants - operator inputs are editable constants, but constant operators are useful for sharing values.
• Modelling - create and manipulate primitives (cube, cylinder, paint, distort, etc).
• Space defining - subdividing and specifying spaces to be used for containing objects (Frames).

Some of the operators available so far

There are hundreds more of these I need to support (something for a future post), but this is plenty for me to test and prove out the principals.  In fact this current limitation means I have to be inventive and really means I push the capabilities of the procedure system to see what I can achieve.

Modelling

Currently there are only two triangle primitives (Cube and Cylinder) and two line primitives (Line and Grid).  The only reason I haven't written more yet is that I have managed to achieve a surprising amount with just these.  All the screenshots you can see so-far are mostly built with the cube operator and an occasional cylinder.  As we will see though they do provide a fair bit of control over how each can actually be used.

Once I got to the point where the geometry synthesis was basically working and I started building shapes I found that a large part of building up objects is actually splitting up the space it is going to occupy into smaller spaces.  This happens at many depths and in many different ways.  There are parallels here to laying out elements on a page or in a user interface, so many concepts like centring, distribution, and offsetting apply equally to 3D space. To facilitate this in Apparance I found a data type to describe an oriented cuboid in space was ideal for this.  These I call 'frames' and operations on them form a large part of the object construction process.

Space partitioning operators in use (highlighted yellow)

Starting with a frame describing the location, orientation, and dimensions of the object being created, you break it down into sub-frames until you reach a point where a single primitive fits exactly, at which point you feed the frame into it generating the geometry needed there.  This aspect of modelling needs a post to its self really :)

Geometry generated by a primitive operator is passed around the graph using a 'Model Segment' data type.  This rather esoteric type is just a way of remembering where in the modelling buffers the vertex and triangle information for that primitive has been put.  A 'combine' operator is available to merge two segments of geometry together so they can be treated as one.  All modelling should result in a single Model Segment output at the top level and it is the geometry enclosed within it that will be displayed.

Part of the appeal (to me at least) of procedural generation is parameterisation.  Anything we build this way can have any aspect of its form exposed as a tweakable parameter.  This may just be the desired size of the object, it might be the thickness of the frame on window, the colour of a building's roof tiles, or the probability of a wonky brick in a wall.  In order for a given parameter to affect the modelling process its value will usually need to be massaged into some other form by using mathematical, conditional, and logic operators.

Synthesis

The process of turning procedures into models that can be rendered it called 'synthesis'.  Starting with a root procedure to be viewed in a 3D scene the synthesis engine starts by instantiating it in memory with any input values needed and requests the geometry via the appropriate output.  This triggers instantiation of all the operators within and their interconnections.  Following the 'flow' of the data connections back from the required output and digging down into procedure within procedure all the functionality needed to produce it is executed.  Requests for output values from leaf operators, ones with actual code behind them causes that code to be executed.  Procedures and operators also call upon their inputs which then cause the evaluation to elevate back up to the level above and follow the connections already in place when the containing procedure was evaluated.

Evaluation tree for the table example

Because procedures are instantiated as they are needed, it can support recursion, i.e. a procedure can include instances of itself.  As long as there are 'exit conditions' defined to limit the recursion depth this turns out to be a really useful way to build a lot of structures.  I discovered early on that this can be used to implement arrays of objects by progressively subdividing until the required object size was reached.  I thought I would need array support explicitly but so far recursion has served well in its absence.

A recursive procedure called "Recursive" that includes itself.

Output of the recursion example

To help with scalability and performance, multiple synthesis runs can be performed in parallel on several separate synthesiser instances.

Four synthesisers running in parallel, busy building geometry

Each has its own pre-allocated chunk of memory as working buffer, used in a non-freeing manner and only reset at the end of each run.  This makes allocation of parameters, values, operator state, and any intermediate data extremely fast and all values effectively immutable, simplifying the operator graph evaluation logic.

A breakdown of how memory was allocated during synthesis

Next

Quite a lot to absorb I'm sure.  I'm happy to answer any questions.  Next time I'll talk about the renderer, some of the less glamorous code supporting everything, and how the project is set up.

What is project "Apparance"?

Videogames

I have been making videogames for over twenty years, and the process always seems to go something like this:

1. Design game
2. Create assets
3. Write code
4. Test game
5. Repeat above in various combinations until done (ship or bust)

These steps are quite separate, mostly performed by different people (or groups of people), and the bigger the game the more of all this you need.

Some time ago I came to the conclusion that above a certain project size this model isn't very efficient for a variety of reasons. Firstly, because assets need to be worked on by multiple people, the friction associated with moving them around (conversion, source control, management, communication, builds, deployment, and loading) can mean lots of additional effort and add up to long and frustrating iteration times.  Secondly, as projects have become more ambitious, with larger scopes and game-play areas where the work involved in creating them increases exponentially (with area/volume), the cost also ramps up this way because the work a team puts in can only, at best, grow linearly with size.  In addition to this, a natural consequence of growing teams is the need for increasing communication, management, and support.  All of these pressures usually mean compromises being made to the game's detail, richness, variety, polish, quality, or seemingly low priority support work is neglected, in order to get it to fit within the budget.

Pondering this problem for several years I have formed a vision of how I think we can improve this construction and authoring process, and what the technology and tools it requires would look like.  I'm fed up with conventional approaches, the waste, the sluggish pace, the repetition, the tighter squeeze on the more fuzzy aspects of creativity like experimentation, play, fun, emergence, and instinct. I want to try working a different way, using novel and interesting techniques, powerful technology and tools, to empower smaller teams to create bigger things.  I believe we can do better, and whilst it won't be everyone's cup of tea, it's something I believe passionately in.

"I'm fed up with conventional approaches, the waste, the sluggish pace, the repetition, the tighter squeeze on the more fuzzy aspects of creativity like experimentation, play, fun, emergence, and instinct."

I've been working on this project in my spare time for around five years, with the original concept rolling around my brain for about five years before that.  As I described in my previous post I am now in the fortunate position to be able to invest all my time in this project.  So, what exactly is it I am proposing?

Vision

I want to enable rapid creation of rich and varied worlds through fully dynamic procedural generation.  I want to encourage play and exploration through fully interactive live editing of content and behaviours.  I want to make games in creative and exciting ways:

• Rich worlds full of the variety that parameterisation and procedural techniques can enable.
• Vast structures with intricate detail from highly scalable dynamic modelling.
• Endless continuous worlds with no loading and no transitions required.
• Don't repeat to save time, every instance can be unique in design and appearance.
• Instant feedback and fast iteration from rapid and responsive tools.
• The play and exploration that real-time control systems with live editing allows.
• Freedom from bulk assets and offline asset pipelines by going fully run-time procedural.
• Increasing the efficiency of creating content through asset modularity and easy reuse.
• Making extensive use of visual editing paradigms (data-flow, node-graph).
• Removing hard coded content, everything data driven.
• Small footprint applications yet with all the richness of today's larger titles.

There is nothing out there that does all this, the technology just don't exist. I see this project as an adventure where we are specialising in procedural generation and real-time interactive experiences that can be applied to a variety of applications; especially games, education, and interactive visualisations. If we aspire to travel to new lands but the vehicles to do so do not exist, we must first build them.

"If we aspire to travel to new lands but the vehicles to do so do not exist, we must first build them."

What it is

This is an authoring environment in which to create your worlds, populate them, and bring them to life. The real-time synthesis engine uses procedural definitions, their input parameters, and the surrounding environment to dynamically fill out the world in more and more detail as you explore.

Apparance Editor - Procedure graph for roof section of an old house

The editor provides authoring and testing of the world through a visual flow based editing model (nodes and wires) to create the procedure definitions describing the geometry, texturing, effects, physics, behaviour, logic, and much, much more.

A standalone player application provides a non-editable view of world, for test, play, and release.
The underlying engine contains the technology driving both editor and game, using highly concurrent operation, with dynamic detail management, largely stateless data-flow control and AI systems, and all with live editing/monitoring during authoring and play.

What it is not

It is not a silver bullet and won't solve everyone's problems; Some games won't fit the development model and technology being created here.  It may be that a game relies heavily on areas that are a particular weakness for this technology. Some games are based on very tightly controlled and designed environments and behaviours. But I believe that working toward its strengths and away from its weaknesses is no different to any existing games development, more so historically as hardware and software capabilities have applied their own unique constraints on possibility.

I don't see it as a plugin for other engines; using a lot of unique techniques to achieve its aims means it may not work well in more conventional engine scene graphs and rendering.  The lack of scalability in many existing systems can cause problems too as non-scaling systems 'crowd-out' the scalable ones as a project grows.  This is a disruptive technology, and relies a lot on the interaction between the various novel techniques at play.

It is not finished yet.  This is a project that I've been working on in my spare time, and whilst showing great promise already, there is still a lot to do to.  I see many stages to its development and I will be building demonstrations and even games to show-case it at various points in the future.

Direction

This is an ongoing project and I want to involve people as it develops.  Through blogging and sharing the results of my work I hope to involve like-minded and creative people to help shape the tech and tools into something that many developers could benefit from.  I am working towards releasing versions of the tools for you all to play with to see what we can create together.  If you are interested in these ideas and the prospect of trying something different please get in touch via email, via my Twitter feed, or just follow progress here on my blog.

The Name

Logo design idea - featuring one of the first models built with the system.

Apparance (or often Apparence) refers to the outward manifestation of a person or thing, the act of being visible, what others see, the process of creating this external impression. The original inspiration for this comes from "Against A Dark Background" by Iain M Banks; The Solipsists don't subscribe to the concept of the individual with no use of the word 'me', referring to, and in a way justifying, the existence of other members of their group as 'apparences'.

Next

Next time I will bring you a full project status report, showing the current state of play and what I'm working towards at the moment.  It should be a very visual post as there is lots to see.

New Beginnings

Tragedy

Nearly three months ago now, everyone at Lionhead was called into a company meeting at short notice to be told that Fable Legends was probably going to be shut down and the studio along with it.  Roll forward to the end of April and the axe finally fell leaving over a hundred developers, a four year project, a smaller side project, and a 20 year old studio, gone.  I don't expect we will ever know the true logic applied by Microsoft to drive this decision, and I'm sure many will speculate on this for a long time. However, due to both the huge amount of talent among the staff and a great rally of  support from the industry I am please to say most (if not all) of my displaced colleagues have found new roles, pursuits, challenges, and adventures to move on to.

The last Lionhead team photo

Opportunity

In a way I have a little difficulty coming to terms with, such tragedy has led to the opening up of a significant opportunity for me.  It has become an enabler, in the form of both a financial safety net and huge prod, to do something new.  I had been at Lionhead for over eleven years and seen a lot of change, but I was still happy there building tools and technology to support the team in making the fantastic Fable visions come to life.  I have mused about quitting to develop my personal projects before, but either the numbers didn't stack up, or the time wasn't good, or I was enjoying work.  This however presents an unavoidable boot up the backside to give it a go... and that is what I am doing.

Cogitation

This infrequently updated (and lesser read) blog that you are reading now has been my attempt to slowly document development of a spare-time project over the last five years.  It is based around a collection of ideas I've been pondering since around 2001, gradually crystallising all the different elements of what is needed to produce some exciting procedural technology that I feel should exist.  I will go into the project details in my next post, trying to finally bring together and explain all the many aspects of my vision, for now you can refer to some of my previous posts for an overview.

Introspection

During the redundancy process we all felt a bit in limbo, not really sure what was happening, or what we should be doing.  With no work to do, we were largely at a bit of a loss.  Some dusted off their CVs and job hunted, some taught themselves new things, some tinkered, some played games, some just didn't come into work.  I had some loose ends to tie up but after a while started spending more time at home on some long overdue bathroom renovation.  This served as a productive use of my time, keeping me busy, and also providing some much needed thinking space, to mull over the possibilities and directions to go.

Preparation

Redundancy included various consultation sessions and training opportunities, which I decided to give a try.  These turned out to be very useful, providing me with a good overview of self-employment, and advice on using social media and the web to drive business.  Conversations with a few friends and family members about my thoughts, concerns, and plans helped too.  Going forward I decided (at least to start with) that a self-employed, one-man-band operation would serve me best.  I have a nice office set up in our spare room (a pleasant environment with desk, chair, PC, light, supplies, etc).  Office hours have been decided (9 to 4:30, flexible).  House rules have been negotiated ('Daddy will still be working when you get home from school'). I am good to go.

Action

This weekend marks the end of my first week of my new job; working for myself, developing my project full-time.  It was always going to be a strange experience, having been an employee all my working life, but I can see myself getting used to it fairly quickly.  I like the fact that I am working on something I am truly passionate about, something that can only come from being the source of the ideas. It is weird that my spare-time project has replaced my work project and that my spare-time will need to become something else.  I will have to be careful to dedicate spare time to more varied interests as I now more than ever need to keep myself mentally balanced.

Prediction

It's a hugely exciting prospect to develop my project properly, investing the time it deserves, and giving it my full focus.  I am pretty certain that it will result in some impressive and useful results and I can't wait to share with you fruits of my labour and the process by which I achieve them.  Please join me as I embark on this adventure. I will take you through some of the business, and personal sides of it but mostly this will be a technical blog (and there is a lot of technical involved).  Exciting things are on the horizon; watch this space...

Procedural Closures II

I now have enough of the four elements described in the last post implemented, to try out the dynamic hierarchical detail generation described.  Here's a (rather extreme) test case in action:

Menger Sponge - extreme subdivision test case

Here you can see successive sub-procedure expansions being made as the camera approaches the model.  Apart from the complexity involved in splitting up the space into 27 parts and adding 20 sub- cubes each time, the procedures - being recursive - are fairly simple.  This test case rapidly runs out of memory; a) because the object complexity escalates by a factor 20 times with each depth level, but also because b) I don’t recycle generated blocks yet. :)

After playing with it for a while and trying to build more conventional objects this way (house/street, simple landscape), I've decided that my approach to the authoring of them leaves a lot to be desired and will have to change.  To summarise the process, it works in the following way:

1. Build a hierarchy of procedures to describe your object, parameterised by (at least) the space it is to occupy and the detail level required, and with a secondary geometry output for capture. Each sub-procedure should be parameterised in the same way where possible. 
2. Procedures suitable for capturing (i.e meeting some criteria) are explicitly marked as such in the editor by selecting which inputs and outputs are to be assigned each meaning required (above).

 My main gripe with this process is the amount of extra mess this introduces when building procedural models.  Secondary (and somewhat related) is the fact that it is proving very hard to explain how to author models in this way.  It is just too explicit, too manual if you like.  I need a better way, and it needs to be as automatic as possible. 

I have been hypothesising that it is possible to gather information about a procedural hierarchy as it is being synthesised and perform an analysis of the structure such that suitable capture points can then be extracted and kept for subsequent detail generation automatically.  To help better understand the problem space, i.e. what exactly does the synthesis process look like, I decided to get Apparance to spit out enough information to visualise the process.  This would then allow me to browse the evaluation tree, looking for patterns, surprises, and insight into the rules that might be used to extract what was needed.

Visualising data you say. Trees you say. Well, a tweet from a friend of mine reminded me of GraphViz which I'd seen used to analyse large asset database dependency graphs in the past but not had cause to investigate myself…  a quick download, documentation read, and 'hello world' graph and I was ready to give it a go.  In a couple of evenings I'd got the evaluation tree for a test house displayed nicely.

House synthesis graph (click to enlarge)

 Salient features you can see here are: 

• Procedures are shown as boxes, operators as plain text, each with, any sub-procedure/operator evaluations at the end of arrows heading to the right. 
• Branches that yield geometry are solid, dotted branches are for any other types of value. 
• Branches that lead to at least one procedure are black, those that only lead to operators are grey. 
• You can see operators (or procedures) that appear to be missing further evaluations where those evaluations have already been performed by an earlier evaluation and are effectively cached. 
• Procedures that contain both procedures and operators that generate geometry are the lowest point that can be captured since operators aren't designed to be captured (as they won't sub-divide).  I have (manually) marked such procedures in red.
• It is clear that this house structure (as it stands) isn't best suited for sub-division as there are only a few suitable capture candidates high up the synthesis tree.

Initial thoughts lead me along the following lines:

1. It should be possible to evaluate and automatically capture procedures. 
2. Some means of hinting at 'bad' structure should be presented in the tools to guide model development. 
3. There may be more rules to be added later.
4. Improvements in the tools can be made to aid the construction process, for example: quick sub-procedure creation from selected elements.

Anyway, I've waffled on for quite long enough.  I shall endeavour to work on this.

Next time I'd like to talk about 'detail'; what it means and how to quantify it.

Procedural Closures

Previously I've been working on proving the synthesis and editing process of a single 'object' within a fixed geometry/calculation memory budget (the synthesis buffer).

Since this doesn't scale and won't let me achieve the ambition of large procedural worlds, we need something more.  This is where Phase II comes in; a system to hierarchically synthesise multiple objects of successively reducing scale and increasing detail when and where it's needed.  This is based around a kind-of 3D subdivision of the world with more subdivision and detail near the observer and less further away.  There are a number of parts to this investigation:

• The synthesis system needs to support the concept of closures; snapshots of a procedure and its input values that can independently drive further synthesis.
• Graphical bodies need to support sub-bodies, each capable of containing procedure closures to drive successive synthesis steps.
• Procedures need to be able to generate higher detail closures from sub-sections of a procedural model to be used to populate these sub-bodies.
• The renderer needs to be able to switch between low density bodies and equivalent high density sub-bodies according to distance from camera (and when they are available to draw).

This is the bare minimum needed to effectively prove the principals of phase II and enable generation of scenes much much larger than a single synthesis step can generate.
Wish me luck, I'm going in...

Approaching Phase 2

Oh my, yet another year.  Not big on progress news are we.  Well, I have not been idle (except where blogging is concerned), and a lot has happened.  Let's start with a screenshot:

This is the current state of my experimental procedural modelling system, called 'Apparance'.  Here you can see a number of significant improvements over my initial prototype.  I have visual editing of the procedures, fairly advanced property editing panel, and a bigger list of operators to build with.  This represents the final stage of what I have come to call 'Phase 1'.  This is where I prove the first key element of my project; that of visual editing to drive procedural modelling.  I also wanted to take it to the point that it could stand up as a 'minimal viable product' that someone could play with without any need for tricks, manual hackery and workarounds.  I am considering releasing the binaries for people to play with to get some feedback and test the water a bit.  I've collected a few more pictures from its development history over the last year in an Apparance development gallery.
No promises when, but next time I'll talk about Phase 2...