Project Progress Report

Sorry for the lull, it's been a bit of a mixed-up month and this will be a similarly mixed up blog post.  Having reached the deadline for demo, web site, and release preparations at the end January things have changed pace and direction a little.  I've rolled an Alpha build to a few people, the web-site has gone live, and I've been out job hunting too!

Goals

The end of January was my deadline for a number of aspects of the project, and this has meant February has been a bit of a flurry of activity releasing things and talking to lots of people about the project (and work prospects).

Demo

As with most things, you never get as much done as you'd like to.  No exception with Future City, but I am mostly happy with how it looks.  It demonstrates the core premises of the project; large environments, varied content, wide-scale parameterisation, high detail, smooth blending, and powerful modelling system.

I'm pleased with how the demo "Future City" has developed

I will continue to develop the city as there are many things I'd like to do with it, but it is also going to be the basis for the first Apparance game.  It will be taking a back seat to other development for now though.

Web Site

The Apparance website, bringing together information about all aspects of the project, is now live at www.apparance.uk.

The new Apparance web-site (procedurally generated too!)

I'm pleased with how it's come out, and the procedural generation aspect is proving very powerful.  A good test of this sort of thing is how easy you find maintenance on returning to it after a while away.  I've extended the site a bit, added new templates and image features with ease.  Updating content is just a question of editing the markdown documents and running the Apparance Processor.  I'm enjoying working with it this way, and looking forward to adding new pages and features as needed.

Release

The 1st of February saw the start of the Alpha release of the project.  The first time anyone has got their hands on Apparance for a couple of years.  It's really been too long coming, but I wanted to get to a particular point and combine it with the website and the city demo.

Apparance editor, currently in (closed) Alpha

It's not a huge audience, and only some of my more technical friends, but I've been getting some solid feedback.  Plenty to go on and lots to talk about.  I've been working a bit of it back into the editor, and site, ready for the closed beta due to start in a few days.  The beta is a larger group, and taken from a wider circle of acquaintances (mostly via Twitter).  This group contains a higher concentration of people who are specifically interested in procedural generation, and should provide a deeper level of feedback.  I can't wait to hear how you folks get on.

Project

So, where does this find us now?

Status

The current set of goals, largely based around proving the more unique aspects of the project, has been proven (in my eyes at least).  Now I have people starting to experiment with, and explore the tools I have a source of objective information to use to potentially steer and drive the project.  I'd like to see a variety of outcomes from the project, but now is time to discuss these and investigate the options.

Support

As an authoring tool, and one with some pretty unusual concepts to learn, support for Apparance users is going to be important at this stage.  This is especially true if I want to nurture a community and have people join me in this adventure.  Documentation, tutorials, and two-way communication are all important parts of this, and typically under provided so I need to be careful to keep feeding this.  I've been asked for a variety of support, from examples, tutorials, videos, features, and improved documentation.

Games

Apparance is a realisation of my vision for building games, and as such needs to continue to move towards an actual game.  I need to show that this is viable, and with the ambitious technical goals too.  I have some exciting game ideas brewing at the moment (several sleepless nights spent designing game-play and story).  Unfortunately, the next step with regard to games (entities, behaviours, and interaction) is quite a hefty piece of work.  This is going to need to be spread out.

Product

I have also begun thinking about the more practical aspects of the project, and other ways the technology could be used.  By creating additional types of product out of the engine and tools it might provide a way to broaden the potential audience (and hence market) and provide a model to generate revenue that isn't game based.  Yes, I am actually considering how Apparance could be modularised so that elements of the technology could be plugged or integrated into other engines, including Unity and Unreal.  I know I haven't been supportive of this approach, but economic pressures do make you think about your options, and there might be ways to do this without compromising the vision of the project.  Again though, this isn't an insignificant amount of work, and requiring a lot of research and ongoing development and then support.

Next

I've really enjoyed the luxury of working full-time on Apparance for (nearly) the past year, but financial pressures are starting to rear their head.  I don't believe I am in a position to pitch what I have technology-wise, and certainly not game-wise, to a third party for funding.  There is too much still to do and I don't have the resources to continue full-time.  As a result I am in the process of negotiating a more traditional source of income so I can continue to support my family properly.

Apparance is most definitely up and running and I'm pleased with what I have achieved so far.  I will continue to support the alpha/beta, blog about the project, work on updates, and start on some of the bigger features over the coming months, but in my spare time (again).

Thank-you for sailing with me through this exciting year of Apparance development.  I look forward to seeing where the wind blows us over the next year.

Sam

What is project "Apparance"?

Videogames

I have been making videogames for over twenty years, and the process always seems to go something like this:

1. Design game
2. Create assets
3. Write code
4. Test game
5. Repeat above in various combinations until done (ship or bust)

These steps are quite separate, mostly performed by different people (or groups of people), and the bigger the game the more of all this you need.

Some time ago I came to the conclusion that above a certain project size this model isn't very efficient for a variety of reasons. Firstly, because assets need to be worked on by multiple people, the friction associated with moving them around (conversion, source control, management, communication, builds, deployment, and loading) can mean lots of additional effort and add up to long and frustrating iteration times.  Secondly, as projects have become more ambitious, with larger scopes and game-play areas where the work involved in creating them increases exponentially (with area/volume), the cost also ramps up this way because the work a team puts in can only, at best, grow linearly with size.  In addition to this, a natural consequence of growing teams is the need for increasing communication, management, and support.  All of these pressures usually mean compromises being made to the game's detail, richness, variety, polish, quality, or seemingly low priority support work is neglected, in order to get it to fit within the budget.

Pondering this problem for several years I have formed a vision of how I think we can improve this construction and authoring process, and what the technology and tools it requires would look like.  I'm fed up with conventional approaches, the waste, the sluggish pace, the repetition, the tighter squeeze on the more fuzzy aspects of creativity like experimentation, play, fun, emergence, and instinct. I want to try working a different way, using novel and interesting techniques, powerful technology and tools, to empower smaller teams to create bigger things.  I believe we can do better, and whilst it won't be everyone's cup of tea, it's something I believe passionately in.

"I'm fed up with conventional approaches, the waste, the sluggish pace, the repetition, the tighter squeeze on the more fuzzy aspects of creativity like experimentation, play, fun, emergence, and instinct."

I've been working on this project in my spare time for around five years, with the original concept rolling around my brain for about five years before that.  As I described in my previous post I am now in the fortunate position to be able to invest all my time in this project.  So, what exactly is it I am proposing?

Vision

I want to enable rapid creation of rich and varied worlds through fully dynamic procedural generation.  I want to encourage play and exploration through fully interactive live editing of content and behaviours.  I want to make games in creative and exciting ways:

• Rich worlds full of the variety that parameterisation and procedural techniques can enable.
• Vast structures with intricate detail from highly scalable dynamic modelling.
• Endless continuous worlds with no loading and no transitions required.
• Don't repeat to save time, every instance can be unique in design and appearance.
• Instant feedback and fast iteration from rapid and responsive tools.
• The play and exploration that real-time control systems with live editing allows.
• Freedom from bulk assets and offline asset pipelines by going fully run-time procedural.
• Increasing the efficiency of creating content through asset modularity and easy reuse.
• Making extensive use of visual editing paradigms (data-flow, node-graph).
• Removing hard coded content, everything data driven.
• Small footprint applications yet with all the richness of today's larger titles.

There is nothing out there that does all this, the technology just don't exist. I see this project as an adventure where we are specialising in procedural generation and real-time interactive experiences that can be applied to a variety of applications; especially games, education, and interactive visualisations. If we aspire to travel to new lands but the vehicles to do so do not exist, we must first build them.

"If we aspire to travel to new lands but the vehicles to do so do not exist, we must first build them."

What it is

This is an authoring environment in which to create your worlds, populate them, and bring them to life. The real-time synthesis engine uses procedural definitions, their input parameters, and the surrounding environment to dynamically fill out the world in more and more detail as you explore.

Apparance Editor - Procedure graph for roof section of an old house

The editor provides authoring and testing of the world through a visual flow based editing model (nodes and wires) to create the procedure definitions describing the geometry, texturing, effects, physics, behaviour, logic, and much, much more.

A standalone player application provides a non-editable view of world, for test, play, and release.
The underlying engine contains the technology driving both editor and game, using highly concurrent operation, with dynamic detail management, largely stateless data-flow control and AI systems, and all with live editing/monitoring during authoring and play.

What it is not

It is not a silver bullet and won't solve everyone's problems; Some games won't fit the development model and technology being created here.  It may be that a game relies heavily on areas that are a particular weakness for this technology. Some games are based on very tightly controlled and designed environments and behaviours. But I believe that working toward its strengths and away from its weaknesses is no different to any existing games development, more so historically as hardware and software capabilities have applied their own unique constraints on possibility.

I don't see it as a plugin for other engines; using a lot of unique techniques to achieve its aims means it may not work well in more conventional engine scene graphs and rendering.  The lack of scalability in many existing systems can cause problems too as non-scaling systems 'crowd-out' the scalable ones as a project grows.  This is a disruptive technology, and relies a lot on the interaction between the various novel techniques at play.

It is not finished yet.  This is a project that I've been working on in my spare time, and whilst showing great promise already, there is still a lot to do to.  I see many stages to its development and I will be building demonstrations and even games to show-case it at various points in the future.

Direction

This is an ongoing project and I want to involve people as it develops.  Through blogging and sharing the results of my work I hope to involve like-minded and creative people to help shape the tech and tools into something that many developers could benefit from.  I am working towards releasing versions of the tools for you all to play with to see what we can create together.  If you are interested in these ideas and the prospect of trying something different please get in touch via email, via my Twitter feed, or just follow progress here on my blog.

The Name

Logo design idea - featuring one of the first models built with the system.

Apparance (or often Apparence) refers to the outward manifestation of a person or thing, the act of being visible, what others see, the process of creating this external impression. The original inspiration for this comes from "Against A Dark Background" by Iain M Banks; The Solipsists don't subscribe to the concept of the individual with no use of the word 'me', referring to, and in a way justifying, the existence of other members of their group as 'apparences'.

Next

Next time I will bring you a full project status report, showing the current state of play and what I'm working towards at the moment.  It should be a very visual post as there is lots to see.

New Beginnings

Tragedy

Nearly three months ago now, everyone at Lionhead was called into a company meeting at short notice to be told that Fable Legends was probably going to be shut down and the studio along with it.  Roll forward to the end of April and the axe finally fell leaving over a hundred developers, a four year project, a smaller side project, and a 20 year old studio, gone.  I don't expect we will ever know the true logic applied by Microsoft to drive this decision, and I'm sure many will speculate on this for a long time. However, due to both the huge amount of talent among the staff and a great rally of  support from the industry I am please to say most (if not all) of my displaced colleagues have found new roles, pursuits, challenges, and adventures to move on to.

The last Lionhead team photo

Opportunity

In a way I have a little difficulty coming to terms with, such tragedy has led to the opening up of a significant opportunity for me.  It has become an enabler, in the form of both a financial safety net and huge prod, to do something new.  I had been at Lionhead for over eleven years and seen a lot of change, but I was still happy there building tools and technology to support the team in making the fantastic Fable visions come to life.  I have mused about quitting to develop my personal projects before, but either the numbers didn't stack up, or the time wasn't good, or I was enjoying work.  This however presents an unavoidable boot up the backside to give it a go... and that is what I am doing.

Cogitation

This infrequently updated (and lesser read) blog that you are reading now has been my attempt to slowly document development of a spare-time project over the last five years.  It is based around a collection of ideas I've been pondering since around 2001, gradually crystallising all the different elements of what is needed to produce some exciting procedural technology that I feel should exist.  I will go into the project details in my next post, trying to finally bring together and explain all the many aspects of my vision, for now you can refer to some of my previous posts for an overview.

Introspection

During the redundancy process we all felt a bit in limbo, not really sure what was happening, or what we should be doing.  With no work to do, we were largely at a bit of a loss.  Some dusted off their CVs and job hunted, some taught themselves new things, some tinkered, some played games, some just didn't come into work.  I had some loose ends to tie up but after a while started spending more time at home on some long overdue bathroom renovation.  This served as a productive use of my time, keeping me busy, and also providing some much needed thinking space, to mull over the possibilities and directions to go.

Preparation

Redundancy included various consultation sessions and training opportunities, which I decided to give a try.  These turned out to be very useful, providing me with a good overview of self-employment, and advice on using social media and the web to drive business.  Conversations with a few friends and family members about my thoughts, concerns, and plans helped too.  Going forward I decided (at least to start with) that a self-employed, one-man-band operation would serve me best.  I have a nice office set up in our spare room (a pleasant environment with desk, chair, PC, light, supplies, etc).  Office hours have been decided (9 to 4:30, flexible).  House rules have been negotiated ('Daddy will still be working when you get home from school'). I am good to go.

Action

This weekend marks the end of my first week of my new job; working for myself, developing my project full-time.  It was always going to be a strange experience, having been an employee all my working life, but I can see myself getting used to it fairly quickly.  I like the fact that I am working on something I am truly passionate about, something that can only come from being the source of the ideas. It is weird that my spare-time project has replaced my work project and that my spare-time will need to become something else.  I will have to be careful to dedicate spare time to more varied interests as I now more than ever need to keep myself mentally balanced.

Prediction

It's a hugely exciting prospect to develop my project properly, investing the time it deserves, and giving it my full focus.  I am pretty certain that it will result in some impressive and useful results and I can't wait to share with you fruits of my labour and the process by which I achieve them.  Please join me as I embark on this adventure. I will take you through some of the business, and personal sides of it but mostly this will be a technical blog (and there is a lot of technical involved).  Exciting things are on the horizon; watch this space...

Approaching Phase 2

Oh my, yet another year.  Not big on progress news are we.  Well, I have not been idle (except where blogging is concerned), and a lot has happened.  Let's start with a screenshot:

This is the current state of my experimental procedural modelling system, called 'Apparance'.  Here you can see a number of significant improvements over my initial prototype.  I have visual editing of the procedures, fairly advanced property editing panel, and a bigger list of operators to build with.  This represents the final stage of what I have come to call 'Phase 1'.  This is where I prove the first key element of my project; that of visual editing to drive procedural modelling.  I also wanted to take it to the point that it could stand up as a 'minimal viable product' that someone could play with without any need for tricks, manual hackery and workarounds.  I am considering releasing the binaries for people to play with to get some feedback and test the water a bit.  I've collected a few more pictures from its development history over the last year in an Apparance development gallery.
No promises when, but next time I'll talk about Phase 2...

Another project takes flight

Goodness, a whole year has passed without a word. Until recently this was also the case with any form of code. I am now however spending most evenings working on something not entirely unrelated to the project. The progress I am making with it is very rewarding and I am programming with passion again. Let me talk about it for a moment...

For a long time I've been thinking about the virtual circuits project, as I hope this blog illustrates, but I have come to find it too hard to work on due to its size. I'm sure I would progress gradually, but it seems to have petered (yet) out again. Along-side this, a second great interest has occupied me with almost equal power, a secondary project idea I have invested a massive amount of thought cycles to over the years; that of procedurally generated worlds. Now without going into it too much at this stage I decided to start prototyping it up and found myself making some significant progress, even re-using some of the virtual circuits codebase I had already laid down. I wouldn't describe this second project as a new direction; it is more a parallel track as there are overlaps and similarities. In fact, I have come to realise that working of this new technology is going to stand me in good stead for tackling the bigger project that now lies underneath on the pile. The following strikes me:

• As I am re-using some of the tech, any work on developing that will benefit both projects.
• The dataflow paradigm is something they both share, all-be-it with some subtle differences, such that development of the model, views and tools is largely transferrable.
• The procedural worlds would be a great test case for the virtual circuits as I envision them being the most powerful way to 'run' the world functionally.

Based on this assumption, I am going to roll the new project into the remit of this blog. It seems like a good idea to keep focussed on a single blog for both rather than start a separate one. In fact, the overlaps between them would just be a pain across two blogs.

So, what have I been busy on? So far I have set up a very basic native DirectX based render system supporting multiple viewports, cameras, scenes, layers, and bodies, with proper resize, device reset, and threading support. This serves as the raw engine into which procedural graphics synthesizers feed meshes, and eventually textures, materials, and more. I have an initial implementation of the synthesis system with multiple synthesizers running on separate threads, the start of an operator library, and data driven procedure definitions. All this is hosted within a managed WPF test app to provide simple viewport/camera/catalogue and textual (XML) procedure editing. Using a system called a Network Bonded Hierarchy I was developing for the VICE project to communicate between the engine and editor the managed-native interop has be reduced to three pinvoke functions. All that’s probably a lot to take in at once; and I hope to expand on this in future posts. For now, though I’ll leave you with a screen shot using just a cube operator and few other basic logic and maths operators to create an interesting recursive shape.

A realisation on progress

I've inadvertantly managed to make some progess, not physical progress, but psychological progress. A couple of nights ago I woke at 3am and spent the rest of the night mulling over some of the problems I face with this project. I decided to apply some very focussed thought to a single particular problem; that of synchronising data flowing between concurrent circuits (between threads of the same process, between processes on the same machine, and between machines via the network). It wasn't the progress I made on mapping out or solving the problem, it was the realisation that after writing it up yesterday that this was actually useful progress. It now seems an obvious thing to say, but this approach which seems to be along the 'divide and conquer' line may actually be the way forward.

I've tried this approach before, but I don't think I had the right framework in which to accomodate the results such that they would remain useful. Part of the problem has been how best to record my thoughts and expanding design; separate files, big word document, web pages, wiki, spreadsheets, one-note, visio diagrams, database, bespoke tools, etc. Having just tackled a very specific task I was looking for the best place to document it and realised it was really just the fleshing out of a small sub-section of the technical design overview. Not wanting to clutter the overview I rearranged the documents slightly so that this tech design doc acted as the to-do list and fill out another document with the details. This will form a technical implementation guide, some design, some solution, some thinking, and act as an execution plan for all the individual parts.

So, I didn't realise it, but my overview gives me a detailed enough skelton that can be fleshed out bit by bit and more-or-less at random. This means several things:
1. If I get stuck on (or bored) of part of the design I can switch to another.
2. I build up an implimentation guide to follow for when I actually write the code.
3. I am able to design a very large and complex system having broken it down into small parts.
4. I get to see really how large and complex a project this is (eek!).
5. It also breaks the deadlock with the hierarchy navigation UI problem I've been stuck on.

I guess none of this is new wisdom, but coming to the realisation myself is more potent than being taught it. Here's to progress!

The longest journal starts with a single word

Welcome to my journal. I hope for this to become a place for me to record, reason, and reflect on thoughts and ideas for an ongoing project of mine. I'd like to have a go at describing, divulging and developing these ideas somewhat to see if it helps build them into a working project, or at least help maintain some forward momentum.
I've always admired great scientists and inventors for the doggedness with which they maintain their notes, but it's something I've never been able to do for very long. I've kept a small journal before for this project, and it did help, but eventually petered out. It took the form of a cross between rambling, lists of ideas, and conversations with myself. This sounds to me like perfect blog material so I thought I'd give it a go.
I'm not sure how public this really needs to be, or if I need any external feedback at all. I think just being able to talk things through with myself will be invaluable, and moments of re-reading my words may give me that external perspective. If people do read this, and are interested, then maybe that will help spur me on to put more work into the project and ultimately fulfil my ambition.
I'll talk about the nature of the project in my next post.
Wish me luck.