Future City: Update 11 - Block Interfacing

Progress on Future City has been slowed by web-site and release preparation work at the moment.  I've split my time about 2:2:1 between Web site, Future City, and Alpha Release preparations.

January push towards Web Site, City demo, and Alpha Release.

This split in attention as well as the pressure of the approaching alpha release means blog posts have been slower forming.  Let's just squeeze out a quick update on the Future City progress...

Future City

At the moment I've been focusing on the support of variable elevation across the city, with each city block having its own height.

Example of blocks with shared connectivity but height discontinuities

This feature results in rather complicated requirements at the interfaces between blocks as well as complicating the process of sub-dividing up the space within them.  As such, it's been quite slow progress working out (a) exactly what is required, (b) how to perform this, and (c) how to implement this.

Requirements

Based on the previous connectivity work which sub-divided the city up into districts/zones and finally blocks, we have already decided on what the connectivity between each block looks like.

Now we need to decide on what the content of the block should be such that it is possible to guarantee that adjacent, separately generated content aligns properly, functions as a seamless transition, and looks plausible.

Inputs

We have the following information about each block boundary:

• Block location/bounds/size.
• Access along each side (left/right of opening within any barrier present).
• Follow the underlying landscape shape, or both share flat ground level at a given height.
• Connectivity via each side (to the outer most city sides).
• Variation seed to drive arbitrary design decisions.

Outputs

From this we need to populate the block with rectangular sub regions.  The main problem we are trying to solve here is to transition from a boundary with zero, one, or two accessibility transitions along each side into spaces for content to go that all have single accessibility along each side, these being simpler requirements to satisfy.  Thus the outputs are of the form of Content Spaces with the following properties:

• Bounds size, orientation and location
• Surface height within the space (or if we are to follow the landscape)
• Is it an interfacing piece, i.e. responsible for handling transition to neighbouring block interface? (Edge - one interface along block boundary, Corner - two interfaces along block boundary, Side - three interfaces along block boundary)
• If not an interface, then an interior piece, with same type all round (flat or landscape)
• For each boundary interface: Specific flat elevation or landscape following. Do we 'own' this side (are responsible for interfacing geometry). Is this a barrier, i.e. will access be blocked? Is there a barrier to the left/right that we may need to provide transition geometry for?

Test Rig

Since there are a lot of combinations to test in forming this system it is worth generating a set of 'unit tests' with clear visualisation of the inputs and outputs.  By creating a hierarchy of test procedures we can quickly generate tests for all the major side and access combinations.

• Ownership - all, none, adjacent, opposite
• Surfaces - all landscape, all flat, mixture
• Access - Closed corners, all open, all closed (bar one), some on left, some narrow, mixture

Unit tests for a lot of the block interfacing cases

For cases where a specific issue is being resolved there is the option to switch out all but one test scenario.

Example test rig showing corner results produced

Algorithms

I spent a lot of time in Excel playing with block boundary examples and possible approaches.

Playing with block algorithm ideas in Excel

From this I've come up with a lot of ideas, but they seem to be heading towards this sequence of calculations:

First pass at the content space and interfacing algorithm

There are bound to be some changes as implementation proceeds, but this is a good starting point.

Lists

Structured Data

The amount of structured information and variability in generated output finally became too much and I had to stop to add some functionality to help support this.  Previously where several properties about a single thing needed to be conveyed through several procedures it had to be passed individually, and explicitly.  This leads to this specific flavour of spaghetti:

A specific flavour of spaghetti: arrays of connections

The algorithms that were forming to perform the interfacing and content-space generation were heading towards an awful lot of this.  From the requirement analysis above we can see each block has at least seven properties that go together, and each potential output content space has at least six.

Traditionally these would be encapsulated in structures, where a single type is used to express an explicit collection of typed data.  This is something that Apparance will natively support in the future, but it's quite a lot of work, especially on the tooling side where composition/decomposition of structures needs to be handled elegantly.

Variable Quantities

Another aspect of the block interfacing implementation problem is that there are several places where we need to pass a variable number of structures.  At the moment there is no concept of this and we would have to pass a fixed (maximum) number of structures (as individual properties) along with an additional flag expressing whether it was in use.  This forms a very, very unwieldy situation where all sorts of operations, such as combining two variable sized collections, become incredibly complicated, expensive and error prone.  We really need some form of list or array to handle this.

Solution

Both these prospects are really going to cause too much of a headache to work on the already complex problem, so we need support to help this.

Looking at other language for inspiration, we find that both these problems can be addressed by using lists of dynamically typed objects.  If a list happens to contain all the same type, then it is effectively a traditional list or array, i.e. a variable sized collection.  If a list happens to contain a fixed number, of different typed elements then it can be used as a structure.  Adding support for lists as just another base data-type to Apparance is fairly straight-forward and then means we can have lists of lists, or put another way, lists of structures.

Implementation

The List data type was added along with an initial set of basic operators to support what we would need, as follows:

• Append - take a list and add another element (any type).
• Get - index a list, returning the element value (any type).
• Set - replace a value at a specific index with a new value (any type).
• Count - query how many elements are in the list.

Part of the implementation that took a bit of effort was that I wanted type information to be maintained with them as the lists are constructed.  With this, a lot of validation can be performed during synthesis that would catch problems that would otherwise be very difficult to diagnose and have potential to crash the engine.

Whilst a bit tricky to implement, lists are proving very useful so far and make a lot of the block interfacing problem much more pleasant to work with.

Passing structures and collections around as lists

I am even now using them in the web site building procedures for all sorts of things.

Progress

I'm about half way through the Block Boundary -to- Content Space processing system and have promising results so far.

Some of the content space generation results so far

I'll post more results as I progress.  In the mean-time I have an Alpha Release to get out!

Procedural Website? - Part 2

Recap

In Part 1 we proposed using Apparance as a text processor and tool automation controller to build our web-site.  So, how exactly would we go building about this?

The Apparance web site prototype.

Page Structure

Browsing a few basic web tutorials on how to set up a fairly standard CSS styled HTML page we are looking at this:

1. Title banner and logo
2. Site navigation bar
3. Page content, different for each page
4. Footer with a few common links and note.

We can encapsulate the shared and custom element construction in a top-level page procedure.  This wraps a procedure for emitting the standard HTML page sections and any extra bits all pages need (metadata, etc).  This is the point that the CSS for the page and the body HTML are passed in as parameters.

The top-level page generation procedures.

This provides a single procedure that can be re-used on each site page, with the page specifics and customisations passed in as parameters.  For example, here it is used on the homepage.

All that is needed for a top level, text content based page.

All that is needed for a page like this is the content source and a couple of constant values.

Title

I wanted the title to use the Apparance logo, and have a background that hinted at both what you can make with Apparance, and how you make it.  This is a combination of background image banner and centred logo.

Title banner background (transparency shown as checker-board)

Title logo (transparency shown as checker-board)

Both of these are images that are prepared by hand (at the moment) and just need to be copied across if they are updated or missing, this is handled by the Image Fetch procedure (see below).

Header background (CSS) and logo (HTML) generating procedures

There is a bit of styling to apply the background, and the logo is an image HTML element over the top.

Navigation

The navigation bar should be a simple row of buttons with hover highlighting and current page indication.  These provide the top-level site choices, the leftmost one being the main homepage and site entry-point.

The main navigation bar.  Current tab and hover highlighting shown.

To do this properly, the idea is to have the items expressed in a simple a way as possible in the HTML and leave the funky appearance to the CSS.  The best of conventional wisdom points at using an un-ordered list (UL tag) with list items (LI tag) for each menu option.  With the appropriate tagging, this can then be 'skinned' into a row of buttons using only CSS.

Again, this can be a single procedure that gets re-used on each page.  In fact, some of the pages will require a second row of buttons to navigate sub-pages (too many pages to have them all in one row).  We can build this procedure (and the CSS) to support this.

The bar and button creation procedures.

The CSS generation procedure for the navigation bar and buttons.

The procedure to generate the CSS is designed to be applied twice, with the menu level as a parameter.  This allows the styling to be slightly different and provide a raked appearance to the menus as they transition into each other and the page content.

Example of a double level navigation bar.

To get the to show the correct selected page in each place it's used, the page it is on is passed as a parameter which causes that button to have the selection highlight styling applied.

Footer

The page footer is very simple, just a few links and some text.

Simple page footer

It is included in the general page procedure as it appears at the foot of every page.

Common footer procedure and where it fits in.

We may need to include some customisation later so that when on pages it actually links to, those links are disabled (for neatness).  There are other cool things we could do with the footer later I'm sure.

Content

Markdown

The main part of every page is the content and comes, for most of them, from external markdown files.  This allows the content to be written in a form that can be transformed into HTML as well as other forms if we need (see later).  Here is an example of what the content looks like in raw markdown and on the actual page.

Markdown source document compared to resulting HTML page output.

Custom

Some pages warrant custom generation and rely on special procedures to generate their content.  A good example of this is with the online user documentation pages I want to include.  Here, instead of hand-writing simple textual descriptions to appear on the Operators manual page, we can build it automatically from the operator definitions that the editor already dumps out on start-up (a simple form of user doc that was added ages ago).  This is in the following form:

Source operator description format generated by the editor

By loading this in with the file read operator, applying some recursive scanning of the content (using find and split operators to parse each section), we can generate web page content for it.  With some HTML table and a sprinkle of CSS magic we can generate fairly good facsimiles of the actual operators in the editor.

Resulting operator documentation example

These have hover-tips for extra detail where available, such as the internal operator name and the IO connection descriptions.  There is also a summary section that appears at the top of the page with links to each section.

Summary section for the operator documentation

Code Generation

The page construction process is broken down into various modular parts, split between the HTML generation and the CSS generation as well as being split into generic (re-usable) operations and ones specific to our site or page.

I've found that by wrapping even the simplest tags with procedures, it keeps the graphs readable, and also allows for site-wide fixes and additions to any of the HTML or CSS constructs.

It's even worth wrapping the simplest of tags.

Most code output (CSS or HTML) is actually just text string concatenation, for example, the heading tag just prepends the opening tag and appends the closing tag:

The heading procedure, parameterised by level (and ID)

HTML & CSS

A library of HTML element procedures and CSS constructs have been accumulated as the site constructed.

Reusable HTML and CSS procedures

The HTML ones are mainly for the tag types that can be used, with a few attribute types to help build them.  The CSS ones are containers for the styling of each section, and a load of property helpers.  Here you can see them in use on the common content styling:

CSS property helpers in use 

Procedures

Some of the re-usable building blocks, specifically for handling files are described in more detail here.

Image Fetch

Using the new file operators, the paths can be prepared, the timestamps checked, and the copy performed if required.  On success we pass out the relative path to the image file, ready for use in the HTML or CSS code.

Image fetching procedures.

Document Read

The markdown documents are read in by invoking Pandoc with the source file as a parameter and accumulating the text output by it to the console stream.  If the return value indicates an error we report a suitable error message.

Document reading and processing procedures.

Some additional document processing is done here to achieve two things:

1. Image tags are parsed and used to drive image thumbnail generation, propagate the main file, and replace the image element with the thumbnail and a link to the full image.
2. Lists have a little bit of formatting added to turn "Title - Description…" into "Title - Description…" to make the headings stand out a bit more.  This is styled and can be updated globally.

Thumbnail Generation

Where image tags are included in any content the processing step applied on load (if enabled) generates the thumbnail automatically.  There are a number of things going on here.

Processing of all image tags in the document using recursion

1. The document is searched (using recursion) for all image tags.
2. The tag is parsed and broken open into its constituants.
3. The image file is located and propagated to the output folder.
4. A name for the thumbnail is generated based on the image name.
5. A thumbnail version of the image is generated in the output folder (using Image Magick).
6. A new image tag is constructed (around the thumbnail file).
7. The image tag is wrapped in a hyperlink tag pointing at the full image.
8. The document around the image tag is re-combined.

There is a lot going on here, but it can all be laid out nicely and the flow of information is clear from the procedure construction.

Viewing the Results

Now we have this all up and running, as changes are made to the procedures the top-level page procedure we are 'viewing' is re-built for us.  This is an HTML page with the required images nearby that can readily be viewed in a browser.

Live Page

One additional requirement to be able to see changes in real-time is for us to get the browser to refresh as pages change.  This could be done with the auto-refresh timer you can add to a web page, but this is rather crude as it will a) be updating all the time, and b) have to be doing so rapidly to get a decent response time to changes.

A bit of digging turned up a lovely little Chrome extension called LivePage.  After setting a few options up, you can click on the icon alongside any page and have it watch the source file for changes.  On detecting one the page is reloaded.

LivePage browser extension in Chrome

It is proving really good for working interactively and has options to track changes to HTML, embedded CSS, and even source image files (experimental, but seems good).

Content

By adding a quick directory watcher feature into the editor I was able to also have generation happen when the source content (markdown) documents or source images changed.  This nicely rounded off the editing experience.

Interaction Examples

The best way to see all this working together is a video, so here you are.

So far this has been a really fun way to develop the web site, and I'm certainly going to continue along this vein.  I hope you liked this slightly tangential application of the technology, I expect there will be more of this sort of thing in the future.

Next

Next time I should be ready to give you an update on height work on Future City and a long overdue engine feature I added to help support it.

Procedural Website? - Part 1

Web Site Needed

Apparance is at the stage now where it needs a proper web-site to bring together all the information accumulating around the project and provide a hub for people to find out all about it.  It's going to need all the usual stuff; summary pages, a gallery, detailed project information, feature lists, technical information, contact information, blog, documentation, and contact information.  Something like this:

A prototype Apparance homepage.

There are many ways to approach creating a web site.  It could be out-sourced, built using an online web-site maker, using offline tools, crafting from the many kits and examples available, or writing it all by hand in notepad.

Not having huge resources available and having hand crafted a few pages in my time I figured it would be best, at least in the short term to code it up myself.  So, given that I'm 'living and breathing procedural' right now, is there any reason not to try and do this procedurally, with Apparance?

I can't see anyone else making websites procedurally!  Am I crazy?

Approach

I'm not familiar with all that is trendy in web development and I'd like to keep things simple so I'm thinking static HTML pages with CSS styling.  I'd like to be able to author page content in some sort of simple format and not have to be editing html or using any web design tools to make changes.  Any image manipulation should preferably be handled automatically or with little interaction. Given this, some process will be needed to effectively compile the content, images, structure, and styling together into each page.

Utilities

Content Processing

Simple format text files with basic formatting information you say?  Sounds like mark-down to me.  A bit of digging found a rather impressive command line utility called Pandoc which actually will convert a plethora of document formats.  By default though passing in simple markdown text it will spit out an HTML fragment.

Pandoc, by default converts markdown into HTML

Perfect for injecting into the structure needed around it to form a complete page.

Image Processing

The most common need for image preparation on a web page has to be scaling to generate thumbnail previews, something we're going to need to show off all the visual goodness of Apparance.  A well know tool that handles this adequately is Image Magick.  Again a command line tools with a huge number of features and functions, it comes with an command specifically designed for generating thumbnails.

Image Magick, great for quickly handling thumbnail generation

Perfect to add to the arsenal of tools for building my pages.

Automation

Now to the question of automation; how best to take all this and build pages out of it?  Let's look at the requirements.

1. Generate the common page HTML and CSS according to the page design.
2. Launch Pandoc to turn the page content into HTML.
3. Launch Image Magick to generate thumbnails for the images.
4. Combine the HTML into pages.
5. Assemble it all in a suitable file/folder structure for serving as a web-site.

Doing it wrong

There are existing static web page compilers available (e.g. Hugo) I could use.  I could write a program to do it, or come up with something in python or similar scripting languages.  But thinking about Apparance and some of the philosophies behind it I realised that these approaches went against it on several levels.

First, there was the baking process, the page needed to be compiled before it could be viewed.  Now, whilst any process of assembling a page requires time to complete, with scripts and command line tools there are still several steps involved in getting from making a change to seeing the results.  You lose the immediacy of a truly interactive editing experience.

Second, committing anything to script or code goes against the data-driven approach, and again, you lose the direct connection between your design and the results.

We've seen this before

Having already worked on interactive editing and text based processing with the material and shader system in Apparance it seemed like a similar approach could be applied here.  Why not use procedures in Apparance to handle the HTML composition, the control of command line utilities, and the manipulation of files and the directory structure?  It would mean any part of the web-site could be editing and you would get to see the results in real-time!  This was clearly the path to follow.

And So; To Procedural

A few things would be needed to be added to Apparance's capabilities for us to be able to support this though:

1. Extra string handling functions - we are going to be doing more fancy things than we did with shader building.
2. File handling operations - we are actually going outside the application, to work with files.
3. An operator for spawning processes and collecting their output - to allow us to use other tools to do our bidding.

With all this in place, we are in a position to start building the site by creating procedures and letting Apparance do the work.  Next time we'll have a look at how this went and what it's like to work with.  Here's a sneak peek at the editor with the procedure for writing out the HTML pages.

Procedure to write out an HTML page