Procedural Website? - Part 2

Recap

In Part 1 we proposed using Apparance as a text processor and tool automation controller to build our web-site.  So, how exactly would we go building about this?

The Apparance web site prototype.

Page Structure

Browsing a few basic web tutorials on how to set up a fairly standard CSS styled HTML page we are looking at this:

1. Title banner and logo
2. Site navigation bar
3. Page content, different for each page
4. Footer with a few common links and note.

We can encapsulate the shared and custom element construction in a top-level page procedure.  This wraps a procedure for emitting the standard HTML page sections and any extra bits all pages need (metadata, etc).  This is the point that the CSS for the page and the body HTML are passed in as parameters.

The top-level page generation procedures.

This provides a single procedure that can be re-used on each site page, with the page specifics and customisations passed in as parameters.  For example, here it is used on the homepage.

All that is needed for a top level, text content based page.

All that is needed for a page like this is the content source and a couple of constant values.

Title

I wanted the title to use the Apparance logo, and have a background that hinted at both what you can make with Apparance, and how you make it.  This is a combination of background image banner and centred logo.

Title banner background (transparency shown as checker-board)

Title logo (transparency shown as checker-board)

Both of these are images that are prepared by hand (at the moment) and just need to be copied across if they are updated or missing, this is handled by the Image Fetch procedure (see below).

Header background (CSS) and logo (HTML) generating procedures

There is a bit of styling to apply the background, and the logo is an image HTML element over the top.

Navigation

The navigation bar should be a simple row of buttons with hover highlighting and current page indication.  These provide the top-level site choices, the leftmost one being the main homepage and site entry-point.

The main navigation bar.  Current tab and hover highlighting shown.

To do this properly, the idea is to have the items expressed in a simple a way as possible in the HTML and leave the funky appearance to the CSS.  The best of conventional wisdom points at using an un-ordered list (UL tag) with list items (LI tag) for each menu option.  With the appropriate tagging, this can then be 'skinned' into a row of buttons using only CSS.

Again, this can be a single procedure that gets re-used on each page.  In fact, some of the pages will require a second row of buttons to navigate sub-pages (too many pages to have them all in one row).  We can build this procedure (and the CSS) to support this.

The bar and button creation procedures.

The CSS generation procedure for the navigation bar and buttons.

The procedure to generate the CSS is designed to be applied twice, with the menu level as a parameter.  This allows the styling to be slightly different and provide a raked appearance to the menus as they transition into each other and the page content.

Example of a double level navigation bar.

To get the to show the correct selected page in each place it's used, the page it is on is passed as a parameter which causes that button to have the selection highlight styling applied.

Footer

The page footer is very simple, just a few links and some text.

Simple page footer

It is included in the general page procedure as it appears at the foot of every page.

Common footer procedure and where it fits in.

We may need to include some customisation later so that when on pages it actually links to, those links are disabled (for neatness).  There are other cool things we could do with the footer later I'm sure.

Content

Markdown

The main part of every page is the content and comes, for most of them, from external markdown files.  This allows the content to be written in a form that can be transformed into HTML as well as other forms if we need (see later).  Here is an example of what the content looks like in raw markdown and on the actual page.

Markdown source document compared to resulting HTML page output.

Custom

Some pages warrant custom generation and rely on special procedures to generate their content.  A good example of this is with the online user documentation pages I want to include.  Here, instead of hand-writing simple textual descriptions to appear on the Operators manual page, we can build it automatically from the operator definitions that the editor already dumps out on start-up (a simple form of user doc that was added ages ago).  This is in the following form:

Source operator description format generated by the editor

By loading this in with the file read operator, applying some recursive scanning of the content (using find and split operators to parse each section), we can generate web page content for it.  With some HTML table and a sprinkle of CSS magic we can generate fairly good facsimiles of the actual operators in the editor.

Resulting operator documentation example

These have hover-tips for extra detail where available, such as the internal operator name and the IO connection descriptions.  There is also a summary section that appears at the top of the page with links to each section.

Summary section for the operator documentation

Code Generation

The page construction process is broken down into various modular parts, split between the HTML generation and the CSS generation as well as being split into generic (re-usable) operations and ones specific to our site or page.

I've found that by wrapping even the simplest tags with procedures, it keeps the graphs readable, and also allows for site-wide fixes and additions to any of the HTML or CSS constructs.

It's even worth wrapping the simplest of tags.

Most code output (CSS or HTML) is actually just text string concatenation, for example, the heading tag just prepends the opening tag and appends the closing tag:

The heading procedure, parameterised by level (and ID)

HTML & CSS

A library of HTML element procedures and CSS constructs have been accumulated as the site constructed.

Reusable HTML and CSS procedures

The HTML ones are mainly for the tag types that can be used, with a few attribute types to help build them.  The CSS ones are containers for the styling of each section, and a load of property helpers.  Here you can see them in use on the common content styling:

CSS property helpers in use 

Procedures

Some of the re-usable building blocks, specifically for handling files are described in more detail here.

Image Fetch

Using the new file operators, the paths can be prepared, the timestamps checked, and the copy performed if required.  On success we pass out the relative path to the image file, ready for use in the HTML or CSS code.

Image fetching procedures.

Document Read

The markdown documents are read in by invoking Pandoc with the source file as a parameter and accumulating the text output by it to the console stream.  If the return value indicates an error we report a suitable error message.

Document reading and processing procedures.

Some additional document processing is done here to achieve two things:

1. Image tags are parsed and used to drive image thumbnail generation, propagate the main file, and replace the image element with the thumbnail and a link to the full image.
2. Lists have a little bit of formatting added to turn "Title - Description…" into "Title - Description…" to make the headings stand out a bit more.  This is styled and can be updated globally.

Thumbnail Generation

Where image tags are included in any content the processing step applied on load (if enabled) generates the thumbnail automatically.  There are a number of things going on here.

Processing of all image tags in the document using recursion

1. The document is searched (using recursion) for all image tags.
2. The tag is parsed and broken open into its constituants.
3. The image file is located and propagated to the output folder.
4. A name for the thumbnail is generated based on the image name.
5. A thumbnail version of the image is generated in the output folder (using Image Magick).
6. A new image tag is constructed (around the thumbnail file).
7. The image tag is wrapped in a hyperlink tag pointing at the full image.
8. The document around the image tag is re-combined.

There is a lot going on here, but it can all be laid out nicely and the flow of information is clear from the procedure construction.

Viewing the Results

Now we have this all up and running, as changes are made to the procedures the top-level page procedure we are 'viewing' is re-built for us.  This is an HTML page with the required images nearby that can readily be viewed in a browser.

Live Page

One additional requirement to be able to see changes in real-time is for us to get the browser to refresh as pages change.  This could be done with the auto-refresh timer you can add to a web page, but this is rather crude as it will a) be updating all the time, and b) have to be doing so rapidly to get a decent response time to changes.

A bit of digging turned up a lovely little Chrome extension called LivePage.  After setting a few options up, you can click on the icon alongside any page and have it watch the source file for changes.  On detecting one the page is reloaded.

LivePage browser extension in Chrome

It is proving really good for working interactively and has options to track changes to HTML, embedded CSS, and even source image files (experimental, but seems good).

Content

By adding a quick directory watcher feature into the editor I was able to also have generation happen when the source content (markdown) documents or source images changed.  This nicely rounded off the editing experience.

Interaction Examples

The best way to see all this working together is a video, so here you are.

So far this has been a really fun way to develop the web site, and I'm certainly going to continue along this vein.  I hope you liked this slightly tangential application of the technology, I expect there will be more of this sort of thing in the future.

Next

Next time I should be ready to give you an update on height work on Future City and a long overdue engine feature I added to help support it.

Procedural Website? - Part 1

Web Site Needed

Apparance is at the stage now where it needs a proper web-site to bring together all the information accumulating around the project and provide a hub for people to find out all about it.  It's going to need all the usual stuff; summary pages, a gallery, detailed project information, feature lists, technical information, contact information, blog, documentation, and contact information.  Something like this:

A prototype Apparance homepage.

There are many ways to approach creating a web site.  It could be out-sourced, built using an online web-site maker, using offline tools, crafting from the many kits and examples available, or writing it all by hand in notepad.

Not having huge resources available and having hand crafted a few pages in my time I figured it would be best, at least in the short term to code it up myself.  So, given that I'm 'living and breathing procedural' right now, is there any reason not to try and do this procedurally, with Apparance?

I can't see anyone else making websites procedurally!  Am I crazy?

Approach

I'm not familiar with all that is trendy in web development and I'd like to keep things simple so I'm thinking static HTML pages with CSS styling.  I'd like to be able to author page content in some sort of simple format and not have to be editing html or using any web design tools to make changes.  Any image manipulation should preferably be handled automatically or with little interaction. Given this, some process will be needed to effectively compile the content, images, structure, and styling together into each page.

Utilities

Content Processing

Simple format text files with basic formatting information you say?  Sounds like mark-down to me.  A bit of digging found a rather impressive command line utility called Pandoc which actually will convert a plethora of document formats.  By default though passing in simple markdown text it will spit out an HTML fragment.

Pandoc, by default converts markdown into HTML

Perfect for injecting into the structure needed around it to form a complete page.

Image Processing

The most common need for image preparation on a web page has to be scaling to generate thumbnail previews, something we're going to need to show off all the visual goodness of Apparance.  A well know tool that handles this adequately is Image Magick.  Again a command line tools with a huge number of features and functions, it comes with an command specifically designed for generating thumbnails.

Image Magick, great for quickly handling thumbnail generation

Perfect to add to the arsenal of tools for building my pages.

Automation

Now to the question of automation; how best to take all this and build pages out of it?  Let's look at the requirements.

1. Generate the common page HTML and CSS according to the page design.
2. Launch Pandoc to turn the page content into HTML.
3. Launch Image Magick to generate thumbnails for the images.
4. Combine the HTML into pages.
5. Assemble it all in a suitable file/folder structure for serving as a web-site.

Doing it wrong

There are existing static web page compilers available (e.g. Hugo) I could use.  I could write a program to do it, or come up with something in python or similar scripting languages.  But thinking about Apparance and some of the philosophies behind it I realised that these approaches went against it on several levels.

First, there was the baking process, the page needed to be compiled before it could be viewed.  Now, whilst any process of assembling a page requires time to complete, with scripts and command line tools there are still several steps involved in getting from making a change to seeing the results.  You lose the immediacy of a truly interactive editing experience.

Second, committing anything to script or code goes against the data-driven approach, and again, you lose the direct connection between your design and the results.

We've seen this before

Having already worked on interactive editing and text based processing with the material and shader system in Apparance it seemed like a similar approach could be applied here.  Why not use procedures in Apparance to handle the HTML composition, the control of command line utilities, and the manipulation of files and the directory structure?  It would mean any part of the web-site could be editing and you would get to see the results in real-time!  This was clearly the path to follow.

And So; To Procedural

A few things would be needed to be added to Apparance's capabilities for us to be able to support this though:

1. Extra string handling functions - we are going to be doing more fancy things than we did with shader building.
2. File handling operations - we are actually going outside the application, to work with files.
3. An operator for spawning processes and collecting their output - to allow us to use other tools to do our bidding.

With all this in place, we are in a position to start building the site by creating procedures and letting Apparance do the work.  Next time we'll have a look at how this went and what it's like to work with.  Here's a sneak peek at the editor with the procedure for writing out the HTML pages.

Procedure to write out an HTML page

Future City: Update 5 - The Fencepost Problem

Fence-posts

A lot of structures have a regularity to them that is useful to emulate, whether it be fences, walls, windows, rows of houses, tree, or chimney pots.  As with the fence-post problem you can't implement them as a straight repeat of a single object though since you need one more post than fence panels.

The Fence-post Problem: N panels but N+1 posts.

Before we look at solving this however, we will start simpler; looking at how we can handle instancing of the more-straightforward repeating structures.

Example of a simple array of objects.

Recursion

Apparance doesn't yet support arrays as a data-type, arraying of objects, or any form of looping, but an effective alternative that is supported well is recursion.  Here we can break down a repeating problem into successive; sub-division, geometry instantiation, and recursion, until the space is filled.

Basic recursion used for repeating geometry.
The process of split+generate is invoked on the left split area each time until there is no more to split.

The process of sub-division can be improved somewhat in a couple of ways; first, instead of creating a chain of recursions, one-deeper-per-instance, we can tackle it with successive sub-division and two recursion paths instead.  This reduces the stack depth needed from N to log2 N (or thereabouts).

Improved recursion used for repeating geometry.
Recursion is applied to both parts of a central split until there is no more to split.

The other thing we can do is to wrap up the splitting calculations into helper procedures that make it easier to implement these arrays of objects.  The general problem here is that we need to be able to have our own geometry within the recursion process which is tricky to support.  Instead of having to support embedding of procedures (akin to passing function pointers in code) however, we can provide two procedures, with which we 'book-end' our own geometry generation.  These handle the Splitting Logic, and the Combining Logic needed to implement the array process. The general form for this is:

Layout of a procedure for arraying geometry using the two helper procedures to wrap the recursion and geometry instantiation.

NOTES: Any external parameters the procedure needs (to customise the generation have to be passed down to the recursion calls too.  The Control channel is used to select which of the three outputs are to be combined before returning to the outer procedure, this could be a pair of recursions, or some generated geometry.  A further improvement to this process might be to use all three at once when an odd number of elements are present since we can make the splitting even by immediately instantiating geometry for the centre region.s

Potential further improvement to recursion efficiency.

The Divider

These helper procedures I've named 'The Divider' as it divides up space in one direction into equal size pieces for our procedure to build geometry into.

Basic recursion helper procedure: The Divider (splitting part)

This is used in conjunction with a 'Merger' which handles aggregation of the elements.

Basic recursion helper procedure: The Divider (merge part)

This is a really useful tool to building and distributing content around the world.  For example, when a long thin strip of land is encountered in the business district, multiple buildings are placed in a row instead of a single long thin building.  Here we see the Divider and Merge stages in use in the Business Block procedure that handles this:

Divider approach used to create rows of buildings in the business district.

And here it is in action as the space available changes.

Array of buildings filling a narrow strip of land using the recursive Divider approach.

The Stacker

To tackle the fence-post problem, and some of the more sophisticated use-cases, a significantly more powerful version of the divider approach is needed.  Here are a couple of examples:

This piece of railing can be sectioned up in a couple of ways, either as the classic fence-post arrangement (circular design 'panels' between flower 'posts'), or by treating the flowers a separators between the circular parts and two end pieces connecting to the walls.

These railings can be divided up in a couple of ways.

This building facade can similarly be broken up in to ways, but these are a bit more involved:

This building facade can be divided up in a couple of ways.

Here, however we divide it up we have two end-caps (the corners of the building), but the way the wall/windows are divided up can be approached in two ways:

1. The window sections are wider and include a bit of wall.
2. The wall sections are much cleaner and are present between the end caps and the windows as well as between pairs of window sections.

These requirements can be generalised into the following (optional) sections with what I'm calling 'The Stacker':

1. Start cap
2. Multiple repetitions of:
1. Spacer
2. Object
3. Spacer
4. End cap

This can be constructed in a similar fashion to the Divider, but with more user supplied elements in between the helper procedures.  This happened to turn out to be the most complicated procedure I'd ever built!  I actually tried to tackle this problem a while ago, but decided that it would be impossible without the grouping and notation feature so I set about adding that instead.  I'm glad I did now as notes are used extensively to partition the logic and document the processing involved:

The most complex procedure I've created: 'The Stacker' (splitting part)

This isn't the most efficient implementation, nor the simplest, I'm sure.  But it works, and can be revisited later if it needs speeding up or reducing in size.  I could even be re-implemented in code as a build-in operator if needed.

As with the Divider, there is a corresponding re-combination procedure seen here:

Much simpler companion to The Stacker; the merge part.

As this was a very complicated procedure, and with many combinations of inputs and options it was imperative to have good test cases.  Here we see a large set of instances for most of the various configurations it supports.

Test cases for The Stacker.  Many combinations of the various configurations are tested in one place for easy visual inspection.

Examples

Now we have this powerful tool at our disposal I had a chance to explore what can be done with it.

Wall

A straightforward test case is a wall with regular support pillars and larger end columns.

The Stacker used to implement a wall with regular support and end detail.

Wrapping the different wall elements in a Stacker and corresponding combiner gives us this procedure:

Interesting brick wall structure implemented with the Stacker recursion helper procedures.

Which produces this satisfactory construction.

Wall design constant parameterisation.

Brick Wall procedure expanding to use the space available.

Steel Truss

One of the real test cases I wanted to try out was to build steel-work structures out of girders that can be used as support structures around many industrial components.

Steel structures of the kind I'd like to use in the industrial district of Future City.

More specifically, I want to re-create this configuration:

Example of steel beam configuration I want to produce.

This is made from steel I-Beam sections and has two stacking aspects to it, horizontally we have:

1. (optional) End upright
2. Set of cross-members
1. separated by uprights
3. (optional) End upright

Vertically the cross-members are stacked thus:

1. Cross-members
1. Separated by gaps

This was implemented with a pair of nested procedures, one for horizontal, one for vertical, all contained within a main entry-point procedure.

Nested horizontal and vertical stacking procedures to create steel-work side wall.

Here is a selection of constructs built using this set of procedures:

Random structure variation based around truss, tower, and side procedures.

Steel constructs based around a steel-work side section.

Truss or gantry built from girders.

Close of up construction detail.  As you get closer, the bolts are revealed.

Next

Next I should be able to start building some more interesting structures for the industrial district using all these new-found constructional powers.