In an article in 3D World, Environment artist Tamas Gyerman discusses the merits of using a CG base for a matte painting. They argue that various structural elements can be tested in combinations and moved around until a satisfying composition is achieved. (2022, pp.48-53)
The premise here is that simple images of the respective buildings, or image sets, are rendered from the 3D application from each projection camera identified during the strategizing phase and are used to assemble the base projection. I chose to render just the diffuse pass for this purpose in order to keep file sizes and render time to a minimum.
The very nature of 3D is that, once a layout is achieved, all perspectives are correct and consequently the representation of the building’s height, width, depth, and position in relation to each other is guaranteed to be correct. Moreover, these elements can later be used to create a perspective grid that can be used to guide alignment of other parts of the matte painting that are not part of the primary projection. An example in my project would be elements in the background cyclorama, which will be added later.
The base projection was applied in accordance with the strategy, established by the checkerboard testing. This involved rendering a still image of each individual image set from the projection cameras identified in the process. Then a shader is developed from each image, which is projected onto the geometry using the appropriate camera.
The image below shows the first pass projection of all image sets in 3D perspective view. Note the untextured areas of the 3D scene, is geometry that is not seen by the shot camera at any point along its motion path and, because the projection cameras are derived from the shot camera, do not have any texture in their range in which to apply to the geometry. Of course, from the shot camera view this is arbitrary as these areas of geometry are out of range and therefore never seen. Indeed there is no actual requirement for geometry to exist in these areas, although removing it would be time-consuming, impractical, and unnecessary.
Application and Analysis of the First Pass Projection
A second ‘low level’ analysis of the base projection is necessary to establish areas requiring corrective action such as adjustment of the frames used to build the layered hierarchical base system or application of patch projections.
Image Set A
This set is visible for only the first 52 frames of the sequence and is therefore the simplest and most uneventful projection.
The initial strategy identified potential texture smearing on both buildings shortly before they go out of shot but it is barely noticeable during analysis with the texture applied. There is no noticeable loss of resolution.
However, this analysis reveals a significant doubling artefact around the roof and window area of the rightmost building of the pair.
A patch projection is required to correct this.
The image below) is taken from the shot camera on frame 38 but with a reduced focal length of 35. This projection setup would allow the affected area to be corrected and also patch the extreme elements of both buildings where texture smearing and potential loss of resolution s present.
The following video articulates the analysis process for this image set.
Image Set B
This set is visible for the first 90 frames of the sequence and uses two projection cameras. One from frame 0 and the other from frame 60.
Analysis found a slight loss of resolution from around frame 75, which can be resolved with a third projection layer using a camera taken from this frame.
There is a small area of missing coverage on far edge of the upper left side of the building which may also be corrected by the additional projection layer.
There is a small area of missing coverage on the underside surface of the arch of the right section of the building. Given that this area is in plain sight of the projection camera on frame 1, this cannot be an artefact and is more likely to be a result of a hole in the geometry and therefore an extremely quick repair.
The following video articulates the analysis process for this image set:
Image Set C
This set is visible for the first 121 frames of the sequence and uses three projection cameras, from frames 0, 60 and 90.
Analysis found a slight loss of resolution from around frame 105, which can be resolved with a further projection layer using a camera taken from this frame.
In terms of texture doubling, this image set is the most affected.
The doubling from the stone texture onto the steps on building 4, and the area of doubling on the pillars on building 5 (right) should partially resolve with the additional projection layer previously discussed. However, as the camera moves from frame 105 to 121, further doubling is likely to occur.
A solution worthy of investigation to overcome all three problems would be to use a single patch instead of a layered projection and base this on a projection camera from a frame later in the sequence (say 120). Then use an Overscanning technique to increase the coverage area by opening the cameras focal length. A resolution calculation will be needed to compensate for the wider view.
Earlier in the sequence there is a major issue on building 4 (left) where the textures on the stone pillars, with mounted lamps are doubling onto the structures behind. This could be resolved with patch projections but the extreme changes in perspective may necessitate three or four separate patches across all the affected frames. An alternative approach would be to split the pillars/lamps from the main geometric structure and project onto them separately. Both approached would involve a similar number of additional projections, but the latter option is more likely to yield a cleaner overall result.
The following video articulates the analysis process for this image set:
Image Set D
Other than the aforementioned resolution loss, there are no doubling or smearing artefacts apparent on building 7 (right).
Given that these elements were part of the same image set when the first pass resolution tests were undertaken, the same projection cameras as those used to project the main building can be deployed.
This method proved to be so effective, that the same approach was used to separate and reproject other problematic geometric elements, in particular the tower on the front corner of the leftmost building on image set D, which is creating similar doubling issues when part of the same shader network as the main building.
The image below shows the elements separated from the main geometric structure projected using temporary shaders.
As there is overlap between the patio elements and the protruding structures, these could not use the same shader setup so would need separate networks.
Having separated the geometry, the image (below) shows the five-layer hierarchical projection applied to the main structure of both buildings with tints applied temporarily to the shaders to highlight a potential problem. With this setup, the points on the geometry where the images join are in places where masking will be difficult to blend.
Previous image sets have revealed changes in lighting through the full camera move and the image (below) shows that this set is also affected. However there are no natural boundary areas within the geometry where masks could be drawn and simple alpha blending is likely to be problematic due to the extent of the light change and also the camera rotation, which has caused the reflections to move position and therefore misalign. The clearest example is the high window which cannot possibly be recovered with and alpha blending without removing all the texture from the uppermost image and therefore exposing a section of lower resolution pixels.
A second iteration of this set is needed, maintaining the resolution threshold but where the image boundaries offer better termination points when masking.
The objective will be to stick closely to the original projection frames as these have already been proven to maintain resolution threshold. However, in this iteration, Overscanning will be used to extend the coverage area, using the geometry to establish good termination points for masking. Each section of the artwork will need a commensurate resolution increase to mitigate any resolution loss arising from the change in focal length.
Starting with he first hierarchical layer, which is the projection from frame 80 (pink) overlapping the projection from frame 0 (yellow). Note that, on the leftmost building, Overscanning allows for natural termination points for the mask around the lines of the domes and the base of the small spire. Moreover, as this projection completely covers the surface of the rightmost building, there is no need for a projection on frame 0 for this building.
The image below shows the projection with the adapted mask
The third projection, from frame 115 (blue), shows how the Overscan reaches high enough to allow good termination points along the roof lines of both buildings
The image below shows the effect of the adapted mask
The fourth projection, on the leftmost building from frame 140 (orange) shows the Overscan extends the coverage to provide natural termination point for the mask along the roofline. At frame 140, only the main façade is in shot so the mask can be terminated at the intersection between both. The rightmost building is out of shot by frame 140 so does not require a projection here.
The image below shows the effect of the adapted mask
Finally, the projection from frame 200, with Overscan to allow the mask to be terminated along the inner recess of the façade.
The image (below) shows the mask adapted to blend at the termination points.
The following video articulates the analysis process for this image set.
Due to the rotation of the camera angle, building 6 (left) is visible for the full sequence but building 7 (right) is out of shot by frame 140. This set uses five projection cameras, from frames 0, 60, 90, 140 and 200. Consideration could be given to projecting both buildings separately, but a more efficient and pragmatic approach would simply be to remove building 7 from the image set taken from frame 200. Even this could be left but its removal would serve as a visual reminder that no overpainting work is needed on the building after frame 140.
Analysis found a loss of resolution from around frame 110, particularly evident on building 7 (right) and the frontmost facades on building 6 (left). The gap in projection frames between 90 and 140 is too large so a further projection layer using a camera taken from a mid-point between the frames (say 115) should resolve this issue.
There are major doubling issues on building 6 (left), most evident in the bollards and chains, and the ornamental orbs on the staircase. A similar approach to image set C will be taken here by separating this geometry from the main structure and projecting separately.
Other than the aforementioned resolution loss, there are no doubling or smearing artefacts apparent on building 7 (right).
The following video articulates the analysis process for this image set:
Image Set E
Due to the rotation of the camera angle, building 8 (left) is visible for the full sequence but building 9 (right) is out of shot by frame 156. It therefore presents the same considerations to Image Set D and requires the identical approach to resolving.
Both buildings in this set are maintaining good resolution throughout and appear to be unaffected by doubling and smearing projection artefacts. There would therefore appear to be very little further processing. However there is a hard join visible between the projection from 115 sitting over the projection from frame 000, which is most noticeable on the pipes.
A soft alpha transition is painted into the top edge of the mask.
When reprojected, the transition is smooth.
The projection from frame 200 presented similar and even more obvious issues when layered over the frame 115 projection.
The mask was adjusted by paining along the window join line and also by creating a gradient transition alpha on the right side to soften the edge and blend into the shadows in the recesses.
When reprojected, the blend is invisible
The following video articulates the analysis process for this image set:
The Ground
The approach to the ground was to render base projections from each of the six cameras in the set and at the resolution designated following Overscanning. Then project each onto the ground geometry using the respective camera/texture pair.
The image (below) shows the second projection masked off along the leading edge of the far-most curb on the street intersection, which presents as a good termination point.
The image below is the alpha channel on the second projection, seen in Photoshop. Here we can see how the mask follows the curb line along the bottom edge, this being the transition between projections 1 and 2. Note the feathered alpha in the street to blend with the intersecting area on projection 1.
A similar combination of hard and soft matte lines can be seen at the top, which is managing the transition between projections 2 and 3.
Managing this process involved working up the layer stack and an interplay between the alphas on the two intersecting textures until a seamless transition, from the perspective of the shot camera, was achieved.
The final projection can be seen (below) in perspective view
Visible areas without texture are occluded from view by the various buildings.
Peter Holmes PhD Journal 