About
Tom Kelly is a research scientist and software engineer in subjects around graphics, machine learning, and urban environments. He does things like generative AI, procedural modelling, software engineering, and creating synthetic data. Get in touch.
Publications
WinSyn: A High Resolution Testbed for Synthetic Data
Tom Kelly; John Femiani; Peter Wonka
in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
@article{kelly2023winsyn,
title = {WinSyn: A High Resolution Testbed for Synthetic Data},
author = {Tom Kelly and John Femiani and Peter Wonka},
url = {https://twak.github.io/winsyn/
https://github.com/twak/winsyn_metadata
https://github.com/twak/winsyn},
year = {2024},
date = {2024-03-01},
journal = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
}
We present WinSyn, a unique dataset and testbed for creating high-quality synthetic data with procedural modeling techniques. The dataset contains high-resolution photographs of windows, selected from locations around the world, with 89,318 individual window crops showcasing diverse geometric and material characteristics. We evaluate a procedural model by training semantic segmentation networks on both synthetic and real images and then comparing their performances on a shared test set of real images. Specifically, we measure the difference in mean Intersection over Union (mIoU) and determine the effective number of real images to match synthetic data’s training performance. We design a baseline procedural model as a benchmark and provide 21,290 synthetically generated images. By tuning the procedural model, key factors are identified which significantly influence the model’s fidelity in replicating real-world scenarios. Importantly, we highlight the challenge of procedural modeling using current techniques, especially in their ability to replicate the spatial semantics of real-world scenarios. This insight is critical because of the potential of procedural models to bridge to hidden scene aspects such as depth, reflectivity, material properties, and lighting conditions.
FacadeNet: Conditional Facade Synthesis via Selective Editing
Yiangos Georgiou; Marios Loizou; Tom Kelly; Melinos Averkiou
in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5384–5393, 2024.
@inproceedings{georgiou2024facadenet,
title = {FacadeNet: Conditional Facade Synthesis via Selective Editing},
author = {Yiangos Georgiou and Marios Loizou and Tom Kelly and Melinos Averkiou},
url = {https://ygeorg01.github.io/FacadeNet/},
doi = {https://tinyurl.com/2ezat87y},
year = {2024},
date = {2024-01-01},
booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
pages = {5384–5393},
}
We introduce FacadeNet, a deep learning approach for synthesizing building facade images from diverse viewpoints. Our method employs a conditional GAN, taking a single view of a facade along with the desired viewpoint information and generates an image of the facade from the distinct viewpoint. To precisely modify view-dependent elements like windows and doors while preserving the structure of view-independent components such as walls, we introduce a selective editing module. This module leverages image embeddings extracted from a pre-trained vision transformer. Our experiments demonstrated state-of-the-art performance on building facade generation, surpassing alternative methods.
Large-Scale Auto-Regressive Modeling Of Street Networks
Michael Birsak; Tom Kelly; Wamiq Para; Peter Wonka
in: arXiv preprint, 2022.
@article{birsak2022large,
title = {Large-Scale Auto-Regressive Modeling Of Street Networks},
author = {Michael Birsak and Tom Kelly and Wamiq Para and Peter Wonka},
doi = {https://arxiv.org/abs/2209.00281},
year = {2022},
date = {2022-01-01},
journal = {arXiv preprint},
}
We present a novel generative method for the creation of city-scale road layouts. While the output of recent methods is limited in both size of the covered area and diversity, our framework produces large traversable graphs of high quality consisting of vertices and edges representing complete street networks covering 400 square kilometers or more. While our framework can process general 2D embedded graphs, we focus on street networks due to the wide availability of training data. Our generative framework consists of a transformer decoder that is used in a sliding window manner to predict a field of indices, with each index encoding a representation of the local neighborhood. The semantics of each index is determined by a dictionary of context vectors. The index field is then input to a decoder to compute the street graph. Using data from OpenStreetMap, we train our system on whole cities and even across large countries such as the US, and finally compare it to the state of the art.
Generative Layout Modeling using Constraint Graphs
Wamiq Para; Paul Guerrero; Tom Kelly; Leonidas Guibas; Peter Wonka
in: Proceedings of the IEEE/CVF international conference on computer vision, pp. 6690–6700, 2021.
@article{para2020generative,
title = {Generative Layout Modeling using Constraint Graphs},
author = {Wamiq Para and Paul Guerrero and Tom Kelly and Leonidas Guibas and Peter Wonka},
doi = {https://tinyurl.com/4nb59s9a},
year = {2021},
date = {2021-01-01},
journal = {Proceedings of the IEEE/CVF international conference on computer vision},
pages = {6690--6700},
}
We propose a new generative model for layout generation. We generate layouts in three steps. First, we generate the layout elements as nodes in a layout graph. Second, we compute constraints between layout elements as edges in the layout graph. Third, we solve for the final layout using constrained optimization. For the first two steps, we build on recent transformer architectures. The layout optimization implements the constraints efficiently. We show three practical contributions compared to the state of the art: our work requires no user input, produces higher quality layouts, and enables many novel capabilities for conditional layout generation.
Seamless Satellite-image Synthesis
Jialin Zhu; Tom Kelly
in: Computer Graphics Forum, pp. 193–204, 2021.
@inproceedings{zhu2021seamless,
title = {Seamless Satellite-image Synthesis},
author = {Jialin Zhu and Tom Kelly},
doi = {https://doi.org/10.1111/cgf.14413},
year = {2021},
date = {2021-01-01},
booktitle = {Computer Graphics Forum},
volume = {40},
number = {7},
pages = {193–204},
}
We introduce Seamless Satellite-image Synthesis (SSS), a novel neural architecture to create scale-and-space continuous satellite textures from cartographic data. While 2D map data is cheap and easily synthesized, accurate satellite imagery is expensive and often unavailable or out of date. Our approach generates seamless textures over arbitrarily large spatial extents which are consistent through scale-space. To overcome tile size limitations in image-to-image translation approaches, SSS learns to remove seams between tiled images in a semantically meaningful manner. Scale-space continuity is achieved by a hierarchy of networks conditioned on style and cartographic data. Our qualitative and quantitative evaluations show that our system improves over the state-of-the-art in several key areas. We show applications to texturing procedurally generation maps and interactive satellite image manipulation.
CityEngine: An Introduction to Rule-Based Modeling
Tom Kelly
in: Urban Informatics, pp. 637–662, Springer, 2021.
@incollection{2021cityengine,
title = {CityEngine: An Introduction to Rule-Based Modeling},
author = {Tom Kelly},
doi = {https://doi.org/10.1007/978},
year = {2021},
date = {2021-01-01},
booktitle = {Urban Informatics},
pages = {637–662},
publisher = {Springer},
}
CityEngine is a rule-based urban modeling software package. It offers a flexible pipeline to transform 2D data into 3D urban models. Typical applications include processing 2D urban cartographic geographic information system (GIS) data to create a detailed 3D city model, creating a detailed visualization of a proposed development, or exploring the design space of a potential project. The rule-based core of Esri’s CityEngine has some unique advantages: Huge cities can be created as easily as small ones, while the quality of the models is consistent throughout. Additionally, this rule-based approach means that large design spaces can be explored quickly, interactively, and analytically compared. Such advantages must be carefully balanced against the increased time to create and parameterize the rules and the sometimes stylistic or approximate models created; coming from more traditional workflows, CityEngine’s pipeline can be initially overwhelming. We introduce the principal workflows and the flexibility they afford, sketch the procedural programming language used, and discuss the export pathways available.
SketchGen: Generating Constrained CAD Sketches
Wamiq Reyaz Para; Shariq Farooq Bhat; Paul Guerrero; Tom Kelly; Niloy Mitra; Leonidas Guibas; Peter Wonka
in: Advances in Neural Information Processing Systems, vol. 34, pp. 5077–5088, 2021.
@article{para2021sketchgen,
title = {SketchGen: Generating Constrained CAD Sketches},
author = {Wamiq Reyaz Para and Shariq Farooq Bhat and Paul Guerrero and Tom Kelly and Niloy Mitra and Leonidas Guibas and Peter Wonka},
doi = {https://proceedings.neurips.cc/paper/2021/hash/28891cb4ab421830acc36b1f5fd6c91e-Abstract.html},
year = {2021},
date = {2021-01-01},
journal = {Advances in Neural Information Processing Systems},
volume = {34},
pages = {5077--5088},
}
Computer-aided design (CAD) is the most widely used modeling approach for technical design. The typical starting point in these designs is 2D sketches which can later be extruded and combined to obtain complex three-dimensional assemblies. uch sketches are typically composed of parametric primitives, such as points, lines, and circular arcs, augmented with geometric constraints linking the primitives, such as coincidence, parallelism, or orthogonality. Sketches can be represented as graphs, with the primitives as nodes and the constraints as edges. Training a model to automatically generate CAD sketches can enable several novel workflows, but is challenging due to the complexity of the graphs and the heterogeneity of the primitives and constraints. In particular, each type of primitive and constraint may require a record of different size and parameter types. We propose SketchGen as a generative model based on a transformer architecture to address the heterogeneity problem by carefully designing a sequential language for the primitives and constraints that allows distinguishing between different primitive or constraint types and their parameters, while encouraging our model to re-use information across related parameters, encoding shared structure. A particular highlight of our work is the ability to produce primitives linked via constraints that enables the final output to be further regularized via a constraint solver. We evaluate our model by demonstrating constraint prediction for given sets of primitives and full sketch generation from scratch, showing that our approach significantly out performs the state-of-the-art in CAD sketch generation.
Yiangos Georgiou; Melinos Averkiou; Tom Kelly; Evangelos Kalogerakis
in: 2021 International Conference on 3D Vision (3DV), pp. 1034–1043, IEEE 2021.
@inproceedings{georgiou2021projective,
title = {Projective Urban Texturing},
author = {Yiangos Georgiou and Melinos Averkiou and Tom Kelly and Evangelos Kalogerakis},
doi = {https://dx.doi.org/10.1109/3DV53792.2021.00111},
year = {2021},
date = {2021-01-01},
booktitle = {2021 International Conference on 3D Vision (3DV)},
pages = {1034–1043},
organization = {IEEE},
}
This paper proposes a method for automatic generation of textures for 3D city meshes in immersive urban environments. Many recent pipelines capture or synthesize large quantities of city geometry using scanners or procedural modeling pipelines. Such geometry is intricate and realistic, however the generation of photo-realistic textures for such large scenes remains a problem. We propose to generate textures for input target 3D meshes driven by the textural style present in readily available datasets of panoramic photos capturing urban environments. Re-targeting such 2D datasets to 3D geometry is challenging because the underlying shape, size, and layout of the urban structures in the photos do not correspond to the ones in the target meshes. Photos also often have objects (e.g., trees, vehicles) that may not even be present in the target geometry. To address these issues we present a method, called Projective Urban Texturing (PUT), which re-targets textural style from real-world panoramic images to unseen urban meshes. PUT relies on contrastive and adversarial training of a neural architecture designed for unpaired image-to-texture translation. The generated textures are stored in a texture atlas applied to the target 3D mesh geometry. To promote texture consistency, PUT employs an iterative procedure in which texture synthesis is conditioned on previously generated, adjacent textures. We demonstrate both quantitative and qualitative evaluation of the generated textures.
A Differentiable CGA Shape Procedural System
T Kelly; V Ceballos Inza; Y Yang
in: Proceedings of the 16th ACM SIGGRAPH European Conference on Visual Media Production, CVMP 2019.
@inproceedings{kelly2019differentiable,
title = {A Differentiable CGA Shape Procedural System},
author = {T Kelly and V Ceballos Inza and Y Yang},
url = {https://www.cvmp-conference.org/files/2019/short/35.pdf},
year = {2019},
date = {2019-01-01},
booktitle = {Proceedings of the 16th ACM SIGGRAPH European Conference on Visual Media Production},
organization = {CVMP},
}
We introduce in this paper the concept of differentiable procedural modelling, which enhances inverse modelling with tools from interactive methods. Procedural models have lots of confusing parameters which only the original programmer understands. We introduce an interactive tool which lets users perform direct modifications on the geometry of models, while it fits the parameters of a grammar to match those modifications.
Simplifying Urban Data Fusion with BigSUR
Tom Kelly; Niloy Mitra
Architecture Media Politics Society, 2018.
@conference{littlesur,
title = {Simplifying Urban Data Fusion with BigSUR},
author = {Tom Kelly and Niloy Mitra},
url = {https://architecturemps.com/wp-content/uploads/2018/04/Tom_Kelly_and_Niloy_Mitra_Simplifying-Urban-Data-Fusion-with-BigSUR_Abstract-ADU.pdf},
doi = {https://arxiv.org/pdf/1807.00687},
year = {2018},
date = {2018-07-01},
booktitle = {Architecture Media Politics Society},
}
Our ability to understand data has always lagged behind our ability to collect it. This is particularly true in urban environments, where mass data capture is particularly valuable, but the objects captured are more varied, denser, and complex. To understand the structure and content of the environment, we must process the unstructured data to a structured form. BigSUR is an urban reconstruction algorithm which fuses GIS data, photogrammetric meshes, and street level photography, to create clean representative, semantically labelled, geometry. However, we have identified three problems with the system i) the street level photography is often difficult to acquire; ii) novel façade styles often frustrate the detection of windows and doors; iii) the computational requirements of the system are large, processing a large city block can take up to 15 hours. In this paper we describe the process of simplifying and validating the BigSUR semantic reconstruction system. In particular, the requirement for street level images is removed, and greedy post-process profile assignment is introduced to accelerate the system. We accomplish this by modifying the binary integer programming (BIP) optimization, and re-evaluating the effects of various parameters. The new variant of the system is evaluated over a variety of urban areas. We objectively measure mean squared error (MSE) terms over the unstructured geometry, showing that BigSUR is able to accurately recover omissions from the input meshes. Further, we evaluate the ability of the system to label the walls and roofs of input meshes, concluding that our new BigSUR variant achieves highly accurate semantic labelling with shorter computational time and less input data.
FrankenGAN: guided detail synthesis for building mass-models using style-synchonized GANs
Tom Kelly; Paul Guerrero; Anthony Steed; Peter Wonka; Niloy J Mitra
in: ACM Transactions on Graphics, vol. 37, iss. 6, 2018.
@article{kelly2018frankengan,
title = {FrankenGAN: guided detail synthesis for building mass-models using style-synchonized GANs},
author = {Tom Kelly and Paul Guerrero and Anthony Steed and Peter Wonka and Niloy J Mitra},
doi = {https://dx.doi.org/10.1145/3272127.3275065},
year = {2018},
date = {2018-01-01},
journal = {ACM Transactions on Graphics},
volume = {37},
issue = {6},
}
Coarse building mass models are now routinely generated at scales ranging from individual buildings to whole cities. Such models can be abstracted from raw measurements, generated procedurally, or created manually. However, these models typically lack any meaningful geometric or texture details, making them unsuitable for direct display. We introduce the problem of automatically and realistically decorating such models by adding semantically consistent geometric details and textures. Building on the recent success of generative adversarial networks (GANs), we propose FrankenGAN, a cascade of GANs that creates plausible details across multiple scales over large neighborhoods. The various GANs are synchronized to produce consistent style distributions over buildings and neighborhoods. We provide the user with direct control over the variability of the output. We allow him/her to interactively specify the style via images and manipulate style-adapted sliders to control style variability. We test our system on several large-scale examples. The generated outputs are qualitatively evaluated via a set of perceptual studies and are found to be realistic, semantically plausible, and consistent in style.
On Realism of Architectural Procedural Models
J. Beneš; T. Kelly; F. Děchtěrenko; J. Křivánek; P. Müller
in: Computer Graphics Forum, vol. 36, no. 2, pp. 225-234, 2017.
@article{benevs2017realism,
title = {On Realism of Architectural Procedural Models},
author = {J. Beneš and T. Kelly and F. Děchtěrenko and J. Křivánek and P. Müller},
doi = {https://doi.org/10.1111/cgf.13121},
year = {2017},
date = {2017-01-01},
journal = {Computer Graphics Forum},
volume = {36},
number = {2},
pages = {225-234},
}
Abstract The goal of procedural modeling is to generate realistic content. The realism of this content is typically assessed by qualitatively evaluating a small number of results, or, less frequently, by conducting a user study. However, there is a lack of systematic treatment and understanding of what is considered realistic, both in procedural modeling and for images in general. We conduct a user study that primarily investigates the realism of procedurally generated buildings. Specifically, we investigate the role of fine and coarse details, and investigate which other factors contribute to the perception of realism. We find that realism is carried on different scales, and identify other factors that contribute to the realism of procedural and non-procedural buildings.
BigSUR: Large-scale structured urban reconstruction
Tom Kelly; John Femiani; Peter Wonka; Niloy J Mitra
in: ACM Transactions on Graphics, vol. 36, no. 6, 2017.
@article{kelly2017bigsur,
title = {BigSUR: Large-scale structured urban reconstruction},
author = {Tom Kelly and John Femiani and Peter Wonka and Niloy J Mitra},
doi = {https://dx.doi.org/10.1145/3130800.3130823},
year = {2017},
date = {2017-01-01},
journal = {ACM Transactions on Graphics},
volume = {36},
number = {6},
publisher = {Association for Computing Machinery},
}
The creation of high-quality semantically parsed 3D models for dense metropolitan areas is a fundamental urban modeling problem. Although recent advances in acquisition techniques and processing algorithms have resulted in large-scale imagery or 3D polygonal reconstructions, such data-sources are typically noisy, and incomplete, with no semantic structure. In this paper, we present an automatic data fusion technique that produces high-quality structured models of city blocks. From coarse polygonal meshes, street-level imagery, and GIS footprints, we formulate a binary integer program that globally balances sources of error to produce semantically parsed mass models with associated façade elements. We demonstrate our system on four city regions of varying complexity; our examples typically contain densely built urban blocks spanning hundreds of buildings. In our largest example, we produce a structured model of 37 city blocks spanning a total of 1,011 buildings at a scale and quality previously impossible to achieve automatically.
Interactive Dimensioning of Parametric Models
Tom Kelly; Peter Wonka; Pascal Müller
in: Computer Graphics Forum, vol. 34, no. 2, pp. 117-129, 2015.
@article{kelly2019interactive,
title = {Interactive Dimensioning of Parametric Models},
author = {Tom Kelly and Peter Wonka and Pascal Müller},
doi = {https://dx.doi.org/10.1111/cgf.12546},
year = {2015},
date = {2015-01-01},
journal = {Computer Graphics Forum},
volume = {34},
number = {2},
pages = {117-129},
publisher = {The Eurographics Association and John Wiley & Sons Ltd.},
}
We propose a solution for the dimensioning of parametric and procedural models. Dimensioning has long been a staple of technical drawings, and we present the first solution for interactive dimensioning: a dimension line positioning system that adapts to the view direction, given behavioral properties. After proposing a set of design principles for interactive dimensioning, we describe our solution consisting of the following major components. First, we describe how an author can specify the desired interactive behavior of a dimension line. Second, we propose a novel algorithm to place dimension lines at interactive speeds. Third, we introduce multiple extensions, including chained dimension lines, controls for different parameter types (e.g. discrete choices, angles), and the use of dimension lines for interactive editing. Our results show the use of dimension lines in an interactive parametric modeling environment for architectural, botanical, and mechanical models.
Unwritten Procedural Modeling with the Straight Skeleton
Tom Kelly
University of Glasgow, 2014, (PhD thesis).
@phdthesis{kelly2014unwritten,
title = {Unwritten Procedural Modeling with the Straight Skeleton},
author = {Tom Kelly},
url = {https://theses.gla.ac.uk/4975/},
year = {2014},
date = {2014-01-01},
school = {University of Glasgow},
note = {PhD thesis},
}
Creating virtual models of urban environments is essential to a disparate range of applications, from geographic information systems to video games. However, the large scale of these environments ensures that manual modeling is an expensive option. Procedural modeling is a automatic alternative that is able to create large cityscapes rapidly, by specifying algorithms that generate streets and buildings. Existing procedural modeling systems rely heavily on programming or scripting - skills which many potential users do not possess. We therefore introduce novel user interface and geometric approaches, particularly generalisations of the straight skeleton, to allow urban procedural modeling without programming. We develop the theory behind the types of degeneracy in the straight skeleton, and introduce a new geometric building block, the mixed weighted straight skeleton. In addition we introduce a simplifcation of the skeleton event, the generalised intersection event. We demonstrate that these skeletons can be applied to two urban procedural modeling systems that do not require the user to write programs. The first application of the skeleton is to the subdivision of city blocks into parcels. We demonstrate how the skeleton can be used to create highly realistic city block subdivisions. The results are shown to be realistic for several measures when compared against the ground truth over several large data sets. The second application of the skeleton is the generation of building's mass models. Inspired by architect's use of plan and elevation drawings, we introduce a system that takes a floor plan and set of elevations and extrudes a solid architectural model. We evaluate the interactive and procedural elements of the user interface separately, finding that the system is able to procedurally generate large urban landscapes robustly, as well as model a wide variety of detailed structures.
Procedural Generation of Parcels in Urban Modeling
Carlos A. Vanegas; Tom Kelly; Basil Weber; Jan Halatsch; Daniel G. Aliaga; Pascal Müller
in: Computer Graphics Forum, vol. 31, no. 2pt3, pp. 681-690, 2012.
@article{vanegas2012procedural,
title = {Procedural Generation of Parcels in Urban Modeling},
author = {Carlos A. Vanegas and Tom Kelly and Basil Weber and Jan Halatsch and Daniel G. Aliaga and Pascal Müller},
doi = {https://doi.org/10.1111/j.1467-8659.2012.03047.x},
year = {2012},
date = {2012-01-01},
journal = {Computer Graphics Forum},
volume = {31},
number = {2pt3},
pages = {681-690},
}
Abstract We present a method for interactive procedural generation of parcels within the urban modeling pipeline. Our approach performs a partitioning of the interior of city blocks using user-specified subdivision attributes and style parameters. Moreover, our method is both robust and persistent in the sense of being able to map individual parcels from before an edit operation to after an edit operation – this enables transferring most, if not all, customizations despite small to large-scale interactive editing operations. The guidelines guarantee that the resulting subdivisions are functionally and geometrically plausible for subsequent building modeling and construction. Our results include visual and statistical comparisons that demonstrate how the parcel configurations created by our method can closely resemble those found in real-world cities of a large variety of styles. By directly addressing the block subdivision problem, we intend to increase the editability and realism of the urban modeling pipeline and to become a standard in parcel generation for future urban modeling methods.
Interactive Architectural Modeling with Procedural Extrusions
Tom Kelly; Peter Wonka
in: ACM Transactions on Graphics (TOG), vol. 30, no. 2, pp. 1–15, 2011.
@article{kelly2011interactive,
title = {Interactive Architectural Modeling with Procedural Extrusions},
author = {Tom Kelly and Peter Wonka},
url = {https://peterwonka.net/Publications/pdfs/2011.TOG.Kelly.ProceduralExtrusions.TechreportVersion.final.pdf},
doi = {https://dx.doi.org/10.1145/1944846.1944854},
year = {2011},
date = {2011-01-01},
journal = {ACM Transactions on Graphics (TOG)},
volume = {30},
number = {2},
pages = {1–15},
publisher = {ACM New York, NY, USA},
}
We present an interactive procedural modeling system for the exterior of architectural models. Our modeling system is based on procedural extrusions of building footprints. The main novelty of our work is that we can model difficult architectural surfaces in a procedural framework, for example, curved roofs, overhanging roofs, dormer windows, interior dormer windows, roof constructions with vertical walls, buttresses, chimneys, bay windows, columns, pilasters, and alcoves. We present a user interface to interactively specify procedural extrusions, a sweep plane algorithm to compute a two-manifold architectural surface, and applications to architectural modeling.