BigSUR: Large-scale Structured Urban Reconstruction

Siggraph Asia 2017 [pdf, other, doi, tex, code (tba)]
Tom Kelly, John Femiani, Peter Wonka, & Niloy Mitra

Our urban reconstruction project was accepted to SGA!

The creation of high-quality semantically parsed 3D models for dense metropolitan areas is a fundamental urban modeling problem. Although recent advances in acquisition techniques and processing algorithms have resulted in large-scale imagery or 3D polygonal reconstructions, such data-sources are typically noisy, and incomplete, with no semantic structure. In this paper, we present an automatic data fusion technique that produces high-quality structured models of city blocks. From coarse polygonal meshes, street-level imagery, and GIS footprints, we formulate a binary integer program that globally balances sources of error to produce semantically parsed mass models with associated façade elements. We demonstrate our system on four city regions of varying complexity; our examples typically contain densely built urban blocks spanning hundreds of buildings. In our largest example, we produce a structured model of 37 city blocks spanning a total of 1,011 buildings at a scale and quality previously impossible to achieve automatically.

Symposium On Geometry Processing 2017

It was great to help out with SGP 2017 this year! Especially since I got to spend 5 days playing with photos and videos. Some photos are here, and the videos I edited from the grad school follow (more info here)...

On Realism of Architectural Procedural Models

Eurographics 2017 [ pdf | doi | author | supporting ]

Jan Beneš, Tom Kelly, Filip Děchtěrenko, Jaroslav Křivánek and Pascal Müller  

Congratulations to Jan on his hard-earned Eurographics paper! The concept for this work was developed in an old blog post, the work was started while we were at Esri R&D in Zurich, and continued back in Prague. Abstract follows.

The goal of procedural modeling is to generate realistic content. The realism of this content is typically assessed by qualitatively evaluating a small number of results, or, less frequently, by conducting a user study. However, there is a lack of systematic treatment and understanding of what is considered realistic, both in procedural modeling and for images in general. We conduct a user study that primarily investigates the realism of procedurally generated buildings. Specifically, we investigate the role of fine and coarse details, and investigate which other factors contribute to the perception of realism. We find that realism is carried on different scales, and identify other factors that contribute to the realism of procedural and non-procedural buildings.

building better video

Strobee made me think quite hard about video, how we do it at the moment, and how would be done if we we started from a blank slate. What would video look like if it were invented for the first time today? What if there was only video?

Video has a lot of legacy components that need re-examining. Video used to be a hardware things, with tape and discs. Now it's software; we all watch video on computers, but many of the old conventions are still with us. Even worse, the web was built for text (hTtp), and has bought it's own conventions with it.

While trying to build a video platform, Strobee came up against many of these conventions, some of them helpful, others a hindrance. I've been collecting a list of them here:

• Why is every video the same every time, when our Facebook page is different every time we return to it? One of Strobee's design challenges has been allowing this, as well as allowing visitors to revisit and link to videos they've seen before.
• Why do we always have to scroll down to see more video on Instagram or Vine? We are used to our web-browsers working as electronic-book-reading-machines. To accommodate this, the current web-video idiom is magical moving pictures on each page of the video book. Video is video, and should be fullscreen. The interface should be built around this ideal. 
• Square videos are a fad, the world is starting to get over them. Our monitors, cinema, phones, and cameras seems to have settled on 16:9.
• Why does every video player need a play and pause button? They are legacy interfaces left over from hardware devices with actual buttons. There is no cost or side effect of not pausing a video. GIFs are a great example showing that we don't need these controls any more.
• Long form video is a different media than social video. It has different conventions, constraints, and ways of viewing. My parents make appointments with TV - they've got to get home on time to watch 60 minutes of Inspector Morse at 9pm. To justify this kind of effort, video has to be 10s of minutes long, and once you've gone to all that effort, and watched something for 10 minutes, you're trapped into to staying and watch the other 50 minutes. 
• Strobee attempts to bridge the gap between long form and short form (micro-video) video - our video can be viewed in small sections, but also keeps playing endlessly, letting people who want to watch for 10s of minutes.
• Video navigation is a hard problem. It was very simple with cable TV: just pick a channel. Today we have many more variables possible (user, video clips, position in those clips, order of those clips, etc...). We experimented a bit, and just now Strobee has a fine-grain history (the clip gallery) along the bottom of the screen, and a course-grain explorer (the story navigator) up the right hand side of the screen. 
• We should share video effortlessly and without thinking - in the same way we Instagram or tweet. We shouldn't be worried that the video isn't interesting to others or that the quality is lower than what we see on TV. We want to share our experiences using video with the click of a single button. Strobee is built to encourage this casual sharing. Clips are short, but what they lack in quality, they make up for that by both variety, and the interface's ability to skip and explore.
• As with photos, some video clips are one-offs, but sometimes sequences of them tell a bigger story. Albums of photos can be more interesting than individual clips. When you look a photo album, you have to do the mental work to reconstruct the event - there's no reason this shouldn't work in video; there's no reason, given the right format, why this shouldn't be the case for video. Generating a perfect narrative (as we see in cinema), requires exact planning to capture every event - a totally different philosophy to the photo album.
• Unwatched video is a massive opportunity! Today, some of us have lots of video, in the future all of us will have lots of video. People with action-cameras (GoPros), are notorious for recording footage they never watch or share. They return from holiday with full memory cards, few have the patience to watch it all, fewer share any, and fewer still take the time to create a video that others would actually want to watch. It should be easy to share bulk footage like this, Strobee makes it a little easier, but there's a long way to go...

I don't think Strobee addresses half the issues that it could, and I expect to see many more imaginative video sharing platforms in the next few years. Weaning people off their old platforms and conventions will be tough, but progress is being made everyday. Strobee has a long way to find the best way to explore video, and we're totally stoked to be at the forefront of these challenges!

but what does it actually do?

Now that the social features are finalized, I finally got around to adding a new "what is Strobee?" video. So we go - a super-brief overview of the whole system:

And, (because I forgot all the marketing points), here's the higher level overview with more buzzwords ("say action cam more", "don't forget facebook"!):

in-browser video splicing

This is a short post about one of the coolest technologies in Strobee - in-browser video splicing & editing. Editing and streaming video requires a lot of processing power, and introduces lots of latency, but here at Strobee we're able to build custom video streams quickly and cheaply - inside every user's browser. The secret is to do this in Javascript without the user even knowing that their web-browser is doing the video splicing (editing short clips together).

In the old days, video editing happened entirely offline in a desktop app. I originally learnt to edit using Media100, but today most people use Adobe Premier, Apple iMovie, Microsoft Movie Maker or Sony Vegas. This is how people who need total control create video - people who want to fill the screen with thousands of options.

So many controls in Lightworks

But these apps take a lot of time to learn, and most people don't have time to do that just to make a holiday video - my Gran is never going to be able to use Movie Maker, however motivated she is. So we needed simpler video editing.

In answer to this, video editing platforms have become automated, and massively simplified, such as Magisto or WeVideo. These are online applications that enable easy video upload, and sharing from your phone, tablet or desktop.

The problem with these online video editors is that they aren't as interactive as the old desktop apps -  you upload your footage to your provider in the cloud, and then wait to preview every action that you see. The waiting is caused by the expensive video splicing happening in the cloud, and causes a really ragged user experience - not what we want for Strobee at all:

Clips being made into a steam in the cloud, before being delivered to your phone

With the advent of HTML 5 video, it's possible to splice video in the browser. At Strobee we hijacked browser's tools that were originally designed for variable quality streaming, to instead perform browser-side video splicing.

The way this works is that each bee (our short video clips) is stored as a static file on the server, and is requested by the browser. The browser uses our api to request file-names, based on the users's preferences. The required static files are then requested from the server, and patches so they can be inserted into the video stream. This means that we can use all the usual static file caching techniques to reduce cost (Memcache, Cloudflare etc...), but give the user custom video streams:

Clips delivered straight to your phone, with much less expensive work done in cloud

It's these custom streams that allows Strobee to give users a different video every-time they come to the site, even every-time they perform an action (fast-forward, repeat a bee, change the length of a bee, preview a user's stories). Even better it means that the video never has to end - we can keep adding bees as fast as the user watches them, even recycling already downloaded clips to save bandwidth.

Strangely, the most complicated part has been creating a simple UI that people understand - letting people edit video while they think they are just navigating a friend-list.

One final feature that we're proud of, is that the user can even download their video. This uses just the same process - the composition happens in the user's browser, splicing the video in memory. We look forward to finding out how this scales - both on the server side (when a user requests all their bees as a single video file), and client side (when browser tries to build the user's video in memory).

Strobee, the story so far

For the past few months I've been working on a re-imagining of video-sharing. But I'll set someone smarter than me summarize it:

You really have to play with Srrobee[sic] to understand how cool it really is. pic.twitter.com/rr4sufrqXC

— jon bradford (@jd) February 16, 2016

Since Jon's announced that (even with slightly dodgy spelling), I might as well write a post a little about the project...
 
This has been a long road, starting with a GCSE (a high school qualification) in Media Studies many years ago. I spent a very happy few days with a bunch of friends creating a trailer for "Starship Troopers". We discovered that by far the most exciting approach (for 15 year olds without any video editing experience), was to find all the short clips of people shooting guns (it was quite a violent film), and put them together. We could get away with this in-class as we'd just been taught about a technique called "montage". We got a good grade for the project, but moved onto other things. (Apologies for video/sound quality, it was recovered from VHS tapes).

More recently, and due to a lot of travelling and cycling, I bought an early action-camera. Compared to a SLR camera or smartphone, these are great because they're robust and easy to use. After a trip to Thailand I was left with loads (32Gb) of footage that was mostly wobbly, with noisy audio, and generally a bit shit, so I tried editing it down into something people would want to watch (here's an old blog post). I tried a few attempts at "professional" editing by carefully constructing a narrative, but I came to the conclusion that there wasn't enough footage to build one. Plus, there was no chance to go back and shoot more. In the end I found that taking many short clips was a very robust and fast way of manually editing a movie from bad footage.

The greatest observation that came out of this process was that selecting the short clip to use wasn't very hard. As long as there was sufficient variety, and the clips kept moving, people wanted to watch more. Much more than if I tried to show them a wobbly 40 minute clip of me trying to feed peanuts to an elephant. What I thought was interesting video, wasn't interesting at all to other people - they just wanted to see all the different things I did on my holiday.

When I found myself with some free time, I decided to automate the process - so Strobee was started as a bit of a startup. This started off as a desktop software project, but it became clear that the project belonged on the web.