Background

We decided to work on techniques involving reverse engineering graphic designs. We discussed possible goals, possible techniques and some background work.

We started by investigating ways to parse a graphic design into its elements. Parsing graphic designs can be defined in various ways. We decided to start from a good application, then define the parsing task according to the requirements of the application.

We looked at Theo that helps beginner users to good graphical design. Theo is supposed to give suggestions to the user to be able to make a professional-looking graphic design.

Dataset

We made a dataset of 107 designs that are simple enough to experiment with. We annotated them using mechanical turk. (I did the annotations in a mechanical turk page.) We annotated text fields and graphics fields using bounding boxes.

Pipeline

A query input design would contain a few graphics and a few text elements. Given the query, the pipeline consists of two major steps:

1- Search for relevant designs for suggestion
2- Transfer the Layout, Fonts and Colors to user Query

Nearest Neighbor Search

We performed nearest neighbor search using a few different distance measures. Some measures consider the image as a whole while some others try to perform a binary matching between pairs of elements. We used the following measures:
1- Intersection over Union
2- Aspect ratio of elements
3- Color distribution
4- Density and Moments of elements
5- Saliency map

Transfer

Our goal is to transfer layout, fonts and colors from a database design to the query design. The elements in the query design are parsed; the extend of graphics and text elements are known. However, Database designs are raw JPG images; they have no annotation by default. We need to process database designs using MTurk and computer vision techniques. Our goal is to extract layout, fonts and colors as well as to certain auxiliary annotation that helps improving search.

We directly generate SVG files and view them in a browser.

Preliminary Results

Preliminary results show various limitations. One major contributing factor to the quality of results is the size of the design dataset, so we need to expand our dataset from 107 images to thousands of images. Another contributing factor is how well dataset images are processed.