Aestheticodes projects aim to make such patterns interactive by associating them with digital materials while also preserving their aesthetic value. From initial application of painting carefully designed aestheticodes in tableware to enhance the dining experience [1], to more recent applications of crafting gift cards and fabricating of Carolan guitar [2], aestheticodes has been successfully applied in different materials (ceramics, card, fabric and wood, etc.) using different techniques and tools (painting, drawing, cutting, embroidery, inlaying and laser cutting, etc.) to augment the common objects with digital technology. As aestheticodes are being used in different scenarios, we discover two typical situations: first, each time when opening up a new application, we may face the same challenges on design and recognition; second, we often encounter completely new challenges when we start a new application or even just use different design material or techniques. For example, from ceramics applications to Carolan project, reflections and shadows are still a persistent challenge for vision recognition technology. However, the influence degree of reflections and shadows may differ between different applications. In ceramics applications, specular reflection from lights would blot out areas of the patterns and therefore break the topological structure of the patterns, especially on glazed surface [1]; nevertheless shadows is the biggest challenge for reliable recognition in Carolan guitar as guitar may often be played at the places with pretty non-uniform lighting conditions, such as concert and bar. For example, cast shadows in ceramics application are mainly caused by 3D structure of ceramic object itself and other associated objects like chopstick [1]; similarly, in Carolan guitar components (e.g. tuning knots) may cast their shadows on codes that highly affects the recognition. [1, 2] resolved reflection problems by choosing appropriate design materials and finishing techniques, it compromises aesthetic and puts additional constraints on designers despite it mitigates surface reflections.
Additionally, under current recognition technology, wood grains and texture are new challenges for guitar project where dark grains can often be mistreated as a part of code [2]. Since soundholes have been designed as a part of pattern, treating as solid blobs, once the situations that allow light enter the body of the guitar being encountered, the topological structure of pattern will be broken and recognition fails. Moreover, the edges of soundholes have a 3D depth that might also confuse recognition, potentially requires them to be stained with dark colour for reliably scan. In order to reliably recognise aestheticodes, compromises on design has been made. Although three robustness mechanisms have been introduced to overcome recognition challenges in ceramics application, however, checksum and validation regions mechanism works only when aestheticodes has been recognised as a valid d-touch [3], and redundancy mechanism, strictly speaking, is still a type of design compromises for compensate the shortcomings of vision technology.
The core of recognition method in aestheticodes projects is histogram based thresholding algorithm [4] which depends on intensity to filter out the recognisable patterns with backgrounds, then uses border following algorithm proposed in [5] to find topological structure and construct a hierarchical region adjacency tree [3]. Generally speaking, light and dark colour will be converted into white and black, respectively. This means that design decisions will be partly restricted by colour contrast, for example, Carolan guitar finally chooses spruce and flamed maple as design materials rather than wenge because of the non-constrasting between wenge and celtic knotwork aestheticodes [2]. The use of color is of importance in graphic design [59] and current computer vision techniques limits the selection of colour combinations. Correspondingly, one of research aims is to design and develop recognition method that puts constraints on designers as little as possible; therefore designers are capable of making design decisions relatively freely without taking so many recognition challenges into consideration.
However, digitally augmenting everyday objects with aestheticodes is a challenging and multi-displinary field. An understanding of the context of an aestheticodes application is then central to a reliable recognition software. For different aestheticodes applications, recognition software may encounter different materials, techniques, design aspects when scanning patterns using distinctive models of camera under different lighting conditions. With a wide range of aestheticodes application situations, it needs to enable recognition software adapt the context variations intelligently, to best support augmentation of everyday objects with aestheticodes.
Visual Marker - The idea of scanning visual codes to trigger digital interactions is well established. The first use of specifically designed codes that contains bits of information is the use of barcodes, initially patented in 1952 [7] and now becomes an universal feature on retail products. More recently, QR codes [8] has become popular and even all the mainstream mobile operating system can read QR codes. Bar codes and QR codes are designed to be robustly identified by machine, where no mistake encountered when you scan a bar code or QR codes. This reliability comes at the cost of a limited aesthetic: they are not visually meaningful to human and difficult to distinguish different markers from each other by looking at them. This reliability comes at the cost of a limited aesthetic: they are not visually meaningful to human and difficult to distinguish different markers from each other by looking at them. Costanza and Huang [3] introduces d-touch, a visual system based on the topological structure of visual codes, and therefore allows users to design their own visual markers, control their aesthetic qualities. d-touch opens up the possibility of creating visually aesthetic and artistic codes and meanwhile can be reliably recognised by machine. Meese et al. [1] present a specific implementation of d-touch, called aestheticodes, making interactive decorative patterns on the surface of ceramics. Benford et al. [3] further extend the application of aestheticodes. Other typical visual markers include reacTIVision [9] and ARTag [10], similarly, they encode information based on topological structure of the image.
Recognition Technology - The heart of any camera-based interaction is a “classifier” that takes an image and identifies those pixels or groups of pixels that are of interest to the interaction [11]. The simplest “classifier” in image processing should be thresholding methods, which is based on threshold value to turn a grayscale image into a binary image. Typical thresholding algorithms include histogram-based [12, 13], clustering-based [14, 15, 16], entropy-based [17] and locally adaptive thresholding [18, 19, 20], etc. A detailed and comprehensive review of the state-of-the-art thresholding algorithms can be found in [21].
As aestheticodes will be used to decorate surfaces made from different materials such as wood, textile or leather, classification materials from their imaged appearance becomes a critical step for aestheticodes design and recognition. Texture is a material property of which we have only an intuitive and not a sound mathematical understanding [22]. According to [22], texture research is generally divided into five canonical problems: synthesis, classification, segmentation, compression, and shape from texture. Here, we just introduce the research work in the areas of classification and segmentation as they are closely related to aestheticodes projects. The success in these areas was largely due to learning a fuller statistical representation of filter bank responses, e.g., S filter [23], Maximum Responses filters (BFS, MR8, MR4 and MRS4) and LM filter [24].
Design Aspects - Now, let us turn attention to the research work on aestheticodes design. Generally speaking, to the best of my knowledge, the work on general guidelines of crafting aestheticodes patterns is lacking. Meese et al. [1] report two design approaches "structure-first" and "sketch-first" are being adopted by participants to draw aestheticodes on paper. They also point out designers are keen to add embellishment and backgrounds around the scannable patterns in order to enhance the aesthetic of patterns. Additionally, colour selections have been realised by designers since light colour will be thresholded to white and so would be indistinguishable by recognition technology.
aestheticodes, as a specific implementation of interactive decorative patterns, may expect to be used in a wide range of scenarios; therefore, aestheticodes design and recognition will face both opportunities and challenges. As decoration patterns is a universal features in everyday world, reliably read out the digital media would be a key step for making decoration patterns interactive. The interactive affordances, properties and aesthetics of materials; techniques and tools; the form, function and context of an artefact; and physical environments (e.g., lighting conditions) may change from case to case. How to make aestheticodes recognition and design adapt those changes and work reliably in different situations becomes a new problem to solve. To address this challenge, we borrow the concept of “context-aware” computing from Ubiquitous Computing to describe our recognition software which is capable of understanding and making adaptions to the changes, in order to maximise the recognition reliability and the aesthetic value of decorative patterns.
The aim is to build a context-aware aestheticodes recognition platform which is able to understand the application context and adapt it appropriately in order to achieve optimal recognition reliability without compromising the object’s integrity. This requires to study the designers’ feedbacks on the aestheticodes design (e.g. the rules of aestheticodes), the application context, the interactive affordances and properties of design materials, and recognition algorithm and its influence on designing or crafting aestheticodes. Generally speaking, at the end of my PhD, we will deliver a context-aware App for recognizing aestheticodes and the general guidelines for aestheticodes design.
This work was carried out at the International Doctoral Innovation Centre (IDIC). The authors acknowledge the financial support from Ningbo Education Bureau, Ningbo Science and Technology Bureau, China's MOST, and the University of Nottingham. The work is also partially supported by EPSRC grant no EP/G037574/1.