Adobe AI learns painting styles to reproduce works of art in less than a minute

If last year’s international RobotArt competition winners are any indication, algorithms aren’t bad for painting, drawing, and sculpting with human precision. A recently published article (“LPaintB: Learning to Paint from Self-Supervision”) on Arxiv.org gives evidence: researchers from the University of Maryland and Adobe Research describe a new machine learning system – LPaintB – which can reproduce hand painted canvases in the style of Leonardo da Vinci, Vincent van Gogh and Johannes Vermeer in less than a minute.

“With the development of non-photorealistic rendering techniques, including stroke-based rendering and painterly rendering, specially designed or hand-crafted methods can increasingly simulate the painting process by applying heuristics,” have writes the co-authors. “[T]These algorithms can generate convincing results, but it is difficult to extend them to new or novel styles… In this article, we focus on building a smart paint agent that can reproduce a reference image in an identical style. or transformed with a sequence of painting actions. “

The researchers’ approach took advantage of self-supervised learning, in which unlabeled data is used in conjunction with small amounts of labeled data to produce improved learning accuracy, to train an agent from zero out of a limited number of reference images. By mathematically modeling the action states of the system (i.e., brush configurations such as brush length, orientation, and size) and replacing the goal state of failures with its state In the end, the team generated a matched corpus with positive rewards, which they provided to the model AI in such a way that they learned to paint reference images in the desired artistic style.

The navigation was not smooth, at least not at first. The researchers note that generally only a small portion of actions sampled by the system had positive rewards, a problem they solved with a reinforcement learning technique that used goal state as matched data to train. a policy (a set of actions in response to states). But the policy generated was not particularly robust, as the paired data used to train it consisted only of actions with positive rewards (which made it difficult to retrieve unwanted actions that returned negative rewards) and d ‘states which were the result of successive series of Actions. Solving this problem again required reinforcement learning: it added noise to the action, which helped generalize the model, and optimized the model’s actions with rewards.

The final result ? An AI framework that could perform paint actions with parameters describing stroke size, color, and position information and update a canvas accordingly, with a reward function that evaluated the distance between state current and target state. To compile a training dataset, the team drew random patches from reference images in a specific style at different scales and sampled the patches at a fixed size. They fed them to the model, which after an hour of training was able to reproduce a 1,000 x 800 image with 20,000 strokes on a PC with a 16-core processor and an Nvidia GTX 1080 graphics chip in less than a minute.

The researchers note that the generalization of the trained model is highly dependent on training data and their method is based on a fairly basic painting environment, but they say the combination of self-supervised and reinforcement learning significantly improves performance. effectiveness and performance of the policy. The team looks to future work incorporating stroke parameters such as brush size, color, and position, as well as creating a model-based reinforcement learning framework that can be incorporated into a painting simulator.