Weaving a Neural Net
We now have the card names and the card stats (as outlined in previous blog posts), but how can we match the two together? The challenge is that the card name is not sufficient for identifying a single trading card, as multiple editions of each trading card exist. The colour property of each card is consistent, so when we are sorting by colour it doesn’t matter which edition we are looking at. However, when sorting by value we must differentiate between them.
One option is to recognize the artwork of each card, as different editions of a card have different pieces of art associated with them. For example, see three different versions of the card Arcbound Ravager below.
However, the trouble becomes that there is over 44,000 Magic: The Gathering trading cards (counting different editions of the same card). That means differentiating between 44,000 different pieces of art. Even storing all the different images at a good resolution would be a memory-consuming task, not to mention the run time difficulty of actually matching a picture of a card to the one in the stack. So, an alternate solution is required.
The answer lies in the set symbols. Each edition, also called a set, has a unique icon that is present on every card. It is easier to recognize the set symbol, because there are only ~210 distinct icons to recognize, as opposed to ~44,000. In addition, each set symbol is in the same relative location of the card, making it possible to quickly identify its location without using any advanced algorithms.
To recognize the card, it was decided that Tensorflow would be used. Tensorflow is a widely used Machine Learning framework for Python that allows you to quickly build and train your own neural networks for image recognition.
The most difficult part of designing these neural nets was acquiring the data. For the foil recognition, it was necessary to take numerous photos of foiled and regular cards using the mechanical setup. One challenge there was that we only had around 30 foiled cards in total. While it is possible to increase that number with data augmentation, we decided to ask friends for some of their foiled cards, so that the original dataset would be larger and we could avoid the potential pitfall of overfitting on some feature which might be present in our original 30 foiled cards.
For the set symbol recognition, Scryfall’s API had a collection of every card’s art available to download, which is where we acquired the set icon data. In addition, Scryfall also had the high quality vector images of the set icons available, so those were downloaded and manipulated to augment our dataset.
After collecting all the data, it was on to designing the neural networks. Thankfully, Tensorflow’s extensive documentation proved very useful in learning how to design a good system. The biggest stumbling block for the set icon neural net was that the original setup was overfitting to a significant degree, and the original images weren’t being parsed very well when being fed into the neural net, resulting in initially poor results. However, these problems were eventually solved and we now have a pair of working neural nets.
Above is the test performance of the foil classifier on several cards. The true card behaviour is in the brackets, while the network’s guess and it’s confidence in that guess is shown on the left. As can be seen, it classified all images correctly, and to a reasonable degree of confidence.