Danceformer - End to End Dance Step Prediction on Chart Difficulty

Master Thesis by Cassandra Grzonkowski

Installation

• Clone and setup repository

$ git clone https://gitlab.com/cassandra.grzonkowski/danceformer.git
$ cd ../danceformer/
$ pip install requirements.txt

• Load some songs from https://zenius-i-vanisher.com/v5.2/simfiles.php, for example the following pack: https://zenius-i-vanisher.com/v5.2/viewsimfilecategory.php?categoryid=127.

First Preliminary Experiment - Check Note Distances and BPMs

Will show some statistics of the (downloaded) songs prepared by given path.

ToDo: also make use of multiple bpms

$ preliminary_exp_main.py

Create (Mel-)Spectogram images from music

Given a .ogg file or multiple (a path to the folder), create (mel-) spectogram.

$ audio_to_image.py

Methodology

Given songs including difficulty and melspectogram images, train a CNN to preprocess the images and a MLP to preprocess the difficulty (getting a tensor). After that input the preprocessed image to a transformers encoder and the preprocessed difficulty to the transformers decoder.

ToDo: encode/decode chart as a sequence of tokens with a dictionary and give it to the decoder

$ main.py