Music Generation Using Deep Learning

Can machines generate music ?

Real World Problem

• The objective of music generation is to explore deep learning regarding the field of music composition using artificial intelligence.
• The case study focuses on generating music automatically using Recurrent Neural Networks(RNN).
• We do not need to be an expert to generate music. Even non experts like me can generate a descent quality creative music using RNN.
• Creating music was something unique to humans until now, but some great advancements have been made using deep learning.

Objective

• Building a model that takes existing music data as input learns the pattern and generates “new” music.

• The model-generated music need not be professional, as long as it is melodious and good to hear.
• It cannot simply copy and paste from the training data. It has to learn the patterns from the existing music that is enjoyed by humans.

Now, what is music?

• Basically, music is a sequence of musical components/events.
• Input- Sequence of musical events/notes
• Output- New sequence of musical events/notes
• In this case study, I have limited myself to single instrument music. You can extend this to multiple instrument music.

Some of the music generated example using RNNs

Folk RNN - Generate folk tunes with a recurrent neural network

Representation of Music

1. Sheet Music

sheet music

Basically, sheet music represents music in a specific notation and musicians use this play music. It can be used for both single instrument and multi instrument. Sheet music is a kind of visual file.

2. ABC Notation

This is an example of the use of ABC notation from Wikipedia

• The first part is the metadata of the ABC notation of the actual music generated.
• X : When there are more than one tune in a file
  T : The title
  M : The time signature
  L : The default note length
  R : The type of tune
  K : the key
• The second part follows the metadata and represents the tune.
• If you know the ABC notation of music, you can write the sheet music and vice versa.
• You can convert abc notation into MIDI too and download it.

3. MIDI Representation

MIDI format

• we can see an excerpt from a midi file that has been read using Music21.
• The data split into two object types: Notes and Chords.
• Note objects contain information about the pitch, octave, and offset of the Note.
• Chord objects are essentially a container for a set of notes that are played at the same time.
• Each line is basically events — event 0, event 1, event 2…event n and it gives us information about the timestamps.

In this case study, we will focus more on ABC notation as it is the simplest one and just uses aplha numeric characters.

Why MP3 is not considered as music representation?

• Mp3 contains frequency, amplitude, and timestamp.
• Musicians use ABC notation or sheet music as a representation which is much more efficient because they don't generate music by thinking in terms of frequency.
• So, it’s better to leverage thousands of years of music notation that phenomenal musicians have designed.
• Hence, we will not use mp3 files. We will compose our music in the space of notations.

Char-RNN Model (High-Level Overview)

So, we have some domain knowledge by now and we need not be an expert. We’ll now ground this in and know about char-RNN. Since our music is a sequence of characters, therefore our obvious choice will be RNNs.

• There is a special type of RNN called “Char-RNN”.
• We will be using many to many RNN. Here, we will feed the RNN with our characters of the sequence one by one and it will out the next character in the sequence.

• This will then allow us to generate a new tune one character at a time.
• As a working example, suppose we only had a vocabulary of four possible letters “helo”, and wanted to train an RNN on the training sequence “hello”.

• This training sequence is in fact a source of 4 separate training examples:
  1. The probability of “e” should be likely given the context of “h”.
  2. “l” should be likely in the context of “he”.
  3. “l” should also be likely given the context of “hel”.
  4. finally “o” should be likely given the context of “hell”.
• we will encode each character into a vector using one-hot encoding(i.e. all zero except for a single one at the index of the character in the vocabulary) and will feed them into RNN one at a time with the step function.
• We will then observe a sequence of 4-dimensional output vectors (one dimension per character), which we interpret as the confidence of the RNN currently assigns to each character coming next in the sequence.
• We want the green numbers to be high and the red numbers low in the output layer, giving us the target characters.
• Remember, as this many to many model, the number os inputs is equal to number of outputs.

Data

• The first part is obtaining data itself.
• The second part is pre-processing the data and generating each of the individual batches which we will use along with it’s construction as we will be applying a variation of the Batch-SGD while training our Char-RNN.

Data Obtaining:

• Refer : http://abc.sourceforge.net/NMD
• It says “ABC version of the Nottingham Music Database” which contains over 1000 folk tunes stored in a special text format.
• Of course, it takes a lot of time to train the model with larger data like 1000 tunes. So I will use the jigs dataset which contains about 340 tunes.
• You get a txt file with multiple tunes here.
• Simply copy and paste into a txt file as input.txt.
• Each tune is having a meta data section and music section.

Data Preprocessing:

We want to preprocess the input.txt file into such a format that we can feed it into the RNN because the way we build our dataset will impact the model heavily and RNNs can be tricky.

• Batch Size = 16
  Sequence length = 64
  Total length of characters in input.txt file = 129,665
  No of unique characters = 86

char_to_idx = { ch: i for (i, ch) in enumerate(sorted(list(set(text)))) }

• Here, in char_to_idx, char-to-idx is converting every character to a index or numerical value where ch(character) is the key and index or numerical value, which is similar to a json file:
  which is jusk like json file :
  {“\n”: 0, “ “: 1, “!”: 2, “\””: 3, “#”: 4, “%”: 5, “&”: 6, “‘“: 7, “(“: 8, “)”: 9, “+”: 10, “,”: 11, “-”: 12, “.”: 13, “/”: 14, “0”: 15, “1”: 16, “2”: 17, “3”: 18, “4”: 19, “5”: 20, “6”: 21, “7”: 22, “8”: 23, “9”: 24, “:”: 25, “=”: 26, “?”: 27, “A”: 28, “B”: 29, “C”: 30, “D”: 31, “E”: 32, “F”: 33, “G”: 34, “H”: 35, “I”: 36, “J”: 37, “K”: 38, “L”: 39, “M”: 40, “N”: 41, “O”: 42, “P”: 43, “Q”: 44, “R”: 45, “S”: 46, “T”: 47, “U”: 48, “V”: 49, “W”: 50, “X”: 51, “Y”: 52, “[“: 53, “\\”: 54, “]”: 55, “^”: 56, “_”: 57, “a”: 58, “b”: 59, “c”: 60, “d”: 61, “e”: 62, “f”: 63, “g”: 64, “h”: 65, “i”: 66, “j”: 67, “k”: 68, “l”: 69, “m”: 70, “n”: 71, “o”: 72, “p”: 73, “q”: 74, “r”: 75, “s”: 76, “t”: 77, “u”: 78, “v”: 79, “w”: 80, “x”: 81, “y”: 82, “z”: 83, “|”: 84, “~”: 85}
• How batches are constructed ?

→ Let’s say at index “0” in X is character index 82.
→ So, I want my Y to be predicted as 80 as the next character index number.
→ The tensors in the “Y” are one hot encoded and considered as much of classification .
→ Character index 82 at index 0 of X is fed into RNN, which gives us an one hot encoded vector with 80 as “1” .
→ As we know, we have assigned index to each of the character.
→ In reality, the batches will contain index of the corresponding character.

• I’m trying to generate new batch everytime using the function called “read_batches”.
  → One batch contains 16 rows(BATCH_SIZE) and 64 columns(SEQUENCE_LENGTH) making a total of (16*64) 1024 characters in one batch, and similarly there are 126 total batches for 1 epoch.
  → Why aren’t we simply using 1st 1024 characters in batch_1, next 1024 characters in batch_2 and so on till batch_126 ?
  It is sequence data so if we combine two sequences means we are giving wrong data. So we have to give all sequences in a batch differently.
  → Given seq length of 64 and batch size of 16, we need to construct 126 batches, in such a way that in the first batch, we input first 63 characters, in batch 2, we continue from 64 onwards and so on such that the sequence is maintained.

The above code snippet is a function to create batches. Here, there are three nested loops. The first loop denotes the batch number. It runs every time when a new batch is created. The second loop denotes a row in a batch and the third loop denotes a column in a batch.

Many to Many RNN

• In the hidden layer, instead of having single lstm, we can have multiple lstm for production purposes. In reality, we can have setup like the below figure.

→ Let’s say, we have four lstms in the hidden layer.
→ I have an input “Ci” at time “t” in the input layer.
→ Now, all four of the lstms will together generate an output for me, which should be ideally “Ci+1” character.
→ The output from the first time step will go to the next time time step, where my input is “Ci+1”.
→ Again, there will be four lstms working together to give me output “Ci+2”.
→ The process will continue so on.
→ Each of the lstm unit will learn different aspect of our character.

• Now to be able to construct a network like this, we need to learn about return sequences and time distributed dense layer.

def build_model(batch_size, seq_len, vocab_size):
    model = Sequential()
    model.add(Embedding(vocab_size, 512, batch_input_shape (batch_size, seq_len)))
    for i in range(3):
        model.add(LSTM(256, return_sequences=True, stateful=True))
        # It creates 256 lstms layers in hiden layers
        model.add(Dropout(0.2))
    model.add(TimeDistributed(Dense(vocab_size))) 
    model.add(Activation('softmax'))
    return model

• Return sequences:

→ For every time step, we give an input and we want an output → Here you can have return-sequence = True .This is what,we want here.
→ For every time step, we keep giving input and at the end, only after consuming the whole sequence, it gives us output → Here you can have return_sequence = False.

• Time distributed dense layer:
  → We have 87 unique characters in our dataset and we want that the output at each time-stamp will be a next character in the sequence which is one of the 87 characters.
  → Time-distributed dense layer contains 87 “Softmax” activations and it creates a dense connection at each time-stamp.
  → Finally, it will generate 87 dimensional output at each time-stamp which will be equivalent to 87 probability values. It helps us to maintain Many-to-Many relationship.

• Stateful RNN:
  - This has got to do with batch mode of operation.
  - Here, we have given “stateful = True”.
  - If True, then the last state for each batch at index ‘i’ in a batch will be used as initial state for the sample of index ‘i’ in the following batch.
  - This is used because all of the batches contains rows in continuation and our model will learn more longer sequences.
  - The default is stateful = False

Model Architecture & Training

Model Architecture

def built_model(batch_size, seq_length, unique_chars):
 model = Sequential()
 
 model.add(Embedding(input_dim = unique_chars, output_dim = 512, batch_input_shape = (batch_size, seq_length))) 
 
 model.add(LSTM(256, return_sequences = True, stateful = True))
 model.add(Dropout(0.2))
 
 model.add(LSTM(256, return_sequences = True, stateful = True))
 model.add(Dropout(0.2))
 
 model.add(LSTM(256, return_sequences = True, stateful = True))
 model.add(Dropout(0.2))
 
 model.add(TimeDistributed(Dense(unique_chars)))
 model.add(Activation(“softmax”))
 
 return model

• X is a matrix of (BATCH_SIZE,SEQ_LENGTH) = (16,64)
• Y is a 3D tensor of (BATCH_SIZE,SEQ_LENGTH,vocab_size) = (16,64,86). The vocab size is considered because of one-hot encoding.
• After embedding, (BACTH_SIZE,SEQ_LENGTH,embedding_dim) = (16,64,512)
• We encoded “Y” as one hot encoded because we will be applying softmax on top of it.
• Now, we want to predict the next character which should be one of the 86 unique characters. So, it’s a multi-class classification problem. Therefore, our last layer is the softmax layer of 86 activations.
• So, I will generate each of my batches and train them. For every training epoch, I will print the categorical cross-entropy loss and accuracy.
• Here is the summary of my model. So, I have 1,904,214 total parameters.

• As we are having so many parameters, so we are using dropouts with a keep probability of 0.2.
• By the time we reach 100 epochs while training, roughly around 90% + times, the model is able to predict what the next character is. So, our model is doing a pretty good job.
• At the end of 10 epochs, we are storing the weights of the model. We will use these weights to reconstruct the model and predict.

Music Generation

• Now that we have the trained model and found our best weights. let’s look at how we will generate new music.
• Our model is ready to predict.
• In order to make prediction, we will give any of the 87 unique characters as input to our model, and it will generate 87 probability values through the softmax layer. From these returned 87 probability values, we will choose the next character probabilistically and finally, we will again feed the chosen character back to the model and so on.
• We will keep concatenating the output characters and generate music of the desired length. The typical number is between 300–600. A too small number will hardly generate any sequence.
• We will errors, when we try to predict music using weights of smaller epochs.

• Open the following link and Paste your generated music in the given space in order to play.

abcjs editor

• I generated music using weights.10.h5 model of epoch = 10 and using weights.100.h5 model of epochs = 100.

The above music sequences are some examples that i have generated from the model.

Generating Tabla Music

You can use the char-RNN model to generate tabla music too. Refer to these blogs for a better understanding.

Machines learn to play Tabla

Machines learn to play Tabla, Part - 2

MIDI Music Generation

Here, Music21 library is used to read the MIDI files and convert it into the sequence of events. You can through the below blogs for a better understanding.

How to Generate Music using a LSTM Neural Network in Keras

Skuldur/Classical-Piano-Composer

Further Scope

Well, we got pretty good results, but we can improve our model by training it with more tunes of multi-instruments.

Here, I trained my model with just 350 tunes. So, we can expose our model to more instruments and varieties of musical tunes, which will result in more melodious tunes with varieties. We can add a method, which can handle unknown notes in the music by filtering unknown notes and replacing them with known notes.

As of now, my model is trained on a single instrument, so the music generated is of only one instrument. By adding the above method and training the model on large varieties of data while increasing the number of classes, we can generate more robust quality and melodious music.

Code Link for this project

https://github.com/Swetadas-1718/Music-Generation