Build an AI Composer – Machine Learning for Hackers #2

Build an AI Composer – Machine Learning for Hackers #2

I actually didn’t play anything. You just
heard AI generated music. Hello World, welcome to Sirajology! In this
episode, we’re going to train a neural network to compose music all by itself. Machine generated
music! the technical term for this is ‘music language modeling’ and it has a long history
of research behind it. Markov Models and restricted bolztman machines. Which kind of sounds like
something out of half life or bioshock. Hold on babe. I’ve got to go save the world
using my restricted boltzman machine. Music is how we communicate our emotions and
passions and its completely based on mathematical relationships. Octaves, Chords, Scales, keys,
all of it is math. At the lowest level, music is a series of sound waves that create pockets
of air pressure, and the pitch we hear depends on frequency of changes in this air pressure.
We’ve created annotation to help us map these sounds
into an instruction set. So if machine learning is all about feeding data into models to find
patterns and make predictions, could we use it to generate music all by itself? absofruitly. We’re going to be build an app that learns
how to compose british folk music by training on a dataset of british folk music. We’ll
be using Tensorflow, the sickest machine learning library ever, to do this in just 10 lines
of Python. We’ll be following the tried and true 4 step machine learning methodology
to do this. Collect a dataset, build the model, train the model, and test the model. To start
off, we’ll want to collect our dataset. So let’s import the urllib module, which
will let us download a file from the web. Once we import it we can call the URLretrieve
method to do just that. We’ll set the parameters to the link to the dataset and the name we’ll
call the downloaded file. We’re using the nottingham dataset for this demo, which is
a collection of 1000 british folk songs in MIDI format. MIDI format is perfect for us
since it encodes all the note and time information exactly how it would be written in music annotation.
It comes in a zip file, so we’ll want to unzip it as well. We can do this programmatically
using the zipfile module. We’ll extract the data from the zip and place it in the
data directory. We’ve got our data, it’s time to create
the model. But before we do that, we need to think about how we want to represent our
input data.There are 88 possible pitches in a MIDI file so we could do one vector representation
per note. But lets be more specific. At each time step in the music, there are two things
happpening. Theres the main tune or melody and then there are the supporting notes or
harmony. Let’s represent each as a vector. And to make things easier we’ll make two
assumptions. The first is that the melody is monophonic. That means only one note is
played at each time stamp. The second is that the harmony at each stamp can be classified
into a chord class. So thats two different vectors one for melody and one for harmony.
We’ll then combine them into one vector for each stamp. We can just import our ML helper class and
then call the create model method to do this. Music plays out over a period of time, its
a sequence of notes. So we need to use a sequence learning model – it has to accept a sequence
of notes as an input and output a new sequence of notes. Plain old neural nets can’t do
this. They accept fixed sized inputs like an image or a number. We’ll need a special
kind of neural network, a recurrent neural network. Yeah! Those can deal with sequences
since data doesn’t just flow one way, it loops. This allows the network to have a kind
of short term memory. Yeah, that’ll work. But wait. We want our network to not just
remember the most recent music its heard, but all the music its heard. Like, a piece
of music can have multiple themes in different parts of it (hopeful, melancholic, angry)
and if the network only remembers the most recent part which was cheery, it’s just
going to compose cheery stuff. We need a special type of recurrent neural network called a
Long short term memory network. Super specific i know. This type of network has a short term
memory that is LONG, like it can remember things from way back in the sequence of data,
and it uses everything it remembers to generate new sequences. We can add this model in our code with just
one line using our helper class. It’ll generate the sequences and chord mapping file to a
file called ‘’ in the data folder. This is just a serialized byte stream representation
of our music that we’re going to train our model with it. Now that we have our model
we can go ahead and train it. You might be thinking wait this is a little too easy, isn’t
there more to it? Well yeah, every machine learning model has a set of what are called
‘hyperparameters’. These are the parameters that we humans set for how our model operates,
like knobs on a control panel. How many layers do we want? how many iterations for training?
How many neurons? You could play around with these, turning all the knobs in different
ways to perfect your end-result, but chances someone somewhere has solved the problem you’re
working and and you can just use an existing model with pre-tuned hyperparameters to build
something awesome So now we’re ready to train our model. We
can just call the train_model method of our recurrent neural net class to do this. This’ll
get the the network to start collecting the input data piece by piece. It took me about
2 hours to train it on my 2013 macbook pro. But you don’t have to wait until its competely
done training to test it out. Just wait until you see the “Best loss so far encountered,
saving model.” message. Once you see that you can type ‘rnn_sample’ into terminal
with the flag —config file and point it to the newly generated config file in your
models folder. That will generate a new song using the newly trained model you’ve just
created. To generate music we just sample the melody and harmony at each time step and
plug it into our trained model. The model will then predict what the next notes will
be. The collection of all the predicted notes is our newly generated song. Let’s listen in to what I’ve generated.
So it sounds nice, it could better but it gives off that british folk vibe. There are
definitely some improvements that could be made. The time signature is kinda sporadic
and in terms of long-term structure, there seems to be a lack of repeated themes and
phrases. The solution may well be more data and more computing power. It usually is when
it comes to machine learning with deep neural nets. Machine learning can help us learn the
fundamental nature of how music works in ways that we haven’t even thought about. I’ve
got links below, check em out. and I’ve gotta go fix a runtime error so thanks for

Comments (78)

  1. how would you do this with TensorFlow? I know you say it early in the video but, import rnn doesn't just work by it's self. and where is the import of tensorFlow?

  2. I am having a problem when creating the midi. It just creates a 2,2kb midi file that gives an error everytime i try to listen to it. I've given the program like 15 epoch to train and keeps doing the same. I am really new at this and noob haha. I am using ubuntu 14 on virtual box running in Windows 7. Any help is gratefully welcome.

  3. Is there any chance you could make a video about prediction let's say stock prices and/or a sports contest? I am working on something similar and I would like to have your point of view on these kinds of models :).

  4. Can you share a config sample file?

  5. I just got introduced to image processing, machine learning and neural network. I just wanted some good intro videos to take the feel of the subject before i actually start learning it. These videos are really good, the content feels legit and the way you present is awesome (except the intro). Please suggest some good beginner tutorials and documentation.

  6. great video. thank you!

  7. +1 for the Donald Trump joke

  8. cant install tensorflow. pip says: could not find version that satisfies the requirement tensorflow (from versions: )
    No matching distribution found for tensorflow
    Please help!

  9. I installed the latest version of tensorflow, It seems they have changed the implementation of the RNN module, I am getting module deprecated error

  10. Is it accidental that the AI generated song resembles a lot Scott Joplins "the entertainer"? (same harmonic progression, ragtime stride bass on the "left hand" vs octaves on the "right hand")

  11. was able to program something like this on TIBASIC but in a different way. There is what I call "controlled randomness" as well as some parameters for chord progression. Not AI, but a similar function, I just think this whole genre is cool 😀

  12. haha, you always got an error at the end of the video

  13. +Sirajology When running the base code it returns "raise ImportError("This module is deprecated. Use tf.nn.rnn_cell instead.")" when trying to import rnn. I tried to modify the script in a few ways to make it work by replacing "rnn.train_model()" with "tensorflow.nn.rnn_cell.train_model()" and added import tensorflow. Now it returns "AttributeError: 'module' object has no attribute 'train_model'" How do I properly import rnn? I'm on the latest version of tensorflow and python 2.7 I believe

  14. Great videos…and oh, you look like Adam Goldberg!

  15. Awesome Please upload more tuts on tenserflow. Thanks alot

  16. Siraj! Can I implore you to help me remember the name of the song you're using in the background at around 0:10? I used to listen to it constantly but I'm forgetting the name and it's lack of lyrics makes it difficult to look up. <3

  17. You should lock exposure before recording video especially in controlled lighting environment such as in front of a green screen. Also, do some color correction on yourself, you always seem to be a different color per video XD

  18. that violin/Double bass??? must have been on some shit, looks like tried to get a undergrad in comp sci

  19. How would I go about training the model on a different dataset? I was thinking of applying something like this to electronic music, or is it limited to only monophonic music?

  20. zipfile doesn't exist anymore…

  21. wow.. you seem very skilled.. cool

  22. Hey, I really love your videos dude, but I'm getting this problem, every time I try to train by running on my Ubuntu machine, it hangs (the mouse cursor can still move but like reeaaaaally slowly) and eventually the terminal outputs the message 'killed' and thats it

  23. Really impressive. What classes would I need to take to learn how to do this?

  24. Is there an api/library that is this good or comes close to this in c#? I haven't found any good libraries on the web that come close to tensor flow. Feelsbadman

  25. Hey man! Awesome videos…really learning a lot. When I run, my process stalls for a bit after "Num Layers: 2, Hidden Size: 200, Melody Coeff: 0.5, Dropout Prob: 0.5, Input Dropout Prob: 0.8, Cell Type: lstm, Time Batch Len: 128, Learning Rate: 0.005, Decay: 0.9", and then "Killed" shows up on the terminal. I tried looking around the comments + GitHub for a solution but no solutions came up. Any idea why this happens? Thank you!

  26. hey dude thanks for everything not everybody appreciate people like u thanks for ur time and effort

  27. is midi python 3 compatible ?

  28. 1:08 is GOLD 😀

  29. Abso-fruit-ly.

    You are a legend.

  30. how can i make my own dataset for this program?

  31. some should put a bunch of black midi files into this app

  32. Awesome videos, you got me interested in deep learning! You should make a similar video explaining how to make hip-hop beats

  33. "I'm gonna make this super advanced AI in just 10 lines of code"
    Imports several libraries and only writes 3 lines of his own code…

    Cool, sure, but you can't really say YOU did the coding, when there's thousands of lines spread across dozens of files that you just make a few calls to.

    With that said, I'm totally gonna do this and try and input 90's game music as training data.


  35. How to get more powerful hardware for cheap. DON'T BUY A MAC, BUILD YOUR OWN COMPUTER AND IF WINDOWS IS NOT YOUR THING GET LINUX.

  36. you rock but you speak so fast

  37. Your hair looks like a feather.

  38. Hi…i have no idea of programing stuff…. but i want a software that i feed with midi files i choose..wich than composes original stuff similar to what i put in …isnt there anything like that ..or a simple way to do that without me having to learn for years? here is allready chosen what it learns. and i have no idea how to achive anything with that video =O need a exe or so xD

  39. Oh Mannn – Who are you – Awesome videos 🙂

  40. Nice one, 2-3-4……………. never finish it!

  41. The output of this code is really nice! i have no idea if its over fitting or not …but its enjoyable music ….can i input my own midis? i tried and it didn't work so i wonder.

  42. man u r awsm!! wht do u do fr a living??

  43. Wow, the shortest, and yet most informative, music theory class ever!

  44. hyper reductionist

  45. @siraj How can we choose the Model over the input data ?

  46. im very new to ML…can anyone tell me how do i make a dataset…that is….if i want to feed my program the music i made , how do i do that?

  47. Very interesting video

  48. Deep nueral net that reads in an image of sheet music and translate that into a data file of the music, into pdf or auto player.

  49. I have a mistake , like this:
    Traceback (most recent call last):
    File "", line 3, in <module>
    import nottingham_util
    ModuleNotFoundError: No module named 'nottingham_util'
    Where can I download this?
    Thank for all!


  51. Once the learning process is started, how do I stop it? Do I just need to force kill it each time?

  52. how would I point to a local folder instead of an url, so that I can build my own music folder instead of using British songs?

  53. >>> urllib.urlretrieve(url, "")
    Traceback (most recent call last):
    File "<pyshell#8>", line 1, in <module>
    urllib.urlretrieve(url, "")
    AttributeError: module 'urllib' has no attribute 'urlretrieve'

  54. awesome video, thanks!

  55. Great.
    But it lacks of the configuration file in the Git repo, where is the file new_config_file.config ?

  56. Adding real value to YouTube!

  57. at first these videos got me so hyped to learn it….but now I'm so disappointed by these videos because it seems to all be taken from someone else's idea/project and not even changed. If you try to customize it at all things fall apart. 🙁
    I'm half convinced there's so many holes in the videos as traps to challenge you to find the answer. I really wish these tutorials weren't put together so……..sloppily. If the code is perfect at the time of release at least mention how you think it could change or whatever. My brain needs to learn.

  58. Love your videos, your energy & your way of teaching. But the music that was made isn´t making the billboards tomorrow I am afraid. Which tells me, there is more to music then just math, all though I agree we can model it etc…

  59. sax and violins !

  60. in python3 urllib has been deprecated to several smaller packages. for the context of the above exercise use ` import urrlib.request ' and `` as the url

  61. Traceback (most recent call last):
    File "C:/Users/User/Desktop/Pyhton/AIMusic", line 3, in <module>
    import nottingham_util
    ModuleNotFoundError: No module named 'nottingham_util'……………………showing me this error…how can i solve it

  62. If you do this now change import to 'from tensorflow.python.ops import rnn, rnn_cell' . The old rnn_cell is deprecated.

  63.¬boulanni/” This is not there at all.

    Not Found

    The requested URL /¬boulanni/ was not found on this server.
    Apache/2.2.15 (Oracle) Server at Port 80

    Would you mind giving us the correct syntax and address for the Your instructs don't seem to work.

  64. why dont u get flagged for using bittersweet symphony in the back?

  65. Sax and Violins. That pun!

  66. Best DT roast ever

  67. New innovations are making it so much less demanding to make music. Nowadays, music scoring software like Dorico ( have even started integrating AI to make it easier and more intuitive to create music.

  68. Dude your are one of the great tutors on youtube.

  69. Thank you for sharing this informative video! 🐻🖐🏻

  70. How can i create my owm datadet

  71. For the noobs: I have been trying to run the code for 4 days. There are some things that the noobs should know before starting to code this. These are the significant problems I had:
    1. You should install the required tensorflow version. Otherwise the code does not work. (The official site does not have the required version so you have to install from github.)
    2. Know your directories
    3. Use the required python and update your operating system. I have been trying to make this work using python 3. but turns out it is written in python 2. I have wasted a lot of time trying to figure out what was wrong. Also most of the installing did not work because my operating system was not up to date

  72. Thanks for this great video..!! But can you make an in detail video for the same so as it helps for beginners like me 🙂

  73. Can you speak slowly…
    Its my request!!

  74. Holy balls you have just gave me an epic quest. Quest accepted.

  75. thanks siraj i appreciate your channel content too much

  76. It isn't working , where do I get the Nottingham library?

Comment here