Can a computer generate realistic music?

Many people would probably answer this question with a clear “no”. Is it because they really believe computers cannot? Or maybe there is an inner voice that refuses to believe that musical creativity is not exclusive to humans? In the meantime, AI systems such as DALL-E and ChatGPT have already proven unprecedented performance for generating images and textual artefacts, and are now becoming ubiquitous in our society. What about AI-generated music?

26 April 2023

by Jacopo de Berardinis & Max Tiel

Many people would probably answer this question with a clear “no”. Is it because they really believe computers cannot? Or maybe there is an inner voice that refuses to believe that musical creativity is not exclusive to humans? In the meantime, AI systems such as DALL-E and ChatGPT have already proven unprecedented performance for generating images and textual artefacts, and are now becoming ubiquitous in our society. What about AI-generated music?

The automatic composition of music dates back to the ancient Greeks, and remained up to and beyond Mozart with the “Dice Game”. Even Ada Lovelace, an esteemed mathematician, speculated that the “calculating engine” might compose elaborate and scientific pieces of music of any degree of complexity or extent. In recent years, AI systems have achieved remarkable results on symbolic and audio music [1]. The variety of computationally creative methods for music is quite broad and diversified, and has already enabled the exploration of novel forms of artistic co-creation [2]. These range from the automatic generation, completion, and alteration of chord progressions and melodies, to the creation of mashups, and audio snippets from textual prompts [3].

However, music AI systems are still far from generating full musical pieces that can be deemed realistic across several musical dimensions (harmony, form, instrumentation, etc.). First, most systems require hours and hours of human-composed (real) music before they can start generating interesting artefacts. Most importantly, although the machine can explore thousands of musical ideas, the resulting compositions are rarely used as final outputs. To make these “musical drafts” realistic, human intervention is often needed to correct, adapt, and extend the generations – depending on the creative workflow put forth by the  artist. Hence, human involvement is at least required upstream (data curation) and downstream (music adaptation) of the generation process.

Human participation is also needed to evaluate music generation systems in a variety of forms, ranging from Turing tests to musicological evaluations. Recently, researchers are also starting to devise new computational methods to automate this process. This way, every time a new music generation system is developed, its outputs can be coherently measured and compared in a controlled framework. Towards this direction, Dr. Jacopo de Berardinis, postdoc at King’s College London, has recently published a new method on evaluating the structural complexity of machine-generated music: by letting an algorithm decide on whether a piece of sound has realistic musical structure or not. De Berardinis is part of the Polifonia consortium, an AI-music project funded by the European Union’s Horizon 2020 research and innovation programme.

De Berardinis: ‘’composing musical ideas longer than phrases is still an open challenge in computer-generated music, a problem that is commonly referred to as the lack of long-term structure in the generations. In addition, the evaluation of the structural complexity of artificial compositions is still done manually – requiring expert knowledge, time and involving subjectivity which is inherent in the perception of musical structure.’’

AI can create short pieces of music or make variations and interpolations on existing pieces. But at some point the music will start diverting because machine learning models still struggle with long term dependencies. By automating the process of the evaluation of musical output, a lot of resources that are in human analysis of the music data, can be saved. To address this, de Berardinis detects musical structures from the sound [4], and describes their decomposition process [5]. This then allows the system to ‘judge’ the structural complexity of music on a scale from ‘real music’ to ‘random music’. In other words: you provide a music dataset and let the system match the input music on a continuum between these two complexity classes.

In conclusion, the automatic generation of music that cannot be distinguished from human compositions still remains an open challenge for AI music research. In the meantime, opening a debate on the eventual implications of these methods is also necessary – as having a system that can realistically generate music raises ethical concerns. Instead of designing tools that could potentially replace artists and composers (for certain commissions), de Berardinis argues that research should focus on leveraging the generative capabilities of AI models to design new systems that can enhance and augment the creative potential of artists – thereby enabling novel opportunities for Artificial Intelligence Augmentation (AIA) [6]. With these concerns and objectives, a team in Polifonia is currently working towards the creation of resources and algorithms to promote more transparent, fair, and reliable paradigms in music AI.

[1] Briot, J. P., Hadjeres, G., & Pachet, F. D. (2020). Deep learning techniques for music generation (Vol. 1). Heidelberg: Springer.

[2]  Huang, C. Z. A., Koops, H. V., Newton-Rex, E., Dinculescu, M., & Cai, C. J. (2020). AI song contest: Human-AI co-creation in songwriting. arXiv preprint arXiv:2010.05388.

[3] Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., … & Frank, C. (2023). Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325.

[4] de Berardinis, J., Vamvakaris, M., Cangelosi, A., & Coutinho, E. (2020). Unveiling the hierarchical structure of music by multi-resolution community detection. Transactions of the International Society for Music Information Retrieval, 3(1), 82-97.

[5] de Berardinis, J., Cangelosi, A., & Coutinho, E. (2022). Measuring the structural complexity of music: from structural segmentations to the automatic evaluation of models for music generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30, 1963-1976.[6] Carter, S., & Nielsen, M. (2017). Using artificial intelligence to augment human intelligence. Distill, 2(12), e9.

Recent News

From the beginning of the project, Podiumkunst.net and Polifonia have been in close contact and looked to each other as role models. Our stakeholder Podiumkunst.net reflects on this synergy with a positive outlook. Remco de Boer and Monique in het Veld on the importance of the collaboration and the impact. 

From the beginning of the project, Podiumkunst.net and Polifonia have been in close contact and looked…

11 June 2024

TONALITIES, IReMus’ pilot for musical heritage data project Polifonia, develops tools for the modal-tonal identification, exploration and classification of monophonic and polyphonic notated music from the Renaissance to the twentieth century. Now, the tools are available for use within the TONALITIES Interface for music analysis. Additionally, a patent was recently acquired for this collaborative interface by the IReMus lab.

TONALITIES, IReMus' pilot for musical heritage data project Polifonia, develops tools for the modal-tonal…

29 May 2024

From April 8 to May 6 Polifonia organised their own version of the Eurovision Song Contest, the Polifonia Song Contest: musicians of all levels were challenged to create the ‘soundtrack of our history’ by using samples from the rich collections in the Polifonia project. Today we can announce the winning song.

From April 8 to May 6 Polifonia organised their own version of the Eurovision Song Contest, the Polifonia…

13 May 2024

After four years of development work, the Polifonia project team is excited to present the results. The consortium, consisting of 10 partners from Italy, the Netherlands, France, England and Ireland launches the music discoverability platform ‘Polifonia Web Portal’. In addition, the researchers and developers have also unlocked and linked other music data, developed tools and software that will help musicologists take steps forward in their research on European musical heritage.

After four years of development work, the Polifonia project team is excited to present the results.…

8 May 2024

The Polifonia project formally ended on April 30, which means that the tools and software developed within this 4-year-project are released and ready for use. Today we look at ‘Patterns UI’.

The Polifonia project formally ended on April 30, which means that the tools and software developed…

3 May 2024

Polifonia Song Contest is two weeks in, and will continue for another two weeks. Have you downloaded the sample pack yet?

With two weeks to go until the deadline, the "Polifonia Song Contest" beckons all musicians who find…

22 April 2024

Are you the type of musician that is inspired by old sounds, such as cheerful Irish folk melodies, the majestic resonance of pipe organ concerts, and the timeless chimes echoing from century-old Italian bell towers? Then ‘Polifonia Song Contest’ is your challenge!

Are you the type of musician that is inspired by old sounds, such as cheerful Irish folk melodies, the…

8 April 2024

The consortium is preparing for the last face-to-face consortium meeting of the Polifonia project in April 2024.

The consortium is preparing for the last face-to-face consortium meeting of the Polifonia project in…

4 April 2024

Polifonia is known for its strong links with academia and is pleased to present some highlights in its involvement in research and associated conferences.

Polifonia is known for its strong links with academia and is pleased to present some highlights in its…

29 February 2024

In 2024, Paul Mulholland, Naomi Barker and Paul Warren (The Open University, U.K) are continuing their experiment investigating how different kinds of music influence the appreciation of an artwork; and to what extent the same kind of sense-making processes are used when viewing artwork and when listening to music. To do this, the researchers are looking for more participants. They have now automated the process so that participants can complete the experiment online without the involvement of an experimenter.

Music instrument with music notes on white background illustration In 2024, Paul Mulholland, Naomi…

17 January 2024

This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement N. 101004746