Can a computer generate realistic music?

Many people would probably answer this question with a clear “no”. Is it because they really believe computers cannot? Or maybe there is an inner voice that refuses to believe that musical creativity is not exclusive to humans? In the meantime, AI systems such as DALL-E and ChatGPT have already proven unprecedented performance for generating images and textual artefacts, and are now becoming ubiquitous in our society. What about AI-generated music?

26 April 2023

by Jacopo de Berardinis & Max Tiel

Many people would probably answer this question with a clear “no”. Is it because they really believe computers cannot? Or maybe there is an inner voice that refuses to believe that musical creativity is not exclusive to humans? In the meantime, AI systems such as DALL-E and ChatGPT have already proven unprecedented performance for generating images and textual artefacts, and are now becoming ubiquitous in our society. What about AI-generated music?

The automatic composition of music dates back to the ancient Greeks, and remained up to and beyond Mozart with the “Dice Game”. Even Ada Lovelace, an esteemed mathematician, speculated that the “calculating engine” might compose elaborate and scientific pieces of music of any degree of complexity or extent. In recent years, AI systems have achieved remarkable results on symbolic and audio music [1]. The variety of computationally creative methods for music is quite broad and diversified, and has already enabled the exploration of novel forms of artistic co-creation [2]. These range from the automatic generation, completion, and alteration of chord progressions and melodies, to the creation of mashups, and audio snippets from textual prompts [3].

However, music AI systems are still far from generating full musical pieces that can be deemed realistic across several musical dimensions (harmony, form, instrumentation, etc.). First, most systems require hours and hours of human-composed (real) music before they can start generating interesting artefacts. Most importantly, although the machine can explore thousands of musical ideas, the resulting compositions are rarely used as final outputs. To make these “musical drafts” realistic, human intervention is often needed to correct, adapt, and extend the generations – depending on the creative workflow put forth by the  artist. Hence, human involvement is at least required upstream (data curation) and downstream (music adaptation) of the generation process.

Human participation is also needed to evaluate music generation systems in a variety of forms, ranging from Turing tests to musicological evaluations. Recently, researchers are also starting to devise new computational methods to automate this process. This way, every time a new music generation system is developed, its outputs can be coherently measured and compared in a controlled framework. Towards this direction, Dr. Jacopo de Berardinis, postdoc at King’s College London, has recently published a new method on evaluating the structural complexity of machine-generated music: by letting an algorithm decide on whether a piece of sound has realistic musical structure or not. De Berardinis is part of the Polifonia consortium, an AI-music project funded by the European Union’s Horizon 2020 research and innovation programme.

De Berardinis: ‘’composing musical ideas longer than phrases is still an open challenge in computer-generated music, a problem that is commonly referred to as the lack of long-term structure in the generations. In addition, the evaluation of the structural complexity of artificial compositions is still done manually – requiring expert knowledge, time and involving subjectivity which is inherent in the perception of musical structure.’’

AI can create short pieces of music or make variations and interpolations on existing pieces. But at some point the music will start diverting because machine learning models still struggle with long term dependencies. By automating the process of the evaluation of musical output, a lot of resources that are in human analysis of the music data, can be saved. To address this, de Berardinis detects musical structures from the sound [4], and describes their decomposition process [5]. This then allows the system to ‘judge’ the structural complexity of music on a scale from ‘real music’ to ‘random music’. In other words: you provide a music dataset and let the system match the input music on a continuum between these two complexity classes.

In conclusion, the automatic generation of music that cannot be distinguished from human compositions still remains an open challenge for AI music research. In the meantime, opening a debate on the eventual implications of these methods is also necessary – as having a system that can realistically generate music raises ethical concerns. Instead of designing tools that could potentially replace artists and composers (for certain commissions), de Berardinis argues that research should focus on leveraging the generative capabilities of AI models to design new systems that can enhance and augment the creative potential of artists – thereby enabling novel opportunities for Artificial Intelligence Augmentation (AIA) [6]. With these concerns and objectives, a team in Polifonia is currently working towards the creation of resources and algorithms to promote more transparent, fair, and reliable paradigms in music AI.

[1] Briot, J. P., Hadjeres, G., & Pachet, F. D. (2020). Deep learning techniques for music generation (Vol. 1). Heidelberg: Springer.

[2]  Huang, C. Z. A., Koops, H. V., Newton-Rex, E., Dinculescu, M., & Cai, C. J. (2020). AI song contest: Human-AI co-creation in songwriting. arXiv preprint arXiv:2010.05388.

[3] Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., … & Frank, C. (2023). Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325.

[4] de Berardinis, J., Vamvakaris, M., Cangelosi, A., & Coutinho, E. (2020). Unveiling the hierarchical structure of music by multi-resolution community detection. Transactions of the International Society for Music Information Retrieval, 3(1), 82-97.

[5] de Berardinis, J., Cangelosi, A., & Coutinho, E. (2022). Measuring the structural complexity of music: from structural segmentations to the automatic evaluation of models for music generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30, 1963-1976.[6] Carter, S., & Nielsen, M. (2017). Using artificial intelligence to augment human intelligence. Distill, 2(12), e9.

Recent News

Polifonia is known for its strong links with academia and is pleased to present some highlights in its involvement in research and associated conferences.

Polifonia is known for its strong links with academia and is pleased to present some highlights in its…

29 February 2024

In 2024, Paul Mulholland, Naomi Barker and Paul Warren (The Open University, U.K) are continuing their experiment investigating how different kinds of music influence the appreciation of an artwork; and to what extent the same kind of sense-making processes are used when viewing artwork and when listening to music. To do this, the researchers are looking for more participants. They have now automated the process so that participants can complete the experiment online without the involvement of an experimenter.

Music instrument with music notes on white background illustration In 2024, Paul Mulholland, Naomi…

17 January 2024

During the last project meeting, the Polifonia consortium extensively discussed how to foster the impact of the project in academia and beyond. How to make the output of Polifonia sustainable after the lifetime of the project is one important aspect. But fostering re-usability does not end by long-term preservation of certain assets (such as data and tools). In Polifonia Research Ecosystem – Impact of a project. A webinar on Data re-use and workflows, we will discuss how we ensure that more fluid assets such as interfaces, but also experiences in setting up and executing workflows via those interfaces, become reproducible and reuseable.

During the last project meeting, the Polifonia consortium extensively discussed how to foster the impact…

15 January 2024

For the Polifonia project, the Central Institute for Cataloging and Documentation (ICCD) of the Italian Ministry of Culture is carrying out activities on the historical bell heritage. The ICCD has also initiated a process of documentation of the practices and knowledge associated with bell production through collaboration with historical Italian foundries.

The bell casting process performed by the Pontifical Marinelli Foundry. Photo courtesy of ICC For…

9 January 2024

One of the tools Polifonia will release is MELODY. It stands for ‘Make mE a Linked Open Data StorY’ and is a place where you can make sense of Linked Open Data and publish text-based as well as visual data stories. Earlier this year, students of the University of Bologna explored data through this tool. Let’s see what they have found and learned about… rock music.

One of the tools Polifonia will release is MELODY. It stands for 'Make mE a Linked Open Data StorY'…

13 December 2023

Music libraries currently lack well-founded information retrieval tools. While it is relatively easy to find music based on metadata, content-based music retrieval still remains as a challenge. The Polifonia FACETS pilot aims to tackle this challenge by building a faceted search engine (FSE) for large collections of music documents.

Music libraries currently lack well-founded information retrieval tools. While it is relatively easy…

24 November 2023

This is a week of major importance to the Polifonia team, as its researchers join both the conference of the International Society for Music Information Retrieval (ISMIR) and the conference for the International Semantic Web and Linked Data Community (ISWC): venues of significant importance for both research and industry. Read more about Polifonia’s contributions below.

This is a week of major importance to the Polifonia team, as its researchers join the conference of…

7 November 2023

On Oct, 13 an explorative workshop took place in a school in Milton Keynes (UK) as part of the Polifonia project. The “Music Meets Machines workshop” gave a look into cutting-edge technologies used to represent music history.

On Oct, 13 an explorative workshop took place in a school in Milton Keynes (UK) as part of the Polifonia…

3 November 2023

Between 16 and 20 October, the Polifonia consortium met in the Italian city of Bologna, home of the project coordinator University of Bologna (UNIBO). During an intensive week, the project team took steps in the development of the pilots, including the long-awaited web portal. Read more about the 7th project meeting here.

Between 16 and 20 October, the Polifonia consortium met in the Italian city of Bologna, home of the…

27 October 2023

The sensory journey “Data Wanderings” is a new project of Polifonia. The art installation will open on Friday, Oct. 13, in Bologna, Italy and you can visit it until the 28th of the month.

The sensory journey "Data Wanderings" is a new project of Polifonia. The art installation will open…

12 October 2023

This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement N. 101004746