LYRELY

MUSIC PORTFOLIO SOFTWARE

ABOUT LYRELY

Lyrely was created to fill a gap in online music portfolio software as it is specifically aimed at video game composers. It needed to do three things well:

Have a web player that sits on an artist’s page nicely (no iframes, mobile support)
Stream high quality music that would support gapless looping
Secure media files and password protected portfolios due to NDAs and the nature of game development.

AUDIO QUALITY

Music is the heart of the platform and there are many different music file formats all with various ups and downs, OS support, etc. I was working with a friend of mine, Tommy Pedrini to help me determine how to maximize sound quality while attempting to keep the file size low. At first we started with AAC, designed to be the successor to MP3 files. AAC, while better than MP3s in terms of file size and audio quality was still not reaching the standard that we hoped to achieve.

OPUS was the answer to our audio quality problems, it supports near-lossless audio wrapped in a reasonable file size (5-8mb typically). However it is not supported on Apple devices unless wrapped as a CAF file, this added some additional complexity in how audio files are converted and served (more on this in the Tech Stack area).

WEB PLAYER

The web player takes advantage of Audio Streaming, HTML Canvas and the Web Audio API to deliver quick load times, advanced visualizer features and gapless looping. The audio visualization features (dynamic waveform and particle effects) and gapless looping required use of the Web Audio API. However, the Web Audio API meant that users would have to download the entire song before any music would begin to play. This delay was extremely noticeable and made the player feed sluggish and broken.

The fix was a hybrid approach: when a user presses play on a song the player will use native HTML Audio Streaming to begin playback. In the background it begins downloading the entire song file and once that download is complete it begins playing the song again at 0% volume using the Web Audio API. Finally, it fades the audio stream out and the Web Audio player in. By fading one audio source out and the other in at the same time, the user doesn’t notice the switch. With full access to the Web Audio API stream we can use FFTs to analyze the incoming audio (to determine “energy” for the particle systems / visualizers) and setup loop points for seamless loops.

BACK END TECH

EXPRESS REST(ISH) API

The back end is primarily powered by Node/Express running on EC2 on AWS. The express API uses Sequelize, an ORM for MySQL to handle communication with the RDS DB and help ease query writing. Most of the back end operations are simple creates, updates and deletes however it is also responsible for dynamically generating the web player script when a user places one on their website.

Due to the nature of game audio (NDAs, etc): the public facing APIs must use a combination of local ids and UUIDs (long, non-sequential ids) which cannot be guessed in order to reference any files. For example, when a user presses play on a web player it will ask the API for track #3 (a local id for that player). The API will then return a temporary URL pointing at the file which expires in a couple of minutes. Due to how the player works by downloading song buffers, these links only need to be briefly available until the download completes. These techniques ensure that files will not be leaked.

FILE PROCESSING

File processing and conversion is an expensive (consumes a lot of CPU cycles) operation. It was important to find a way to avoid doing this on the server that’s running the API to avoid any slowdowns as it’s needed to serve the web players to people swiftly. I was able to use AWS’s Lambda service to handle the conversions without impacting the API server. It automatically triggers when any audio files are added to a specific Amazon S3 folder.

For best results and to avoid weird conversion issues, using FFMPEG to convert whatever file was uploaded to .wav first was the safest course of action. Next, it needs to generate the waveform data for the front end to use via a library by the BBC called “audiowaveform”. Finally, it converts the audio files to OPUS and CAF, moves them to the “complete” folder on S3 and flags the conversion as complete in the database.

Using Lambda was not without issues however, in order to support converting to both OPUS and CAF, I had to build a custom version of FFMPEG with that support added and upload it to the Lambda layer. The next problem was difficulty talking to the database server once the conversion was complete. It needs to update the track row in the database and tell the rest of the system that the conversion completed successfully but for some reason I was unable to get the AWS permissions to allow access to S3 (the files) and RDS (the database) at the same time. As a result, I ended up setting up a second Lambda service that runs after the file conversion which has access to the database but not the files. It receives the conversion status and is able to update the correct database row.

FRONT END APPLICATION

REACT, RTK-QUERY AND TAILWIND

The front end management application is built on React, uses RTK-Query to fetch data from the backend and Tailwind for quick and easy styling. React is my bread and butter when it comes to web development. I enjoy working with it significantly more than Angular or VueJS so it of course made sense to use it for this project. It was my first time using RTK-Query, all of my previous projects involved manually fetching API data and managing it with local state or Redux. RTK-Query simplifies this significantly and makes using Redux a lot easier to manage. I will be using RTK-Query going forward for API and app state without a doubt.

Tailwind has probably been one of the biggest quality of life changes to front end development for me. I know it has cons but the simple fact that I no longer need to think of class names is such a blessing. Trying to name something a useful name one of the hardest parts of development and it gets much worse on the front end when several container elements are required to achieve a certain layout. Tailwind removes all those several minute blockers and makes styling quick and easy.

A screenshot of Lyrely's track edit page

Editing a Track

LIBRARY MANAGEMENT QOL

Library management is a big part of creating and publishing playlists (web players) and portfolios. It’s very important that this aspect of the application feel good for users. One of the first major debates I had with myself was surrounding how to store playlist data. A couple of composers that I spoke to expressed interest in being able to easily move folders and tracks between playlists. Given that order is important, I felt the easiest way to achieve this was going to be a JSON blob representing all playlists so playlists, folders and items can be easily rearranged without multiple rows manually managing “order”. There are some potential long term concerns with storing large datasets in JSON, however I tested inserting 100k rows into the database with the proposed data structure and there was no noticeable degradation in speed.

Once users were using the platform it became obvious how important multi-selection features were. Most applications I had built in the past had click being the default action to go places. However, once I realized how important it would be to select tracks, folders, etc I changed all the interfaces to have click select instead of “open”. Additionally I added options for CTRL to select additional items and Shift to group select into the Tracks and Playlist areas with support to drag/drop multiple as well. While these were major quality of life improvements, the final step was to add some rudimentary copy / paste features to the playlist areas to allow for easier duplication of data across players.

Managing your library with multiselect

PORTFOLIO EDITOR WITH TIPTAP

Portfolio’s allow users to quickly create a landing page with a web player on it that they can send to prospective clients. These pages can be password protected to prevent any sensitive data (tracks, personal info, etc) from leaking out. The portfolio editor allows users to fully manage and style their portfolio page. Users can set passwords, adjust styles (colors, backgrounds), change the layout, build a web player and add their own content via a Tiptap WYSIWYG. Portfolios are served at https://{username}.lyrely.com/{custom-slug}. Users can set custom-slug to whatever they want or just leave it with the default randomly generated url.

I chose Tiptap due to its ability to output both HTML and JSON for storage / rendering and because it is a headless editor making it easy to integrate and style without significant CSS overrides. Tiptap also has an extensive API for writing custom plugins, I took advantage of this to add image uploads with easy inline resizing to the portfolio editor.

A screenshot of Lyrely's portfolio editor

Creating a portfolio