Music Streaming Backend and Recomendation Engine

Overview

Navimix is a program that sits between a normal (vanilla) subsonic server and the client, acting as a proxy that can add features.

Background

I have been running my own music server with media I own through an open source program called Navidrome for some time now. It works great, but of course only serves media I have added to my library. I also have a premium subscription to Deezer. As a result, I constantly find myself switching between my subsonic server and the Deezer app.

The aim of this project is to bridge the two services together, exlusively on the backend. My main goal is to learn about backends and the GO programming language. Rather than modify Navidrome's source code directly, which would likely be tedious to learn its architecture, and also prevent my server from receiving future security and feature updates, I decided to write a program to sit between my Navidrome server and the client (my phone).

This was much simpler because Navidrome implements the Subsonic API. By essentially intercepting each API function and "injecting" features adhering to the same format, this program acts as a bridge between Deezer and Navidrome.

The name "Navimix" is derived from Navidrome and Deemix, which is the engine used to download files in a Subsonic-friendly format from Deezer.

Current Features

Navimix is written entirely in Go, and runs exclusively as a backend. Its purpose is to transparently download any (copyright-free) music and stream it on the fly via Subsonic. Navimix essentially sits between a normal Subsonic server and the client, proxying unchanged API functionality and modifying certain API requests.

(In addition to a standard Subsonic server)

• Search any song outside of the library (available on Deezer) and have it playback via streaming
• Stream any album
• Fetch cover art for songs and albums
• Find similar songs external to the library in some clients through the getSimilar API function (unmodified)
• Server-side caching of Deezer API requests

Navimix acts as a translation and integration layer between the Deezer API, Deemix, Navidrome, and Subsonic. It is designed to run on top of Navidrome (or any compatible Subsonic server) and add functionality. It is still very much under development, with many new features currently in progress.

See below for a comparison of a search with/without Navimix

Basic architecture

The basic way the program works is as follows:

When Navimix receives a request, it first decides whether to forward it or to parse it and inject information. If it is a request like search, stream, or something to do with media retrieval, it will parse it. Otherwise, it forwards. In the subsonic server architecture, every track, album and artist is assigned a server-specific unique ID. These IDs are long alphanumeric strings. This works out well for Navimix as Deezer IDs are fully unique numeric codes. This both ensures there is no overlap between Deezer tracks and Subsonic tracks and also gives Navimix a simple way to check whether to forward an ID request to subsonic (if the ID contains letters), or whether to retreive the track from Deezer.

For most API endpoints, Navimix essentially retrieves information from the Subsonic server, then matched information from Deezer, and combines them, discarding overlap. For example, if the user searched "Taylor Swift," Navimix would search the local Subsonic server for "Taylor Swift," then search Deezer for "Taylor Swift," and remove any results found in both searches. Then, it would format the results into a JSON file in the Subsonic-specified format, and return it to the client.

When the user attempts to perform an action on a track in the Subsonic library, Navimix simply proxies traffic; the client makes a request to the given API endpoint with the given ID. Since the ID will be alphanumeric, Navimix determines that the track is already in the library, and should just be proxied to the Subsonic server. If the track is not in the library (ID only contains integers), Navimix uses a local Deemix server to download the track from Deezer to the server, then serve it to the client. Navimix also keeps tracks downloaded so that future requests are quicker.

Since most API requests are downloading metadata about albums, artists and tracks from Deezer, Navimix implements SQLite to cache any API requests made. Since metadata is static content, no API request has to be made more than once. Navimix is still a work in progress. So far, upward of 8 API endpoints have been implemented, but there are a total of ~30 endpoints.

What I built and challenges

• Built a system using SQLite for efficient local storage
• Implemented high-throughput data ingestion (rate-limited crawling / API handling)
• Designed query logic for fast retrieval of songs, artists, and albums

This was my first project working with both GO, and this depth of backend code.

One of the main challenges was balancing high request throughput with external API limits. I implemented rate control and batching strategies to maximize data collection speed without exceeding limits. Rather than have my program constantly probe the Deezer API for information that has likely already been requested, navimix maintains a local database. This both made requests a lot quicker, and prevented me from running into limits with the API.

Future plans

While I am pretty happy with how this project turned out, there are some features I still want to add:

• Semi-custom recomendation engine for songs by injecting a playlist
• Buffering of media to increase responsiveness (ie stream partial files)