// AUDIO AI & MACHINE LEARNING
Audio AI & Machine Learning
Development
Audio AI development puts machine learning inside a real-time audio product, from sound event detection to generative audio. We train in PyTorch or TensorFlow, then run inference in C++ with ONNX Runtime inside your JUCE plugin, app, or device. We measure latency and memory on your hardware.
// WHAT WE BUILD
Audio AI Capabilities
Sound Event Detection
Real-time classification and detection of audio events using neural networks, from onset detection to full scene analysis. Mel spectrogram front ends, compact CNN and recurrent architectures, latency measured in buffers.
Generative Audio Systems
Neural synthesis, sample generation, and hybrid DSP/ML architectures for creative tools. Models that produce coherent, musically useful output.
On-Device ML Deployment
Model quantization, pruning, and ONNX/C++ inference integration for embedded hardware and resource-constrained platforms. We've shipped real-time inference on devices with 500MB RAM.
Audio Analysis & Intelligence
Pitch detection, timbre analysis, tempo tracking, and content-aware processing. Features that let a tool react to what it is actually hearing.
C++ & JUCE Integration
We bridge Python model development and C++ production deployment with ONNX Runtime or hand-written inference, so ML features run inside your JUCE plugin or application within the real-time budget.
Dataset Curation & Training
End-to-end model development from raw data to production-ready weights. Dataset cleaning, augmentation, architecture design, training, and evaluation on your specific audio domain.
// THE HARD PART
Why is audio AI different from other machine learning?
The real-time constraint
Audio AI is different because inference has to finish inside a buffer of 64 to 512 samples. A model that takes 50ms to infer is useless in a plugin. Getting ML to work within this constraint requires specialized architecture choices, quantization, and sometimes custom C++ inference code.
Your dataset sets the ceiling
Architecture matters less than the data behind it. Labels recorded in one room, at one sample rate, through one microphone produce a model that works in that room. We audit what you already have, define the labelling scheme, and say early if the dataset cannot support the feature you intend to sell.
Why we're equipped for it
We combine signal processing depth with ML engineering. We know where a neural model earns its cost and where a classical DSP approach is faster and more predictable, and we write that judgment down before the budget is committed. Our neural vocal restoration case study shows what a bounded feasibility sprint produces when the question is whether a route exists at all.
// ENGAGEMENT
From Feasibility to Shipped Inference
Each stage ends in a document, a measured number, or a build you can run.
Feasibility note
A written note covering what the model has to predict, the accuracy that would make the feature worth shipping, the latency and memory budget on your target hardware, and whether DSP would do the job cheaper.
Dataset and baseline
A labelled dataset with documented splits, plus a baseline model and its scores, so every later improvement is measured against a number you have already seen.
Model and evaluation
Trained weights in PyTorch or TensorFlow and an evaluation report with precision, recall, and F1 per class on held-out data, including the failure cases we did not solve.
Real-time integration
A plugin or application build running the model on your machine or device, with inference time per buffer and CPU usage measured under load.
Deployment and handover
The quantized model, the C++ inference code inside your product, the training scripts that reproduce the weights, and the measurement report, all in your repository.
// WHAT YOU RECEIVE
What do you deliver on an audio ML project?
Models and inference code
You receive trained weights with an ONNX export where that fits the target, C++ inference wired into your JUCE plugin or application, and the Python pipeline that produced them. PyTorch and TensorFlow for training; ONNX Runtime or hand-written inference in production.
Evaluation you can audit
Per-class precision, recall, and F1 on a held-out split, the confusions that matter to your users, and inference time per buffer on your hardware. Our embedded sound event detection work ran in real time on a device with 500MB of RAM while retaining a 95% F1 detection score.
Dataset tooling
Labelling conventions, augmentation scripts, and the splits themselves, so retraining on next year's recordings is a task your team can run without reconstructing how the model was built.
// WHAT GOES WRONG
Pitfalls We Design Around
Inference on the Audio Thread
A model that allocates, locks, or takes a variable time per call cannot live in processBlock. Inference runs on a worker thread behind a lock-free queue, or with a fixed cost small enough to measure.
Lookahead and Reported Latency
A model that needs 1024 samples of context adds latency the host has to be told about. Report it wrong and every track in a Pro Tools or Cubase session drifts against the plugin.
Model Output Breaking the DSP
Predictions usually drive a gain, a filter, or a parameter. One NaN or a denormal value crossing that boundary silences the mix, so outputs are clamped and checked where inference meets audio.
Evaluation That Flatters the Model
A random split leaks the same recording into training and test, and the score looks excellent until release. Splitting by session, performer, or device produces the number you can actually promise.
Memory on the Target Device
A float model of a few hundred megabytes does not fit an embedded board or an AUv3 extension on iPadOS. Quantization and pruning are design decisions taken early, and they change the accuracy you can claim.
Weights You Cannot Ship
Pretrained models and public datasets carry licences, and several of the popular ones forbid commercial use. We check provenance before anything goes into a product you intend to sell.
// SHIPPED WORK
Audio ML in Production
CLIENT: CONFIDENTIAL
Embedded Neural Network
Model quantization, dataset curation, and a neural network architecture designed for the client's embedded hardware. It runs in real time on a device with 500MB of RAM while retaining a 95% F1 detection score. The client, the product, and the domain stay undisclosed by contract, which is normal for this kind of work.
CLIENT: AUDIOMODERN
Playbeat 4
End-to-end development of the full application and the sequence randomization engine in JUCE and C++. Released 5 March 2025, Playbeat 4 offers AI genre-based pattern and sound randomization across eight independent sequencers and ships on Windows, macOS, and iPadOS.
// FIT
When is machine learning the wrong answer?
Many features sold as AI are better served by signal processing. A transient detector, a pitch tracker, or a loudness matcher built from DSP is smaller, faster, easier to support, and predictable in a way a neural network is not. When that is the honest answer, it goes in the feasibility note and we quote the DSP work instead. We also decline to promise an accuracy figure before seeing your data.
// FAQ
Common Questions
Can ML models run in real-time inside an audio plugin?
Yes, with the right architecture. We use model quantization, pruning, and efficient C++ inference to deploy lightweight neural networks that meet real-time audio latency requirements. Our embedded neural network work has achieved real-time inference on devices with 500MB RAM.
What frameworks do you use for audio machine learning?
We work with PyTorch and TensorFlow for model development and training, and deploy using ONNX Runtime or custom C++ inference for integration into JUCE-based plugins and standalone applications.
Can you train a model on our proprietary audio dataset?
Yes. We handle dataset curation, preprocessing, training, evaluation, and deployment. We work under NDA and all data remains confidential.
What kinds of audio AI projects have you shipped?
Our portfolio includes a Sound Event Detection (SED) system deployed on embedded hardware, achieving 95% F1 score at real-time on a resource-constrained device. We also work on generative audio systems and intelligent audio analysis tools.
Do you work with generative audio models?
Yes. We build generative audio systems including sample generation, procedural audio engines, and hybrid DSP/ML approaches that combine traditional signal processing with neural models for creative tools.
What if we have no dataset yet?
Then the first milestone is data. We define the labelling scheme, help you record or source material, and build a baseline on a small set before anyone commits to a full training run.
Does the model need an internet connection?
No. Inference runs locally in C++ on the user's machine or device, so there is no per-user API cost, no network latency in the audio path, and no customer audio leaving their computer.
// RELATED SERVICES