A Model Converting a Model: Core AI with GPT 5.6 Sol
One of the coolest things I did with GPT 5.6 Sol was ask one model to convert another.
I was still figuring out how far I could push Sol when I trusted it with several gigabytes of unfamiliar speech-model weights and the brand-new framework from Apple, CoreAI.
The project started in a chat called “I just want to vent out.” Nothing about that title suggests compiler work. I was talking about validation, shipping too many projects, and wanting to build something people would remember.
Then I remembered Ichi, my old conversational AI app, and asked whether Apple’s new Core AI framework could replace its speech pipeline.
I wanted a text-to-speech model that is something expressive enough to make people stop and listen, running through Core AI.
Sol searched Apple’s model repositories, recent pull requests, community conversions, and the speech projects already on my Mac. It brought back Whisper and Parakeet for speech recognition, but the text-to-speech side was thin. Apple’s public Core AI examples did not include a ready TTS model back in June.
Then Sol pulled official Chatterbox Turbo samples and put them in a folder for me to hear.
My honest reaction to it:
“holy shit this is just so good”
Chatterbox did not sound like a polite system voice. It laughed, breathed, paused, and performed the sentence. An MLX implementation already existed, including native Swift work, but I wanted to know whether the same model could run through Core AI.
So I sent the line that changed the initial chat’s direction from validation to building:
“Can you do a project with CoreAI? not mlx”
What “Convert It” Actually Meant
Chatterbox Turbo is not one model file that accepts text and returns audio. Its inference path has distinct learned stages, plus tokenization, sampling, cache management, and audio handling around them.
The Core AI version ended up with four production assets:
- T3 embeddings turn text and generated speech tokens into embeddings.
- The T3 transformer generates speech tokens autoregressively and keeps a KV cache between steps.
- S3Gen turns those tokens into 512 mel-spectrogram frames.
- The HiFT vocoder turns the mel representation into 245,760 waveform samples at 24 kHz.
All learned neural inference runs through Core AI. Swift still owns the work that should stay ordinary code: tokenization, sampling, KV-cache updates, WAV writing, playback, and the interface. PyTorch appears only in the offline conversion tools.
That distinction mattered because I did not want a demo where Core AI loaded one tiny submodel while Python or MLX quietly did the difficult work. The app had to synthesize a complete clip with no server, no Python process, and no downloaded runtime model.
The Converter Said No
Apple’s coreai-torch package accepts a PyTorch exported program and lowers it into a .aimodel. The small examples look almost suspiciously easy:
converter = coreai_torch.TorchConverter() converter.add_exported_program(exported_program) model = converter.to_coreai() model.optimize() model.save_asset(“model.aimodel”)
Chatterbox was not a small example.
Its vocoder used seven operations the converter did not accept: FFT, ELU, internal random generation, scalar remainder, nearest-neighbor upsampling, tensor unfolding, and reflection padding. Sol could not fix that with a flag. It had to preserve the learned network while expressing those operations through primitives Core AI understood.
FFT became fixed Hann-window DFT filters built from convolution and transposed convolution. Random phase and noise moved outside the graph and became explicit inputs. ELU, modulo, upsampling, and reflection padding received equivalent primitive implementations. Weight-normalization parameters were folded into their learned weights.
Then Xcode 27 found a different problem. A Conv1d layout that converted successfully failed during MPS specialization, so the exporter moved those layers into an equivalent trailing-axis Conv2d layout.
This was the real work.
Sol had to read the source model, inspect tensor shapes, change the exported graph, run PyTorch parity tests, load the result through Core AI, listen to the generated audio, and return to the graph whenever one of those checks disagreed.
Conversion is a compiler problem wearing an ML hat.
Sol Took Wrong Turns
The first answers were not perfect. Sol initially told me the native Swift Chatterbox port was missing, then inspected the current tree and corrected itself. It recommended Core ML during one part of the discussion even though I had asked for Core AI. I pushed back.
The first app scaffold could import a .aimodel, but it could not synthesize speech because the converted stages did not exist yet. The iOS Simulator could not import Core AI at all. We switched to macOS 27 with Xcode 27 and kept going.
Later, a generated clip cut off before the final words. The first S3Gen graph only allowed 128 speech tokens, which produced exactly 5.12 seconds of audio.
Sol rebuilt it for 256 tokens, traced the model until it emitted its real stop token, and used Whisper to confirm that the final “Core AI” survived.
Even the pull request caught real mistakes. Bugbot found that two vocoder commands used 100 or 256 mel frames despite the exported graph expecting 512, and the Python tokenizer could exceed the T3 prefill limit. Sol fixed the contracts, added focused tests, ran 13 Python tests and six Xcode tests, then waited until every review thread closed.
This is the part of Sol I trust most. It can be wrong halfway through a long run and still recover before I return.
Making It Small Enough
The first T3 transformer asset used FP16 weights and occupied roughly 591 MB. Sol exported an INT4 block-16 version that brought the transformer down to about 222 MB without changing the rest of the pipeline.
I asked it to record both versions and “see the frequency” because I wanted proof beyond a successful function call. It generated the same prompt through FP16 and INT4, compared their spectrograms, checked for clipping, and transcribed both outputs with Whisper. The speech structure remained intact and the transcription matched.
The four production assets plus tokenizer now occupy about 600 MiB. That is still large for a voice feature, but it is a real local pipeline rather than a placeholder graph.
On my M5 Mac, the warm Release app generated 5.68 seconds of speech in 4.85 seconds, an RTF of 0.85. S3Gen took around 0.29 seconds and the vocoder around 0.37 seconds. Most of the remaining time sat inside the autoregressive T3 decoder.
That result immediately caused another argument. MLX could process only the active KV-cache prefix, while our first Core AI transformer attended over a fixed 768-token cache on every generated token. Sol measured the cost and explained why moving the same fixed cache into Core AI state would not solve it; the graph itself needs active-prefix attention or cache buckets.
I asked how much work.
Its answer was three to five focused days for a serious Mac optimization, and one to two weeks for something I would trust with long text and regression coverage. I stopped asking for a magic flag.
What Runs Today
The current Chatterbox path in Core AI Framework Lab is honest about its borders.
I just got a reset while writing this post, so you know the repository will be update by this weekend with the production implementation.
It runs on macOS 27, bundles one fixed voice, and supports up to 253 generated speech tokens. Longer passages still need sentence chunking and waveform stitching. The current assets favor the GPU; they are not the separately authored FP16, static-shape, Neural Engine-friendly graphs an iPhone version would need.
Asking Core AI to prefer the Neural Engine did not make the model compatible with it. Xcode’s validator rejected FP32 intermediates and part of the transposed-convolution layout. That failure saved me from writing “runs on the Neural Engine” because a preference enum returned without crashing.
Core AI is also still beta software. Converter support and specialization behavior can change between Xcode seeds.
None of that reduces what worked: Chatterbox Turbo’s embeddings, transformer, speech generator, and vocoder run inside a native Swift app through Core AI.
I can type a sentence, press Generate, and hear the result without Python, MLX, Core ML, or a network request.
The Model Was Sol
I wrote A Sol’s Work about the walk-away test: give an agent a real lane, leave, and see whether its report still matches reality when you return.
https://x.com/rudrank/status/2075300423377019295
The Chatterbox conversion was one of my earliest Sol lanes, and I ran it with the highest reasoning.
Sol searched for candidates, changed direction when I rejected Kokoro, wrote the conversion harness, rewrote unsupported operators, built the macOS app, generated audio, profiled it, handled review comments, and got the pull request merged.
I changed direction while I was learning about Core AI myself during the chat but the corrections did not collapse the session. Sol retained the technical thread and kept narrowing the problem until the app spoke.
That is a better description of agentic engineering than “one-shot.” I supplied taste, impatience, and a refusal to accept a fake boundary. Sol supplied the magic to inspect every layer between a PyTorch checkpoint and sound coming out of a Mac.
One model helped me convert another, and both of them surprised me.