A System of LLM Config and Code Optimization

The era of simple LLM optimization — using inference engines, caching, quantization schemes, etc. testing multiple model API Providers — is over. Those sets of optimizations are now the baseline expectation. The new strategic challenge is about architecting resilient cost-quality-latency-throughput aware hybrid systems that manage a volatile and chaotic portfolio of model provider APIs and local models.

The ground has shifted on both API and self-hosted fronts. We operate in a world of opaque provider APIs where unannounced model updates (e.g., -0125 vs. -1106) can silently degrade performance or explode costs. Winning in this environment means moving beyond one-off tactics and embracing system-level optimization thinking. We may then evolve from slow ad-hoc analysis followed by "one-off" tweaks to architecting and supporting a continuous optimization platform integrated with other APIs, Agents, IDEs, and analysis systems.

The Current Optimization Landscape

Let's think about the way humans work and make decisions to solve optimization problems for LLM systems (not only the literal implementation). The current way to solve this boils down to relatively manual data analysis efforts (slow), expert knowledge, lots of WAGs ("wild a** guesses"), and "gut feels", with a sprinkling of "vibe feels" and YOLO GPT copy-pasta.

Data (including production data) is needed to optimize these workloads. This is because there is no one size fits all approach that consistently works.

Many paradoxes exist from the high level down to the nuts and bolts. For example the data type Normal Float (e.g.NF8), which in theory is optimized for the normal distribution of model weights, seems like it should naturally be better then the classic FP8.

Yet, in reality, it is sometimes better and sometimes not, depending on the hardware, the load on the hardware, the model, the system and energy usage goals etc. So let's reflect on how that example has parallels across the rest of the choices, when something that seemingly obvious and that small has so much complexity.

Of course over time some of these things will standardize, yet it seems as though that will be many years into the future, and the chaotic nature of the variable user token input and model output may take decades to standardize at industry scale.

No longer just ask, "Which optimization approach is best?" but instead, "Which optimization substrate - meaning the holistic set of configurations and system choices - is optimal for this specific workload, at this specific point in its lifecycle?" This is not reasonable manually, but is when using a system to manage it.

The system makes reasonable to consistently find the minimum engineering effort required to reach an acceptable intersection of cost, latency, and required intelligence. This is naturally extended to AI guided, workflow, or integrated implementation of said config and code. It's changing one-off, difficult to explain and replicate, decisions into a continuous data-backed lifecycle that starts with development and extends through the entire lifecycle.

Going back in time for a moment, consider the Alexa architecture of having a local "wake word" classifier. Now, there are many ways to do this using lightweight local models. E.g. PII, categorizes intent, and summarizes the history before routing a complex issue, to a big model.

This concept works for fallbacks too, for example a user-facing generation task first tries a premium model like GPT-4o. If it times out, the request is automatically re-routed to a faster model like Claude 3 Haiku or a self-hosted Mixtral-8x7B.

The risks - financial, engineering, etc. - of misusing Provider APIs, hosting under-optimized models, and the engineering effort to understand and optimize work manually, are staggering. Of course AI models can be used naively - but no one wants their production logs thrown ad-hoc into context windows directly - nor is that a reaslitic system.

Therefore, it's important to start applying systems thinking to how we build these optimizations and provide the next rung of users with more consistent and automatic optimizations across the entire system, while the underlying hardware, software, and environment continue to evolve.

These optimization strategies, implementations, and history are difficult without a layer that decouples your application logic from the underlying sets of hardware and software concepts. This new config and code "meta" optimization layer is a critical piece of infrastructure, a platform to embrace.

Essential use of existing technology: First and foremost, we are not encouraging building yet another inference engine, chip, eval tools, fine-tuning, all-in-one, trust-us-it works etc. The focus is on evidence based + workload specific data supported + system generated + human controlled configuration change and code recommendations that make better utilization of existing (and new) prerequisites. The latest of these exploding sets of optimization options including model routing, observability, etc.

Any system here should assist teams in maintaining stability of the "choatic-input-output" plus the "cost-quality-latency-throughput" quadrilemma. This includes things like "undocumented" API/hardware provider optimizations, such as providers silently changing quantization methods during peak load. A resilient architecture assumes this will happen and defends against it proactively.

If you have an urgent need please contact us to start a pilot of our system for this. It can analyze your existing logs and show you the opportunities in under 48 hours at no cost.