BaseRT

BaseRT is a high-performance Large Language Model (LLM) runtime specifically engineered for Apple Silicon. Its primary function is to enable users to run AI models locally on their own devices, offering a solution that prioritizes speed, privacy, and cost-effectiveness by eliminating per-token fees. The product is targeted at individuals and developers seeking to leverage the power of AI without relying on cloud-based services. The proliferation of AI models has created a demand for efficient and accessible ways to run them. However, traditional cloud-based inference can be expensive due to per-token costs and raises concerns about data privacy. For users with powerful local hardware, particularly Apple's M-series chips, there's a significant opportunity to run these models directly on their devices. BaseRT aims to bridge the gap between the capabilities of Apple Silicon and the software needed to fully exploit it for local LLM inference. One of the core features of BaseRT is its exceptional speed. It is reported to be up to 6.4 times faster than llama.cpp and 3.9 times faster than MLX in prompt processing (prefill). This performance is achieved by optimizing for Apple's M-series chips, including the new tensor core architecture exposed through the Metal 4 tensor API. This optimization allows BaseRT to maximize the computational power of the hardware for faster LLM operations. BaseRT simplifies the process of running local AI models. It can be installed with a single command, making it accessible even to users who are not deeply familiar with complex software setups. This ease of installation is crucial for encouraging wider adoption of on-device AI inference. The product is designed to run local models directly on the user's device, ensuring that data remains private and under the user's control. The product supports a range of LLM models, with benchmarks conducted on configurations using Qwen, Llama 3, and Gemma models, varying in parameter size from 0.6B to 35B. This broad compatibility ensures that users can experiment with various models to find the best fit for their needs. The ability to run these models locally means users can process AI tasks without incurring ongoing per-token costs, making it a more economical solution for frequent or intensive use. BaseRT's approach focuses on maximizing the performance of Apple Silicon. By tuning the runtime to utilize the specific hardware capabilities, such as the tensor cores and Metal API, it achieves a new performance ceiling for LLMs on these devices. This specialized optimization is key to its speed advantage over more general-purpose runtimes. The benefits for users include significantly faster AI model performance, enhanced data privacy due to local processing, and cost savings from the absence of per-token charges. This combination makes on-device AI inference more practical and affordable, especially for applications requiring continuous or background AI processing. Potential use cases for BaseRT include developing always-on background AI agents, enabling real-time voice processing pipelines on-device where low latency is critical, and facilitating small-business AI solutions that require both privacy and cost-effectiveness. Users can also test and switch between different models for varying latency or quality requirements. BaseRT is an open-source project, indicating a commitment to community development and transparency. It is designed for Apple Silicon hardware and is available for installation via a simple command. While specific pricing tiers are not detailed, its open-source nature and focus on local execution suggest a cost-effective solution. The product is categorized under AI Infrastructure Tools and LLM Developer Tools. In summary, BaseRT provides a powerful and efficient solution for running LLMs locally on Apple Silicon, delivering unparalleled speed, privacy, and cost benefits for developers and AI enthusiasts.