SDK empowers on-premises LLM development for edge AI innovations

Advantech has announced a new software offering – GenAI Studio for the Advantech Edge AI SDK, addressing the increasing demand for cost-effective, on-premises large language model (LLM) solutions. This initiative aims to make LLM adoption more accessible for developers and SMBs by greatly decreasing the reliance on GPUs, lowering costs and facilitating broader utilisation of generative AI technologies.

As one of the software offerings, GenAI Studio satisfies industry pain points, such as lowering factory operator wait times for critical information and easing the documentation workload of healthcare professionals. Its no-code, cost-effective platform streamlines LLM adoption, helping businesses to deploy AI solutions quickly and efficiently, enhancing productivity and operational efficiency. Built for the Advantech Edge AI SDK, GenAI Studio employs a versatile LLM platform with excellent integration capabilities for local and cloud-based LLMs, including OpenAI, Gemini, Anthropic, and Ollama. It also presents full-parameter fine-tuning functionality, optimised for environments with limited GPU resources, enabling broader accessibility and performance enhancements.

The GenAI Studio offers a comprehensive suite of tools for LLM development:

• Integrated Fine-Tuning and Inference Services: GenAI Studio combines fine-tuning and inference capabilities to maximise hardware utilisation, allowing more flexible and efficient resource allocation.
• Advanced GPU Resource Management and Task Scheduling: These features allow users to optimise AI hardware performance, improving the cost-effectiveness of high-value equipment.

The rapid rise of AI has underscored the necessity for accessible LLM solutions, but limited resources hinder many companies. The SDK addresses this challenge by delivering a toolset that allows efficient evaluation, development, and deployment of edge AI applications. For instance, fine-tuning a 70-billion-parameter LLM, which traditionally needs over 30 GPUs with 48GB memory each, can be achieved with just four GPUs using the SDK. This represents an 87% reduction in resource requirements, dramatically lowering costs and making LLM solutions more attainable.

The company's AIR-520 edge AI server supplies a robust hardware platform equipped with NVIDIA RTX GPUs and Phison AI SSDs to complement GenAI Studio. This integration delivers reliable, high-efficiency computing capabilities designed to satisfy the demands of AI applications across industries such as manufacturing, healthcare, and retail.

The SDK is a fully integrated platform developed for seamless edge AI development. With pre-configured hardware and optimally tuned software, it provides a plug-and-play experience, enabling cost-effective LLM customisation, seamless toolkit compatibility, and effortless management of large-scale edge deployments. Designed with reliability, scalability, and user-friendliness, the SDK simplifies the path to AI innovation. It now includes three core components:

• GenAI Studio: Facilitates cost-effective creation, evaluation, and integration of custom LLMs on-premises.
• Inference Kit: Enables rapid optimisation and assessment of efficient AI runtimes compatible with embedded operating systems.
• Orchestration Platform: This platform efficiently manages AI model and application updates across large-scale edge deployments, incorporating MLOps for streamlined operations.