Practical LLM Inference in Modern Java

Explore practical approaches to implementing local Large Language Model (LLM) inference using modern Java in this 51-minute Devoxx conference talk. Learn how to leverage the latest Java features to implement local inference for various open-source LLMs, starting with Llama 2 and 3 (Meta). Discover how to extend this approach to run other popular open-source models on standard CPUs without specialized hardware. Delve into key topics such as implementing efficient LLM inference engines for local execution, utilizing Java 21+ features for optimized CPU-based performance, creating a flexible framework adaptable to multiple LLM architectures, and maximizing standard CPU utilization without GPU dependencies. Gain insights on integrating with LangChain4j for streamlined local inference execution and optimizing performance using the Java Vector API for accelerated matrix operations. Learn how to leverage GraalVM to reduce latency and memory consumption. Acquire the knowledge to implement and optimize local LLM inference for open-source models in Java projects, enabling the creation of fast and efficient AI applications using cutting-edge Java technologies.

Practical LLM Inference in Modern Java by Alfonso² Peterssen, Alina Yurenko