| This is the first of a three-part series by Markus Eisele. Stay tuned for the follow-up posts. |
AI is everywhere right now. Every conference, keynote, and internal meeting has someone showing a prototype powered by a large language model. It looks impressive. You ask a question, and the system answers in natural language. But if you are an enterprise Java developer, you probably have mixed feelings. You know how hard it is to build reliable systems that scale, comply with regulations, and run for years. You also know that what looks good in a demo often falls apart in production. That’s the dilemma we face. How do we make sense of AI and apply it to our world without giving up the qualities that made Java the standard for enterprise software?
The History of Java in the Enterprise
Java became the backbone of enterprise systems for a reason. It gave us strong typing, memory safety, portability across operating systems, and an ecosystem of frameworks that codified best practices. Whether you used Jakarta EE, Spring, or later, Quarkus and Micronaut, the goal was the same: build systems that are stable, predictable, and maintainable. Enterprises invested heavily because they knew Java applications would still be running years later with minimal surprises.
This history matters when we talk about AI. Java developers are used to deterministic behavior. If a method returns a result, you can rely on that result as long as your inputs are the same. Business processes depend on that predictability. AI does not work like that. Outputs are probabilistic. The same input might give different results. That alone challenges everything we know about enterprise software.
The Prototype Versus Production Gap
Most AI work today starts with prototypes. A team connects to an API, wires up a chat interface, and demonstrates a result. Prototypes are good for exploration. They aren’t good for production. Once you try to run them at scale you discover problems.
Latency is one issue. A call to a remote model may take several seconds. That’s not acceptable in systems where a two-second delay feels like forever. Cost is another issue. Calling hosted models is not free, and repeated calls across thousands of users quickly adds up. Security and compliance are even bigger concerns. Enterprises need to know where data goes, how it’s stored, and whether it leaks into a shared model. A quick demo rarely answers those questions.
The result is that many prototypes never make it into production. The gap between a demo and a production system is large, and most teams underestimate the effort required to close it.
Why This Matters for Java Developers
Java developers are often the ones who receive these prototypes and are asked to “make them real.” That means dealing with all the issues left unsolved. How do you handle unpredictable outputs? How do you log and monitor AI behavior? How do you validate responses before they reach downstream systems? These are not trivial questions.
At the same time, business stakeholders expect results. They see the promise of AI and want it integrated into existing platforms. The pressure to deliver is strong. The dilemma is that we cannot ignore AI, but we also cannot adopt it naively. Our responsibility is to bridge the gap between experimentation and production.
Where the Risks Show Up
Let’s make this concrete. Imagine an AI-powered customer support tool. The prototype connects a chat interface to a hosted LLM. It works in a demo with simple questions. Now imagine it deployed in production. A customer asks about account balances. The model hallucinates and invents a number. The system has just broken compliance rules. Or imagine a user submits malicious input and the model responds with something harmful. Suddenly you’re facing a security incident. These are real risks that go beyond “the model sometimes gets it wrong.”
For Java developers, this is the dilemma. We need to preserve the qualities we know matter: correctness, security, and maintainability. But we also need to embrace a new class of technologies that behave very differently from what we’re used to.
The Role of Java Standards and Frameworks
The good news is that the Java ecosystem is already moving to help. Standards and frameworks are emerging that make AI integration less of a wild west. The OpenAI API turns into a standard, providing a way to access models in a standard form, regardless of vendor. That means code you write today won’t be locked in to a single provider. The Model Context Protocol (MCP) is another step, defining how tools and models can interact in a consistent way.
Frameworks are also evolving. Quarkus has extensions for LangChain4j, making it possible to define AI services as easily as you define REST endpoints. Spring has introduced Spring AI. These projects bring the discipline of dependency injection, configuration management, and testing into the AI space. In other words, they give Java developers familiar tools for unfamiliar problems.
The Standards Versus Speed Dilemma
A common argument against Java and enterprise standards is that they move too slowly. The AI world changes every month, with new models and APIs appearing at a pace that no standards body can match. At first glance, it looks like standards are a barrier to progress. The reality is different. In enterprise software, standards are not the anchors holding us back. They’re the foundation that makes long-term progress possible.
Standards define a shared vocabulary. They ensure that knowledge is transferable across projects and teams. If you hire a developer who knows JDBC, you can expect them to work with any database supported by the driver ecosystem. If you rely on Jakarta REST, you can swap frameworks or vendors without rewriting every service. This is not slow. This is what allows enterprises to move fast without constantly breaking things.
AI will be no different. Proprietary APIs and vendor-specific SDKs can get you started quickly, but they come with hidden costs. You risk locking yourself in to one provider, or building a system that only a small set of specialists understands. If those people leave, or if the vendor changes terms, you’re stuck. Standards avoid that trap. They make sure that today’s investment remains useful years from now.
Another advantage is the support horizon. Enterprises don’t think in terms of weeks or hackathon demos. They think in years. Standards bodies and established frameworks commit to supporting APIs and specifications over the long term. That stability is critical for applications that process financial transactions, manage healthcare data, or run supply chains. Without standards, every system becomes a one-off, fragile and dependent on whoever built it.
Java has shown this again and again. Servlets, CDI, JMS, JPA: These standards secured decades of business-critical development. They allowed millions of developers to build applications without reinventing core infrastructure. They also made it possible for vendors and open source projects to compete on quality, not just lock-in. The same will be true for AI. Emerging efforts like LangChain4j and the Java SDK for the Model Context Protocol or the Agent2Agent Protocol SDK will not slow us down. They’ll enable enterprises to adopt AI at scale, safely and sustainably.
In the end, speed without standards leads to short-lived prototypes. Standards with speed lead to systems that survive and evolve. Java developers should not see standards as a constraint. They should see them as the mechanism that allows us to bring AI into production, where it actually matters.
Performance and Numerics: Java’s Catching Up
One more part of the dilemma is performance. Python became the default language for AI not because of its syntax, but because of its libraries. NumPy, SciPy, PyTorch, and TensorFlow all rely on highly optimized C and C++ code. Python is mostly a frontend wrapper around these math kernels. Java, by contrast, has never had numerics libraries of the same adoption or depth. JNI made calling native code possible, but it was awkward and unsafe.
That is changing. The Foreign Function & Memory (FFM) API (JEP 454) makes it possible to call native libraries directly from Java without the boilerplate of JNI. It’s safer, faster, and easier to use. This opens the door for Java applications to integrate with the same optimized math libraries that power Python. Alongside FFM, the Vector API (JEP 508) introduces explicit support for SIMD operations on modern CPUs. It allows developers to write vectorized algorithms in Java that run efficiently across hardware platforms. Together, these features bring Java much closer to the performance profile needed for AI and machine learning workloads.
For enterprise architects, this matters because it changes the role of Java in AI systems. Java isn’t the only orchestration layer that calls external services. With projects like Jlama, models can run inside the JVM. With FFM and the Vector API, Java can take advantage of native math libraries and hardware acceleration. That means AI inference can move closer to where the data lives, whether in the data center or at the edge, while still benefiting from the standards and discipline of the Java ecosystem.
The Testing Dimension
Another part of the dilemma is testing. Enterprise systems are only trusted when they’re tested. Java has a long tradition of unit testing and integration testing, supported by standards and frameworks that every developer knows: JUnit, TestNG, Testcontainers, Jakarta EE testing harnesses, and more recently, Quarkus Dev Services for spinning up dependencies in integration tests. These practices are a core reason Java applications are considered production-grade. Hamel Husain’s work on evaluation frameworks is directly relevant here. He describes three levels of evaluation: unit tests, model/human evaluation, and production-facing A/B tests. For Java developers treating models as black boxes, the first two levels map neatly onto our existing practice: unit tests for deterministic components and black-box evaluations with curated prompts for system behavior.
AI-infused applications bring new challenges. How do you write a unit test for a model that gives slightly different answers each time? How do you validate that an AI component works correctly when the definition of “correct” is fuzzy? The answer is not to give up testing but to extend it.
At the unit level, you still test deterministic components around the AI service: context builders, data retrieval pipelines, validation, and guardrail logic. These remain classic unit test targets. For the AI service itself, you can use schema validation tests, golden datasets, and bounded assertions. For example, you may assert that the model returns valid JSON, contains required fields, or produces a result within an acceptable range. The exact words may differ, but the structure and boundaries must hold.
At the integration level, you can bring AI into the picture. Dev Services can spin up a local Ollama container or mock inference API for repeatable test runs. Testcontainers can manage vector databases like PostgreSQL with pgvector or Elasticsearch. Property-based testing libraries such as jqwik can generate varied inputs to expose edge cases in AI pipelines. These tools are already familiar to Java developers; they simply need to be applied to new targets.
The key insight is that AI testing must complement, not replace, the testing discipline we already have. Enterprises cannot put untested AI into production and hope for the best. By extending unit and integration testing practices to AI-infused components, we give stakeholders the confidence that these systems behave within defined boundaries. Even when individual model outputs are probabilistic.
This is where Java’s culture of testing becomes an advantage. Teams already expect comprehensive test coverage before deploying. Extending that mindset to AI ensures that these applications meet enterprise standards, not just demo requirements. Over time, testing patterns for AI outputs will mature into the same kind of de facto standards that JUnit brought to unit tests and Arquillian brought to integration tests. We should expect evaluation frameworks for AI-infused applications to become as normal as JUnit in the enterprise stack.
A Path Forward
So what should we do? The first step is to acknowledge that AI is not going away. Enterprises will demand it, and customers will expect it. The second step is to be realistic. Not every prototype deserves to become a product. We need to evaluate use cases carefully, ask whether AI adds real value, and design with risks in mind.
From there, the path forward looks familiar. Use standards to avoid lock-in. Use frameworks to manage complexity. Apply the same discipline you already use for transactions, messaging, and observability. The difference is that now you also need to handle probabilistic behavior. That means adding validation layers, monitoring AI outputs, and designing systems that fail gracefully when the model is wrong.
The Java developer’s dilemma is not about choosing whether to use AI. It’s about how to use it responsibly. We cannot treat AI like a library we drop into an application and forget about. We need to integrate it with the same care we apply to any critical system. The Java ecosystem is giving us the tools to do that. Our challenge is to learn quickly, apply those tools, and keep the qualities that made Java the enterprise standard in the first place.
This is the beginning of a larger conversation. In the next article we will look at new types of applications that emerge when AI is treated as a core part of the architecture, not just an add-on. That’s where the real transformation happens.