PHP AI Inference: Ship RAG Without a Python Service

A client asks for semantic search over a catalogue of forty thousand products. The answer arrives before the question finishes: stand up a Python service, load a sentence-transformer, put an HTTP wrapper around it, call it from PHP.

Code: FerryAI, MIT licensed.

That plan works. It also adds a second runtime, a second deploy pipeline, a process manager, a health check, and a network hop on every query. For a feature that returns twelve products, the infrastructure is bigger than the feature.

PHP AI inference removes the sidecar. FerryAI loads ONNX Runtime and llama.cpp through FFI, in the same process that serves the page, and the model call becomes a function call.

What the sidecar actually costs

The bill is not the model. It is everything around it.

Two runtimes mean two sets of dependencies to patch, and a Python environment that has to match the production image exactly. A failed pip install during a deploy is an outage with an unusual error message. Debugging spans two processes, so a slow response needs timing from both sides before anyone can say where the time went.

Memory is the quiet cost. A sidecar serving eight PHP workers either scales its own worker pool or serialises requests behind one model instance. Either way you pay for the model twice, once in Python and once in the page that called it.

An HTTP hop per inference is the third cost. On a local network it is a millisecond, which sounds small until a page makes twenty embedding calls to rank results.

What runs inside the process

Embeddings come from ONNX Runtime. Any .onnx model works, and a sentence-transformer such as all-MiniLM-L6-v2 returns 384-dimension vectors for semantic search. GPU providers are detected when present, and the call falls back to CPU when a CUDA install is incomplete. Silent fallback is the right default for a web request, because a slow answer beats a 500.

Generation comes from llama.cpp through a wrapper library, with the same FFI bridge. Chat, token-by-token streaming, and grammar-constrained output all run locally on CPU or CUDA. The grammar part matters more than it sounds: passing a JSON schema constrains the sampler so the model cannot return invalid JSON, which removes a class of parsing bugs that a regex and a retry loop never fully fix.

Vectors live in SQLite or PostgreSQL with pgvector. SQLite uses sqlite-vec for a native nearest-neighbour index where available and brute force otherwise, which is fine below a few hundred thousand rows. PostgreSQL with pgvector handles HNSW indexes and concurrent writers, and that is the option for a production catalogue.

Nothing above needs Python at runtime.

Where PHP AI inference usually breaks

Native libraries are the first wall. A shared object compiled for the wrong architecture, a missing CUDA DLL, a tokenizer that expects a different format than the one in the model folder. The library ships a diagnostic command, ferry-ai check, that reports PHP version, extensions, available backends and the cache state before the first request fails in production.

Model download is the second. Fetching a 90 MB model on boot is not acceptable, so models are pulled once from HuggingFace, verified by SHA-256 and kept in a size-limited cache. When a download fails mid-way, the retry is the default rather than the exception.

Memory is the third. Loading weights per request will exhaust any pool. A bounded LRU pool keeps instances alive, and workers can share weights through shared memory when the extension is available. That is the difference between a warm page and a memory error at peak traffic.

When the sidecar is still the right answer

Fine-tuning at scale, heavy GPU batch jobs, or a team that already runs Python services with real observability. If the model needs a distributed training pipeline, PHP is the wrong place to put it.

The boundary we use is simple: if the work fits inside a request and the model is already trained, run it in the process that serves the request. If the work takes minutes or needs a GPU queue, keep it out.

What we ship for clients

For RAG features, that means the retrieval layer runs in PHP next to the content, and the vector store is a table in the database the site already has. Backups cover it, the staging copy behaves the same, and a client’s host does not need a new service.

FerryAI is MIT licensed, requires PHP 8.3 with the FFI extension, and reports 793 passing tests with PHPStan at level 8. It is an early release, so pin a version and read the changelog on upgrade.

More on how we build the retrieval side: RAG development. For the stack underneath it: PHP development. The smallest version of the same architecture, one JSON tree behind an MCP endpoint, is described in a self-hosted link in bio page.

Further reading: ONNX Runtime and llama.cpp.