LLM gateways: what they add between your app and the model API, what they cost and where they become a risk (October 2026)

What I checked, and what I left out

For each gateway I opened the pricing page, the documentation on routing, logging and production setup, and the vendor's own security or status posts. The sources are vendor pages and documentation, two Palo Alto Networks press releases and, for the LiteLLM incident, BleepingComputer and Arthur as cross-checks against the vendor's account. Anything I could not find on a primary page stayed out.

Of the 7 entries in the niche I left out Artificial Analysis, a benchmark site, and Msty, a desktop app. Dataiku's LLM Mesh is in the explorer below, but I ran no claim check on it: the vendor publishes no price and I did not review its gateway documentation in this run. That leaves LiteLLM, Portkey, OpenRouter and Orq.ai for the five claims. Each gets a verdict, Holds, Partly or Does not hold, and I corrected one entry in our own data along the way.

Sources: [1][2][3][4][5][6]

Claim 1, one API for every model: Partly

The call looks the same everywhere. What happens behind it does not. LiteLLM states that it calls 100+ LLM APIs in the OpenAI format. OpenRouter's public model list returned 466 models and its provider list 112 providers when I pulled both on October 4, 2026. Our own entry said more than 500 models from more than 80 providers, so I corrected it.

The differences show up in parameters. When a provider does not support a parameter you send, LiteLLM raises an exception by default and drops the parameter only if you set drop_params. Its own documentation adds that this drops only unsupported OpenAI parameters and passes anything else through as provider specific. OpenRouter does the opposite by default: a provider that lacks a parameter can still receive the request and ignores what it does not know, unless you set require_parameters.

So one codebase can behave differently depending on the gateway and the setting. A request with a tool option or a sampling parameter may fail loudly in one setup and run quietly without that parameter in another. The unified interface saves integration work, and it moves the compatibility question into configuration.

Sources: [7][8][9][10][11]

Claim 2, it adds almost no latency: Holds, on the vendors' own numbers

LiteLLM's benchmark page reports its own overhead at 1,170 requests per second on four instances with 4 CPUs and 8 GB of memory each. The median is 2 ms, 8 ms at the 95th percentile and 13 ms at the 99th. Total request latency in the same run was 100 ms, 150 ms and 240 ms. Portkey's open-source gateway claims under 1 ms of latency with a 122 kb footprint. Next to a model call that takes seconds, a few milliseconds is small.

Both numbers are the vendors' own. The only head-to-head I found sits on LiteLLM's documentation: LiteLLM v1.79.1-stable against Portkey v1.14.0, with a 95th percentile of 150 ms against 230 ms. The page itself notes that Portkey ran without a database and LiteLLM with one, and it names no independent operator. I counted it as LiteLLM's claim about a competitor and did not use it to rank anything.

The claim holds for the proxy hop. It says nothing about what a gateway adds when it is also your cache, your guardrail check or your logger. I found no published measurement for those.

Sources: [12][13]

Claim 3, open source means no gateway bill: Does not hold

Does not hold, and the reason is different for each product. The gateways differ most here, so I wrote the prices out as the vendors publish them.

LiteLLM's main project is MIT licensed. Single sign-on, SCIM, audit logs on every request and 24/7 support with response times belong to the Enterprise edition, which has no published price and a 30-day trial key. The production guide asks for PostgreSQL, a pod of 1 vCPU and 4 Gi of memory, and Redis as soon as you run more than one instance. The vendor publishes no server cost, so I cannot give you a number for self-hosting.

Portkey's gateway is MIT licensed as well. The hosted service starts free with 10,000 recorded logs a month and 3 days of log retention. Production costs $49 a month with 100,000 logs, $9 per additional 100,000 and 30 days of retention. Enterprise is priced on request.

OpenRouter says it adds no markup on inference pricing, and it also charges a platform fee: 5.5% on Standard and 8% on Business, taken when you buy credits. With your own provider keys the first $25,000 of list-price inference a month carries no fee and 5% applies after that. Orq.ai takes 4.5% on credits when you use its keys, and with your own keys it is free up to 1 million requests a month, then 4%. I found a mismatch on OpenRouter's free tier too: the pricing page promises 25+ free models, while the public model list holds 17 IDs ending in :free. I left both figures out.

Sources: [1][14][2][13][3][15][10][4]

Claim 4, a gateway makes you more reliable: Partly, until the gateway fails

Fallbacks work as described. OpenRouter's FAQ says that if a provider returns an error it falls back to the next provider. The routing guide shows allow_fallbacks on by default, with providers that had significant outages in the last 30 seconds ranked behind the others. Portkey's gateway lists automatic retries, fallbacks and load balancing, and LiteLLM's proxy lists load balancing.

A gateway is also one more thing that can be down. OpenRouter published two incidents in February 2026. On February 17 API requests failed for 38 minutes from 5:27 UTC, on February 19 for 35 minutes from 7:36 UTC. About 20% of requests failed at first and later 80% to 90%. The cause was an external caching dependency, and a denial-of-service attack on the 17th hid it at first. Authorization lookups that failed for infrastructure reasons came back as 401, which looks like a bad key. OpenRouter now returns 503 there.

Self-hosting makes the gateway's uptime your job. LiteLLM's production guide says that without its database, models defined only in the database become inaccessible and uncached virtual key lookups fail. Keys already in the cache keep working for 60 seconds by default.

Sources: [15][9][16][13][7][14]

Claim 5, self-hosting removes the vendor risk: Does not hold

On March 24, 2026 a maintainer's PyPI account for LiteLLM was compromised and versions 1.82.7 and 1.82.8 went out carrying a credential stealer. LiteLLM says they were published at 10:39 UTC and stayed up for about 40 minutes before PyPI quarantined them. Customers on the official Docker proxy image were not affected, because that path pins its dependencies. Anyone who installed the package without pinning a version was told to treat every credential on that system as compromised. Arthur's write-up puts the window at under five hours from about 08:00 UTC. I follow the vendor's account here and note that the two differ.

The vendor traced the cause to a compromised Trivy scanner in its build pipeline that could reach release secrets. A clean version, 1.83.0, followed on March 30 with a rebuilt release pipeline. A gateway holds a key for every provider it routes to, which is why one compromised host leaked so much at once.

It was not the only notice. LiteLLM published guardrail logging that exposed plaintext Authorization headers on March 18. It followed with a command injection on April 21 that needs an admin key, an unauthenticated SQL injection in key verification on April 29 and a Host header authentication bypass on June 1. I read those five notices, which fall into 75 days, and the vendor's security index lists two more posts in that window, a townhall update and a hardening summary. To its credit, LiteLLM publishes them and names the affected versions. The SQL injection post mentions no exploitation.

Ownership is a second kind of vendor risk. Palo Alto Networks announced on April 30, 2026 that it would acquire Portkey and completed the purchase on May 29. Portkey's site now reads Prisma AIRS AI Gateway. The press release promises continued support for existing customers and does not mention the open-source gateway.

Sources: [17][18][19][20][21][22][23][24][25][26][2]

What the gateway sees: prompts, logs and where they travel

A gateway sees every prompt, so I read what each one stores by default. OpenRouter keeps metadata such as token counts and latency and, by default, neither prompts nor responses. Logging them is an opt-in, and a second opt-in lets OpenRouter use your inputs and outputs in exchange for a 1% discount. The pricing FAQ says it does not train on your data. The routing default is different: data_collection is set to allow, which permits providers that store data non-transiently and may train on it. Setting it to deny, or turning on zdr, narrows routing to providers with matching policies.

Portkey records the raw request and response objects by default and shows them in its logs. Retention is 3 days on the free plan and 30 days on Production. The DO NOT TRACK setting reduces a log entry to tokens, cost and latency. The logging documentation I read says nothing on data residency.

For European data, Orq.ai says it is EU-built and EU-hosted. OpenRouter offers an EU endpoint, eu.openrouter.ai, where requests are decrypted and processed in the region and fail with a 404 if no provider there serves the model. That runs only on the Business and Enterprise plans. Our directory shows 1 product in the niche with an EU region. A hosted gateway that handles prompts with personal data acts as your processor under Article 28 GDPR, which asks for a contract with documented instructions and conditions for sub-processors. OpenRouter's terms refer to a data processing agreement for commercial users.

Sources: [27][9][3][28][29][30][2][4][31][6]

When a gateway earns its place

With one provider and one application, a gateway adds a hop, a dependency and a second place where keys live, and it has no job yet. With several providers, several teams or a need for budgets per key, it starts to pay for itself, and the question becomes who runs it.

A hosted gateway costs a percentage or a monthly fee and moves uptime and data handling to a vendor you have to read a contract with. A self-hosted one costs nothing in license and an operations routine, including fast patching. Our directory lists 43 products across AI infrastructure, and the 7 in the gateway niche are only the start of that list.

Sources: [14][19][28][6]