Want the bigger picture first? Start with our hubs on private AI infrastructure for business and private LLMs for business. This post focuses on the hardware.

How much memory does an on-premise LLM need?

Roughly, a model needs memory for its weights plus room to work. At 16-bit precision each parameter takes about 2 bytes, so a 7-billion-parameter model needs about 14 GB. Compressed to 4-bit, it takes about a quarter of that. Add headroom for longer documents and several users, then size up from there.

Rule-of-thumb math (parameters × bytes per parameter), weights only. Real needs vary by software, context length and number of users.

Is a bigger model always better for a small business?

No. For drafting, summarizing and answering questions from your own documents, a well-tuned mid-size model with good search over your files often beats a giant model with poor setup. Start with the smallest model that does the job well in testing, then move up only if the answers aren’t good enough.

Desk-side, tower or cloud: which on-premise setup fits?

Desk-side AI computers fit small teams that want private use without a server room. Towers and rack servers fit heavier, shared use with more users or bigger models. Cloud fits bursty work and testing, though data leaves your building. Many small businesses start desk-side, keep some cloud for experiments and grow from there.

What can a desk-side unit like NVIDIA DGX Spark handle?

NVIDIA says DGX Spark offers up to 1 petaFLOP of AI performance at FP4 precision and up to 128 GB of unified system memory, in 64 GB or 128 GB configurations. NVIDIA lists inference on AI models up to 200 billion parameters and fine-tuning of models up to 70 billion parameters on one unit.

NVIDIA also says two units can be linked to work with models up to 400 billion parameters. For most small businesses, one unit running a mid-size model is a realistic starting point. Treat these as NVIDIA’s upper limits, not what every workload will reach.

What else does an on-premise LLM need besides hardware?

You need software to run and serve the model, a way for staff to use it, access controls, logging and backups. Plan who can reach the system, which documents it can search and how updates happen. The box is the easy part. Setup, security and day-to-day care decide whether it’s useful and safe.

Example: how might a small law firm size on-premise LLM hardware?

Picture a ten-person law firm that wants to search and summarize its own files without sending them to a public chatbot. A single desk-side unit running a mid-size model, connected to a document index and limited to firm accounts, could cover drafting and research questions. They’d test first and grow only if needed.

In this illustrative setup, the firm would decide which matters and folders the system can search, keep logs of who asked what, and have a lawyer check every output before it goes to a client. Professional duties still apply, so the firm should check its own rules and obligations before relying on any AI tool.

On-premise LLM: frequently asked questions

Leave a Reply

Your email address will not be published. Required fields are marked *