Mohamed Yasser explains why workload assessment, model selection, and performance testing should guide enterprise AI infrastructure investment
Enterprise AI infrastructure discussions often begin with hardware. How many GPUs do we need? Should we buy a bigger server? Will 24 GB of GPU memory suffice, or should we move straight to 48 GB?
Workload requirements should come first. What must the AI system do, and how many requests must it handle simultaneously?
Consider a local language model whose weights occupy approximately 4 GB of GPU memory on a machine with 12 GB available. When five AI agents submit requests simultaneously to a shared inference service, they do not necessarily require five separate copies of the model. Sharing model weights across requests is an established approach. Modern inference engines improve efficiency through techniques such as continuous batching and advanced KVcache management, allowing greater concurrency without proportionally increasing model memory requirements.
Requests using the same model can share one loaded copy. Each request maintains its own working context, including a key-value (KV) cache that stores previously computed attention keys and values to avoid redundant calculations during token generation. The inference engine schedules and batches requests, allowing multiple independent inputs to be processed using shared model weights rather than loading a separate model for each request.
Understanding this distinction changes the economics of inference. A model that fits comfortably in memory may serve multiple requests without duplicating its weights. However, sufficient memory does not guarantee acceptable performance for every user.
Memory determines whether a model and its concurrent workloads can fit within the available hardware resources. Performancedetermines whether the system can meet acceptable response times and throughput requirements. A GPU may accommodate five active requests yet take too long to return the first token or complete each response. Long document prompts increase prefill computation, while concurrent token generation shares available compute and memory bandwidth. Batching can improve overall throughput, although individual request latency may increase.
Enterprise AI planning therefore needs the same discipline applied to databases, APIs and transaction systems. Inference introduces additional variables, including input and output tokens, context length, model architecture, KV-cache size and concurrency.
Licensed user numbers provide an incomplete picture of demand. Twenty employees might generate only a few overlapping requests during normal work, while one employee using a multi-agent workflow could trigger five or ten calls in parallel. Sizing infrastructure by headcount can overprovision one application and underprovision another.
Model selection requires similar scrutiny. Bigger models are not automatically better. A smaller model that reliably performs a defined task, supports the required languages and runs efficiently may offer more value than a much larger general-purpose model.
Quantisation can substantially reduce model memory requirements, but it does not automatically improve inference speed. Its impact depends on model architecture, hardware capabilities and inference engine support.
Organisations must also evaluate its effect on output quality and verify licensing requirements before enterprise deployment.
A practical process starts with representative tasks and acceptable accuracy and response-time targets. Organisations can then shortlist models, estimate memory requirements for weights and cache, and evaluate hardware. GPU decisions should consider memory bandwidth and sustained performance alongside VRAM capacity. CPU, RAM, storage, power, cooling and operational support complete the assessment.
For a small deployment, I would favour validating one carefully configured inference server before purchasing a cluster based on assumptions. Test one, two, five and ten simultaneous requests, combining short conversations with realistic documents. Measure time to first token, complete response time, throughput, queueing and peak GPU memory.
Benchmark results provide the evidence needed to investigate missed performance targets. Additional profiling may be required to determine whether the bottleneck lies in model selection, inference engine configuration, GPU memory capacity, compute performance or memory bandwidth. Sustained testing helps identify these limitations before committing to additional hardware investments.
Enterprise readiness extends beyond speed. Internal hosting may support data-residency objectives, but on-premises deployment does not automatically secure a system. Authentication, network restrictions, access controls, logging, model provenance, monitoring and recovery require deliberate design. A single GPU server may suit a pilot, while a business-critical service may justify redundancy even with modest usage.
AI infrastructure increasingly demands a predictable service built around the model. Agentic systems make demand less directly dependent on user numbers because applications can generate work in parallel, sometimes without a person waiting for each step.
My view is straightforward: begin with the workload, prove the model meets the need, calculate memory requirements, measure concurrency and establish clear performance targets. Buy what the evidence supports.
The objective is an inference service that is useful, secure, and appropriately sized for the job.
This opinion piece is authored by Mohamed Yasser, Solution Architect, Sharjah City Municipality.
Source: Tahawul Tech