NVIDIA NCP-AII NVIDIA AI Infrastructure Exam Practice Test

NVIDIA Spectrum-X is the world’s first high-performance Ethernet fabric designed specifically for AI, combining Spectrum-4 switches with BlueField-3 SuperNICs. To achieve the high-throughput, low-latency requirements of East-West (server-to-server) AI traffic, the BlueField-3 hardware must be correctly provisioned. The first requirement is ensuring the physical ports are operating in Ethernet mode; LINK_TYPE_P1=2 (and P2=2 for the second port) toggles the hardware from InfiniBand to Ethernet mode. Secondly, the BlueField-3 can operate in several "modes." While "DPU mode" (Option D) offloads the entire OS to the ARM cores, "NIC mode" (or SuperNIC mode) allows the host to manage the networking while leveraging theInternal CPU Offload Engine. Setting INTERNAL_CPU_OFFLOAD_ENGINE=1 is a specific configuration step for Spectrum-X that allows the SuperNIC to handle RoCE (RDMA over Converged Ethernet) and Congestion Control (CC) algorithms more efficiently at 400G speeds. This setup is vital for AI workloads like LLM training, where consistent latency and zero-packet-loss are non-negotiable for collective communications.

Interpreting NCCL (NVIDIA Collective Communications Library) results requires understanding the difference between theoretical link rate and effective bus bandwidth. A 400G (NDR) InfiniBand link has a theoretical unidirectional capacity of 50 GB/s. In an 8-GPU DGX system, where each GPU is typically mapped to its own 400G HCA, the aggregate theoretical bandwidth is 400 GB/s. However, NCCL performance benchmarks reportBus Bandwidth, which accounts for the mathematical overhead of collective operations like all_reduce. Achieving350 GB/s(Option A) is widely considered optimal and representative of a healthy, well-configured fabric. This indicates that the system is effectively utilizing GPUDirect RDMA to bypass host memory and that the network switches are routing traffic without significant congestion or packet drops. Expecting >390 GB/s (Option C) is unrealistic once protocol headers and algorithmic overhead are subtracted from the raw bit rate. Rerunning with CPU stress (Option D) would not provide insights into the InfiniBand/GPU fabric health.

The nvidia-ctk runtime configure command is a crucial step that modifies the Docker daemon configuration file (/etc/docker/daemon.json) to register the nvidia runtime. However, the Docker daemon only reads this configuration file during its initialization phase. Even though the toolkit is installed and the configuration file is updated, Docker will not be able to spawn GPU-accelerated containers until the service is refreshed. Executing sudo systemctl restart docker (or the equivalent for your container engine) is the mandatory final step. This forces Docker to reload its settings and recognize the NVIDIA Container Runtime as a valid option. Without this restart, attempting to run a container with the --gpus all flag will result in an error stating that the "nvidia" runtime is not found or is unconfigured. This is a common point of failure in automated AI infrastructure deployments where the configuration script finishes, but the service state remains stale.

The NVIDIA Container Toolkit (formerly nvidia-docker2) is the essential middleware that allows Docker, Podman, or Containerd to "see" and utilize the host's GPU hardware. The standard installation workflow on Debian-based systems like Ubuntu involves three core phases: repository configuration, package installation, and runtime configuration. Once the GPG key is added (to ensure package integrity) and the .list file is placed in /etc/apt/sources.list.d/ (to point to the NVIDIA production servers), the local package index must be refreshed via apt-get update. Immediately following this, the administrator must install the toolkit using the command sudo apt-get install -y nvidia-container-toolkit. Rebooting (Option A) is unnecessary at this stage because no kernel modules have been modified yet. Downloading the CUDA Toolkit (Option D) is a separate step; notably, the Container Toolkit allows containers to run CUDA applications even if the host only has the NVIDIA driver installed, making the driver—not the host CUDA toolkit—the primary prerequisite.

The physical installation of high-performance NVIDIA components, such as H100 PCIe GPUs or BlueField DPUs, requires strict adherence to data center safety and hardware preservation standards. Option D is the only "100% verified" procedure because it covers three critical pillars: Power, Compatibility, and Safety. First, high-end GPUs can draw up to 300W-450W individually; verifying the server's PDU and internal PSU capacity is essential to prevent over-current shutdowns. Second, verifying cable compatibility (such as 12VHPWR or specific PCIe power 8-pin layouts) is vital to avoid electrical damage. Third, "Cold Service" (ensuring the server is powered down and cables are removed) is the standard for non-hot-plug PCIe components to prevent short circuits. Finally, wearing an ESD (Electrostatic Discharge) bracelet is non-negotiable when handling NVIDIA hardware, as static charges can destroy the sensitive HBM (High Bandwidth Memory) or the GPU die itself. Skipping ESD protection (as suggested in Option A) or performing the install while the system is "up and running" (as suggested in Option C) are leading causes of hardware infant mortality in AI infrastructure.

Managing a large-scale AI fabric requires centralized visibility into the physical layer. The NVIDIAUnified Fabric Manager (UFM)provides a comprehensive Dashboard for InfiniBand networks. To check transceiver firmware—which is critical for ensuring feature parity and stability across the fabric—the administrator can use the UFM Enterprise GUI. By navigating to the "Devices" section and selecting a specific switch, the "Cables" tab will aggregate telemetry for every occupied port. This view displays the manufacturer, part number, and the specific firmware version of the transceivers (LinkX) or Active Optical Cables (AOC). This consolidated view is far more efficient than manual CLI queries (Option C) for 200+ ports. Maintaining uniform firmware across transceivers ensures that optimizations like Adaptive Routing and Congestion Control perform consistently across the entire 400G or 200G fabric.

High-Performance Linpack (HPL) is a memory-intensive benchmark that allocates a large portion of available GPU memory to store the matrix $N$. While a server may have 2TB of physical system RAM, the "not enough memory" error usually refers to theHBM (High Bandwidth Memory)on the GPUs themselves. In a DGX H100 system, each GPU has 80GB of HBM3. If the problem size ($N$) specified in the HPL.dat file is too large, the required memory for the matrix will exceed the aggregate capacity of the GPU memory. Reducing the problem size ($N$) while maintaining the optimal block size ($NB$) ensures that the problem fits within the GPU memory limits while still pushing the computational units to their peak performance. Increasing the block size (Option C) would actually increase the memory footprint of certain internal buffers, potentially worsening the issue. Reducing $N$ is the standard procedure to stabilize the run during the initial tuning phase of an AI cluster bring-up.

Updating firmware on an NVIDIA DGX cluster is a critical operation that involves multiple sensitive components, including the GPU baseboard, the BMC, the motherboard tray (SBC), and the InfiniBand HCAs. In a production environment, "Batching" is the industry standard to prevent a single corrupted firmware image or update failure from taking down the entire AI factory. The process must begin with "Draining" the nodes in the workload scheduler (like Slurm or Kubernetes) to ensure no active training jobs are interrupted. Running pre-update diagnostics—using tools like nvsm show health or dcgmi diag—is vital to establish a baseline and ensure the hardware is stable before applying changes. Once the firmware is applied in a controlled batch, post-update verification is required to confirm the system returns to a "Healthy" state and that all versions match the target manifest. This "Rolling Update" strategy allows the engineer to pause the automation if a specific node fails to return to service, protecting the overall availability of the cluster. Skipping diagnostics (Option D) or leaving nodes on mismatched versions (Option C) creates "configuration drift," which leads to unpredictable performance in collective communication libraries.

UEFI Secure Boot is a security standard that ensures only digitally signed code is allowed to execute during the boot process. Since NVIDIA GPU drivers include kernel modules (nvidia.ko), they must be signed by a key trusted by the system's firmware. When drivers are installed on a DGX system with Secure Boot active, the installation process generates a uniqueMachine Owner Key (MOK). However, the Linux kernel will not trust this key until the user manually authenticates it at the "Shim" level before the OS loads. Upon the first reboot after installation, the system enters the "MOK Management" blue screen. The administrator must select "Enroll MOK" and enter the temporary password created during the driver installation. Failing to do this results in the kernel rejecting the nvidia module, leading to an "Unable to determine the device handle for GPU" error in nvidia-smi. Disabling Secure Boot (Option A) would resolve the symptom but violates the security posture of the AI infrastructure.

In a Bright Cluster Manager HA setup, the database (MySQL/MariaDB) must remain perfectly synchronized between the active and standby head nodes to allow for a seamless transition. This synchronization typically occurs over a dedicated management or heartbeat network. If cmsh status shows the database service as [FAIL] on the secondary node, it almost always points to a communication breakdown. Without a stable network path, the secondary node cannot receive the binary logs from the primary node to keep its local copy up to date. While licensing (Option A) is important, a license failure usually disables management capabilities entirely rather than just the MySQL sync. Furthermore, head nodes are management servers and do not require GPU drivers (Option C) for their primary function. Ensuring low-latency, reliable connectivity between the two head nodes is the primary troubleshooting step for resolving "MySQL FAIL" states in BCM.

In NVIDIA networking environments, specifically those utilizing InfiniBand or high-speed Ethernet via ConnectX adapters, monitoring the physical link quality is critical for preventing packet loss and RDMA retransmissions. The mlxlink tool is part of the NVIDIA Firmware Tools (MFT) package and is the primary utility for checking the status and health of the physical link. Using the -d flag specifies the device (e.g., /dev/mst/mt4123_pciconf0), while the -c (counters) and -e (error counters/BER) flags provide a detailed readout of the link's performance. Bit Error Rate (BER) is a fundamental metric for signal integrity. NVIDIA systems typically distinguish between "Raw BER" (errors before Forward Error Correction) and "Effective BER" (errors remaining after FEC). A high BER often points to a failing transceiver, a dirty fiber connector, or a marginal DAC cable. While ethtool can show general statistics in Ethernet mode, mlxlink is the verified method for granular BER measurement across InfiniBand and high-speed fabrics, allowing engineers to determine if a link meets the "Error-Free" operation standards required for large-scale AI collective communications like NCCL.

On systems where UEFI Secure Boot is enabled, the Linux kernel strictly enforces a "Signature Verification" policy for all loaded kernel modules. The NVIDIA driver consists of several modules (like nvidia.ko and nvidia-uvm.ko) that provide the interface between the hardware and the NVIDIA Container Runtime. If these modules are not signed by a key trusted by the system's firmware, the kernel will block them from loading, leading to "Permission Denied" errors when the container attempts to access /dev/nvidiactl or the GPU device nodes. To resolve this without compromising the security of the host, the administrator must utilize aMachine Owner Key (MOK). The modules must be signed using this key, and the key itself must be "Enrolled" into the system's Secure Boot database via the MOK management interface during the boot process. This establishes a "Chain of Trust" that allows the drivers to function. Disabling SELinux (Option C) or using sudo (Option D) will not resolve this, as the blockage is happening at the kernel-firmware interface level, not at the user-space permission level.

In an InfiniBand fabric, the Subnet Manager (SM) is the "brain" of the network, responsible for discovering the topology, assigning Local Identifiers (LIDs), and calculating routing tables. In a multi-node fabric, there is typically one Master SM and several Standby SMs for high availability. To identify the master, the sminfo command is first used; it queries the fabric and returns the LID of the current Master SM. Once the LID is obtained, the engineer must map that numerical LID to a physical server name or Node Description. The smpquery ND (Node Description) command is then executed, targeting that specific LID. This sequence is vital for troubleshooting fabric-wide issues, as logs on the Master SM server provide the definitive record of sweeps, traps, and topology changes. Using smpquery NI (Node Info) would provide hardware-level details like the GUID and device ID, but it does not return the human-readable string (server name) defined in the Node Description, which is necessary for rapid identification in a crowded data center.

In a large-scale Spectrum-X Ethernet fabric, "East-West" traffic refers to the cross-rack communication between compute nodes. To validate the "Bisectional Bandwidth" (the throughput between two halves of the cluster), administrators use NCCL tests with specific environment variables to control traffic patterns. The NCCL_TESTS_SPLIT variable is used to partition the GPUs into distinct groups for the benchmark. Setting NCCL_TESTS_SPLIT="DIV 8" is a standard configuration for multi-node testing on 8-GPU systems. It effectively divides the total number of GPUs by the node count, creating a test environment where each GPU communicates with its corresponding rank on other nodes. By combining this with -g 1 (one GPU per process) across multiple nodes, the engineer can force data to travel across the leaf-and-spine switches rather than staying within the NVLink fabric of a single node. This isolates the physical network performance from the internal GPU-to-GPU bandwidth, providing a true measurement of the fabric's ability to handle high-speed AI traffic.

When a DGX system exhibits high fan speeds and performance degradation, it is typically engaging inThermal Throttling. High-performance GPUs like the A100 or H100 will automatically reduce their clock speeds (and thus performance) if they exceed safe temperature thresholds. The first and most critical diagnostic step is to run nvidia-smi. This utility provides immediate, real-time telemetry on GPU temperatures, power draw, and "Clocks Throttle Reasons." By reviewing the output, an administrator can see if "Thermal" is listed as the reason for reduced clocks. This identifies whether the issue is environmental (blocked airflow/hot aisle temperature) or hardware-specific (a failed GPU thermal interface or a dead internal fan). Running more workloads (Option A) would exacerbate the heat, while a power drain (Option C) is a "last resort" that doesn't provide diagnostic data. nvidia-smi provides the evidentiary data needed to determine if an RMA (Return Merchandise Authorization) is required for the GPU tray.

Traditional data center metrics like PUE (Power Usage Effectiveness) only measure how much energy is "wasted" by cooling and power delivery relative to the IT load; they say nothing about how efficiently that IT load is performing its task. In an AI Factory, "Efficiency" is better defined by the amount of AI training or inference performed per watt. NVIDIA advocates for the use of workload-specific benchmarks, such asMLPerf, to quantify this. MLPerf measures the time and energy required to complete standardized AI tasks (like training a ResNet-50 model or an LLM). By prioritizing these benchmarks (Option C), an organization can compare the energy efficiency of different hardware architectures (e.g., A100 vs. H100) or different software optimizations (e.g., FP8 vs. FP16). For example, even if an H100 system draws more peak power than an older system, its ability to complete a training job 9x faster results in a significantly lower "Total Energy Consumed per Job". This shift from "infrastructure efficiency" (PUE) to "computing efficiency" (MLPerf-per-watt) is essential for modern AI data centers aiming for sustainability and cost-effective scaling.

To validate the robustness of theNVLink Switch fabricin a DGX H100, engineers must test how the switches handle traffic when the cluster is logically partitioned. While a standard all_reduce_perf test (Option D) shows aggregate throughput, it may not reveal issues with specific internal switch paths. Using the NCCL_TESTS_SPLIT environment variable allows for more granular stress testing. Specifically, using a bitwise mask like "AND 0x1" (Option B) creates specific traffic subsets that force data through the internal NVLink switch bisection. This ensures that even when only half the GPUs are communicating—or when specific patterns are used—the switches can maintain full wire speed without internal contention. This is a critical validation step during the "Bring-up" phase to ensure there are no manufacturing defects in the NVSwitch baseboard or the high-speed traces connecting the GPU modules.

In NVIDIA BlueField DPU architectures, theRShim (Remote-Shim)interface provides a vital communication channel between the DPU and the host or BMC, typically used for early-stage provisioning, console access, and firmware loading. While the DPU is usually managed via the host's PCIe bus, certain data center configurations require the DPU to be managed out-of-band via the server's Baseboard Management Controller (BMC). To enable this capability, a specific low-level command must be sent to the BMC to toggle the RShim functionality over the internal USB-to-BMC bridge. The command ipmitool raw 0x32 0x6a 1 is the verified raw IPMI hex code used in NVIDIA DGX and certified systems to enable the BMC-to-DPU RShim path. Once enabled, the BMC can "see" the DPU as a USB device, allowing the administrator to push a BlueField Boot (BFB) image to /dev/rshim0/boot for OS installation even if the host CPU is powered off or unresponsive. Option B and C are host-side service commands that assume the driver is already loaded and the hardware path is active, whereas the raw IPMI command is required to enable the hardware path itself.

The ibstat command is a fundamental diagnostic tool in the NVIDIA InfiniBand stack used to query the status of local Host Channel Adapters (HCAs). In the provided output, the most critical data points are the Physical state, the State, and the SM lid.

The Physical state: Linkup confirms that the electrical or optical connection between the server's HCA and the neighboring switch port is established and healthy at the physical layer. This immediately rules out a disconnected cable (Option D) or a completely dead hardware port (Options A and C). However, the State: Initializing indicates that while the "wires" are connected, the logical InfiniBand protocol has not finished its handshake.

In an InfiniBand fabric, the Subnet Manager (SM) is the centralized "brain" responsible for discovering nodes, assigning Local Identifiers (LIDs), and configuring routing tables. The output shows Base lid: 0 and SM lid: 0, which signifies that the port has not been assigned a LID and cannot find an active Subnet Manager to talk to. Without a running SM to transition the port from "Initializing" to "Active," no RDMA traffic can pass through the fabric. This scenario typically occurs if the SM service has crashed on the management node, or if the SM is disabled on the managed switches. Therefore, the root cause is the absence of an operational Subnet Manager in the fabric to complete the logical link initialization.

A 24-hour stress test (using tools like HPL or NCCL) is designed to push the thermal and electrical limits of a DGX system. To verify a "Pass," the administrator must ensure that the hardware maintained its performance targets without degradation.Consistent GPU utilization >95%confirms that the workload successfully saturated the compute cores for the entire duration. Crucially, the absence ofthermal throttling events(verified via nvidia-smi -q -d PERFORMANCE) ensures that the system's cooling solution (fans and heatsinks) is adequate for the environment; if throttling occurred, the GPUs would have slowed down to protect themselves, indicating a potential cooling failure or environmental heat issue. While power consumption (Option D) and CPU usage (Option A) are interesting, they are not the primary indicators of "Stability" under extreme AI training loads. System stability is defined by the ability to run at peak speeds indefinitely without hardware-level interventions or slowdowns.

NVIDIA Run:ai is an advanced orchestration platform designed to optimize GPU resource allocation within Kubernetes environments. Because Run:ai is cloud-native, its control plane and worker agents are deployed as Kubernetes resources. Therefore, the absolute first prerequisite is arunning Kubernetes cluster(Option B) to host the services. Secondly, Run:ai utilizesHelm, the package manager for Kubernetes, to manage its complex installation charts, deployments, and service configurations. Without Helm installed on the administrative machine (Option A), the installation scripts will fail to execute. While having GPUs (Option C) is the ultimate goal for the worker nodes, thecontrol planeitself can be installed on a cluster before all GPU hardware is physically present. Disabling NTP (Option D) is never recommended; in fact, accurate time synchronization is vital for the TLS certificates and logging used by Run:ai and Kubernetes.