Machine Learning on Custom AI Chips: What Software Engineers Need to Know

Section 1: Why Custom AI Chips Are Changing Machine-Learning Software Engineering

From General-Purpose Compute to Specialized AI Acceleration

For many years, machine-learning software could be developed with relatively little concern for the exact processor executing it because CPUs and GPUs provided flexible computational environments that supported a wide variety of workloads. Custom AI chips change that relationship by designing hardware around the operations that dominate specific machine-learning workloads, particularly tensor operations, matrix multiplication, specialized data movement, and lower-precision arithmetic. Application-specific accelerators such as Google TPUs illustrate this design philosophy by emphasizing matrix processing through specialized compute units rather than attempting to serve every possible software workload.

This specialization can provide important advantages because the accelerator does not need to devote as much silicon, power, or memory bandwidth to operations that are irrelevant to the target workload. An AI accelerator can dedicate more resources to parallel multiply-accumulate operations, tensor movement, or other computations that occur repeatedly during training and inference. For software engineers, however, specialization means that the model is no longer completely independent from the execution platform. Operators, tensor shapes, numerical formats, memory layouts, and execution schedules can all influence how effectively the accelerator is utilized.

The consequence is a shift from hardware-agnostic model development toward hardware-aware engineering. Engineers increasingly need to understand not only whether a model produces the correct prediction, but also whether its computational graph maps efficiently onto the available accelerator. Two mathematically equivalent implementations can have very different execution characteristics when one aligns naturally with the hardware's supported operations while the other requires additional conversions, synchronization, or fallback execution.

Why Hardware Architecture Now Influences Model Design

Custom AI silicon makes the relationship between architecture and model design more direct because each accelerator exposes a particular collection of strengths and constraints. Some architectures emphasize dense matrix multiplication, while others may prioritize sparse computation, low-precision inference, memory bandwidth, or specialized operations. Consequently, the model architecture selected during development can influence not only predictive accuracy but also the amount of computation, memory traffic, and synchronization required during execution.

Tensor dimensions can become especially important because accelerators often execute most efficiently when workloads align with their internal processing structures. An operation that produces awkward tensor shapes may leave portions of the hardware underutilized, while a mathematically similar operation with more suitable dimensions can achieve substantially higher throughput. Engineers therefore need to think about batch sizes, sequence lengths, channel dimensions, and operator selection as performance variables rather than purely algorithmic choices.

Numerical precision adds another layer to this relationship. Training and inference increasingly use reduced-precision formats when supported by the hardware because lower precision can reduce memory requirements and increase computational throughput. However, changing precision can affect numerical stability or model quality, meaning that software engineers need to validate the complete workload rather than assuming that lower precision automatically produces a better system.

This hardware-aware perspective is closely connected to the broader principles discussed in “Resource-Aware Machine Learning: Designing Models Around Compute and Energy Limits,” because custom accelerators make resource constraints part of the model-design process itself. Instead of optimizing accuracy first and hardware performance later, teams increasingly need to evaluate both dimensions together.

Compute Is Only Half the Problem: Memory and Data Movement

One of the most important lessons in accelerator engineering is that theoretical compute capability does not automatically translate into application performance. Neural networks continually move parameters, activations, gradients, and intermediate results between different levels of memory, and those transfers can become a limiting factor when computation proceeds faster than data can be supplied. Modern accelerator architectures therefore devote substantial engineering attention to high-bandwidth memory, on-chip storage, interconnects, and dataflow. Google's TPU documentation, for example, describes high-bandwidth memory feeding specialized matrix units, illustrating how compute and memory are designed as one system rather than separate components.

For software engineers, reducing arithmetic operations is therefore not always enough. A model can have relatively low theoretical compute but still execute slowly if it repeatedly moves large tensors through memory or forces frequent synchronization between processing units. Operator fusion can help by combining multiple operations so intermediate results remain closer to compute units instead of being repeatedly written to and reloaded from memory. Choosing appropriate tensor layouts and minimizing unnecessary copies can similarly improve effective throughput without changing the model's high-level behavior.

Large-scale training introduces additional data-movement challenges because multiple accelerators must exchange gradients, activations, or other state during distributed execution. In such environments, network topology and interconnect bandwidth can influence scaling efficiency just as strongly as raw compute capacity. A cluster with extremely powerful chips can therefore underperform when communication overhead becomes dominant, making computation and communication a combined optimization problem rather than independent concerns.

Key Takeaway

Custom AI chips are changing machine-learning software engineering because performance increasingly depends on how closely model computation aligns with the underlying hardware. Compute units, memory systems, numerical precision, tensor shapes, compiler transformations, and interconnects all contribute to real-world performance, making hardware-aware optimization and hardware–software co-design essential skills for engineers building high-performance machine-learning systems.

Section 2: The Software Stack Behind Custom AI Chips

Compilers Translate Machine-Learning Graphs Into Hardware Operations

Custom AI accelerators become useful to software engineers only when the hardware can be accessed through a software stack that translates high-level machine-learning code into operations the processor can execute efficiently. A model may be defined through frameworks such as PyTorch or TensorFlow, but the accelerator does not directly execute the original framework code because that code represents a higher-level computational abstraction. Between the model and the physical processor, compiler infrastructure analyzes the computation graph, selects supported operations, transforms tensor layouts, manages precision, schedules execution, and generates instructions or intermediate representations appropriate for the target hardware.

This compiler layer becomes particularly important because the same model can produce very different performance depending on how effectively its operations are lowered to the accelerator. An operation may be theoretically supported but still require costly conversions, additional memory movement, or synchronization when its implementation does not align with the hardware architecture. Compiler optimizations can combine compatible operations, eliminate redundant transformations, reuse intermediate values, and select execution strategies that reduce unnecessary overhead. Software engineers therefore need to understand not only how to express a model in a framework but also how that model is transformed before it reaches the accelerator.

Compiler maturity also influences how portable machine-learning software can remain across different accelerator platforms. A strong compiler abstraction can allow engineers to preserve much of their high-level model code while changing the underlying execution target, whereas significant gaps in operator support may require platform-specific modifications. This makes the compiler one of the most important interfaces between software productivity and hardware specialization because it determines how much accelerator complexity developers must understand directly.

Runtimes, Kernels, and Operator Support Determine Real Performance

The compiler alone does not determine the final behavior of a production workload because the runtime system controls how compiled operations are loaded, scheduled, synchronized, and executed on the accelerator. Runtime libraries can manage device memory, execution queues, communication, model initialization, and interaction with host processors, while optimized kernels implement performance-critical mathematical operations. The difference between a functional integration and a highly optimized integration often depends on how efficiently these components work together.

Operator coverage becomes especially important when engineers move models from one hardware environment to another. A neural network may contain hundreds of operations, but a custom accelerator may only provide highly optimized implementations for a subset of them. Supported operators can execute directly on the accelerator, while unsupported or poorly supported operations may require alternative implementations or execution on another processor. Such fallbacks can create synchronization points and additional memory transfers that significantly reduce overall performance, even when most of the model runs efficiently.

Kernel optimization therefore becomes a specialized software-engineering skill. Engineers may need to choose implementations based on tensor dimensions, numerical precision, memory layout, and expected workload characteristics, while lower-level developers may write or tune custom kernels for operations that dominate execution time. The objective is not merely to minimize the number of operations but to ensure that the operations remaining in the graph are executed efficiently under the accelerator's actual computational and memory architecture.

This layer is also where seemingly minor implementation choices can produce substantial differences in latency and throughput. Repeated tensor conversions, unnecessary synchronization, poorly aligned shapes, and inefficient memory access patterns can prevent the accelerator from reaching high utilization. Understanding these details allows engineers to diagnose why a model that appears computationally efficient at the algorithmic level may still perform poorly in production.

Framework Compatibility Does Not Mean Hardware Optimization Is Automatic

Modern accelerator ecosystems increasingly provide integrations that allow software engineers to execute familiar machine-learning frameworks on specialized hardware, reducing the amount of low-level code required to begin using a new accelerator. This abstraction is valuable because it preserves developer productivity and allows existing models to be tested on new platforms without rewriting every component. However, framework compatibility should not be interpreted as evidence that the model is automatically optimized for the underlying chip.

A model can execute successfully while leaving substantial performance on the table because the compiler may choose conservative transformations, certain operators may use generic implementations, or tensor shapes may not align with the accelerator's preferred execution patterns. Engineers therefore need to distinguish between functional portability and performance portability. The first means the model produces the expected outputs on the new platform, while the second means the model achieves appropriate latency, throughput, memory efficiency, and cost under the target workload.

This distinction becomes particularly important when organizations evaluate multiple accelerator options because benchmark comparisons based only on peak theoretical compute can produce misleading conclusions. The meaningful question is whether the actual workload reaches efficient utilization through the complete software stack, including the framework integration, compiler, runtime, kernels, memory system, and communication infrastructure. The broader engineering principles discussed in “The Hidden Engineering Work Behind Every Successful Machine Learning Product” are highly relevant here because much of the effort required to achieve production performance exists below the model API and is easy to overlook during initial experimentation.

The strongest accelerator strategy therefore combines high-level framework productivity with enough hardware awareness to identify when deeper optimization is necessary. Software engineers do not need to write assembly for every model, but they increasingly need to understand how graph compilation, operator support, runtime scheduling, kernels, memory movement, and profiling interact so that they can recognize when a model is limited by the accelerator stack rather than by the underlying machine-learning algorithm.

Key Takeaway

The software stack behind custom AI chips determines whether theoretical hardware capability becomes useful production performance. Compilers, runtimes, kernels, operator coverage, profiling tools, and framework integrations form a chain in which weaknesses at any layer can limit the efficiency of the complete workload, making it essential for software engineers to distinguish simple hardware compatibility from genuine hardware-aware optimization.

Section 3: How Engineers Optimize Models for Custom AI Accelerators

Numerical Precision Changes Performance and Memory Requirements

One of the most important optimization decisions for custom AI accelerators is the numerical precision used throughout training and inference because the representation chosen for weights, activations, gradients, and intermediate values directly affects memory consumption, computational throughput, and energy efficiency. Traditional machine-learning development often begins with numerical formats that prioritize stability and generality, but specialized accelerators can provide substantially greater efficiency when workloads use lower-precision representations that match the hardware's native capabilities. Engineers therefore need to consider numerical precision as part of model architecture and deployment design rather than treating it as a final optimization applied after the model has been completed.

Reduced precision can decrease the amount of memory required to store parameters and activations, which can allow larger portions of a model to remain closer to compute units and reduce the volume of data transferred through the memory hierarchy. Lower-precision arithmetic can also increase the number of operations performed within a fixed hardware budget when the accelerator contains specialized units designed for those formats. These benefits can be particularly valuable for large neural networks where memory capacity and bandwidth frequently become limiting factors before raw arithmetic capacity is fully utilized.

The challenge is that reducing precision can affect numerical behavior, and the acceptable trade-off depends on the model, workload, and accelerator. Some layers may tolerate aggressive precision reduction while others may be more sensitive to accumulated numerical error, requiring engineers to preserve higher precision selectively. Production optimization therefore involves testing the complete model rather than assuming that one numerical format can be applied uniformly. Engineers need to compare accuracy, latency, memory use, throughput, and stability under realistic data because a reduction in arithmetic cost provides limited value if the resulting model loses capabilities that matter to the application.

Kernel Fusion and Efficient Data Movement Reduce Overhead

Custom accelerators can execute mathematical operations extremely quickly, but the overall system can still become bottlenecked by the cost of moving data between operations. Neural networks often contain sequences of transformations in which one operation generates an intermediate tensor that is immediately consumed by another operation. If each stage writes its output to external memory and then reloads it for the next stage, the workload can spend substantial time and energy moving data rather than performing useful computation.

Kernel fusion addresses this problem by combining compatible operations into a larger execution unit so that intermediate values can remain closer to the compute resources. For example, a sequence of element-wise transformations may be executed together rather than requiring multiple separate launches and memory transfers. The benefit depends on the specific accelerator and compiler, but the underlying principle is broadly important because minimizing unnecessary data movement can improve effective performance even when the theoretical amount of computation remains unchanged.

Tensor layout also matters because memory access patterns influence how efficiently an accelerator can retrieve and process data. A model that frequently requires reshaping, transposing, or copying tensors may introduce overhead that is invisible when looking only at the high-level computational graph. Engineers can therefore optimize data representations so that consecutive operators use compatible layouts and avoid unnecessary transformations. This kind of optimization requires understanding the relationship between model graphs, compiler transformations, memory hierarchy, and hardware execution rather than focusing exclusively on the mathematical operations.

The same principle becomes more important during distributed training or large-scale inference because data may have to move not only between memory levels within one accelerator but also across multiple devices. When communication becomes dominant, adding more accelerators does not necessarily improve performance proportionally. Engineers therefore need to minimize synchronization, overlap communication with computation where possible, and organize workloads so that the accelerator spends as much time as possible performing useful operations instead of waiting for data.

Model Architecture Must Match the Accelerator's Strengths

Hardware-aware optimization can begin much earlier than compilation because model architecture itself determines which operations, tensor shapes, and memory behaviors will dominate execution. A software engineer choosing between alternative architectures should therefore consider not only predictive quality and parameter count but also whether the model's computation maps naturally onto the target accelerator. Architectures that rely heavily on well-supported matrix or tensor operations may achieve higher utilization than mathematically similar designs that depend on unsupported or inefficient operators.

Tensor dimensions can have a surprisingly large effect because accelerators often contain fixed processing structures that achieve their highest utilization when workloads align with particular shapes. Poorly aligned dimensions can leave compute resources partially idle or require additional transformations before operations can execute efficiently. Engineers can sometimes improve performance simply by selecting compatible hidden dimensions, channel counts, sequence lengths, or batch configurations while maintaining essentially the same modeling approach.

Sparse computation provides another opportunity when the target accelerator can exploit it effectively. Removing unnecessary computation can reduce memory traffic and arithmetic requirements, but theoretical sparsity does not automatically create a practical speedup because the hardware and compiler must be capable of processing the sparse structure efficiently. This is why hardware-aware sparsity is different from simply creating a sparse model, with the real benefit depending on whether the accelerator can translate that sparsity into fewer operations or lower data movement.

The broader principle is that architecture and hardware should increasingly be considered together, particularly for workloads that operate at significant scale. The ideas discussed in “Sparse Machine Learning: Why Doing Less Computation Can Produce Better AI Systems” become especially relevant when engineers can design models around accelerators that explicitly benefit from reduced computation. A model with slightly greater theoretical complexity can sometimes outperform a smaller model when its operations align better with the available hardware, demonstrating why parameter count alone is an incomplete measure of efficiency.

Key Takeaway

Optimizing machine-learning models for custom AI accelerators requires engineers to consider numerical precision, memory movement, kernel execution, model architecture, sparsity, and inter-device communication as interconnected parts of one system. The strongest results come from hardware-aware co-design in which models are structured around the capabilities of the target accelerator, unnecessary data movement is reduced, numerical formats are chosen deliberately, and distributed execution is optimized so that additional hardware translates into meaningful improvements in throughput, latency, or cost.

Section 4: Building Production Systems Around Custom AI Chips

Cost per Inference Becomes a First-Class Engineering Metric

Deploying machine-learning workloads on custom AI chips changes the definition of performance because production engineering must consider not only whether a model can execute efficiently but also whether the resulting system delivers the required business value at an acceptable cost. Accelerator comparisons based solely on theoretical compute throughput can be misleading because the effective economics of a workload depend on utilization, memory capacity, data movement, software licensing, power consumption, model size, batch characteristics, and the amount of infrastructure required to support the deployment. A custom accelerator may provide excellent performance for one workload while delivering limited benefits for another whose operators or memory-access patterns do not align with the architecture.

Inference cost becomes especially important for high-volume applications in which even small efficiency differences accumulate across millions or billions of requests. Engineers therefore need to evaluate metrics such as cost per inference, throughput per accelerator, latency at production concurrency, accelerator utilization, and energy consumed per prediction rather than relying on peak benchmark figures. This approach also changes optimization priorities because reducing memory movement or increasing accelerator utilization can sometimes create greater economic value than reducing the model's raw parameter count.

Training workloads require a different economic analysis because training jobs may run for hours or days across large accelerator clusters. In these environments, engineers must consider scaling efficiency, checkpointing overhead, interconnect utilization, storage requirements, and idle capacity in addition to raw compute performance. The practical objective is to maximize useful model progress for each unit of infrastructure consumed, which makes performance profiling and workload scheduling as important as selecting the accelerator itself.

Portability and Vendor Lock-In Become Architectural Concerns

Custom AI hardware can deliver strong performance when software is optimized for its specific architecture, but specialization can also introduce portability challenges because models, kernels, compiler configurations, and deployment pipelines may become dependent on vendor-specific capabilities. A team that achieves excellent performance through specialized operators or low-level optimizations may find that moving the workload to another accelerator requires significant engineering work. This does not make specialization undesirable, but it makes portability an explicit architectural consideration.

Software engineers can reduce unnecessary coupling by preserving clear boundaries between model logic, hardware-specific optimizations, compilation configuration, and deployment infrastructure. High-level framework representations can provide a portable baseline, while specialized kernels and accelerator-specific transformations can be isolated behind stable interfaces. This allows teams to maintain a common model implementation while introducing hardware-specific paths when the expected performance or cost benefits justify the additional complexity.

Portability is also affected by operator coverage because a model that uses broadly supported operations is easier to move between platforms than one that depends heavily on proprietary implementations. Engineers therefore need to evaluate not only the current accelerator's performance but also the future cost of changing hardware, expanding to additional environments, or supporting multiple accelerator types simultaneously. The broader principles in “From Experiment to Production: The Decisions That Shape an ML System” are particularly relevant because infrastructure decisions made during experimentation can create long-term operational consequences when the model becomes a production dependency.

Monitoring Must Expose Hardware Utilization as Well as Model Behavior

Production monitoring for custom AI chips cannot stop at machine-learning metrics such as accuracy, loss, or prediction quality because a model may remain mathematically correct while the underlying accelerator becomes increasingly inefficient. Hardware utilization, memory bandwidth, device temperature, power consumption, host-to-device transfers, compilation behavior, queue depth, and inference latency can reveal bottlenecks that are invisible from model-level telemetry. Engineers therefore need observability that connects model behavior with the hardware execution path responsible for producing each prediction.

This becomes particularly important after model updates because apparently small changes to architecture, tensor dimensions, operator selection, or numerical precision can alter compiler behavior and accelerator utilization. A new model version may maintain the same predictive quality while requiring significantly more memory or producing lower throughput, creating higher infrastructure costs without an obvious change in machine-learning metrics. Continuous benchmarking and performance regression testing can help identify these changes before they become expensive production problems.

Monitoring also needs to account for workload composition because accelerator efficiency may vary with request size, sequence length, concurrency, and batch structure. An inference service that performs efficiently during large scheduled workloads may behave very differently under low-volume interactive requests. Engineers should therefore evaluate performance across the operating conditions that the production service is expected to encounter rather than treating one benchmark configuration as representative of every workload.

Key Takeaway

Custom AI chips make production machine-learning engineering increasingly dependent on the relationship between models and hardware because cost, portability, utilization, monitoring, and scalability are determined by the complete execution stack rather than by model accuracy alone. The future therefore favors engineers who can connect machine-learning architecture with compiler behavior, accelerator capabilities, memory systems, distributed execution, and production economics to build systems that are not only accurate but also efficient, maintainable, and operationally sustainable.

Conclusion

Custom AI chips are changing machine-learning engineering because the performance of an ML system is increasingly determined by the interaction between model architecture, compiler behavior, memory systems, runtime execution, and accelerator hardware. For many years, engineers could think primarily in terms of algorithms and model architectures because the underlying compute platform was relatively standardized and software abstractions hid much of the hardware complexity. Specialized AI accelerators make those boundaries less distinct, creating an environment in which software decisions can directly influence how efficiently physical hardware is utilized.

The most important shift is that machine-learning performance can no longer be understood through model accuracy or theoretical compute alone. A model may achieve strong predictive results while performing poorly in production because it generates excessive memory traffic, depends on inefficient operators, uses unsupported operations, requires frequent synchronization, or fails to utilize the accelerator effectively. Conversely, a carefully designed model with slightly greater algorithmic complexity can sometimes provide better production performance when its computation aligns naturally with the strengths of the target hardware.

This makes hardware-aware development increasingly important.

Software engineers working with custom AI chips need to understand how a high-level computational graph is transformed into hardware-specific execution. Frameworks provide the abstraction required to build models productively, but compilers determine how those models are lowered to accelerator operations, runtimes coordinate execution and memory, and optimized kernels determine how important operations are actually performed. The result is a multi-layer software stack in which a bottleneck at one level can prevent the system from realizing the theoretical capabilities of the hardware.

Frequently Asked Questions

1. What is a custom AI chip?

A custom AI chip is a processor or accelerator designed specifically, or highly specifically, for machine-learning workloads rather than for broad general-purpose computing. Its architecture may prioritize operations such as matrix multiplication, tensor processing, specialized memory access, or reduced-precision computation.

2. How are custom AI chips different from GPUs?

GPUs are highly parallel processors originally designed for graphics but widely adopted for machine learning because their architecture is well suited to parallel numerical workloads. Custom AI chips are designed more specifically around selected AI workloads and may provide stronger efficiency for those workloads when the software stack and model architecture align with the accelerator.

3. Why do software engineers need to understand AI hardware?

Model performance depends increasingly on hardware characteristics such as memory bandwidth, supported numerical formats, tensor dimensions, operator availability, and communication architecture. Understanding these factors helps engineers identify why a model may be inefficient and determine which optimizations are likely to produce meaningful production improvements.

4. What is hardware-aware machine-learning optimization?

Hardware-aware optimization is the process of designing or modifying models, computation graphs, kernels, and execution strategies so that they use the actual capabilities of a target accelerator efficiently. It considers computation, memory movement, numerical precision, tensor shapes, operator support, and device communication together.

5. Why are compilers important for custom AI chips?

Compilers translate high-level machine-learning computations into operations that the accelerator can execute. They can optimize tensor layouts, fuse operations, select supported kernels, manage execution schedules, and reduce unnecessary computation or memory movement, making the compiler a critical link between model code and hardware performance.

6. What role do kernels play in accelerator performance?

Kernels are implementations of computational operations that execute on the accelerator. Efficient kernels can take advantage of the accelerator's architecture, memory system, numerical formats, and parallelism, while inefficient or unsupported kernels can create bottlenecks even when the overall model architecture appears efficient.

7. Why is memory bandwidth important for machine-learning accelerators?

Machine-learning workloads repeatedly move large amounts of parameters, activations, and intermediate data. When an accelerator can perform arithmetic faster than the memory system can supply data, memory movement becomes the bottleneck, making data locality, tensor layouts, caching, and memory-aware execution important parts of optimization.

8. How does quantization help custom AI accelerators?

Quantization reduces the numerical precision used to represent model values, which can lower memory consumption and increase computational efficiency when the target accelerator supports the selected format effectively. Engineers must still validate model quality because aggressive precision reduction can affect numerical behavior.

9. What is kernel fusion?

Kernel fusion combines multiple compatible operations into a single execution unit so that intermediate results can remain closer to compute resources and unnecessary memory transfers or execution overhead can be reduced. The effectiveness of fusion depends on the accelerator architecture and compiler capabilities.

10. Why can a model with fewer parameters still perform poorly on specialized hardware?

Parameter count does not capture all sources of execution cost. Tensor shapes, memory movement, operator support, synchronization, data layout, and accelerator utilization can have substantial effects on actual latency and throughput, meaning a smaller model is not automatically a faster model.

11. How do engineers profile custom AI chip workloads?

Engineers use profiling and benchmarking tools to measure metrics such as accelerator utilization, operator execution time, memory bandwidth, data transfers, device synchronization, latency, throughput, and resource consumption. Profiling helps identify the actual bottleneck before engineers decide which optimization to apply.

12. Why does distributed training require hardware-aware optimization?

Distributed training requires multiple accelerators to exchange information such as gradients, activations, or parameters. Communication can become a bottleneck when synchronization costs are large, so engineers must consider network topology, collective operations, workload partitioning, and communication–computation overlap alongside raw accelerator performance.

13. What is the biggest risk of optimizing for one custom AI chip?

Deep hardware-specific optimization can increase dependency on vendor-specific compilers, runtimes, kernels, and capabilities. This can make future migration to another accelerator more expensive, so organizations need to balance the benefits of specialization against their long-term portability requirements.

14. How should companies evaluate custom AI chips?

Evaluation should use the organization's actual workloads rather than relying exclusively on theoretical specifications. Important considerations include end-to-end latency, throughput, model compatibility, memory capacity, accelerator utilization, energy consumption, infrastructure cost, software maturity, development effort, scalability, and the long-term maintainability of the platform.

15. What skills do software engineers need to work with custom AI chips?

Engineers benefit from understanding machine-learning frameworks, computational graphs, compilers, runtimes, numerical precision, memory hierarchy, profiling, distributed systems, performance engineering, and model architecture. They do not need to become chip designers, but they increasingly need enough hardware knowledge to reason about how software decisions translate into real accelerator performance.