This change enables the repack stage to utilize the user-specified
thread count, ensuring that both the logical thread IDs and the total
number of threads remain consistent between the repack and inference
stages.
In a NUMA architecture where the `--numa distribute` parameter is used,
logical threads are pinned to specific physical NUMA nodes. By aligning
the thread configuration across these two stages, we can fully leverage
the operating system's "first-touch" memory allocation policy:
1. Repack Stage: Logical thread i (bound to NUMA node j) is responsible
for repacking and writing the weight data. Since the "first touch"
occurs within this thread, the corresponding physical memory is
allocated on node j.
2. Inference Stage: The same logical thread i (still bound to node j)
reads these weights. Since the data already resides on the local
node, low-latency local memory access is achieved.
Without ensuring consistency in the number of threads, data may be
randomly allocated to mismatched nodes, resulting in significant
cross-node access overhead during inference.
Signed-off-by: Jianhui Zhou <jonaszhou@zhaoxin.com>
5714d4b86e