Ascend Direct Transport

Ascend Direct Transport#

Ascend Direct Transport (referred to as Transport throughout this document) is located at Mooncake/mooncake-transfer-engine/src/transport/ascend_transport/ascend_direct_transport.

Overview#

Transport is a transmission adapter layer built on CANN’s ADXL capabilities, providing direct compatibility with the Mooncake Transfer Engine. With Transport, users can perform Host-to-Device, Device-to-Host, and Device-to-Device transfers in Ascend environments using various communication protocols such as HCCS and RDMA. Future enhancements will enable data transfer between heterogeneous Ascend and GPU environments.

To build and use the Transport library, please follow the process described in build.md and enable the USE_ASCEND_DIRECT option (the legacy USE_ASCEND option now selects the same Ascend Direct transport).

Additional Dependencies#

Ascend Compute Architecture for Neural Networks (CANN)

Please install the latest version of Ascend Compute Architecture for Neural Networks.

The TENT transport backend for Ascend (MC_USE_TENT=1) uses HIXL’s AutoConnect session mode and keeps peer sessions open until teardown, so it requires CANN/HIXL 9.2.0 or newer. On 9.1.x and earlier, deregistering a memory region while AutoConnect sessions to that peer are still open fails; use the classic ascend transport on those versions.

Test Cases#

Transport provides a performance test file at mooncake-transfer-engine/example/transfer_engine_ascend_direct_perf.cpp.

When the metadata_server is configured as P2PHANDSHAKE, Mooncake randomly selects listening ports from the new RPC port mapping to avoid port conflicts. Therefore, testing requires the following steps:

  1. Start the target node first and observe the logs printed in mooncake-transfer-engine/src/transfer_engine.cpp. Look for a statement in the following format:

    Transfer Engine RPC using <protocol> listening on <IP>:<actual_port>

    Record the actual listening port number of the target node.

  2. Modify the startup command for the initiator node: Change the --segment_id parameter value to the target node’s IP + actual listening port number (format: <IP>:<port>).

  3. Start the initiator node to complete the connection test.

Complete command format is shown below:

Start target node:

./transfer_engine_ascend_direct_perf --metadata_server=P2PHANDSHAKE --local_server_name=127.0.0.1:12345 --operation=write --device_logicid=0 --mode=target --block_size=16384 --batch_size=32 --block_iteration=10

Start initiator node:

./transfer_engine_ascend_direct_perf --metadata_server=P2PHANDSHAKE --local_server_name=127.0.0.1:12346 --operation=write --device_logicid=1 --mode=initiator --block_size=16384 --batch_size=32 --block_iteration=10  --segment_id=127.0.0.1:real_port

Environment Variables Configuration#

The following environment variables can be configured to control Ascend Direct Transport behavior:

Variable

Description

Default Value

Example

ASCEND_AUTO_CONNECT

Enable automatic connection management

1 (enabled)

ASCEND_AUTO_CONNECT=0

ASCEND_ENABLE_USE_FABRIC_MEM

Enable fabric memory transfer mode in Mooncake Store (A3 only)

0 (disabled)

ASCEND_ENABLE_USE_FABRIC_MEM=1

ASCEND_USE_ASYNC_TRANSFER

Enable asynchronous transfer mode

0 (disabled)

ASCEND_USE_ASYNC_TRANSFER=1

ASCEND_GLOBAL_RESOURCE_CONFIG

Global resource configuration

-

ASCEND_GLOBAL_RESOURCE_CONFIG="{\"fabric_memory.max_capacity\":32}"

ASCEND_CONNECT_TIMEOUT

Link establishment timeout in milliseconds

10000

ASCEND_CONNECT_TIMEOUT=5000

ASCEND_TRANSFER_TIMEOUT

Data transfer timeout in milliseconds

10000

ASCEND_TRANSFER_TIMEOUT=10000

ASCEND_THREAD_POOL_SIZE

Number of worker threads in the transfer thread pool

16 (1 for buffer pool mode)

ASCEND_THREAD_POOL_SIZE=32

ASCEND_USE_SHORT_CONNECTION

Enable short connection mode (disconnect after each transfer)

0 (disabled)

ASCEND_USE_SHORT_CONNECTION=1

ASCEND_BUFFER_POOL

Buffer pool configuration for intermediate transfer mode (BUFFER_NUM:BUFFER_SIZE_MB)

“0:0” (disabled)

ASCEND_BUFFER_POOL=4:8

ASCEND_BASE_PORT

Base port for ADXL engine port allocation

20000

ASCEND_BASE_PORT=20000

HCCL_INTRA_ROCE_ENABLE

Enable RDMA protocol for intra-node communication

0 (disabled)

HCCL_INTRA_ROCE_ENABLE=1

HCCL_RDMA_TIMEOUT

RDMA packet retransmission timeout coefficient

-

HCCL_RDMA_TIMEOUT=14

HCCL_RDMA_RETRY_CNT

RDMA packet retransmission count

-

HCCL_RDMA_RETRY_CNT=7

Detailed Descriptions:

  • ASCEND_AUTO_CONNECT: Requires CANN 9.0 or later. Default is 1: link can be automatically disconnected when the remote end goes offline abnormally. Set ASCEND_AUTO_CONNECT=0 to disable.

  • ASCEND_ENABLE_USE_FABRIC_MEM: Requires CANN 9.0+ and HDK 26.0+. Recommended when using Mooncake Store on supported A3 platforms: it can significantly improve transmission performance.

  • ASCEND_USE_ASYNC_TRANSFER: Requires CANN 8.5+. Enables HIXL asynchronous transfer mode, defaults to synchronous mode.

  • ASCEND_GLOBAL_RESOURCE_CONFIG: Configures HIXL global resources. Refer to HIXL documentation for OPTION_GLOBAL_RESOURCE_CONFIG settings.

  • MC_CUSTOM_TOPO_JSON: Path to a NIC topology JSON file; Transfer Engine also uses it to configure HIXL’s comm_resource_config.nic_topo_path (npu:<user_device_id> key) for Host RoCE endpoint generation. Under TENT, this is read from topology/custom_json_path (set via MC_TENT_CONF or MC_CUSTOM_TOPO_JSON); when both are set, MC_CUSTOM_TOPO_JSON takes precedence.

Important Notes#

  1. Device Setup Required: Before calling TransferEngine.initialize(), you must set the device (e.g., torch.npu.set_device(0)).

  2. TCP Connection: Transport internally establishes a host-side TCP connection and automatically searches for available ports. An error will be reported if no ports are available.

  3. Memory Alignment for HCCS: When using the HCCS communication protocol, registered memory addresses must be aligned with page tables (device memory: 2MB alignment). Registering a remote-accessible device buffer whose base address is not 2MB-aligned (for example, a sub-block from the torch_npu caching allocator) fails immediately with ERR_INVALID_ARGUMENT instead of failing later at connection time.

  4. Configuration File: Ensure /etc/hccn.conf exists, especially in containers. Mount /etc/hccn.conf or copy the host file to the container’s /etc path.

  5. Timeout Configuration:

    • Use the ASCEND_CONNECT_TIMEOUT environment variable to control link establishment timeout (default: 10 seconds)

    • Use the ASCEND_TRANSFER_TIMEOUT environment variable to control data transfer timeout (default: 10 seconds)

    • When the caller drives transfers through the Python API, keep the outer MC_TRANSFER_TIMEOUT (seconds, default 30) strictly greater than twice ASCEND_TRANSFER_TIMEOUT (milliseconds, default 10000). A transfer that hits the inner timeout is retried, so the inner budget can be spent twice; the outer deadline has to cover both attempts. If the outer timeout fires first, the transfer can still be in flight inside the Ascend transport, and the caller cannot tell whether it has quiesced — so do not reuse or unregister the target buffer until the transfer is known to have drained. Ascend Direct checks this when it initializes a transfer executor and logs a warning naming both values if the outer deadline is too small.

  6. RDMA Timeout and Retry Configuration:

    • Use HCCL_RDMA_TIMEOUT to configure the RDMA NIC packet retransmission timeout coefficient. The actual packet retransmission timeout is 4.096us * 2 ^ $HCCL_RDMA_TIMEOUT

    • Use HCCL_RDMA_RETRY_CNT to configure the RDMA NIC retransmission count

    • It is recommended to configure ASCEND_TRANSFER_TIMEOUT to be slightly larger than retransmission_timeout * HCCL_RDMA_RETRY_CNT

  7. Communication Protocol Selection: Within A2 servers/A3 supernodes, the default communication protocol is HCCS. You can specify RDMA by setting export HCCL_INTRA_ROCE_ENABLE=1.

  8. RDMA Configuration: When using the RDMA communication protocol, in scenarios where switch and NIC default configurations are inconsistent or traffic planning is required, you may need to modify the RDMA NIC’s Traffic Class and Service Level configuration:

    • Set Traffic Class using the ASCEND_RDMA_TC environment variable

    • Set Service Level using the ASCEND_RDMA_SL environment variable

  9. Buffer Pool Configuration: Under constrained scenarios, transmission can be achieved by using intermediate buffers. The specific way to enable this is by configuring the ASCEND_BUFFER_POOL environment variable in the format BUFFER_NUM:BUFFER_SIZE (in MB). The recommended size is 4:8, which can be adjusted to the most suitable configuration based on actual scenarios.

  10. Async transfer: The asynchronous transfer mode can be enabled by configuring the ASCEND_USE_ASYNC_TRANSFER environment variable.

  11. Fabric Memory mode: On the A3, with the latest drivers and CANN installed, when using Mooncake store, the ASCEND_ENABLE_USE_FABRIC_MEM environment variable can be set to enable fabric memory transfer mode (which allows direct access remote HOST memory).

  12. Auto Connect: The auto connect feature is enabled by default (ASCEND_AUTO_CONNECT=1). Set ASCEND_AUTO_CONNECT=0 to disable.

Contents