U yh @sddlZddlZddlZddlmZddlmZddlmZ ddl m Z ddl m Z dddgZe eZd Zd Zd ZdeeeeeeeedddZddZddZejdddZdS)N)closing)Optional) get_logger)barriercreate_c10d_store get_free_portget_socket_with_portzAddress already in usezSocket Timeoutz_tcp_store/num_membersXT) is_server server_addr server_port world_sizetimeoutwait_for_workers use_libuvc CsJ|dkr$|dkr$td|d||dkr8td||dkrD|nd}|dkrV|} nt} td|| ||||z^tjtj|| ||tj |d|d} |dkr| } n | |d } |rt | ||d td | WSt k rB} zRt | t kr0||krtd | |||d7}nt d |d| d| nW5d} ~ XYqHXqHdS)Nr r zCserver_port must be specified when world_size > 1, got server_port=z , world_size=z+sever_port: %s, specified, ignoring retriesziCreating c10d store on %s:%s world_size : %s is_server : %s timeout(sec): %s use_libuv : %s )seconds)Z host_nameportrZ is_masterrr)r)rzSuccessfully created c10d storez)port: %s already in use, attempt: [%s/%s]zon z, port: z already in use) ValueErrorloggerinfor functoolspartialdistZTCPStoredatetime timedelta_check_full_rank RuntimeErrorstr_ADDRESS_IN_USEwarning) r rrrrrretriesrattemptrZ store_builderstoreer']/var/www/html/venv/lib/python3.8/site-packages/torch/distributed/elastic/utils/distributed.pyrsh         c Cs^zt||t|dWnDtk rX}z&t|tkrFtd|d|nW5d}~XYnXdS)N)Z key_prefixZbarrier_timeoutztimed out waiting for all z members to join)r_TCP_STORE_INITrr _SOCKET_TIMEOUT TimeoutError)r%rrr&r'r'r(rhs  rc Cs2t}t||dW5QRSQRXdS)Nr )rr getsockname)sockr'r'r(rts )returnc Cstjddtjtjd}|D]x}|\}}}}}t|||}z|d|d|WStk r}z|tj d|dW5d}~XYqXqt ddS) a Returns a free port on localhost that is "reserved" by binding a temporary socket on it. Close the socket before passing the port to the entity that requires it. Usage example :: sock = _get_socket_with_port() with closing(sock): port = sock.getsockname()[1] sock.close() # there is still a race-condition that some other process # may grab this port before func() runs func(port) localhostN)hostrfamilytype)r/rrzSocket creation attempt failed.)exc_infozFailed to create a socket) socket getaddrinfo AF_UNSPEC SOCK_STREAMbindlistenOSErrorcloserr"r)addrsaddrr1r2proto_sr&r'r'r(rzs"   ")r r r Tr N)rrr4 contextlibrtypingrZtorch.distributed distributedrZ'torch.distributed.elastic.utils.loggingrZ%torch.distributed.elastic.utils.storer__all____name__rr!r*r)boolr intfloatrrrrr'r'r'r( s<       K