pool.map 간단한 질문하나만...
조회수 603회
if __name__ == '__main__':
pool = Pool(processes=8)
pool.map(loop1, range(3,10003))
pool.close()
pool.join()
이렇게 해뒀는데 8개의 프로세스가 서로 같은짓을 반복하는게 맞나요? 예를들면 1번~8번 프로세스가 전부 다 3부터 10002까지 대입하면서 똑같은 계산을 하는게 맞나요??
제가 해봤는데 그런거 같아서 질문드립니다.. 만약 맞다면 프로세스 끼리 이미 사용된 argument를 안쓰는 법이 있을까요?
1 답변
-
아니요. 3~10002 까지의 숫자를 워커프로세스에 나누어서 실행시킵니다.
아래 예제를 볼께요.
from multiprocessing import Pool from collections import Counter import multiprocessing as mp import time def f(x): current = mp.current_process() time.sleep(0.0001) return (current.name, x * x) if __name__ == "__main__": with Pool(processes=8) as pool: result = pool.map(f, range(100)) print(result) print(Counter([e[0] for e in result]))
pool 만들고 close 하는 것만 간단하게
with
구문으로 바꾼 거에요. (파이썬 다큐멘테이션의 예제중에 map 부분만 사용해서 조금 변형한 코드입니다.) 0~99 를 8개의 프로세스에 나누어서 실행시키고, 진짜로 그렇게 되는지 확인하기 위해서,multiprocessing.current_process()
로 워커프로세스를 알아내고, 그 이름까지 같이 반환합니다.마지막 print 문에서 Counter 로 결과 중에서 워커프로세스명의 갯수를 세어 봅니다.
결과는 다음과 같았습니다.
[('SpawnPoolWorker-3', 0), ('SpawnPoolWorker-3', 1), ('SpawnPoolWorker-3', 4), ('SpawnPoolWorker-3', 9), ('SpawnPoolWorker-1', 16), ('SpawnPoolWorker-1', 25), ('SpawnPoolWorker-1', 36), ('SpawnPoolWorker-1', 49), ('SpawnPoolWorker-1', 64), ('SpawnPoolWorker-1', 81), ('SpawnPoolWorker-1', 100), ('SpawnPoolWorker-1', 121), ('SpawnPoolWorker-3', 144), ('SpawnPoolWorker-3', 169), ('SpawnPoolWorker-3', 196), ('SpawnPoolWorker-3', 225), ('SpawnPoolWorker-8', 256), ('SpawnPoolWorker-8', 289), ('SpawnPoolWorker-8', 324), ('SpawnPoolWorker-8', 361), ('SpawnPoolWorker-1', 400), ('SpawnPoolWorker-1', 441), ('SpawnPoolWorker-1', 484), ('SpawnPoolWorker-1', 529), ('SpawnPoolWorker-3', 576), ('SpawnPoolWorker-3', 625), ('SpawnPoolWorker-3', 676), ('SpawnPoolWorker-3', 729), ('SpawnPoolWorker-8', 784), ('SpawnPoolWorker-8', 841), ('SpawnPoolWorker-8', 900), ('SpawnPoolWorker-8', 961), ('SpawnPoolWorker-1', 1024), ('SpawnPoolWorker-1', 1089), ('SpawnPoolWorker-1', 1156), ('SpawnPoolWorker-1', 1225), ('SpawnPoolWorker-3', 1296), ('SpawnPoolWorker-3', 1369), ('SpawnPoolWorker-3', 1444), ('SpawnPoolWorker-3', 1521), ('SpawnPoolWorker-8', 1600), ('SpawnPoolWorker-8', 1681), ('SpawnPoolWorker-8', 1764), ('SpawnPoolWorker-8', 1849), ('SpawnPoolWorker-3', 1936), ('SpawnPoolWorker-3', 2025), ('SpawnPoolWorker-3', 2116), ('SpawnPoolWorker-3', 2209), ('SpawnPoolWorker-1', 2304), ('SpawnPoolWorker-1', 2401), ('SpawnPoolWorker-1', 2500), ('SpawnPoolWorker-1', 2601), ('SpawnPoolWorker-8', 2704), ('SpawnPoolWorker-8', 2809), ('SpawnPoolWorker-8', 2916), ('SpawnPoolWorker-8', 3025), ('SpawnPoolWorker-5', 3136), ('SpawnPoolWorker-5', 3249), ('SpawnPoolWorker-5', 3364), ('SpawnPoolWorker-5', 3481), ('SpawnPoolWorker-1', 3600), ('SpawnPoolWorker-1', 3721), ('SpawnPoolWorker-1', 3844), ('SpawnPoolWorker-1', 3969), ('SpawnPoolWorker-3', 4096), ('SpawnPoolWorker-3', 4225), ('SpawnPoolWorker-3', 4356), ('SpawnPoolWorker-3', 4489), ('SpawnPoolWorker-8', 4624), ('SpawnPoolWorker-8', 4761), ('SpawnPoolWorker-8', 4900), ('SpawnPoolWorker-8', 5041), ('SpawnPoolWorker-1', 5184), ('SpawnPoolWorker-1', 5329), ('SpawnPoolWorker-1', 5476), ('SpawnPoolWorker-1', 5625), ('SpawnPoolWorker-3', 5776), ('SpawnPoolWorker-3', 5929), ('SpawnPoolWorker-3', 6084), ('SpawnPoolWorker-3', 6241), ('SpawnPoolWorker-5', 6400), ('SpawnPoolWorker-5', 6561), ('SpawnPoolWorker-5', 6724), ('SpawnPoolWorker-5', 6889), ('SpawnPoolWorker-8', 7056), ('SpawnPoolWorker-8', 7225), ('SpawnPoolWorker-8', 7396), ('SpawnPoolWorker-8', 7569), ('SpawnPoolWorker-1', 7744), ('SpawnPoolWorker-1', 7921), ('SpawnPoolWorker-1', 8100), ('SpawnPoolWorker-1', 8281), ('SpawnPoolWorker-5', 8464), ('SpawnPoolWorker-5', 8649), ('SpawnPoolWorker-5', 8836), ('SpawnPoolWorker-5', 9025), ('SpawnPoolWorker-3', 9216), ('SpawnPoolWorker-3', 9409), ('SpawnPoolWorker-3', 9604), ('SpawnPoolWorker-3', 9801)] Counter({'SpawnPoolWorker-3': 32, 'SpawnPoolWorker-1': 32, 'SpawnPoolWorker-8': 24, 'SpawnPoolWorker-5': 12})
SpawnPoolWorker-3, 1, 8, 5
4개의 워커가 100개를 각각 32, 32, 24, 12개 씩 나누어 수행했습니다. 8개의 워커프로세스로 수행하도록 코딩하였지만, 함수가 오래 걸리지 않아서, 적절히 4개의 워커만 만들어져서 수행된 것으로 보입니다.실제로, f 함수의 time.sleep 을 주지 않으면, 워커 하나만 만들어져서 다 수행하는 결과도 볼 수 있고요. time.sleep 을 좀 더 길게 하거나, range 를 좀 더 늘려서 실행시키면, 8개의 워커의 결과가 모두 나오는 것도 확인할 수 있습니다.
- 제가 짠 스크립트에선 속도가 23.07it/s?? 나오다가 좀지나면 1까지 떨어지는데 Duk Gi Awa Yang 2020.5.19 12:41
- sleep을 주면 오히려 늘어나나요??? 최대한 최적화를 했습니다만.. 레인지는 3~10003이구 openpyxl이랑 shutil로 copy를 쓰고있습니다 Duk Gi Awa Yang 2020.5.19 12:42
- 아 그리고 답변 감사합니다 Duk Gi Awa Yang 2020.5.19 12:43
- sleep는 테스트용 함수 f 가 너무 시간이 안 걸리니까, 오래걸리는 함수를 시뮬레이션 하려는 것 뿐입니다. 그리고 yang님 스크립트에 대해선 단편적인 정보만 가지고 어떻다고 말할 수 없네요. nowp 2020.5.19 12:48
- 아하 의심가는부분 따로 질문 올려두었습니다 감사합니다~! Duk Gi Awa Yang 2020.5.19 13:31
댓글 입력