편집 기록

편집 기록
  • 프로필daewon님의 편집
    날짜2022.01.10

    셀레니움과 BS4 사용시 Max retries exceeded with url & [SSL: CERTIFICATE_VERIFY_FAILED] unable to get local issuer certificate 오류


    안녕하세요, qpldocs.dla.mil/search/parts.aspx?qpl=1780 상기의 url에서 아래 이미지에 나오는 리스트들을 텍스트로 추출하고자 합니다.(이후 각 리스트들을 타고 들어가 추가 크롤링을 할 예정이므로, 단순 BS4만의 코딩이 아닌 셀레니움의 사용이 필수적으로 필요합니다.)

    이미지

    이를 위해 beautifulsoup을 통해 lxml과 for문을 통해 리스트를 추출하려고 했는데요, lxml를 사용 하려고만 하면 하기와 같이 오류가 발생합니다.

    
    >  Max retries exceeded with url: /search/parts.aspx?qpl=1780
      (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate 
    

    beautiful을 쓰지 않고 셀레니움만 사용해서 1개의 리스트만 출력해보면 오류가 뜨지 않고 잘 작동합니다. (코드에 작성되어있는 #오류가 발생하는 3줄 주석 처리하면 작동하는 것 확인 가능합니다.

    Install Certificates.command 다시 실행 해보았으나 문제가 사라지지 않네요. 소중한 답변 부탁드리겠습니다. 감사합니다!

    제 설명이 부족할 경우 댓글 주시면 더욱 자세히 설명 드릴 수 있도록 하겠습니다!

    아래는 저의 코드 입니다.

    import requests
    from bs4 import BeautifulSoup
    import os
    from selenium import webdriver
    from selenium.webdriver.common.by import By
    from selenium.webdriver.common.keys import Keys
    from selenium.webdriver.chrome.service import Service
    from webdriver_manager.chrome import ChromeDriverManager
    import time
    from urllib.request import urlopen
    
    
    def set_chrome_driver():
        chrome_options = webdriver.ChromeOptions()
        driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=chrome_options)
        return driver
    
    driver = webdriver.Chrome("./chromedriver")
    url = "https://qpldocs.dla.mil/search/parts.aspx?qpl=1780"
    driver.get(url)
    #오류가 발생하는 아래 3줄
    res = requests.get(url)
    res.raise_for_status()
    soup = BeautifulSoup(res.text,"lxml")
    
    partlist = driver.find_element(By.ID, "Lu_gov_DG_ctl03_btnGovPartNo")
    print(partlist.text)
    
    
    driver.find_element(By.XPATH,"//*[@id='Lu_gov_DG_ctl03_btnGovPartNo']").click()
    
    partauthorizationcompay = driver.find_element(By.ID, "Lu_man_DG_ctl03_lblCompany")
    print(partauthorizationcompay.text)
    
    partstatus = driver.find_element(By.ID, "Lu_man_DG_ctl03_imgCompanyStatus")
    print(partstatus.text)
    
    

    하기는 오류 출력 결과를 그대로 복사 붙여넣기 한 것입니다!

    > /Users/seoyeonghun/Desktop/Project/WebCrawling_QPD/practice3.py:20: DeprecationWarning: executable_path has been deprecated, please pass in a Service object
      driver = webdriver.Chrome("./chromedriver")
    Traceback (most recent call last):
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 699, in urlopen
        httplib_response = self._make_request(
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 382, in _make_request
        self._validate_conn(conn)
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
        conn.connect()
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connection.py", line 416, in connect
        self.sock = ssl_wrap_socket(
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
        ssl_sock = _ssl_wrap_socket_impl(
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
        return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 512, in wrap_socket
        return self.sslsocket_class._create(
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 1070, in _create
        self.do_handshake()
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 1341, in do_handshake
        self._sslobj.do_handshake()
    ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)
    
    > During handling of the above exception, another exception occurred:
    
    > Traceback (most recent call last):
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/adapters.py", line 439, in send
        resp = conn.urlopen(
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 755, in urlopen
        retries = retries.increment(
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/util/retry.py", line 574, in increment
        raise MaxRetryError(_pool, url, error or ResponseError(cause))
    urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='qpldocs.dla.mil', port=443): Max retries exceeded with url: /search/parts.aspx?qpl=1780 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
    
    > During handling of the above exception, another exception occurred:
    
    > Traceback (most recent call last):
      File "/Users/seoyeonghun/Desktop/Project/WebCrawling_QPD/practice3.py", line 23, in <module>
        res = requests.get(url)
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/api.py", line 75, in get
        return request('get', url, params=params, **kwargs)
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/api.py", line 61, in request
        return session.request(method=method, url=url, **kwargs)
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/sessions.py", line 542, in request
        resp = self.send(prep, **send_kwargs)
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/sessions.py", line 655, in send
        r = adapter.send(request, **kwargs)
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/adapters.py", line 514, in send
        raise SSLError(e, request=request)
    requests.exceptions.SSLError: HTTPSConnectionPool(host='qpldocs.dla.mil', port=443): Max retries exceeded with url: /search/parts.aspx?qpl=1780 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))
    

    감사합니다.

  • 프로필초보님의 편집
    날짜2022.01.09

    셀레니움과 BS4 사용시 Max retries exceeded with url & [SSL: CERTIFICATE_VERIFY_FAILED] unable to get local issuer certificate 오류


    안녕하세요, qpldocs.dla.mil/search/parts.aspx?qpl=1780 상기의 url에서 아래 이미지에 나오는 리스트들을 텍스트로 추출하고자 합니다.(이후 각 리스트들을 타고 들어가 추가 크롤링을 할 예정이므로, 단순 BS4만의 코딩이 아닌 셀레니움의 사용이 필수적으로 필요합니다.) 이미지

    이를 위해 beautifulsoup을 통해 lxml과 for문을 통해 리스트를 추출하려고 했는데요, lxml를 사용 하려고만 하면 하기와 같이 오류가 발생합니다.

    >  Max retries exceeded with url: /search/parts.aspx?qpl=1780 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate 
    

    라며 오류가 뜹니다..ㅠㅠ beautiful을 쓰지 않고 셀레니움만 사용해서 1개의 리스트만 출력해보면 오류가 뜨지 않고 잘 작동합니다. (코드에 작성되어있는 #오류가 발생하는 3줄 주석 처리하면 작동하는 것 확인 가능합니다.

    Install Certificates.command 다시 실행 해보았으나 문제가 사라지지 않네요..ㅠㅠ 소중한 답변 부탁드리겠습니다. 감사합니다!

    제 설명이 부족할 경우 댓글 주시면 더욱 자세히 설명 드릴 수 있도록 하겠습니다!

    아래는 저의 코드 입니다.

    import requests
    from bs4 import BeautifulSoup
    import os
    from selenium import webdriver
    from selenium.webdriver.common.by import By
    from selenium.webdriver.common.keys import Keys
    from selenium.webdriver.chrome.service import Service
    from webdriver_manager.chrome import ChromeDriverManager
    import time
    from urllib.request import urlopen
    
    
    def set_chrome_driver():
        chrome_options = webdriver.ChromeOptions()
        driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=chrome_options)
        return driver
    
    driver = webdriver.Chrome("./chromedriver")
    url = "https://qpldocs.dla.mil/search/parts.aspx?qpl=1780"
    driver.get(url)
    #오류가 발생하는 아래 3줄
    res = requests.get(url)
    res.raise_for_status()
    soup = BeautifulSoup(res.text,"lxml")
    
    partlist = driver.find_element(By.ID, "Lu_gov_DG_ctl03_btnGovPartNo")
    print(partlist.text)
    
    
    driver.find_element(By.XPATH,"//*[@id='Lu_gov_DG_ctl03_btnGovPartNo']").click()
    
    partauthorizationcompay = driver.find_element(By.ID, "Lu_man_DG_ctl03_lblCompany")
    print(partauthorizationcompay.text)
    
    partstatus = driver.find_element(By.ID, "Lu_man_DG_ctl03_imgCompanyStatus")
    print(partstatus.text)
    
    

    하기는 오류 출력 결과를 그대로 복사 붙여넣기 한 것입니다!

    > /Users/seoyeonghun/Desktop/Project/WebCrawling_QPD/practice3.py:20: DeprecationWarning: executable_path has been deprecated, please pass in a Service object
      driver = webdriver.Chrome("./chromedriver")
    Traceback (most recent call last):
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 699, in urlopen
        httplib_response = self._make_request(
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 382, in _make_request
        self._validate_conn(conn)
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
        conn.connect()
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connection.py", line 416, in connect
        self.sock = ssl_wrap_socket(
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
        ssl_sock = _ssl_wrap_socket_impl(
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
        return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 512, in wrap_socket
        return self.sslsocket_class._create(
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 1070, in _create
        self.do_handshake()
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 1341, in do_handshake
        self._sslobj.do_handshake()
    ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)
    
    > During handling of the above exception, another exception occurred:
    
    > Traceback (most recent call last):
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/adapters.py", line 439, in send
        resp = conn.urlopen(
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 755, in urlopen
        retries = retries.increment(
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/util/retry.py", line 574, in increment
        raise MaxRetryError(_pool, url, error or ResponseError(cause))
    urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='qpldocs.dla.mil', port=443): Max retries exceeded with url: /search/parts.aspx?qpl=1780 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
    
    > During handling of the above exception, another exception occurred:
    
    > Traceback (most recent call last):
      File "/Users/seoyeonghun/Desktop/Project/WebCrawling_QPD/practice3.py", line 23, in <module>
        res = requests.get(url)
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/api.py", line 75, in get
        return request('get', url, params=params, **kwargs)
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/api.py", line 61, in request
        return session.request(method=method, url=url, **kwargs)
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/sessions.py", line 542, in request
        resp = self.send(prep, **send_kwargs)
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/sessions.py", line 655, in send
        r = adapter.send(request, **kwargs)
      File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/adapters.py", line 514, in send
        raise SSLError(e, request=request)
    requests.exceptions.SSLError: HTTPSConnectionPool(host='qpldocs.dla.mil', port=443): Max retries exceeded with url: /search/parts.aspx?qpl=1780 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))
    

    감사합니다.

  • 프로필서영훈님의 편집
    날짜2022.01.09

    셀레니움과 BS4 사용시 Max retries exceeded with url & [SSL: CERTIFICATE_VERIFY_FAILED] unable to get local issuer certificate 오류


    안녕하세요, qpldocs.dla.mil/search/parts.aspx?qpl=1780 상기의 url에서 아래 이미지에 나오는 리스트들을 텍스트로 추출하고자 합니다.(이후 각 리스트들을 타고 들어가 추가 크롤링을 할 예정이므로, 단순 BS4만의 코딩이 아닌 셀레니움의 사용이 필수적으로 필요합니다.) 이미지

    이를 위해 beautifulsoup을 통해 lxml과 for문을 통해 리스트를 추출하려고 했는데요, lxml를 사용 하려고만 하면 하기와 같이 오류가 발생합니다.

    Max retries exceeded with url: /search/parts.aspx?qpl=1780 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate

    라며 오류가 뜹니다..ㅠㅠ beautiful을 쓰지 않고 셀레니움만 사용해서 1개의 리스트만 출력해보면 오류가 뜨지 않고 잘 작동합니다. (코드에 작성되어있는 #오류가 발생하는 3줄 주석 처리하면 작동하는 것 확인 가능합니다.

    Install Certificates.command 다시 실행 해보았으나 문제가 사라지지 않네요..ㅠㅠ 소중한 답변 부탁드리겠습니다. 감사합니다!

    제 설명이 부족할 경우 댓글 주시면 더욱 자세히 설명 드릴 수 있도록 하겠습니다!

    아래는 저의 코드 입니다.

    import requests
    from bs4 import BeautifulSoup
    import os
    from selenium import webdriver
    from selenium.webdriver.common.by import By
    from selenium.webdriver.common.keys import Keys
    from selenium.webdriver.chrome.service import Service
    from webdriver_manager.chrome import ChromeDriverManager
    import time
    from urllib.request import urlopen
    
    
    def set_chrome_driver():
        chrome_options = webdriver.ChromeOptions()
        driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=chrome_options)
        return driver
    
    driver = webdriver.Chrome("./chromedriver")
    url = "https://qpldocs.dla.mil/search/parts.aspx?qpl=1780"
    driver.get(url)
    #오류가 발생하는 아래 3줄
    res = requests.get(url)
    res.raise_for_status()
    soup = BeautifulSoup(res.text,"lxml")
    
    partlist = driver.find_element(By.ID, "Lu_gov_DG_ctl03_btnGovPartNo")
    print(partlist.text)
    
    
    driver.find_element(By.XPATH,"//*[@id='Lu_gov_DG_ctl03_btnGovPartNo']").click()
    
    partauthorizationcompay = driver.find_element(By.ID, "Lu_man_DG_ctl03_lblCompany")
    print(partauthorizationcompay.text)
    
    partstatus = driver.find_element(By.ID, "Lu_man_DG_ctl03_imgCompanyStatus")
    print(partstatus.text)
    
    

    하기는 오류 출력 결과를 그대로 복사 붙여넣기 한 것입니다!

    /Users/seoyeonghun/Desktop/Project/WebCrawling_QPD/practice3.py:20: DeprecationWarning: executable_path has been deprecated, please pass in a Service object driver = webdriver.Chrome("./chromedriver") Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 699, in urlopen httplib_response = self.make_request( File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 382, in _make_request self._validate_conn(conn) File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn conn.connect() File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connection.py", line 416, in connect self.sock = ssl_wrap_socket( File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/util/ssl.py", line 449, in ssl_wrap_socket ssl_sock = ssl_wrap_socket_impl( File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/util/ssl.py", line 493, in _ssl_wrap_socket_impl return ssl_context.wrap_socket(sock, server_hostname=server_hostname) File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 512, in wrap_socket return self.sslsocket_class._create( File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 1070, in _create self.do_handshake() File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 1341, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 755, in urlopen retries = retries.increment( File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/util/retry.py", line 574, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='qpldocs.dla.mil', port=443): Max retries exceeded with url: /search/parts.aspx?qpl=1780 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last): File "/Users/seoyeonghun/Desktop/Project/WebCrawling_QPD/practice3.py", line 23, in res = requests.get(url) File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/api.py", line 75, in get return request('get', url, params=params, **kwargs) File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/sessions.py", line 655, in send r = adapter.send(request, **kwargs) File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/adapters.py", line 514, in send raise SSLError(e, request=request) requests.exceptions.SSLError: HTTPSConnectionPool(host='qpldocs.dla.mil', port=443): Max retries exceeded with url: /search/parts.aspx?qpl=1780 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))

    감사합니다.

ᕕ( ᐛ )ᕗ
로그인이 필요합니다

Hashcode는 개발자들을 위한 무료 QnA사이트 입니다. 계정을 생성하셔야만 글을 작성하실 수 있습니다.