안녕하세요, qpldocs.dla.mil/search/parts.aspx?qpl=1780 상기의 url에서 아래 이미지에 나오는 리스트들을 텍스트로 추출하고자 합니다.(이후 각 리스트들을 타고 들어가 추가 크롤링을 할 예정이므로, 단순 BS4만의 코딩이 아닌 셀레니움의 사용이 필수적으로 필요합니다.)
이를 위해 beautifulsoup을 통해 lxml과 for문을 통해 리스트를 추출하려고 했는데요, lxml를 사용 하려고만 하면 하기와 같이 오류가 발생합니다.
> Max retries exceeded with url: /search/parts.aspx?qpl=1780
(Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate
beautiful을 쓰지 않고 셀레니움만 사용해서 1개의 리스트만 출력해보면 오류가 뜨지 않고 잘 작동합니다. (코드에 작성되어있는 #오류가 발생하는 3줄 주석 처리하면 작동하는 것 확인 가능합니다.
Install Certificates.command 다시 실행 해보았으나 문제가 사라지지 않네요. 소중한 답변 부탁드리겠습니다. 감사합니다!
제 설명이 부족할 경우 댓글 주시면 더욱 자세히 설명 드릴 수 있도록 하겠습니다!
아래는 저의 코드 입니다.
import requests
from bs4 import BeautifulSoup
import os
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
import time
from urllib.request import urlopen
def set_chrome_driver():
chrome_options = webdriver.ChromeOptions()
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=chrome_options)
return driver
driver = webdriver.Chrome("./chromedriver")
url = "https://qpldocs.dla.mil/search/parts.aspx?qpl=1780"
driver.get(url)
#오류가 발생하는 아래 3줄
res = requests.get(url)
res.raise_for_status()
soup = BeautifulSoup(res.text,"lxml")
partlist = driver.find_element(By.ID, "Lu_gov_DG_ctl03_btnGovPartNo")
print(partlist.text)
driver.find_element(By.XPATH,"//*[@id='Lu_gov_DG_ctl03_btnGovPartNo']").click()
partauthorizationcompay = driver.find_element(By.ID, "Lu_man_DG_ctl03_lblCompany")
print(partauthorizationcompay.text)
partstatus = driver.find_element(By.ID, "Lu_man_DG_ctl03_imgCompanyStatus")
print(partstatus.text)
하기는 오류 출력 결과를 그대로 복사 붙여넣기 한 것입니다!
> /Users/seoyeonghun/Desktop/Project/WebCrawling_QPD/practice3.py:20: DeprecationWarning: executable_path has been deprecated, please pass in a Service object
driver = webdriver.Chrome("./chromedriver")
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
conn.connect()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connection.py", line 416, in connect
self.sock = ssl_wrap_socket(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 512, in wrap_socket
return self.sslsocket_class._create(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 1070, in _create
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 1341, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/util/retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='qpldocs.dla.mil', port=443): Max retries exceeded with url: /search/parts.aspx?qpl=1780 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
File "/Users/seoyeonghun/Desktop/Project/WebCrawling_QPD/practice3.py", line 23, in <module>
res = requests.get(url)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='qpldocs.dla.mil', port=443): Max retries exceeded with url: /search/parts.aspx?qpl=1780 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))
감사합니다.