Skip to content

refactor: PortalFetch #20

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
uniquePaul wants to merge 5 commits into
base: master
Choose a base branch
from
Open

refactor: PortalFetch #20

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ebd1810
refactor: PortalFetch
uniquePaul Jul 15, 2021
f2444ee
refactor: fixed variable names and adding comments
uniquePaul Jul 18, 2021
37c8cb8
refactor: able to download series of quartervalue
uniquePaul Jul 27, 2021
38d3396
refactor: able to download series of quartervalues
uniquePaul Jul 27, 2021
3e6f0de
refactor: able to download based on series quartervalue
uniquePaul Jul 27, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 33 additions & 23 deletions PortalFetch/crawler_download.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Fetch course information from De Anza myportal.

It requires file 'user.ini' to load the user's own user name and password.
Expand Down Expand Up @@ -206,34 +207,42 @@ def waitUtilPageLoaded(driver, count):
raise ElementNotVisibleException("Could not load the full page!")


def generateQuarterAndFilename(quarterValue):
def generateQuarterAndFilename(quarterValueStr):
"""Return quarter and filename.

Args:
quarterValue:the quarter_value in crawler.config
Returns:
quarter str and filename str
quarter str list and filename str list

"""
year = quarterValue[0:4]
quarterSwitcher = {
"1": "Summer",
"2": "Fall",
"3": "Winter",
"4": "Spring",
}
schoolSwitcher = {
"1": "Foothill",
"2": "De Anza",
}
school = schoolSwitcher.get(quarterValue[5], "")
quarter = quarterSwitcher.get(quarterValue[4], "")
if quarter == "Summer":
year = str(int(year)-1)
quarterOutput = year + " " + quarter + " " + school
if school == "De Anza":
school = "De_Anza"
fileNameOutput = year + "_" + quarter + "_" + school + "_courseData.json"
n = 6

@hk-mp5a3 hk-mp5a3 Jul 15, 2021

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

avoid naming "n". the naming should be meaningful here. moreover, if a number is a constant, please use upper case. reference:https://www.python.org/dev/peps/pep-0008/#constants

quarterValueList = [quarterValueStr[i:i+n] for i in range(0, len(quarterValueStr), n)]
fileNameOutput = []
quarterOutput = []
for quarterValue in quarterValueList:
year = quarterValue[0:4]
quarterSwitcher = {
"1": "Summer",
"2": "Fall",
"3": "Winter",
"4": "Spring",
}
schoolSwitcher = {
"1": "Foothill",
"2": "De Anza",
}
school = schoolSwitcher.get(quarterValue[5], "")
quarter = quarterSwitcher.get(quarterValue[4], "")
if quarter == "Summer":
Comment thread
yifeili98 marked this conversation as resolved.
Outdated
year = str(int(year)-1)

quarterOutput.append(year + " " + quarter + " " + school)
if school == "De Anza":
school = "De_Anza"

fileNameOutput.append(year + "_" + quarter + "_" + school + "_courseData.json")

return quarterOutput, fileNameOutput


Expand Down Expand Up @@ -278,9 +287,10 @@ def main():
# Save searched courses
html = saveResult(driver)
# get quarter and filename based on quarter_value in crawler.config
quarter, filename = generateQuarterAndFilename(value)
quarter_list, filename_list = generateQuarterAndFilename(value)
for i in range(0, len(filename_list)):
Comment thread
yifeili98 marked this conversation as resolved.
Outdated
DataProcess().data_process(html, filename_list[i], quarter_list[i])

DataProcess().data_process(html, filename, quarter)
logging.info("Download Finished!")
except Exception as e:
logger.error(repr(e))
Expand Down