python - Edit* Having trouble writing a crawler that will scrape a directory of folders of websites and write that info to a .csv -


i've included os.walk in still won't output .csv , i'm not sure it's reading html dir. have directory of offline websites need scrape info (e.g. url, email, name, phone etc) , output .csv. when run (i know it's far being executable,) hangs on line 13 permission denied error. 

import os, csv bs4 import beautifulsoup   def main(folder, outputfile):     open(outputfile, "wb") f:         w = csv.writer(f)         header = ("tag", "name", "name", "email", "phone", "location", "url")         w.writerow(header)         rows = crawlhtmls(folder)         w.writerows(rows)  def crawlhtmls(folder):         root, dirs, files in os.walk(folder):     f in files:         if f.lower().endswith(".html"):         soup = beautifulsoup(f.read())         events = soup.findall('div', attrs={'class': 'post'})         headline = x.find('h2')         name = x.find('')         email = x.find('address')          phone = x.find('tel')         description = x.find('div', attrs={'class': 'entry'})          headline2 = str(headline)         name2 = str(name)         email2 = str(name)         phone2 = str(phone)          description2 = str(description)          headline3 = headline2.replace(",", " -")         name3 = name2.replace(",", " -")         email3 = email2.replace(",", " -")         phone3 = phone2.replace(",", " -")         description3 = description2.replace(",", " -")          headline4 = headline3.replace('<h2 class', "")         headline5 = headline4.replace('</h2>', "")         headline6 = headline5.replace('- ', "")         headline7 = headline6.replace("at ", "")          description4 = description3.replace('[<p>', "")         description5 = description4.replace('</p>]', "")         description6 = description5.replace('\n', " ")         description7 = description6.replace('[]', "")          link4 = link3.replace('<a href', "")         link5 = link4.replace('</a>', "")         link6 = link5.replace('h2', " ")         link7 = link6.replace('=', "")          seq = (headline7, name3, email3, phone3, descripton7)         yield seq  if __name__ == "__main__":     folderpath = r"c:\projects\training\html"      output = r"c:\projects\training\about.csv"      main(folderpath, output)


Comments

Post a Comment

Popular posts from this blog

linux - xterm copying to CLIPBOARD using copy-selection causes automatic updating of CLIPBOARD upon mouse selection -

i've tried various ways coerce xterm (versions 285 , 292) copy selection clipboard clipboard whenever press ctrl-shift-c. promising way far has been, in ~/.xresources, put this: xterm*vt100.translations: #override \ ctrl shift <keypress> c: copy-selection(clipboard) \n\ ctrl shift <keypress> v: insert-selection(clipboard) ctrl-shift-v works perfectly, copying has nuance... if restart xterm, highlighting text puts things in primary clipboard; expected, proper, default behavior. if hit ctrl-shift-c, copies current selection clipboard clipboard. the bug, however, if highlight text after first time pressed ctrl-shift-c, you'll see highlighting copies both primary and clipboard. cannot figure out how stop xterm updating clipboard upon selection... , doesn't make sense me in first place. i said, @ specific point in time, copy selection clipboard... yet starts updating automatically upon selection after doing once... anyone have workaround...
Read more

qt - Errors in generated MOC files for QT5 from cmake -

i generated moc files qt5 using set (cmake_automoc on) set(cmake_include_current_dir on) then add moc files src using set(src src/main.cpp src/video_widget_surface.cpp src/video_widget.cpp src/video_player.cpp #moc files moc/moc_video_player.cpp moc/moc_video_widget.cpp moc/moc_video_widget_surface.cpp finally add executable using add_executable(somegui ${src}) but errors in moc files saying : /other/workspace/perception/somestuff/moc/moc_video_widget.cpp:54:6: error: 'videowidget' has not been declared /other/workspace/perception/somestuff/moc/moc_video_widget.cpp:62:19: error: 'videowidget' has not been declared /other/workspace/perception/somestuff/moc/moc_video_widget.cpp:68:20: error: 'videowidget' has not been declared /other/workspace/perception/somestuff/moc/moc_video_widget.cpp:68:46: error: non-member function 'const qmetaobject* metaobject()' cannot have cv-qualifier /other/qt5.0.1/5.0.1/gcc_64/include/qtcore/qo...
Read more