Scraperwiki Twitter Query -


please forgive me, have limited knowledge of scraperwiki , twitter mining.

i have following code scrape twitter data. however, want edit code give me results geotagged new york on particular date (let's say, april 1, 2013). know how should this?

############################################################################### # twitter srcaper term 'hello'. ###############################################################################  import scraperwiki import simplejson  # retrieve page base_url = 'http://search.twitter.com/search.json?q=' q = 'hello' options = '&rpp=10&page=' page = 1  while 1:     try:         url = base_url + q + options + str(page)         html = scraperwiki.scrape(url)         #print html         soup = simplejson.loads(html)         result in soup['results']:             data = {}             data['id'] = result['id']             data['text'] = result['text']             data['from_user'] = result['from_user']             data['created_at'] = result['created_at']             # save records datastore             scraperwiki.datastore.save(["id"], data)         page = page + 1     except:         print str(page) + ' pages scraped'         break

in addition q, use query parameters geocode , until. see page of twitter api documentation. please note cannot use search api find tweets older about week.

besides, it's easier use urllib.urlencode() construct query, example

query_dict = {'q':'search term(s)', 'geocode':'37.781157,-122.398720,25mi', 'until':'2013-05-10'} query = urllib.urlencode(query_dict) response = urllib.urlopen(basic_url + query).read()

update: please see this example scraper can copy , adapt needs.


Comments

Post a Comment

Popular posts from this blog

linux - xterm copying to CLIPBOARD using copy-selection causes automatic updating of CLIPBOARD upon mouse selection -

i've tried various ways coerce xterm (versions 285 , 292) copy selection clipboard clipboard whenever press ctrl-shift-c. promising way far has been, in ~/.xresources, put this: xterm*vt100.translations: #override \ ctrl shift <keypress> c: copy-selection(clipboard) \n\ ctrl shift <keypress> v: insert-selection(clipboard) ctrl-shift-v works perfectly, copying has nuance... if restart xterm, highlighting text puts things in primary clipboard; expected, proper, default behavior. if hit ctrl-shift-c, copies current selection clipboard clipboard. the bug, however, if highlight text after first time pressed ctrl-shift-c, you'll see highlighting copies both primary and clipboard. cannot figure out how stop xterm updating clipboard upon selection... , doesn't make sense me in first place. i said, @ specific point in time, copy selection clipboard... yet starts updating automatically upon selection after doing once... anyone have workaround...
Read more

qt - Errors in generated MOC files for QT5 from cmake -

i generated moc files qt5 using set (cmake_automoc on) set(cmake_include_current_dir on) then add moc files src using set(src src/main.cpp src/video_widget_surface.cpp src/video_widget.cpp src/video_player.cpp #moc files moc/moc_video_player.cpp moc/moc_video_widget.cpp moc/moc_video_widget_surface.cpp finally add executable using add_executable(somegui ${src}) but errors in moc files saying : /other/workspace/perception/somestuff/moc/moc_video_widget.cpp:54:6: error: 'videowidget' has not been declared /other/workspace/perception/somestuff/moc/moc_video_widget.cpp:62:19: error: 'videowidget' has not been declared /other/workspace/perception/somestuff/moc/moc_video_widget.cpp:68:20: error: 'videowidget' has not been declared /other/workspace/perception/somestuff/moc/moc_video_widget.cpp:68:46: error: non-member function 'const qmetaobject* metaobject()' cannot have cv-qualifier /other/qt5.0.1/5.0.1/gcc_64/include/qtcore/qo...
Read more