Python Scrapy Xpath not following url -


i new @ python , having bit of trouble getting scrapy follow urls. suspect may xpath specification, after doing several tutorials on topic, no closer resolving this. loops on urls in referenced table , scrapes content starting page repetitively. doing wrong?

code attached: 

import scrapy   scrapy.selector import selector scrapy.spiders import crawlspider scrapy.spiders import rule scrapy.linkextractors import linkextractor scrapy.http import request  class myspider(crawlspider):     name = 'unespider'     allowed_domains = ['https://my.une.edu.au/']     start_urls = ['https://my.une.edu.au/courses/']     rules = rule(linkextractor(canonicalize = true, unique = true), follow = true, callback = "parse"),          def parse(self, response):         hxs = selector(response)         url in response.xpath('//*'):         yield {             'title': url.xpath('//*[@id="main-content"]/div/h2/a/text()').extract_first(),             'avail': url.xpath('//*[@id="overviewtab-snapshotdiv"]/p[3]/a/text()').extract_first(),                }                 url in hxs.xpath('//tr/td/a/@href').extract():             yield request(response.urljoin(url), callback=self.parse)

** update see wanted , updated code, follows each year , outputs correct**

i apologize i'm not sure trying follow , scrape start page //*[@id="overviewtab-snapshotdiv"]. wasn't able find xpath. i'd out more since i'm new programming , scrapy hard @ first, ended making own class scraper later way, though i'm sure scrapy better :) i've done code scrape titles , urls, commented out rule since don't know trying follow or why.

import scrapy scrapy.spiders import rule  scrapy.linkextractors import linkextractor  overflowquestion2 import items #make sure import items.py   class davesimspider(scrapy.spider):      name = 'davesim'     allowed_domains = ['my.une.edu.au']     start_urls = ['http://my.une.edu.au/courses/2007', ] #start @ 2007     rules = rule(linkextractor(canonicalize=true, unique=true), follow=true, callback="parse")   def parse(self, response): #this scrape links , follow      #grab main div wrapping links     divlinkwrapper = response.xpath('//div[@class="pagination"]')      links in divlinkwrapper: #for every element extract links         thelinks = links.xpath('ul/li/a/@href').extract()         in thelinks: #for every link, follow link             yield scrapy.request(i, callback=self.contentparse)    def contentparse(self, response): #scrape content want      #grab main div wrapper content     divmaincontent = response.xpath('//div[@id="main-content"]')      titles in divmaincontent:         #create item object items.py function         item = items.overflowquestion2item()          thetitles = titles.xpath('div[@class="content"]//a/text()').extract()          #set item scrapy.field in items.py         item['title'] = thetitles          yield item #yield item through pipeline      urls in divmaincontent:         item = items.overflowquestion2item()         theurls = urls.xpath('//table/tr/td/a/@href').extract()         item['url'] = theurls         yield item

now items.py:

import scrapy   class overflowquestion2item(scrapy.item):     # define fields item here like:     # name = scrapy.field()     title = scrapy.field()     url = scrapy.field()      pass

also remember in settings uncomment items pipeline,

item_pipelines = {     'overflowquestion2.pipelines.overflowquestion2pipeline': 300, }

i'm positive coded better , hope here refine ;)


Comments

Post a Comment

Popular posts from this blog

Sort a complex associative array in PHP -

this question has answer here: php sort array subarray value 5 answers i have associative array in php like: $newarray1 = array ( [producta] => array ( [link] => http://sanjosespartan.com/blog/products/agile-java-development/ [visitcount] => 4 ) [productd] => array ( [link] => http://sanjosespartan.com/blog/products/intro-to-go/ [visitcount] => 1 ) [productg] => array ( [link] => http://sanjosespartan.com/blog/products/node-js-for-developers/ [visitcount] => 1 ) [productb] => array ( [link] => http://sanjosespartan.com/blog/products/beginning-mysql/ [visitcount] => 1 ) [productc] => array ( [link] => http://sanjosespartan.com/blog/products/computer-networks/ [visitcount] => 1 ) ...
Read more

vb.net - How to ignore if a cell is empty nothing -

how use roscolor() ignore value (like empty cell) ? color row if value upper 5 when there nothing in cell, want ignore roscolor() apply, how? public sub roscolor() integer = 0 quotedatagridview1.rows.count() - 1 step +1 dim val integer val = quotedatagridview1.rows(i).cells(3).value if val = vbempty quotedatagridview1.rows(i).defaultcellstyle.backcolor = color.white elseif val < 5 quotedatagridview1.rows(i).defaultcellstyle.backcolor = color.red elseif val > 5 , val < 10 quotedatagridview1.rows(i).defaultcellstyle.backcolor = color.lightyellow end if next end sub you can check empty value following public sub roscolor() integer = 0 quotedatagridview1.rows.count() - 1 step +1 dim val = quotedatagridview1.rows(i).cells(3).value if isdbnull(val) or val nothing quotedatagridview1.rows(i).defaultcellstyle.backcolor = color.white ...
Read more

recursion - Can every recursive algorithm be improved with dynamic programming? -

i first year undergraduate csc student looking competitive programming. recursion involves defining , solving sub problems. understand, top down dynamic programming (dp) involves memoizing solutions sub problems reduce time complexity of algorithm. can top down dp used improve efficiency of every recursive algorithm overlapping sub problems? dp fail work , how can identify this? the short answer is: yes. however, there constraints. obvious 1 recursive calls must overlap. i.e. during execution of algorithm, recursive function must called multiple times same parameters. lets truncate recursion tree memoization. can use memoization reduce number of calls. however, reduction of calls comes price. need store results somewhere. next obvious constraint need have enough memory. comes not-so obvious constraint. memory access requires time. first need find result stored , maybe copy location. in cases, might faster let recursion calculate result instead of loading somewher...
Read more