Scrapy 使用Item

Item 对象是Python的常规字典。我们可以使用以下语法来访问该类的属性：

>>> item = DmozItem()
>>> item['title'] = 'sample title'
>>> item['title']
'sample title'

将以上代码添加到以下示例：

import scrapy

from tutorial.items import DmozItem

class MyprojectSpider(scrapy.Spider):
   name = "project"
   allowed_domains = ["dmoz.org"]

   start_urls = [
      "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
      "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
   ]
   def parse(self, response):
      for sel in response.xpath('//ul/li'):
         item = DmozItem()
         item['title'] = sel.xpath('a/text()').extract()
         item['link'] = sel.xpath('a/@href').extract()
         item['desc'] = sel.xpath('text()').extract()
         yield item

上面的蜘蛛的输出：

[scrapy] DEBUG: Scraped from <200
http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
   {'desc': [u' - By David Mertz; Addison Wesley. Book in progress, full text,
      ASCII format. Asks for feedback. [author website, Gnosis Software, Inc.\n],
   'link': [u'http://gnosis.cx/TPiP/'],
   'title': [u'Text Processing in Python']}
[scrapy] DEBUG: Scraped from <200
http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
   {'desc': [u' - By Sean McGrath; Prentice Hall PTR, 2000, ISBN 0130211192,
      has CD-ROM. Methods to build XML applications fast, Python tutorial, DOM and
      SAX, new Pyxie open source XML processing library. [Prentice Hall PTR]\n'],
   'link': [u'http://www.informit.com/store/product.aspx?isbn=0130211192'],
   'title': [u'XML Processing with Python']}

Scrapy跟踪链接：在本章中，我们将研究如何提取感兴趣页面的链接，并关注它们并从该页面提取数据。为此，我们需要对以前的代码进行如下更改，如下所示：import scrapyfrom tutorial.items import DmozIt ...