Tüm sade metinleri Scrapy ile bir web sitesinden nasıl alabilirim?

Question

HTML'nin oluşturulmasından sonra tüm metnin bir web sitesinden görünmesini istiyorum. Python'da Scrapy çatısı ile çalışıyorum. xpath('//body//text()') ile bunu elde edebiliyorum ancak HTML etiketleriyle sadece metni istiyorum. Bunun için herhangi bir çözüm var mı? Teşekkürler !

Answer 1

kolay seçenek bulundu extract//body//text() ve join her şeyi olurdu.

Başka bir seçenek kullanmaktır nltk 'ın clean_html():

>>> import nltk 
>>> html = """ 
... <div class="post-text" itemprop="description"> 
... 
...   <p>I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework. 
... With <code>xpath('//body//text()')</code> I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !</p> 
... 
...  </div>""" 
>>> nltk.clean_html(html) 
"I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.\nWith xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !" 

Başka bir seçenek kullanmaktır BeautifulSoup' ın ​​get_text():

get_text()

If you only want the text part of a document or tag, you can use the get_text() method. It returns all the text in a document or beneath a tag, as a single Unicode string.

>>> from bs4 import BeautifulSoup 
>>> soup = BeautifulSoup(html) 
>>> print soup.get_text().strip() 
I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework. 
With xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks ! 

Başka bir seçenek kullanmaktır lxml.html 'ın text_content() :

.text_content()

Returns the text content of the element, including the text content of its children, with no markup.

>>> import lxml.html 
>>> tree = lxml.html.fromstring(html) 
>>> print tree.text_content().strip() 
I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework. 
With xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks ! 

Answer 2

Denediniz mi?

''.join(sel.select("//body//text()").extract()).strip() 

sel bir Selector örneğidir:

xpath('//body//text()').re('(\w+)') 

VEYA

xpath('//body//text()').extract()