Я вынужден заниматься программированием "между делом", но регулярно... потому здесь об администрировании Windows, Linux... и веб-мастеринге LAMP, WAMP, Java, JScript, jQuery, CSS, Google API, Drupal, Moodle...
пятница, 6 декабря 2013 г.
Осваиваем технологии web-scrapping (Python)
Повезло, наше "теоретический" материал о скорости парсинга (клик по картинке). Сначала "разложим" в меню все, что уже есть, потом начнем изучение, свои файлы будем складывать на GitHub (?) Сначала читаем статью. С учетом того, что я нашел уже изрядное количество видеороликов, статей и справочников..., пожалуй, можно уже сформулировать первое задание. Далеко ходить не надо. Сначала (скачаем по списку)текстовые файлы из интернета, поместим их в папки, потом в каждом файле распарсим строку URL, потом запишем все в один большой csv файл, потом этот файл проверим и почистим, уберем оттуда NA ячейки... Потом все это разобьем на блоки... Потом подумаем о том, как все это запустить в автоматическом режиме... Блок кода Основной текст Блок 2 Другие посты по теме
Подписаться на:
Комментарии к сообщению (Atom)
Сначала о бибиблотеках (из основной статьи http://www.ibm.com/developerworks/ru/library/x-hiperfparse/):
ОтветитьУдалитьPython никогда не страдал от нехватки XML-библиотек. Начиная с версии 2.0 он включает в себя привычную
xml.dom.minidom и связанные с ней модели
pulldom и
Simple API for XML (SAX).
Начиная с версии 2.4 в него входит широко распространенный
API ElementTree.
К тому же всегда есть независимые библиотеки, которые предоставляют более высокоуровневые или более подходящие для Python интерфейсы.
В то время как для простого DOM- или SAX-анализа небольших файлов достаточно любой XML-библиотеки, разработчики все чаще сталкиваются с крупными наборами данных и необходимостью разбирать XML в реальном времени в контексте Web-сервисов.
УдалитьВ то же время опытные XML-разработчики могут предпочесть более XML-ориентированные языки, такие как
XPath или
XSLT,
за их лаконичность и выразительность.
Идеальным выходом из этой ситуации было бы иметь доступ к декларативному синтаксису XPath, сохраняя при этом общую функциональность, доступную в Python.
Хваленую библиотеку lxml использует пакет scrapy...
ОтветитьУдалитьПосмотрел я видео, почитал документацию. И отзывы хорошие..
http://doc.scrapy.org/en/latest/intro/tutorial.html#intro-tutorial
https://www.youtube.com/playlist?list=PL51BA5190961CFEE3
Надо бы попробовать, но "забыл", как устанавливать эти самые дополнительные пакеты. Особенно под Windows... В скриптах есть файлы, которые так и называются install.py ... точнее - setup.py
Но чего-то не очень они устанавливаются... По опыту борьбы с Django... думаю, что основные проблемы в том, что пути надо прописывать, файрвол норовит заблокировать установку... Дабы не разбираться с этим каждый раз есть пакеты
pip ... easy_install
Если уж разбираться с установкой, то этих установщиков...
Начал с того, что нашел в Анаконде pip.bat
Удалить...не думал долго, а запустил с правами администратора в окне cmd.exe
вот, что получилось
C:\Users\kiss\Anaconda\Scripts>pip.bat
Usage:
pip [options]
Commands:
install Install packages.
uninstall Uninstall packages.
freeze Output installed packages in requirements format.
list List installed packages.
show Show information about installed packages.
search Search PyPI for packages.
wheel Build wheels from your requirements.
zip Zip individual packages.
unzip Unzip individual packages.
bundle Create pybundles.
help Show help for commands.
General Options:
-h, --help Show help.
-v, --verbose Give more output. Option is additive, and can be
used up to 3 times.
-V, --version Show version and exit.
-q, --quiet Give less output.
--log Log file where a complete (maximum verbosity)
record will be kept.
--proxy Specify a proxy in the form
[user:passwd@]proxy.server:port.
--timeout Set the socket timeout (default 15 seconds).
--exists-action Default action when a path already exists:
(s)witch, (i)gnore, (w)ipe, (b)ackup.
--cert Path to alternate CA bundle.
C:\Users\kiss\Anaconda\Scripts>
При работе пришлось руководствоваться вот этой инструкцией
Удалитьhttp://doc.scrapy.org/en/latest/intro/install.html#intro-install
Здесь во первых строках
pip install Scrapy
пробуем сразу после выполнения C:\Users\kiss\Anaconda\Scripts>pip.bat
....
Баааа. Грузится.
Итак, получается, что
по команде
pip install Scrapy
ПИП сам находит в своих репозитариях соотетсвующие пакеты. Дальше грузит их ко мне на компьютер в соответсвующие директории...
Посмотрим, что откуда и куда..., но сначала, как
Вот, что в итоге установилось:
УдалитьC:\Users\kiss\Anaconda\Scripts>pip install Scrapy
Downloading/unpacking Scrapy
Downloading Scrapy-0.20.1.tar.gz (745kB): 745kB downloaded
Running setup.py egg_info for package Scrapy
no previously-included directories found matching 'docs\build'
Downloading/unpacking Twisted>=10.0.0 (from Scrapy)
Downloading Twisted-13.2.0.tar.bz2 (2.7MB): 2.7MB downloaded
Running setup.py egg_info for package Twisted
Downloading/unpacking w3lib>=1.2 (from Scrapy)
Downloading w3lib-1.5.tar.gz
Running setup.py egg_info for package w3lib
Downloading/unpacking queuelib (from Scrapy)
Downloading queuelib-1.1.1.tar.gz
Running setup.py egg_info for package queuelib
Requirement already satisfied (use --upgrade to upgrade): lxml in c:\users\kiss\anaconda\lib\site-packages (from Scrapy
Downloading/unpacking pyOpenSSL (from Scrapy)
Downloading pyOpenSSL-0.13.1.tar.gz (254kB): 254kB downloaded
Running setup.py egg_info for package pyOpenSSL
error: Could not find 'openssl.exe'
Complete output from command python setup.py egg_info:
running egg_info
creating pip-egg-info\pyOpenSSL.egg-info
writing pip-egg-info\pyOpenSSL.egg-info\PKG-INFO
writing top-level names to pip-egg-info\pyOpenSSL.egg-info\top_level.txt
writing dependency_links to pip-egg-info\pyOpenSSL.egg-info\dependency_links.txt
writing manifest file 'pip-egg-info\pyOpenSSL.egg-info\SOURCES.txt'
warning: manifest_maker: standard file '-c' not found
error: Could not find 'openssl.exe'
----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in c:\users\kiss\appdata\local\temp\pip_build_kiss\pyOpenSSL
Storing complete log in C:\Users\kiss\pip\pip.log
C:\Users\kiss\Anaconda\Scripts>
Ну вот, иногда метод тупого тыка очень даже полезен.
УдалитьПочти все ясно. Действительно, pip качает из своего репозитория нужные файлы, проверяет сначала наличие, потом обновления...
И все бы было здорово, если бы я "тупо" выполнил инструкции (фрагмент см. ниже) http://doc.scrapy.org/en/latest/intro/install.html#intro-install
Pre-requisites
The installation steps assume that you have the following things installed:
Python 2.7
lxml. Most Linux distributions ships prepackaged versions of lxml. Otherwise refer to http://lxml.de/installation.html
OpenSSL. This comes preinstalled in all operating systems except Windows (see Platform specific installation notes)
pip or easy_install Python package managers
Так что, надо было установить сначала OpenSSL
Вот та часть из инструкции по установке, которую надо (было) выполнить:
ОтветитьУдалитьPlatform specific installation notes
Windows
After installing Python, follow these steps before installing Scrapy:
add the C:\python27\Scripts and C:\python27 folders to the system path by adding those directories to the PATH environment variable from the Control Panel.
install OpenSSL by following these steps:
go to Win32 OpenSSL page
download Visual C++ 2008 redistributables for your Windows and architecture
download OpenSSL for your Windows and architecture (the regular version, not the light one)
add the c:\openssl-win32\bin (or similar) directory to your PATH, the same way you added python27 in the first step`` in the first step
some binary packages that Scrapy depends on (like Twisted, lxml and pyOpenSSL) require a compiler available to install, and fail if you don’t have Visual Studio installed. You can find Windows installers for those in the following links. Make sure you respect your Python version and Windows architecture.
pywin32: http://sourceforge.net/projects/pywin32/files/
Twisted: http://twistedmatrix.com/trac/wiki/Downloads
zope.interface: download the egg from zope.interface pypi page and install it by running easy_install file.egg
lxml: http://pypi.python.org/pypi/lxml/
pyOpenSSL: https://launchpad.net/pyopenssl
Finally, this page contains many precompiled Python binary libraries, which may come handy to fulfill Scrapy dependencies:
http://www.lfd.uci.edu/~gohlke/pythonlibs/
Насколько я понял "download Visual C++ 2008 redistributables for your Windows and architecture" нужно только в том случае, если на компьютере не стоит Visual Studio... Полагаю, что мой express - это то, что надо... так что пока качать не буду...
УдалитьА вот сейчас качаю:
http://slproweb.com/download/Win64OpenSSL-1_0_1e.exe
А потом проверю все пути и пропишу новый
"add the c:\openssl-win32\bin (or similar) directory to your PATH, the same way you added python27 in the first step"
В процессе установки выскочило сообщение, что сначала надо установить
Удалитьhttp://www.microsoft.com/en-us/download/details.aspx?id=15336
The Microsoft Visual C++ 2008 Redistributable Package (x86) installs runtime components of Visual C++ Libraries required to run applications developed with Visual C++ on a computer that does not have Visual C++ 2008 installed.
Устанавливаю
смотрим пути (переменная Path из "нижнего" окна - "Переменные среды")
УдалитьC:\HashiCorp\Vagrant\bin;
C:\Users\kiss\Anaconda\Scripts;
C:\OpenSSL-Win64\bin
Исправил... не было пут в анаконду (а сразу в папку со скриптами)...
УдалитьВот теперь, что у меня в путях:
C:\Program Files (x86)\AMD APP\bin\x86_64;
C:\Program Files (x86)\AMD APP\bin\x86;
%SystemRoot%\system32;
%SystemRoot%;
%SystemRoot%\System32\Wbem;
%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;
C:\Program Files (x86)\ATI Technologies\ATI.ACE\Core-Static;
C:\Program Files (x86)\Windows Live\Shared;C:\Program Files\Java\jdk1.7.0_21\bin;
C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;
C:\HashiCorp\Vagrant\bin;
C:\Users\kiss\Anaconda;
C:\Users\kiss\Anaconda\Scripts;
C:\OpenSSL-Win64\bin
После перезагрузки опять та же ошибка...
ОтветитьУдалитьЗадал поиск по тексту ошибки и нашел
http://stackoverflow.com/questions/4750806/how-to-install-pip-on-windows (слишком запутано)
А вот то, что сработало:
http://stackoverflow.com/questions/10688843/trying-to-install-scrapy-error-could-not-find-openssl-exe
Отсюда https://pypi.python.org/pypi/pyOpenSSL/0.13
загрузил наугад
https://pypi.python.org/packages/2.7/p/pyOpenSSL/pyOpenSSL-0.13.winxp32-py2.7.msi#md5=3438c0e3a87060959d7061f939c6b03d
Установил...
А потом и пип установился вроде бы
ОтветитьУдалитьC:\Users\kiss\Anaconda\Scripts>pip install Scrapy
Downloading/unpacking Scrapy
Downloading Scrapy-0.20.1.tar.gz (745kB): 745kB downloaded
Running setup.py egg_info for package Scrapy
no previously-included directories found matching 'docs\build'
Downloading/unpacking Twisted>=10.0.0 (from Scrapy)
Downloading Twisted-13.2.0.tar.bz2 (2.7MB): 2.7MB downloaded
Running setup.py egg_info for package Twisted
Downloading/unpacking w3lib>=1.2 (from Scrapy)
Downloading w3lib-1.5.tar.gz
Running setup.py egg_info for package w3lib
Downloading/unpacking queuelib (from Scrapy)
Downloading queuelib-1.1.1.tar.gz
Running setup.py egg_info for package queuelib
Requirement already satisfied (use --upgrade to upgrade): lxml in c:\users\kiss\anaconda\lib\site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in c:\users\kiss\anaconda\lib\site-packages (from Scrapy)
Downloading/unpacking cssselect>=0.9 (from Scrapy)
Downloading cssselect-0.9.1.tar.gz
Running setup.py egg_info for package cssselect
no previously-included directories found matching 'docs\_build'
Downloading/unpacking zope.interface>=3.6.0 (from Twisted>=10.0.0->Scrapy)
Downloading zope.interface-4.0.5.zip (173kB): 173kB downloaded
Running setup.py egg_info for package zope.interface
warning: no previously-included files matching '*.dll' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
Requirement already satisfied (use --upgrade to upgrade): six>=1.4.1 in c:\users\kiss\anaconda\lib\site-packages (from w3lib>=1.2->S
crapy)
Requirement already satisfied (use --upgrade to upgrade): distribute in c:\users\kiss\anaconda\lib\site-packages (from zope.interfac
e>=3.6.0->Twisted>=10.0.0->Scrapy)
Installing collected packages: Scrapy, Twisted, w3lib, queuelib, cssselect, zope.interface
Running setup.py install for Scrapy
no previously-included directories found matching 'docs\build'
Running setup.py install for Twisted
C:\Users\kiss\Anaconda\Scripts\gcc.bat -DMS_WIN64 -mdll -O -Wall -IC:\Users\kiss\Anaconda\include -IC:\Users\kiss\Anaconda\PC -c
Удалитьconftest.c -o conftest.o
conftest.c:1:21: fatal error: rpc/rpc.h: No such file or directory
compilation terminated.
building 'twisted.test.raiser' extension
C:\Users\kiss\Anaconda\Scripts\gcc.bat -DMS_WIN64 -mdll -O -Wall -DWIN32=1 -IC:\Users\kiss\Anaconda\include -IC:\Users\kiss\Anac
onda\PC -c twisted/test/raiser.c -o build\temp.win-amd64-2.7\Release\twisted\test\raiser.o
C:\Users\kiss\Anaconda\Scripts\dllwrap.bat -DMS_WIN64 -mdll -static --entry _DllMain@12 --output-lib build\temp.win-amd64-2.7\Re
lease\twisted\test\libraiser.a --def build\temp.win-amd64-2.7\Release\twisted\test\raiser.def -s build\temp.win-amd64-2.7\Release\tw
isted\test\raiser.o -LC:\Users\kiss\Anaconda\libs -LC:\Users\kiss\Anaconda\PCbuild\amd64 -lpython27 -lmsvcr90 -o build\lib.win-amd64
-2.7\twisted\test\raiser.pyd
c:/users/kiss/anaconda/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/bin/ld.exe: warning: cannot
find entry symbol _DllMain@12; defaulting to 000000006c481000
c:/users/kiss/anaconda/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/bin/ld.exe: warning: cannot
find entry symbol _DllMain@12; defaulting to 000000006c481000
c:/users/kiss/anaconda/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/bin/ld.exe: warning: cannot
find entry symbol _DllMain@12; defaulting to 000000006c481000
building 'twisted.internet.iocpreactor.iocpsupport' extension
C:\Users\kiss\Anaconda\Scripts\gcc.bat -DMS_WIN64 -mdll -O -Wall -DWIN32=1 -IC:\Users\kiss\Anaconda\include -IC:\Users\kiss\Anac
onda\PC -c twisted/internet/iocpreactor/iocpsupport/iocpsupport.c -o build\temp.win-amd64-2.7\Release\twisted\internet\iocpreactor\i
ocpsupport\iocpsupport.o
twisted/internet/iocpreactor/iocpsupport/iocpsupport.c: In function '__pyx_pf_11iocpsupport_14CompletionPort___init__':
twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:1157:15: warning: assignment makes integer from pointer without a cast [e
nabled by default]
twisted/internet/iocpreactor/iocpsupport/iocpsupport.c: In function '__pyx_pf_11iocpsupport_14CompletionPort_1addHandle':
twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:1282:65: warning: passing argument 1 of 'CreateIoCompletionPort' makes po
inter from integer without a cast [enabled by default]
In file included from c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/includ
e/windows.h:60:0,
from c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/includ
Удалитьe/winsock2.h:23,
from twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:237:
c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/include/winbase.h:1344:28: n
ote: expected 'HANDLE' but argument is of type '__pyx_t_11iocpsupport_HANDLE'
twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:1282:65: warning: passing argument 2 of 'CreateIoCompletionPort' makes po
inter from integer without a cast [enabled by default]
In file included from c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/includ
e/windows.h:60:0,
from c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/includ
e/winsock2.h:23,
from twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:237:
c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/include/winbase.h:1344:28: n
ote: expected 'HANDLE' but argument is of type '__pyx_t_11iocpsupport_HANDLE'
twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:1282:15: warning: assignment makes integer from pointer without a cast [e
nabled by default]
twisted/internet/iocpreactor/iocpsupport/iocpsupport.c: In function '__pyx_pf_11iocpsupport_14CompletionPort_2getEvent':
twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:1366:17: warning: assignment from incompatible pointer type [enabled by d
efault]
twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:1375:51: warning: passing argument 1 of 'GetQueuedCompletionStatus' makes
pointer from integer without a cast [enabled by default]
In file included from c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/includ
e/windows.h:60:0,
from c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/includ
e/winsock2.h:23,
from twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:237:
c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/include/winbase.h:1345:29: n
ote: expected 'HANDLE' but argument is of type '__pyx_t_11iocpsupport_HANDLE'
twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:1384:3: warning: passing argument 1 of 'PyEval_RestoreThread' from incomp
atible pointer type [enabled by default]
In file included from C:\Users\kiss\Anaconda\include/Python.h:128:0,
from twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:4:
C:\Users\kiss\Anaconda\include/ceval.h:118:18: note: expected 'struct PyThreadState *' but argument is of type 'struct PyThreadS
Удалитьtate *'
twisted/internet/iocpreactor/iocpsupport/iocpsupport.c: In function '__pyx_pf_11iocpsupport_14CompletionPort_3postEvent':
twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:1669:52: warning: passing argument 1 of 'PostQueuedCompletionStatus' make
s pointer from integer without a cast [enabled by default]
In file included from c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/includ
e/windows.h:60:0,
from c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/includ
e/winsock2.h:23,
from twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:237:
c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/include/winbase.h:1346:29: n
ote: expected 'HANDLE' but argument is of type '__pyx_t_11iocpsupport_HANDLE'
twisted/internet/iocpreactor/iocpsupport/iocpsupport.c: In function '__pyx_pf_11iocpsupport_14CompletionPort_4__del__':
twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:1760:24: warning: passing argument 1 of 'CloseHandle' makes pointer from
integer without a cast [enabled by default]
In file included from c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/includ
e/windows.h:60:0,
from c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/includ
e/winsock2.h:23,
from twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:237:
c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/include/winbase.h:1442:29: n
ote: expected 'HANDLE' but argument is of type '__pyx_t_11iocpsupport_HANDLE'
twisted/internet/iocpreactor/iocpsupport/iocpsupport.c: In function '__pyx_pf_11iocpsupport_makesockaddr':
twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:1798:3: warning: passing argument 2 of 'PyObject_AsReadBuffer' from incom
patible pointer type [enabled by default]
In file included from C:\Users\kiss\Anaconda\include/Python.h:133:0,
from twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:4:
C:\Users\kiss\Anaconda\include/abstract.h:501:22: note: expected 'const void **' but argument is of type 'void **'
Удалитьtwisted/internet/iocpreactor/iocpsupport/iocpsupport.c: In function '__pyx_pf_11iocpsupport_4get_accept_addrs':
twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:3170:3: warning: passing argument 2 of 'PyObject_AsReadBuffer' from incom
patible pointer type [enabled by default]
In file included from C:\Users\kiss\Anaconda\include/Python.h:133:0,
from twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:4:
C:\Users\kiss\Anaconda\include/abstract.h:501:22: note: expected 'const void **' but argument is of type 'void **'
twisted/internet/iocpreactor/iocpsupport/iocpsupport.c: In function '__pyx_pf_11iocpsupport_6recv':
twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:3725:19: warning: assignment from incompatible pointer type [enabled by d
efault]
twisted/internet/iocpreactor/iocpsupport/iocpsupport.c: In function '__pyx_pf_11iocpsupport_8send':
twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:4445:3: warning: passing argument 2 of 'PyObject_AsReadBuffer' from incom
patible pointer type [enabled by default]
In file included from C:\Users\kiss\Anaconda\include/Python.h:133:0,
from twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:4:
C:\Users\kiss\Anaconda\include/abstract.h:501:22: note: expected 'const void **' but argument is of type 'void **'
C:\Users\kiss\Anaconda\Scripts\gcc.bat -DMS_WIN64 -mdll -O -Wall -DWIN32=1 -IC:\Users\kiss\Anaconda\include -IC:\Users\kiss\Anac
onda\PC -c twisted/internet/iocpreactor/iocpsupport/winsock_pointers.c -o build\temp.win-amd64-2.7\Release\twisted\internet\iocpreac
tor\iocpsupport\winsock_pointers.
C:\Users\kiss\Anaconda\Scripts\dllwrap.bat -DMS_WIN64 -mdll -static --entry _DllMain@12 --output-lib build\temp.win-amd64-2.7\Re
Удалитьlease\twisted\internet\iocpreactor\iocpsupport\libiocpsupport.a --def build\temp.win-amd64-2.7\Release\twisted\internet\iocpreactor\
iocpsupport\iocpsupport.def -s build\temp.win-amd64-2.7\Release\twisted\internet\iocpreactor\iocpsupport\iocpsupport.o build\temp.wi
n-amd64-2.7\Release\twisted\internet\iocpreactor\iocpsupport\winsock_pointers.o -LC:\Users\kiss\Anaconda\libs -LC:\Users\kiss\Anacon
da\PCbuild\amd64 -lws2_32 -lpython27 -lmsvcr90 -o build\lib.win-amd64-2.7\twisted\internet\iocpreactor\iocpsupport.pyd
c:/users/kiss/anaconda/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/bin/ld.exe: warning: cannot
find entry symbol _DllMain@12; defaulting to 0000000062741000
Running setup.py install for w3lib
Running setup.py install for queuelib
Running setup.py install for cssselect
no previously-included directories found matching 'docs\_build'
Running setup.py install for zope.interface
warning: no previously-included files matching '*.dll' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
building 'zope.interface._zope_interface_coptimizations' extension
C:\Users\kiss\Anaconda\Scripts\gcc.bat -DMS_WIN64 -mdll -O -Wall -IC:\Users\kiss\Anaconda\include -IC:\Users\kiss\Anaconda\PC -c
src\zope\interface\_zope_interface_coptimizations.c -o build\temp.win-amd64-2.7\Release\src\zope\interface\_zope_interface_coptimiz
ations.o
C:\Users\kiss\Anaconda\Scripts\gcc.bat -DMS_WIN64 -shared -s build\temp.win-amd64-2.7\Release\src\zope\interface\_zope_interface
_coptimizations.o build\temp.win-amd64-2.7\Release\src\zope\interface\_zope_interface_coptimizations.def -LC:\Users\kiss\Anaconda\li
bs -LC:\Users\kiss\Anaconda\PCbuild\amd64 -lpython27 -lmsvcr90 -o build\lib.win-amd64-2.7\zope\interface\_zope_interface_coptimizati
ons.pyd
Skipping installation of C:\Users\kiss\Anaconda\Lib\site-packages\zope\__init__.py (namespace package)
Installing C:\Users\kiss\Anaconda\Lib\site-packages\zope.interface-4.0.5-py2.7-nspkg.pth
Successfully installed Scrapy Twisted w3lib queuelib cssselect zope.interface
Cleaning up...
C:\Users\kiss\Anaconda\Scripts>
Все очень сложно...
ОтветитьУдалитьПоэтому просто посмотрим, что будет, если опят установить
C:\Users\kiss\Anaconda\Scripts>pip install Scrapy
Requirement already satisfied (use --upgrade to upgrade): Scrapy in c:\users\kiss\anaconda\lib\site-packages
Requirement already satisfied (use --upgrade to upgrade): Twisted>=10.0.0 in c:\users\kiss\anaconda\lib\site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): w3lib>=1.2 in c:\users\kiss\anaconda\lib\site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): queuelib in c:\users\kiss\anaconda\lib\site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): lxml in c:\users\kiss\anaconda\lib\site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in c:\users\kiss\anaconda\lib\site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in c:\users\kiss\anaconda\lib\site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in c:\users\kiss\anaconda\lib\site-packages (from Tw
isted>=10.0.0->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): six>=1.4.1 in c:\users\kiss\anaconda\lib\site-packages (from w3lib>=1.2->S
crapy)
Requirement already satisfied (use --upgrade to upgrade): distribute in c:\users\kiss\anaconda\lib\site-packages (from zope.interfac
e>=3.6.0->Twisted>=10.0.0->Scrapy)
Cleaning up...
C:\Users\kiss\Anaconda\Scripts>
В итоге запускаем C:\Users\kiss\Anaconda\Scripts>scrapy.bat
ОтветитьУдалитьи получаем ту же ошибку:
File "C:\Users\kiss\Anaconda\lib\site-packages\OpenSSL\__init__.py", line 13, in
from OpenSSL import crypto
ImportError: DLL load failed: %1 эх ты хЄё яЁшыюцхэшхь Win32.
Смотрим в фале строку 13, там
from OpenSSL import crypto
... нугугливаем ответ в неожиданном месте http://twistedmatrix.com/pipermail/twisted-python/2012-February/025240.html
там написано, что скорее всего, мы скачали не ту версию файла...
Какого файла? Вспоминаю, что скачал 32-х разрядную версию (64-х) просто не было
скачал вот отсюда https://pypi.python.org/pypi/pyOpenSSL/0.13
пробую сократить строку и..., о чудо, здесь есть следующая версия как раз для меня
со страницы https://pypi.python.org/pypi/pyOpenSSL
скачиваю 64 разрядную версию
https://pypi.python.org/packages/2.7/p/pyOpenSSL/pyOpenSSL-0.13.1.win-amd64-py2.7.exe#md5=223cc4ab7439818ccaf1bf7f51736dc8
теперь надо деинсталировать старую..., но как?
УдалитьПогуглил, нашел рекомендации снять через штатный сервис Windows...
Удивился, посмотрел на расширение файла (экзешник)..., поискал и не нашел старую установленную версию...
Решил, что она и не должна была устанавливаться..., и установил новую...
И похоже, что все заработало:
C:\Users\kiss\Anaconda\Scripts>scrapy.bat
Scrapy 0.20.1 - no active project
Usage:
scrapy [options] [args]
Available commands:
bench Run quick benchmark test
fetch Fetch a URL using the Scrapy downloader
runspider Run a self-contained spider (without creating a project)
settings Get settings values
shell Interactive scraping console
startproject Create new project
version Print Scrapy version
view Open URL in browser, as seen by Scrapy
[ more ] More commands available when run from project directory
Use "scrapy -h" to see more info about a command
C:\Users\kiss\Anaconda\Scripts>scrapy runspider
Usage
=====
scrapy runspider [options]
Run the spider defined in the given file
Options
=======
--help, -h show this help message and exit
-a NAME=VALUE set spider argument (may be repeated)
--output=FILE, -o FILE dump scraped items into FILE (use - for stdout)
--output-format=FORMAT, -t FORMAT
format to use for dumping items with -o (default:
jsonlines)
Global Options
--------------
--logfile=FILE log file. if omitted stderr will be used
--loglevel=LEVEL, -L LEVEL
log level (default: DEBUG)
--nolog disable logging completely
--profile=FILE write python cProfile stats to FILE
--lsprof=FILE write lsprof profiling stats to FILE
--pidfile=FILE write process ID to FILE
--set=NAME=VALUE, -s NAME=VALUE
set/override setting (may be repeated)
--pdb enable pdb on failure
C:\Users\kiss\Anaconda\Scripts>
Потом догадался запустить
ОтветитьУдалитьC:\Users\kiss\Anaconda\Scripts>scrapy bench
Traceback (most recent call last):
File "C:\Users\kiss\Anaconda\lib\runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "C:\Users\kiss\Anaconda\lib\runpy.py", line 72, in _run_code
exec code in run_globals
File "C:\Users\kiss\Anaconda\lib\site-packages\scrapy\tests\mockserver.py", line 187, in
os.path.join(os.path.dirname(__file__), 'keys/cert.pem'),
File "C:\Users\kiss\Anaconda\lib\site-packages\twisted\internet\ssl.py", line 70, in __init__
self.cacheContext()
File "C:\Users\kiss\Anaconda\lib\site-packages\twisted\internet\ssl.py", line 79, in cacheContext
ctx.use_certificate_file(self.certificateFileName)
OpenSSL.SSL.Error: [('system library', 'fopen', 'No such process'), ('BIO routines', 'FILE_CTRL', 'system lib'), ('SSL r
SL_CTX_use_certificate_file', 'system lib')]
2013-12-11 07:06:32+0400 [scrapy] INFO: Scrapy 0.20.1 started (bot: scrapybot)
2013-12-11 07:06:42+0400 [follow] INFO: Spider opened
2013-12-11 07:06:42+0400 [follow] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2013-12-11 07:06:43+0400 [follow] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2013-12-11 07:06:44+0400 [follow] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2013-12-11 07:06:45+0400 [follow] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2013-12-11 07:06:45+0400 [follow] ERROR: Error downloading : Connection was refused by other side: 10061: ╧юфъы■ўхэшх эх єёЄрэютыхэю, Є.ъ. ъюэхўэ√щ ъюья№■ЄхЁ юЄтхЁу чряЁюё
эшх..
2013-12-11 07:06:45+0400 [follow] INFO: Closing spider (finished)
2013-12-11 07:06:46+0400 [follow] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 3,
'downloader/exception_type_count/twisted.internet.error.ConnectionRefusedError': 3,
'downloader/request_bytes': 804,
'downloader/request_count': 3,
'downloader/request_method_count/GET': 3,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2013, 12, 11, 3, 6, 46, 34000),
'log_count/ERROR': 1,
'log_count/INFO': 6,
'scheduler/dequeued': 3,
'scheduler/dequeued/memory': 3,
'scheduler/enqueued': 3,
'scheduler/enqueued/memory': 3,
'start_time': datetime.datetime(2013, 12, 11, 3, 6, 42, 497000)}
2013-12-11 07:06:46+0400 [follow] INFO: Spider closed (finished)
C:\Users\kiss\Anaconda\Scripts>