пятница, 6 декабря 2013 г.

Осваиваем технологии web-scrapping (Python)

Повезло, наше "теоретический" материал о скорости парсинга (клик по картинке). Сначала "разложим" в меню все, что уже есть, потом начнем изучение, свои файлы будем складывать на GitHub (?) Сначала читаем статью. С учетом того, что я нашел уже изрядное количество видеороликов, статей и справочников..., пожалуй, можно уже сформулировать первое задание. Далеко ходить не надо. Сначала (скачаем по списку)текстовые файлы из интернета, поместим их в папки, потом в каждом файле распарсим строку URL, потом запишем все в один большой csv файл, потом этот файл проверим и почистим, уберем оттуда NA ячейки... Потом все это разобьем на блоки... Потом подумаем о том, как все это запустить в автоматическом режиме... Блок кода Основной текст Блок 2 Другие посты по теме

23 комментария:

  1. Сначала о бибиблотеках (из основной статьи http://www.ibm.com/developerworks/ru/library/x-hiperfparse/):

    Python никогда не страдал от нехватки XML-библиотек. Начиная с версии 2.0 он включает в себя привычную
    xml.dom.minidom и связанные с ней модели
    pulldom и
    Simple API for XML (SAX).

    Начиная с версии 2.4 в него входит широко распространенный
    API ElementTree.
    К тому же всегда есть независимые библиотеки, которые предоставляют более высокоуровневые или более подходящие для Python интерфейсы.

    ОтветитьУдалить
    Ответы
    1. В то время как для простого DOM- или SAX-анализа небольших файлов достаточно любой XML-библиотеки, разработчики все чаще сталкиваются с крупными наборами данных и необходимостью разбирать XML в реальном времени в контексте Web-сервисов.

      В то же время опытные XML-разработчики могут предпочесть более XML-ориентированные языки, такие как
      XPath или
      XSLT,
      за их лаконичность и выразительность.
      Идеальным выходом из этой ситуации было бы иметь доступ к декларативному синтаксису XPath, сохраняя при этом общую функциональность, доступную в Python.

      Удалить
  2. Хваленую библиотеку lxml использует пакет scrapy...
    Посмотрел я видео, почитал документацию. И отзывы хорошие..
    http://doc.scrapy.org/en/latest/intro/tutorial.html#intro-tutorial
    https://www.youtube.com/playlist?list=PL51BA5190961CFEE3
    Надо бы попробовать, но "забыл", как устанавливать эти самые дополнительные пакеты. Особенно под Windows... В скриптах есть файлы, которые так и называются install.py ... точнее - setup.py
    Но чего-то не очень они устанавливаются... По опыту борьбы с Django... думаю, что основные проблемы в том, что пути надо прописывать, файрвол норовит заблокировать установку... Дабы не разбираться с этим каждый раз есть пакеты
    pip ... easy_install

    Если уж разбираться с установкой, то этих установщиков...

    ОтветитьУдалить
    Ответы
    1. Начал с того, что нашел в Анаконде pip.bat
      ...не думал долго, а запустил с правами администратора в окне cmd.exe
      вот, что получилось

      C:\Users\kiss\Anaconda\Scripts>pip.bat

      Usage:
      pip [options]

      Commands:
      install Install packages.
      uninstall Uninstall packages.
      freeze Output installed packages in requirements format.
      list List installed packages.
      show Show information about installed packages.
      search Search PyPI for packages.
      wheel Build wheels from your requirements.
      zip Zip individual packages.
      unzip Unzip individual packages.
      bundle Create pybundles.
      help Show help for commands.

      General Options:
      -h, --help Show help.
      -v, --verbose Give more output. Option is additive, and can be
      used up to 3 times.
      -V, --version Show version and exit.
      -q, --quiet Give less output.
      --log Log file where a complete (maximum verbosity)
      record will be kept.
      --proxy Specify a proxy in the form
      [user:passwd@]proxy.server:port.
      --timeout Set the socket timeout (default 15 seconds).
      --exists-action Default action when a path already exists:
      (s)witch, (i)gnore, (w)ipe, (b)ackup.
      --cert Path to alternate CA bundle.

      C:\Users\kiss\Anaconda\Scripts>

      Удалить
    2. При работе пришлось руководствоваться вот этой инструкцией
      http://doc.scrapy.org/en/latest/intro/install.html#intro-install

      Здесь во первых строках
      pip install Scrapy
      пробуем сразу после выполнения C:\Users\kiss\Anaconda\Scripts>pip.bat
      ....
      Баааа. Грузится.
      Итак, получается, что
      по команде
      pip install Scrapy
      ПИП сам находит в своих репозитариях соотетсвующие пакеты. Дальше грузит их ко мне на компьютер в соответсвующие директории...
      Посмотрим, что откуда и куда..., но сначала, как

      Удалить
    3. Вот, что в итоге установилось:
      C:\Users\kiss\Anaconda\Scripts>pip install Scrapy
      Downloading/unpacking Scrapy
      Downloading Scrapy-0.20.1.tar.gz (745kB): 745kB downloaded
      Running setup.py egg_info for package Scrapy

      no previously-included directories found matching 'docs\build'
      Downloading/unpacking Twisted>=10.0.0 (from Scrapy)
      Downloading Twisted-13.2.0.tar.bz2 (2.7MB): 2.7MB downloaded
      Running setup.py egg_info for package Twisted

      Downloading/unpacking w3lib>=1.2 (from Scrapy)
      Downloading w3lib-1.5.tar.gz
      Running setup.py egg_info for package w3lib

      Downloading/unpacking queuelib (from Scrapy)
      Downloading queuelib-1.1.1.tar.gz
      Running setup.py egg_info for package queuelib

      Requirement already satisfied (use --upgrade to upgrade): lxml in c:\users\kiss\anaconda\lib\site-packages (from Scrapy
      Downloading/unpacking pyOpenSSL (from Scrapy)
      Downloading pyOpenSSL-0.13.1.tar.gz (254kB): 254kB downloaded
      Running setup.py egg_info for package pyOpenSSL

      error: Could not find 'openssl.exe'
      Complete output from command python setup.py egg_info:
      running egg_info

      creating pip-egg-info\pyOpenSSL.egg-info

      writing pip-egg-info\pyOpenSSL.egg-info\PKG-INFO

      writing top-level names to pip-egg-info\pyOpenSSL.egg-info\top_level.txt

      writing dependency_links to pip-egg-info\pyOpenSSL.egg-info\dependency_links.txt

      writing manifest file 'pip-egg-info\pyOpenSSL.egg-info\SOURCES.txt'

      warning: manifest_maker: standard file '-c' not found



      error: Could not find 'openssl.exe'

      ----------------------------------------
      Cleaning up...
      Command python setup.py egg_info failed with error code 1 in c:\users\kiss\appdata\local\temp\pip_build_kiss\pyOpenSSL
      Storing complete log in C:\Users\kiss\pip\pip.log

      C:\Users\kiss\Anaconda\Scripts>

      Удалить
    4. Ну вот, иногда метод тупого тыка очень даже полезен.
      Почти все ясно. Действительно, pip качает из своего репозитория нужные файлы, проверяет сначала наличие, потом обновления...
      И все бы было здорово, если бы я "тупо" выполнил инструкции (фрагмент см. ниже) http://doc.scrapy.org/en/latest/intro/install.html#intro-install

      Pre-requisites
      The installation steps assume that you have the following things installed:

      Python 2.7
      lxml. Most Linux distributions ships prepackaged versions of lxml. Otherwise refer to http://lxml.de/installation.html
      OpenSSL. This comes preinstalled in all operating systems except Windows (see Platform specific installation notes)
      pip or easy_install Python package managers

      Так что, надо было установить сначала OpenSSL

      Удалить
  3. Вот та часть из инструкции по установке, которую надо (было) выполнить:

    Platform specific installation notes
    Windows
    After installing Python, follow these steps before installing Scrapy:

    add the C:\python27\Scripts and C:\python27 folders to the system path by adding those directories to the PATH environment variable from the Control Panel.
    install OpenSSL by following these steps:
    go to Win32 OpenSSL page
    download Visual C++ 2008 redistributables for your Windows and architecture
    download OpenSSL for your Windows and architecture (the regular version, not the light one)
    add the c:\openssl-win32\bin (or similar) directory to your PATH, the same way you added python27 in the first step`` in the first step
    some binary packages that Scrapy depends on (like Twisted, lxml and pyOpenSSL) require a compiler available to install, and fail if you don’t have Visual Studio installed. You can find Windows installers for those in the following links. Make sure you respect your Python version and Windows architecture.
    pywin32: http://sourceforge.net/projects/pywin32/files/
    Twisted: http://twistedmatrix.com/trac/wiki/Downloads
    zope.interface: download the egg from zope.interface pypi page and install it by running easy_install file.egg
    lxml: http://pypi.python.org/pypi/lxml/
    pyOpenSSL: https://launchpad.net/pyopenssl
    Finally, this page contains many precompiled Python binary libraries, which may come handy to fulfill Scrapy dependencies:

    http://www.lfd.uci.edu/~gohlke/pythonlibs/

    ОтветитьУдалить
    Ответы
    1. Насколько я понял "download Visual C++ 2008 redistributables for your Windows and architecture" нужно только в том случае, если на компьютере не стоит Visual Studio... Полагаю, что мой express - это то, что надо... так что пока качать не буду...

      А вот сейчас качаю:
      http://slproweb.com/download/Win64OpenSSL-1_0_1e.exe

      А потом проверю все пути и пропишу новый

      "add the c:\openssl-win32\bin (or similar) directory to your PATH, the same way you added python27 in the first step"

      Удалить
    2. В процессе установки выскочило сообщение, что сначала надо установить
      http://www.microsoft.com/en-us/download/details.aspx?id=15336

      The Microsoft Visual C++ 2008 Redistributable Package (x86) installs runtime components of Visual C++ Libraries required to run applications developed with Visual C++ on a computer that does not have Visual C++ 2008 installed.

      Устанавливаю

      Удалить
    3. смотрим пути (переменная Path из "нижнего" окна - "Переменные среды")
      C:\HashiCorp\Vagrant\bin;
      C:\Users\kiss\Anaconda\Scripts;
      C:\OpenSSL-Win64\bin

      Удалить
    4. Исправил... не было пут в анаконду (а сразу в папку со скриптами)...
      Вот теперь, что у меня в путях:
      C:\Program Files (x86)\AMD APP\bin\x86_64;
      C:\Program Files (x86)\AMD APP\bin\x86;
      %SystemRoot%\system32;
      %SystemRoot%;
      %SystemRoot%\System32\Wbem;
      %SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;
      C:\Program Files (x86)\ATI Technologies\ATI.ACE\Core-Static;
      C:\Program Files (x86)\Windows Live\Shared;C:\Program Files\Java\jdk1.7.0_21\bin;
      C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;
      C:\HashiCorp\Vagrant\bin;
      C:\Users\kiss\Anaconda;
      C:\Users\kiss\Anaconda\Scripts;
      C:\OpenSSL-Win64\bin

      Удалить
  4. После перезагрузки опять та же ошибка...
    Задал поиск по тексту ошибки и нашел
    http://stackoverflow.com/questions/4750806/how-to-install-pip-on-windows (слишком запутано)
    А вот то, что сработало:
    http://stackoverflow.com/questions/10688843/trying-to-install-scrapy-error-could-not-find-openssl-exe

    Отсюда https://pypi.python.org/pypi/pyOpenSSL/0.13
    загрузил наугад
    https://pypi.python.org/packages/2.7/p/pyOpenSSL/pyOpenSSL-0.13.winxp32-py2.7.msi#md5=3438c0e3a87060959d7061f939c6b03d

    Установил...

    ОтветитьУдалить
  5. А потом и пип установился вроде бы

    C:\Users\kiss\Anaconda\Scripts>pip install Scrapy
    Downloading/unpacking Scrapy
    Downloading Scrapy-0.20.1.tar.gz (745kB): 745kB downloaded
    Running setup.py egg_info for package Scrapy

    no previously-included directories found matching 'docs\build'
    Downloading/unpacking Twisted>=10.0.0 (from Scrapy)
    Downloading Twisted-13.2.0.tar.bz2 (2.7MB): 2.7MB downloaded
    Running setup.py egg_info for package Twisted

    Downloading/unpacking w3lib>=1.2 (from Scrapy)
    Downloading w3lib-1.5.tar.gz
    Running setup.py egg_info for package w3lib

    Downloading/unpacking queuelib (from Scrapy)
    Downloading queuelib-1.1.1.tar.gz
    Running setup.py egg_info for package queuelib

    Requirement already satisfied (use --upgrade to upgrade): lxml in c:\users\kiss\anaconda\lib\site-packages (from Scrapy)
    Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in c:\users\kiss\anaconda\lib\site-packages (from Scrapy)
    Downloading/unpacking cssselect>=0.9 (from Scrapy)
    Downloading cssselect-0.9.1.tar.gz
    Running setup.py egg_info for package cssselect

    no previously-included directories found matching 'docs\_build'
    Downloading/unpacking zope.interface>=3.6.0 (from Twisted>=10.0.0->Scrapy)
    Downloading zope.interface-4.0.5.zip (173kB): 173kB downloaded
    Running setup.py egg_info for package zope.interface

    warning: no previously-included files matching '*.dll' found anywhere in distribution
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '*.pyo' found anywhere in distribution
    warning: no previously-included files matching '*.so' found anywhere in distribution
    Requirement already satisfied (use --upgrade to upgrade): six>=1.4.1 in c:\users\kiss\anaconda\lib\site-packages (from w3lib>=1.2->S
    crapy)
    Requirement already satisfied (use --upgrade to upgrade): distribute in c:\users\kiss\anaconda\lib\site-packages (from zope.interfac
    e>=3.6.0->Twisted>=10.0.0->Scrapy)
    Installing collected packages: Scrapy, Twisted, w3lib, queuelib, cssselect, zope.interface
    Running setup.py install for Scrapy

    no previously-included directories found matching 'docs\build'
    Running setup.py install for Twisted

    ОтветитьУдалить
    Ответы
    1. C:\Users\kiss\Anaconda\Scripts\gcc.bat -DMS_WIN64 -mdll -O -Wall -IC:\Users\kiss\Anaconda\include -IC:\Users\kiss\Anaconda\PC -c
      conftest.c -o conftest.o
      conftest.c:1:21: fatal error: rpc/rpc.h: No such file or directory
      compilation terminated.
      building 'twisted.test.raiser' extension
      C:\Users\kiss\Anaconda\Scripts\gcc.bat -DMS_WIN64 -mdll -O -Wall -DWIN32=1 -IC:\Users\kiss\Anaconda\include -IC:\Users\kiss\Anac
      onda\PC -c twisted/test/raiser.c -o build\temp.win-amd64-2.7\Release\twisted\test\raiser.o
      C:\Users\kiss\Anaconda\Scripts\dllwrap.bat -DMS_WIN64 -mdll -static --entry _DllMain@12 --output-lib build\temp.win-amd64-2.7\Re
      lease\twisted\test\libraiser.a --def build\temp.win-amd64-2.7\Release\twisted\test\raiser.def -s build\temp.win-amd64-2.7\Release\tw
      isted\test\raiser.o -LC:\Users\kiss\Anaconda\libs -LC:\Users\kiss\Anaconda\PCbuild\amd64 -lpython27 -lmsvcr90 -o build\lib.win-amd64
      -2.7\twisted\test\raiser.pyd
      c:/users/kiss/anaconda/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/bin/ld.exe: warning: cannot
      find entry symbol _DllMain@12; defaulting to 000000006c481000
      c:/users/kiss/anaconda/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/bin/ld.exe: warning: cannot
      find entry symbol _DllMain@12; defaulting to 000000006c481000
      c:/users/kiss/anaconda/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/bin/ld.exe: warning: cannot
      find entry symbol _DllMain@12; defaulting to 000000006c481000
      building 'twisted.internet.iocpreactor.iocpsupport' extension
      C:\Users\kiss\Anaconda\Scripts\gcc.bat -DMS_WIN64 -mdll -O -Wall -DWIN32=1 -IC:\Users\kiss\Anaconda\include -IC:\Users\kiss\Anac
      onda\PC -c twisted/internet/iocpreactor/iocpsupport/iocpsupport.c -o build\temp.win-amd64-2.7\Release\twisted\internet\iocpreactor\i
      ocpsupport\iocpsupport.o
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c: In function '__pyx_pf_11iocpsupport_14CompletionPort___init__':
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:1157:15: warning: assignment makes integer from pointer without a cast [e
      nabled by default]
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c: In function '__pyx_pf_11iocpsupport_14CompletionPort_1addHandle':
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:1282:65: warning: passing argument 1 of 'CreateIoCompletionPort' makes po
      inter from integer without a cast [enabled by default]
      In file included from c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/includ
      e/windows.h:60:0,

      Удалить
    2. from c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/includ
      e/winsock2.h:23,
      from twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:237:
      c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/include/winbase.h:1344:28: n
      ote: expected 'HANDLE' but argument is of type '__pyx_t_11iocpsupport_HANDLE'
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:1282:65: warning: passing argument 2 of 'CreateIoCompletionPort' makes po
      inter from integer without a cast [enabled by default]
      In file included from c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/includ
      e/windows.h:60:0,
      from c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/includ
      e/winsock2.h:23,
      from twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:237:
      c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/include/winbase.h:1344:28: n
      ote: expected 'HANDLE' but argument is of type '__pyx_t_11iocpsupport_HANDLE'
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:1282:15: warning: assignment makes integer from pointer without a cast [e
      nabled by default]
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c: In function '__pyx_pf_11iocpsupport_14CompletionPort_2getEvent':
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:1366:17: warning: assignment from incompatible pointer type [enabled by d
      efault]
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:1375:51: warning: passing argument 1 of 'GetQueuedCompletionStatus' makes
      pointer from integer without a cast [enabled by default]
      In file included from c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/includ
      e/windows.h:60:0,
      from c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/includ
      e/winsock2.h:23,
      from twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:237:
      c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/include/winbase.h:1345:29: n
      ote: expected 'HANDLE' but argument is of type '__pyx_t_11iocpsupport_HANDLE'
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:1384:3: warning: passing argument 1 of 'PyEval_RestoreThread' from incomp
      atible pointer type [enabled by default]
      In file included from C:\Users\kiss\Anaconda\include/Python.h:128:0,
      from twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:4:

      Удалить
    3. C:\Users\kiss\Anaconda\include/ceval.h:118:18: note: expected 'struct PyThreadState *' but argument is of type 'struct PyThreadS
      tate *'
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c: In function '__pyx_pf_11iocpsupport_14CompletionPort_3postEvent':
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:1669:52: warning: passing argument 1 of 'PostQueuedCompletionStatus' make
      s pointer from integer without a cast [enabled by default]
      In file included from c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/includ
      e/windows.h:60:0,
      from c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/includ
      e/winsock2.h:23,
      from twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:237:
      c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/include/winbase.h:1346:29: n
      ote: expected 'HANDLE' but argument is of type '__pyx_t_11iocpsupport_HANDLE'
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c: In function '__pyx_pf_11iocpsupport_14CompletionPort_4__del__':
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:1760:24: warning: passing argument 1 of 'CloseHandle' makes pointer from
      integer without a cast [enabled by default]
      In file included from c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/includ
      e/windows.h:60:0,
      from c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/includ
      e/winsock2.h:23,
      from twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:237:
      c:\users\kiss\anaconda\mingw\bin\../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/include/winbase.h:1442:29: n
      ote: expected 'HANDLE' but argument is of type '__pyx_t_11iocpsupport_HANDLE'
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c: In function '__pyx_pf_11iocpsupport_makesockaddr':
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:1798:3: warning: passing argument 2 of 'PyObject_AsReadBuffer' from incom
      patible pointer type [enabled by default]
      In file included from C:\Users\kiss\Anaconda\include/Python.h:133:0,
      from twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:4:

      Удалить
    4. C:\Users\kiss\Anaconda\include/abstract.h:501:22: note: expected 'const void **' but argument is of type 'void **'
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c: In function '__pyx_pf_11iocpsupport_4get_accept_addrs':
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:3170:3: warning: passing argument 2 of 'PyObject_AsReadBuffer' from incom
      patible pointer type [enabled by default]
      In file included from C:\Users\kiss\Anaconda\include/Python.h:133:0,
      from twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:4:
      C:\Users\kiss\Anaconda\include/abstract.h:501:22: note: expected 'const void **' but argument is of type 'void **'
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c: In function '__pyx_pf_11iocpsupport_6recv':
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:3725:19: warning: assignment from incompatible pointer type [enabled by d
      efault]
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c: In function '__pyx_pf_11iocpsupport_8send':
      twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:4445:3: warning: passing argument 2 of 'PyObject_AsReadBuffer' from incom
      patible pointer type [enabled by default]
      In file included from C:\Users\kiss\Anaconda\include/Python.h:133:0,
      from twisted/internet/iocpreactor/iocpsupport/iocpsupport.c:4:
      C:\Users\kiss\Anaconda\include/abstract.h:501:22: note: expected 'const void **' but argument is of type 'void **'
      C:\Users\kiss\Anaconda\Scripts\gcc.bat -DMS_WIN64 -mdll -O -Wall -DWIN32=1 -IC:\Users\kiss\Anaconda\include -IC:\Users\kiss\Anac
      onda\PC -c twisted/internet/iocpreactor/iocpsupport/winsock_pointers.c -o build\temp.win-amd64-2.7\Release\twisted\internet\iocpreac
      tor\iocpsupport\winsock_pointers.

      Удалить
    5. C:\Users\kiss\Anaconda\Scripts\dllwrap.bat -DMS_WIN64 -mdll -static --entry _DllMain@12 --output-lib build\temp.win-amd64-2.7\Re
      lease\twisted\internet\iocpreactor\iocpsupport\libiocpsupport.a --def build\temp.win-amd64-2.7\Release\twisted\internet\iocpreactor\
      iocpsupport\iocpsupport.def -s build\temp.win-amd64-2.7\Release\twisted\internet\iocpreactor\iocpsupport\iocpsupport.o build\temp.wi
      n-amd64-2.7\Release\twisted\internet\iocpreactor\iocpsupport\winsock_pointers.o -LC:\Users\kiss\Anaconda\libs -LC:\Users\kiss\Anacon
      da\PCbuild\amd64 -lws2_32 -lpython27 -lmsvcr90 -o build\lib.win-amd64-2.7\twisted\internet\iocpreactor\iocpsupport.pyd
      c:/users/kiss/anaconda/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/bin/ld.exe: warning: cannot
      find entry symbol _DllMain@12; defaulting to 0000000062741000
      Running setup.py install for w3lib

      Running setup.py install for queuelib

      Running setup.py install for cssselect

      no previously-included directories found matching 'docs\_build'
      Running setup.py install for zope.interface

      warning: no previously-included files matching '*.dll' found anywhere in distribution
      warning: no previously-included files matching '*.pyc' found anywhere in distribution
      warning: no previously-included files matching '*.pyo' found anywhere in distribution
      warning: no previously-included files matching '*.so' found anywhere in distribution
      building 'zope.interface._zope_interface_coptimizations' extension
      C:\Users\kiss\Anaconda\Scripts\gcc.bat -DMS_WIN64 -mdll -O -Wall -IC:\Users\kiss\Anaconda\include -IC:\Users\kiss\Anaconda\PC -c
      src\zope\interface\_zope_interface_coptimizations.c -o build\temp.win-amd64-2.7\Release\src\zope\interface\_zope_interface_coptimiz
      ations.o
      C:\Users\kiss\Anaconda\Scripts\gcc.bat -DMS_WIN64 -shared -s build\temp.win-amd64-2.7\Release\src\zope\interface\_zope_interface
      _coptimizations.o build\temp.win-amd64-2.7\Release\src\zope\interface\_zope_interface_coptimizations.def -LC:\Users\kiss\Anaconda\li
      bs -LC:\Users\kiss\Anaconda\PCbuild\amd64 -lpython27 -lmsvcr90 -o build\lib.win-amd64-2.7\zope\interface\_zope_interface_coptimizati
      ons.pyd
      Skipping installation of C:\Users\kiss\Anaconda\Lib\site-packages\zope\__init__.py (namespace package)
      Installing C:\Users\kiss\Anaconda\Lib\site-packages\zope.interface-4.0.5-py2.7-nspkg.pth
      Successfully installed Scrapy Twisted w3lib queuelib cssselect zope.interface
      Cleaning up...

      C:\Users\kiss\Anaconda\Scripts>

      Удалить
  6. Все очень сложно...
    Поэтому просто посмотрим, что будет, если опят установить

    C:\Users\kiss\Anaconda\Scripts>pip install Scrapy
    Requirement already satisfied (use --upgrade to upgrade): Scrapy in c:\users\kiss\anaconda\lib\site-packages
    Requirement already satisfied (use --upgrade to upgrade): Twisted>=10.0.0 in c:\users\kiss\anaconda\lib\site-packages (from Scrapy)
    Requirement already satisfied (use --upgrade to upgrade): w3lib>=1.2 in c:\users\kiss\anaconda\lib\site-packages (from Scrapy)
    Requirement already satisfied (use --upgrade to upgrade): queuelib in c:\users\kiss\anaconda\lib\site-packages (from Scrapy)
    Requirement already satisfied (use --upgrade to upgrade): lxml in c:\users\kiss\anaconda\lib\site-packages (from Scrapy)
    Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in c:\users\kiss\anaconda\lib\site-packages (from Scrapy)
    Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in c:\users\kiss\anaconda\lib\site-packages (from Scrapy)
    Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in c:\users\kiss\anaconda\lib\site-packages (from Tw
    isted>=10.0.0->Scrapy)
    Requirement already satisfied (use --upgrade to upgrade): six>=1.4.1 in c:\users\kiss\anaconda\lib\site-packages (from w3lib>=1.2->S
    crapy)
    Requirement already satisfied (use --upgrade to upgrade): distribute in c:\users\kiss\anaconda\lib\site-packages (from zope.interfac
    e>=3.6.0->Twisted>=10.0.0->Scrapy)
    Cleaning up...

    C:\Users\kiss\Anaconda\Scripts>

    ОтветитьУдалить
  7. В итоге запускаем C:\Users\kiss\Anaconda\Scripts>scrapy.bat
    и получаем ту же ошибку:
    File "C:\Users\kiss\Anaconda\lib\site-packages\OpenSSL\__init__.py", line 13, in
    from OpenSSL import crypto
    ImportError: DLL load failed: %1 эх  ты хЄё  яЁшыюцхэшхь Win32.

    Смотрим в фале строку 13, там
    from OpenSSL import crypto
    ... нугугливаем ответ в неожиданном месте http://twistedmatrix.com/pipermail/twisted-python/2012-February/025240.html
    там написано, что скорее всего, мы скачали не ту версию файла...
    Какого файла? Вспоминаю, что скачал 32-х разрядную версию (64-х) просто не было
    скачал вот отсюда https://pypi.python.org/pypi/pyOpenSSL/0.13
    пробую сократить строку и..., о чудо, здесь есть следующая версия как раз для меня
    со страницы https://pypi.python.org/pypi/pyOpenSSL
    скачиваю 64 разрядную версию
    https://pypi.python.org/packages/2.7/p/pyOpenSSL/pyOpenSSL-0.13.1.win-amd64-py2.7.exe#md5=223cc4ab7439818ccaf1bf7f51736dc8

    ОтветитьУдалить
    Ответы
    1. теперь надо деинсталировать старую..., но как?
      Погуглил, нашел рекомендации снять через штатный сервис Windows...
      Удивился, посмотрел на расширение файла (экзешник)..., поискал и не нашел старую установленную версию...
      Решил, что она и не должна была устанавливаться..., и установил новую...
      И похоже, что все заработало:


      C:\Users\kiss\Anaconda\Scripts>scrapy.bat
      Scrapy 0.20.1 - no active project

      Usage:
      scrapy [options] [args]

      Available commands:
      bench Run quick benchmark test
      fetch Fetch a URL using the Scrapy downloader
      runspider Run a self-contained spider (without creating a project)
      settings Get settings values
      shell Interactive scraping console
      startproject Create new project
      version Print Scrapy version
      view Open URL in browser, as seen by Scrapy

      [ more ] More commands available when run from project directory

      Use "scrapy -h" to see more info about a command

      C:\Users\kiss\Anaconda\Scripts>scrapy runspider
      Usage
      =====
      scrapy runspider [options]

      Run the spider defined in the given file

      Options
      =======
      --help, -h show this help message and exit
      -a NAME=VALUE set spider argument (may be repeated)
      --output=FILE, -o FILE dump scraped items into FILE (use - for stdout)
      --output-format=FORMAT, -t FORMAT
      format to use for dumping items with -o (default:
      jsonlines)

      Global Options
      --------------
      --logfile=FILE log file. if omitted stderr will be used
      --loglevel=LEVEL, -L LEVEL
      log level (default: DEBUG)
      --nolog disable logging completely
      --profile=FILE write python cProfile stats to FILE
      --lsprof=FILE write lsprof profiling stats to FILE
      --pidfile=FILE write process ID to FILE
      --set=NAME=VALUE, -s NAME=VALUE
      set/override setting (may be repeated)
      --pdb enable pdb on failure

      C:\Users\kiss\Anaconda\Scripts>

      Удалить
  8. Потом догадался запустить
    C:\Users\kiss\Anaconda\Scripts>scrapy bench
    Traceback (most recent call last):
    File "C:\Users\kiss\Anaconda\lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
    File "C:\Users\kiss\Anaconda\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
    File "C:\Users\kiss\Anaconda\lib\site-packages\scrapy\tests\mockserver.py", line 187, in
    os.path.join(os.path.dirname(__file__), 'keys/cert.pem'),
    File "C:\Users\kiss\Anaconda\lib\site-packages\twisted\internet\ssl.py", line 70, in __init__
    self.cacheContext()
    File "C:\Users\kiss\Anaconda\lib\site-packages\twisted\internet\ssl.py", line 79, in cacheContext
    ctx.use_certificate_file(self.certificateFileName)
    OpenSSL.SSL.Error: [('system library', 'fopen', 'No such process'), ('BIO routines', 'FILE_CTRL', 'system lib'), ('SSL r
    SL_CTX_use_certificate_file', 'system lib')]
    2013-12-11 07:06:32+0400 [scrapy] INFO: Scrapy 0.20.1 started (bot: scrapybot)
    2013-12-11 07:06:42+0400 [follow] INFO: Spider opened
    2013-12-11 07:06:42+0400 [follow] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
    2013-12-11 07:06:43+0400 [follow] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
    2013-12-11 07:06:44+0400 [follow] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
    2013-12-11 07:06:45+0400 [follow] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
    2013-12-11 07:06:45+0400 [follow] ERROR: Error downloading : Connection was refused by other side: 10061: ╧юфъы■ўхэшх эх єёЄрэютыхэю, Є.ъ. ъюэхўэ√щ ъюья№■ЄхЁ юЄтхЁу чряЁюё
    эшх..
    2013-12-11 07:06:45+0400 [follow] INFO: Closing spider (finished)
    2013-12-11 07:06:46+0400 [follow] INFO: Dumping Scrapy stats:
    {'downloader/exception_count': 3,
    'downloader/exception_type_count/twisted.internet.error.ConnectionRefusedError': 3,
    'downloader/request_bytes': 804,
    'downloader/request_count': 3,
    'downloader/request_method_count/GET': 3,
    'finish_reason': 'finished',
    'finish_time': datetime.datetime(2013, 12, 11, 3, 6, 46, 34000),
    'log_count/ERROR': 1,
    'log_count/INFO': 6,
    'scheduler/dequeued': 3,
    'scheduler/dequeued/memory': 3,
    'scheduler/enqueued': 3,
    'scheduler/enqueued/memory': 3,
    'start_time': datetime.datetime(2013, 12, 11, 3, 6, 42, 497000)}
    2013-12-11 07:06:46+0400 [follow] INFO: Spider closed (finished)

    C:\Users\kiss\Anaconda\Scripts>

    ОтветитьУдалить