How do you install Scrapy in Anaconda?

In this chapter, we will discuss how to install and set up Scrapy. Scrapy must be installed with Python.

Scrapy can be installed by using pip. To install, run the following command −

pip install Scrapy

Windows

Note − Python 3 is not supported on Windows OS.

Step 1 − Install Python 2.7 from Python

Set environmental variables by adding the following paths to the PATH −

C:\Python27\;C:\Python27\Scripts\; 

You can check the Python version using the following command −

python --version

Step 2 − Install OpenSSL.

Add C:\OpenSSL-Win32\bin in your environmental variables.

Note − OpenSSL comes preinstalled in all operating systems except Windows.

Step 3 − Install Visual C++ 2008 redistributables.

Step 4 − Install pywin32.

Step 5 − Install pip for Python versions older than 2.7.9.

You can check the pip version using the following command −

pip --version

Step 6 − To install scrapy, run the following command −

pip install Scrapy

Anaconda

If you have anaconda or miniconda installed on your machine, run the below command to install Scrapy using conda −

conda install -c scrapinghub scrapy 

Scrapinghub company supports official conda packages for Linux, Windows, and OS X.

Note − It is recommended to install Scrapy using the above command if you have issues installing via pip.

Ubuntu 9.10 or Above

The latest version of Python is pre-installed on Ubuntu OS. Use the Ubuntu packages aptgettable provided by Scrapinghub. To use the packages −

Step 1 − You need to import the GPG key used to sign Scrapy packages into APT keyring −

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 627220E7

Step 2 − Next, use the following command to create /etc/apt/sources.list.d/scrapy.list file −

echo 'deb http://archive.scrapy.org/ubuntu scrapy main' | sudo tee 
/etc/apt/sources.list.d/scrapy.list

Step 3 − Update package list and install scrapy −

sudo apt-get update && sudo apt-get install scrapy

Archlinux

You can install Scrapy from AUR Scrapy package using the following command −

yaourt -S scrapy

Mac OS X

Use the following command to install Xcode command line tools −

C:\Python27\;C:\Python27\Scripts\; 

Instead of using system Python, install a new updated version that doesn't conflict with the rest of your system.

Step 1 − Install homebrew.

Step 2 − Set environmental PATH variable to specify that homebrew packages should be used before system packages −

If you already have installed Anaconda or Miniconda, the company Scrapinghub maintains official conda packages for Linux, Windows and OS X.

To install Scrapy using conda, run:

conda install -c scrapinghub scrapy

Ubuntu 9.10 or above

Use the official Ubuntu Packages, which already solve all dependencies for you and are continuously updated with the latest bug fixes.

If you prefer to build the python dependencies locally instead of relying on system packages you’ll need to install their required non-python dependencies first:

sudo apt-get install python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev

You can install Scrapy with

conda install -c scrapinghub scrapy

pip install Scrapy

Archlinux

You can follow the generic instructions or install Scrapy from AUR Scrapy package:

yaourt -S scrapy

Windows

Scrapy with Python 3 is not yet supported on Windows.

Follow This steps to install scrapy on windows:

• Install Python 2.7

• adjust PATH environment variable to include paths to the Python executable and additional scripts. The following paths need to be added to PATH:

  C:\Python27;C:\Python27\Scripts;

• Install pywin32 from here

• let’s install Scrapy:

   pip install Scrapy
  

Mac OS X

Building Scrapy’s dependencies requires the presence of a C compiler and development headers. On OS X this is typically provided by Apple’s Xcode development tools. To install the Xcode command line tools open a terminal window and run:

xcode-select --install

There’s a known issue that prevents

conda install -c scrapinghub scrapy

• (Recommended) Don’t use system python, install a new, updated version that doesn’t conflict with the rest of your system. Here’s how to do it using the homebrew package manager:

  • Install homebrew following the instructions in http://brew.sh/

  • Update your

    conda install -c scrapinghub scrapy
    

    5 variable to state that homebrew packages should be used before system packages (Change

    conda install -c scrapinghub scrapy
    

    6 to

    conda install -c scrapinghub scrapy
    

    7 accordantly if you’re using zsh as default shell):

    echo "export PATH=/usr/local/bin:/usr/local/sbin:$PATH" >> ~/.bashrc
    

  • Reload

    conda install -c scrapinghub scrapy
    

    6 to ensure the changes have taken place:

    source ~/.bashrc
    

  • Install python:

    brew install python
    

  • Latest versions of python have

    conda install -c scrapinghub scrapy
    

    2 bundled with them so you won’t need to install it separately. If this is not the case, upgrade python:

    conda install -c scrapinghub scrapy
    

    0

• (Optional) Install Scrapy inside an isolated python environment.

  This method is a workaround for the above OS X issue, but it’s an overall good practice for managing dependencies and can complement the first method.

  virtualenv is a tool you can use to create virtual environments in python. We recommended reading a tutorial like http://docs.python-guide.org/en/latest/dev/virtualenvs/ to get started.

  Where is Scrapy installed?

  Instead, we recommend that you install scrapy within a so-called “virtual environment” (virtualenv). Virtualenvs allow you to not conflict with already-installed Python system packages (which could break some of your system tools and scripts), and still install packages normally with pip (without sudo and the likes).

  How to use Scrapy in Python?

  While working with Scrapy, one needs to create scrapy project. In Scrapy, always try to create one spider which helps to fetch data, so to create one, move to spider folder and create one python file over there. Create one spider with name gfgfetch.py python file. Move to the spider folder and create gfgfetch.py .

  How to install Scrapy MacOS?

  Follow the below steps to install the SCrapy package on macOS using pip:.

  Step 1: Install the latest Python3 in MacOS..

  Step 2: Check if pip3 and python3 are correctly installed..

  Step 3: Upgrade your pip to avoid errors during installation..

  Step 4: Enter the following command to install Scrapy using pip3..

  How to install scrapy in colab?

  Creating the files and directories for our project. # install scrapy ! pip install Scrapy # create files for learning ! scrapy startproject firstproject..

  Creating quotes_spider.py and save it. Change the current working directory to the spiders directory with os. chdir()..