Modules and Packages ==================== .. toctree:: :maxdepth: 1 Indices and tables ------------------ * :ref:`genindex` * :ref:`modindex` * :ref:`search` Introduction ------------ Python defines the following two containers for code reuse: * Modules, which group related functions together. * Packages, which group related modules (and additional sub-packages) together in a named hierarchy. .. note:: In reality, Python only has one object to represent organizational units of code, which is the module object. As a result, all packages are actually modules (but not all modules are packages). Packages and modules can come from more than just the filesystem; they can reside in zip files, a database, an http endpoint on the network, in the Windows registry etc. In other words, any location you can search and access. .. tip:: It is common for the names of built-in modules/packages to be all lower-case, but, for the first letter of custom modules/packages to be upper-case. .. _section_heading-Modules: Modules ------- Modules are typically source code that is found in a .py file. Programs are also source code that is found in a .py file. Sounds like the same thing... The distinction is in how each type is used * A module is designed to be imported into another program, whereas, * A program is meant to be run standalone. Defining a module is covered in :ref:`section_heading-Program_Structure` Modules can contain a docstring which describes the module. The docstring contents are placed in the ``__doc__`` attribute of the module object that is created. For example, given the following code in ``My_module.py``: .. literalinclude:: My_module.py Importing it will populate the ``__doc__`` attribute as: >>> import my_module >>> my_module.__doc__ 'The docstring for My_module\n' .. _section_heading-Packages: Packages -------- Python defines two types of packages: * Regular packages: typically implemented as a directory containing modules, and optionally, additional sub-packages (as sub-directories). * Namespace packages: a package where the portions of the entire package are spread out and reside in different locations. These will not be discussed in this course. Packages can contain modules and sub-packages. The named hierarchy of a packages uses the ``.`` separator to separate the child modules and subpackages from the parent packages. Python identifies a directory as a package if it contains an ``__init__.py`` file. Code in the packages ``__init__.py`` file is execute when the package is imported for the first time. Any names defined in it are bound to names in the packages namespace (i.e. under the package name). The ``__init__.py`` file can contain one or more of the following: * Nothing (i.e. empty), in which case nothing happens when the package is imported. * Contain a docstring which describes the package (the docstring contents are placed in the ``__doc__`` attribute of the module object that is created). * Contain an attributed called ``__all__`` pointing to a list of string names of sub-packages to import if a specific type of import statement is used. * Contain code to initialize names and other parts of the package when it is first imported. The module object created when the package is imported is a module that has a ``__path__`` attribute defined in it, where-as a regular module object has no such ``__path__`` attribute. * The ``__path__`` attribute provides a list of paths to search for modules and sub-packages. .. _section_heading-Import: Importing --------- You gain access to the code in a module or package by importing it using the import machinery. This machinery is quite sophisticated and extensible. However, the simplest and most canonical way is by using the :py:keyword:`import` statement. Just keep in mind there are other ways. .. tip:: It is customary, but not required, to put all import statements at the top of the file with standard modules first, then 3rd party modules and then your own custom modules. The import system takes the fully qualified name of the module or package (i.e. a hierarchical dotted path to the item) being imported. For example: ``pkg_a.pkg_b.module_b0`` When a module or package is imported, the import system searches for, and then loads, the module or package (including any of the parent packages listed). The search is conducted sequentially in turn according to the following steps until the module / package is found: * Looking in the module cache (:py:data:`sys.modules`) for previously imported modules/packages and their parents. If present, the cached module is used. * Searches for names built into the interpreter (a.k.a 'built-ins') * File system paths on :py:data:`sys.path`, for the source code file or package directory, are searched. If present, the module / package is loaded. The :py:data:`sys.path` name is initialized as follows: * The directory containing the input script (or the current directory when no file is specified) * From the list of directory names given in environment variable, ``PYTHONPATH`` * From an installation-dependent default set of values, which typically gives the locations of the standard library modules / packages available to all users, as well as 3rd part modules that are only available to the current user. When the package or module is found, the import system binds a name, **in the local scope of where the import statement being executed exists**, to point to the item being imported. If a module or package can't be found during import, Python throws an :py:exc:`ImportError` exception. Once a package / module is loaded ANYWHERE in the source code, all subsequent imports use that module/package since the search will be satisfied by the module cache. This means you can't re-load / re-fresh a module / package just by importing it a second time (you must use :py:func:`importlib.reload`). .. _section_heading-Import_Examples: Import Examples ^^^^^^^^^^^^^^^ The import system offers numerous ways to specify what to import. The full rules are given in the :py:keyword:`import` syntax. Using the following example package hierarchy we can look at some common examples of its use: .. literalinclude:: pkg_hier_desc.txt :language: text :emphasize-lines: 2 The above layout has the following pieces: * A standalone module (``My_module``) which is not part of any package. * A top-level package (``Pkg_top``) which is the start of the named hierarchy. * Inside the top-level package there are 3 sub sub-packages (``Sub_pkg_a``, ``Sub_pkg_b``, ``Sub_pkg_c``). * At all levels of the hierarchy, there are ``__init__.py`` files to identify the directories as being Python packages. All the ``__init__.py`` files use the same general format and declarations inside. For example: .. literalinclude:: Pkg_top/Sub_pkg_a/__init__.py :language: python * At all levels of the hierarchy, there are py files, which are the importable modules. The importable modules all use the same general format and declarations inside. For example: .. literalinclude:: My_module.py :language: python Importing a module directly ___________________________ .. code-block:: python :emphasize-lines: 5,12 >>> import My_module Hi from My_module >>> dir() ['In' 'My_module', ..., 'quit'] >>> dir(My_module) ['__builtins__', ..., '__spec__', 'my_module_fn'] >>> My_module.my_module_fn() Hi from my_module_fn Importing a top-level package directly ______________________________________ .. code-block:: python :emphasize-lines: 6, 9 >>> import Pkg_top Hi from Pkg_top >>> dir() ['In', ..., 'Pkg_top', 'quit'] >>> dir(Pkg_top) ['NAME_FROM_PKG_TOP', '__all__', ..., '__spec__'] Importing a module from a package _________________________________ .. code-block:: python :emphasize-lines: 7, 11, 12, 20 >>> import Pkg_top.Module_top Hi from Pkg_top Hi from Pkg_top.Module_top >>> dir() ['In', 'Out', 'Pkg_top', ..., 'quit'] >>> dir(Pkg_top) ['Module_top', 'NAME_FROM_PKG_TOP', '__all__', ..., '__spec__'] >>> dir(Pkt_top.Module_top) ['__builtins__', ..., '__spec__', 'module_top_fn'] >>> Pkg_top.Module_top.module_top_fn() Hi from module_top_fn Importing a sub-package or sub-module directly ______________________________________________ .. code-block:: python :emphasize-lines: 7, 11, 12, 16 >>> import Pkg_top.Sub_pkg_a Hi from Pkg_top Hi from Pkg_top.Sub_pkg_a >>> dir() ['In', 'Out', 'Pkg_top', ..., 'quit'] >>> dir(Pkg_top) ['NAME_FROM_PKG_TOP', 'Sub_pkg_a', ..., '__spec__'] >>> dir(Pkg_top.Sub_pkg_a) ['NAME_FROM_SUB_PKG_A', '__builtins__', ..., '__spec__'] .. code-block:: python :emphasize-lines: 8, 12, 13, 17, 18, 25 >>> import Pkg_top.Sub_pkg_a.Module_a0 Hi from Pkg_top Hi from Pkg_top.Sub_pkg_a Hi from Pkg_top.Sub_pkg_a.Module_a >>> dir() ['In', 'Out', 'Pkg_top', ..., 'quit'] >>> dir(Pkg_top) ['NAME_FROM_PKG_TOP', 'Sub_pkg_a', ..., '__spec__'] >>> dir(Pkg_top.Sub_pkg_a) ['Module_a0', 'NAME_FROM_SUB_PKG_A', ..., '__spec__'] >>> dir(Pkg_top.Sub_pkg_a.Module_a0) ['__builtins__', ..., '__spec__', 'module_a0_fn'] >>> Pkg_top.Sub_pkg_a.Module_a0.module_a0_fn() Hi from module_a0_fn Importing a module into the local namespace ___________________________________________ .. code-block:: python :emphasize-lines: 7, 14 >>> from Pkg_top.Sub_pkg_a import Module_a0 Hi from Pkg_top Hi from Pkg_top.Sub_pkg_a Hi from Pkg_top.Sub_pkg_a.Module_a0 >>> dir() ['In', 'Module_a0', ..., 'quit'] >>> dir(Module_a0) ['__builtins__', '__cached__', ..., 'module_a0_fn'] >>> Module_a0.module_a0_fn() Hi from module_a0_fn The imported module becomes directly available in local scope without needing to be qualified by the preceding package name(s). .. note:: Using this syntax, names in preceding packages are NOT imported. Importing All Names ___________________ Using ``from package import *`` or ``from module import *``, will imports all public names. Public names are as follows: * If ``__all__`` is defined as a sequence of strings, the strings are considered to be the public API names to import, otherwise, * All names defined in the package / module that don't begin with an underscore ``_``. For example, recall what ``Pkg_top.__init__`` looks like: .. literalinclude:: Pkg_top/__init__.py :emphasize-lines: 10 .. code-block:: python :emphasize-lines: 9-11 >>> from Pkg_top import * Hi from Pkg_top Hi from Pkg_top.Sub_pkg_a Hi from Pkg_top.Sub_pkg_b Hi from Pkg_top.Sub_pkg_c >>> dir() ['In', 'Out', 'Sub_pkg_a', 'Sub_pkg_b', 'Sub_pkg_c', ..., 'quit'] Notice: 1) ``NAME_FROM_PKG_TOP`` was not included in ``__all__`` so it wasn't automatically imported! 2) There is no ``Pkg_top`` name in the local scope. The sub-packages were brought directly into the local scope. .. tip:: Using import * is generally frowned upon because you end up importing a set of unknown names, which dirties up your namespace, possibly re-defining names you already had. For example, importing os.path using ``*`` would bring in ~40 names into the local namespace. Changing the Name Bound to the Import Item __________________________________________ Use the ``as`` syntax to change the name referencing the imported module or package: ``from pkg_name as pkg`` ``from pkg_name import module as mod`` This can be used to avoid name conflicts with existing code and to shorten otherwise long names (i.e. to type less). For example: .. code-block:: python :emphasize-lines: 4, 12 >>> import My_module as FOO Hi from My_module >>> dir() ['FOO', 'In', ..., 'quit'] >>> dir(FOO) ['__builtins__', '__cached__', ..., 'my_module_fn'] >>> FOO.my_module_fn() Hi from my_module_fn Relative imports ________________ Use 1 or more preceding ``.``'s to access other packages elsewhere in the package hierarchy (a single leading ``.`` means the current package): ``from "."* import pkg_name`` ``from "."* import module`` For example, consider the module ``Pkg_top.Sub_pkg_c.relative_import`` with the following import statements in it: .. literalinclude:: Pkg_top/Sub_pkg_c/relative_import.py Importing ``relative_import`` would result in the following: .. code-block:: python :emphasize-lines: 11, 13, 14, 15 >>> from Pkg_top.Sub_pkg_c import relative_import Hi from Pkg_top Hi from Pkg_top.Sub_pkg_c Hi from Pkg_top.Sub_pkg_c.Module_c0 Hi from Pkg_top.Sub_pkg_a Hi from Pkg_top.Module_top >>> dir() ['In', ..., 'quit', 'relative_import'] >>> dir(relative_import) ['Module_c0', 'Module_top', 'Sub_pkg_a', ..., '__spec__'] .. tip:: Intra-package names are nice because you don't have to know the names of, or type out, the names of intervening packages to get at other packages in the hierarchy. .. warning:: Keep in mind, you can only use these WITHIN a package (the parent packages have to already be imported)! So using them in a top level program to import a package at the same level as the program, will not work. .. _section_heading-Site_Packages: Site Packages ------------- The ``site-packages`` directory is where Python places 3rd party packages that have been installed. It resides as part of Python's distribution directory unless you are working in a virtual environment. It is part of the module search path by default so it is checked during package import. You can add new modules using Pip, your OS package manager or a standalone installer. On Linux, the location of the site-packages directories are as follows: * All users: ``/usr/lib/python3/site-packages`` ``/usr/lib64/python3/site-packages`` * Local user: ``~/.local/lib/python3/site-packages`` .. _section_heading-Pip: Pip --- The ``pip`` program pulls packages from a package index and places them in the site-packages directory. The ``pip`` program takes care of resolving any additional dependencies that are required by the package being installed. By default ``pip`` fetches from the Python Package Index (`Pypi `_), but, it can be configured to fetch from other indexes or from a version control system. Pre-built packages are in the wheel format, which is essentially a zip archive of the package contents. .. note:: ``pip`` refers to packages as a synonym for a distribution (which is a collection of released software to be installed). ``pip`` comes bundled with Python as of Python v3.4. In previous versions it had to be bootstrapped in. .. warning:: There are 2 versions of the pip program: * ``pip`` - Used with Python2 * ``pip3`` - Used with Python3 Be sure to use ``pip3`` with Python3. On Linux, will need root access to install for all users. To install for just the current user, use the ``--user`` switch. Below are some common uses(remove ``--user`` to install for all users): 1) Install any version of package ``pip3 install --user pkg_name`` 2) Install package from a range of acceptable versions ``pip3 install --user 'pkg_name>=verA,<=verB'`` .. note:: Quotes are required around the pkg_name/version pair when a version specifier is used. 3) Install specific version of package ``pip3 install --user 'pkg_name==ver'`` 4) Install a manually downloaded wheel file ``pip3 install --user wheel_file.whl`` 5) Install all packages contained in a list of requirements If a text file of packages (and optionally their required versions) is created that captures all the packages required to run your app, ``pip`` can fetch them all automatically. For example, consider the following requirements.txt file: .. code-block:: text pytest == 3.4.1 # Need this specific version requests > 2.10 # Anything after 2.10 is ok Which can be passed to ``pip``: ``pip3 install --user -r requirements.txt`` Would cause ``pip`` to install what's inside it and any additional dependencies: .. code-block:: text Collecting pytest==3.4.1 (from -r requirements.txt (line 1)) Downloading pytest-3.4.1-py2.py3-none-any.whl (188kB) 100% |████████████████████████████████| 194kB 1.7MB/s Collecting requests>2.10 (from -r requirements.txt (line 2)) Downloading requests-2.18.4-py2.py3-none-any.whl (88kB) 100% |████████████████████████████████| 92kB 2.7MB/s Collecting py>=1.5.0 (from pytest==3.4.1->-r requirements.txt (line 1)) Downloading py-1.5.2-py2.py3-none-any.whl (88kB) 100% |████████████████████████████████| 92kB 3.0MB/s 6) Generate a requirements file based on installed site-packages: ``pip3 freeze --user > requirements.txt`` 7) Determine which packages are outdated ``pip3 list --outdated`` 8) Upgrading packages ``pip3 install --user --upgrade pkg_name`` 9) Upgrading Pip ``pip3 install --user --upgrade pip`` 10) Remove a package ``pip3 uninstall --user pkg_name`` This is just a quick overview. Refer to the pip `website `_ for the full details.