Python Project Life Cycle - Java to Python
This document is written by keeping experienced java developers in mind. It might be helpful to others but given references only java developers can understand.
This post helps java developers to understand differences between Java and Python project structure, building, installing and publishing packages. It will give an idea of what need to be done in each of these phases. Step by step by instructions were not given in this post.
Installing
https://wiki.python.org/moin/BeginnersGuide/Download
Checking installation
python -V # prints the python and ersion exits. That means Python installed successfully
python --version # also does the samething.
Compiling
Python is interpreter. No need to compile like Java
Running
python filename.py # including extension. For java no need to submit extension .class
python -m module.name # java -jar option takes jar file name. Python take full module name. Full module name is similar to java class name including package
Entry Points
public static void main(String[] args) is java developers favorite method. There is nothing like main method in Python, Because it is interpreter program execution starts from first global line. Global line is a line which is not part of Python class or method.
A note about __main__
Take a look at following code snippet. The if condition is satisfied only when this program is called from interactive prompt, script file or from standard input.
if __name__ == "__main__":
print "If you are running this program, this line will be printed"
print "This is not equivalent to Java main method, you can think it like that"
If a.py has above code and a.py is called from b.py the above code will not be executed
Entry points for a package
Do you remember you used Manifest.txt and
Main-Class: MyPackage.MyClass while creating java jar files. There is a similar concept in Python too. setuptools.setup method takes entry_points argument. This entry_points is a dictionary and you can pass a console_scripts to specify alias. This alias will be pointing to a python module's method. If run that alias corresponding methods is invoked.
sys.path and PYTHONPATH for CLASSPATH
Java's CLASSPATH contains list of location when it is finding jars and classes. sys is Python's library that has access to its runtime. sys.path contains list of directories where it finds its python modules. sys.path includes current directory, PYTHONPATH, and installation dependent entries. If environment variable PYTHONPATH is changed sys.path updates automatically. Just like order in Java CLASSPATH matters order in sys.path also matters
Package
All the java programs first line is package com.mycompany.devision.project. Python does not have keyword like package. Python packages its modules similar to java packages. This package is implicitly derived based on Python project's home directory and python module location.
Let us say you have a a/b.py file. And in b.py you have class C. C's qualified name is a.b.C
Java's common practice is .java file can have only one public class. But Python is different it can more than one classes in one python file.
__init__.py
__init__.py is a special file in Python. If this file exists then only Python and its tools consider corresponding directory as a package and all its .py files will be part of Python package
Importing
Importing happens in Python similar to Java. But it has two varieties.
- import x.y.z
- from x.y import z
Python Project
Java developers are using maven to create java projects. Before maven they used to create every folder on their own as per their need/design. Github has a lot of python skeleton projects those helps to create a Python project from scratch. I observed openstack projects like neutron, nova. Openstack project structure is little different from tools like openstack skeleton. Openstack came up with some tools/python modules like
run_tests.sh. These tools works good with OpenStack project structure only.
tox is also working good with openstack project structure
Following directories/files copied from nova project
- AUTHORS # this file contents authors names and email addresses
- babel.cfg
- ChangeLog # for each release release notes
- contrib
- CONTRIBUTING.rst # Ways to contribute this project
- doc #
- etc # location for .conf files
- HACKING.rst # Coding guide lines
- LICENSE
- MANIFEST.in # instruction to setup.py to include and exclude file in package distribution
- nova # Actual project diretory. The only directory that has .py files.
- openstack-common.conf
- PKG-INFO
- plugins
- README.rst
- requirements.txt # convenient file for user or third party tools to install required dependencies
- run_tests.sh # this is openstack specific. to check pep8 standards and to run test cases.
- setup.cfg # Provides more configuration information to setup.py
- setup.py # builds and installs this project.
- test-requirements.txt # similar to requirements.txt used by run_tests.sh and tox
- tools
- tox.ini # gives inputs to tox
If you look at the above file list to create a core Python project you need only file/folders in red color and some of them are optional. Let us see what is required at what stage. I am choosing a project name j2p (java to python) for this article.
- First you will be creating a directory my-j2p. This is the project name on git repository. It contains all python, conf, doc files.
- Then you create j2p This is the python project name. This is the root for all python modules.
- __init__.py to know more about this please refer package section above.
And this point of time you can add python modules to folder j2p and run them from the directory
my-j2p. Not from any where else.
Now question is how to use this my-j2p in other projects? How did it work on Java projects?
- Include my-j2p in CLASSPATH so that on the same system you can use it in other projects
- Create jar file and include that jar file in other projects lib
- Create a jar file and copy it to JAVA_HOME/lib/ext ( This is NOT a common practice)
- Publish jar file to maven repository and others will download from there.
In Python world too you have simalar options.
- Include my-j2p in PYTHONPATH. On the same system you can use it in other projects
- Create .tar and distribute it. Users can unzip it update PYTHONPATH to point to it and use it.
- Create a package and install it in /usr/../python.x/dist_packages (How to do? It will come in next sections)
- Create a package and publish it to http://pypi.python.org. How to do it? That is not in the scope of this article. Then others can download from there. More details on installing packages. Synopsis will be given in this article.
Creating a package
Python projects are distributed as package. They are not called as distributions to avoid confusion with Linux distributions. To build a python project you need setup.py python module in your my-j2p folder. python setup.py install builds and installs your python package to /usr/local/lib/pythonx.x/dist_packages. That is part of sys.path so all other python programs running on same version of python should be able to access this package. You can do more research setup.py and setup.cfg to utilize other available options for this step. Installing to your own system is not sufficient for you in real world. You need to publish it to http://pypi.python.org so that other Python developer can reuse your hardwork.
Publishing Packages
The above section explained how to build and install Python packages to local system. How to publish them to http://pypi.python.org is out of scope of this document and more details available
here.
Installing Packages
Once you publish your packages other can install your packages as explained
here The
pip (Python Install Packages) tool used for installing python packages. Most frequently used commands are as follows
pip install <<packge_name>> # it download and install the package from http://pypi.python.org. The magic while installing one package it also installs all required packages.
pip list # lists all the packages installed on your system
pip list | grep <<package_name>> easy way to find is required package installed or not
pip search <<query>> # easy way to search available package on http://pypi.python.org
pip uninstall <<package_name>>
pip install -r requirements.txt # explained more in next section.
Resolving Dependencies
All looks simple so far. But once project is growing bigger it has to depend on so many other python packages. This kind of issues are not new to java developer before maven. Maven made java developers life easy to resolve jar dependency issues.
There is similar concept in Python too and that is requirements.txt file. This file will be under my-j2p. Examples contents of requirements.txt are as follows
pbr>=0.6,!=0.7,<1.0
SQLAlchemy>=0.9.7,<=0.9.99
boto>=2.32.1
decorator>=3.4.0
requirements.txt supposed to follow certain syntax. When it is passed to pip install -r pip takes each line and installs that package including dependency packages too. That gives complete environment required to run your package (j2p)
Configuring the projects
By this time you understood how to code, build, install to local, publish, download and instll. But while coding your project you need to pass some configuration files. Java developer use properties files to do this. Python projects use similar concept. Python uses
.conf files. All the openstack projects keep these configuration files under
my-j2p/etc/ folder. Common .conf file names are
j2p.conf, j2p-api.conf, j2p-db.conf. How to read these .conf files will be explained in next posts . For quick reference you can have a look at
oso_config
Python Project Structure
So a simple Python project structure looks like this.
In this image you can see main_like.py. That could be like your java project's java program with main method.
To see how setup is using main_like.py look at previous section creating package.
