1. Introduction

1.1. Overview

At the most fundamental level, an “execution” or a “run” of any data-processing can be thought like that:

      .--------------.     _____________        .-------------.
     ;  DataTree    ;    |             |      ;   DataTree   ;
    ;--------------; ==> |  <cfunc_1>  | ==> ;--------------;
   ; /some/data   ;      |  <cfunc_2>  |    ; /some/data   ;
  ;  /some/other ;       |     ...     |   ;  /some/other ;
 ;   /foo/bar   ;        |_____________|  ;   /foo/bar   ;
'--------------'                         '--------------.
  • The data-tree might come from json, hdf5, excel-workbooks, or plain dictionaries and lists. Its values are strings and numbers, numpy-lists, pandas or xray-datasets, etc.

  • The component-functions must abide to the following simple signature:

    cfunc_do_something(pandelone, datatree)
    

    and must not return any value, just read and write into the data-tree.

  • Here is a simple component-function:

    def cfunc_standardize(pandelone, datatree):
        pin, pon = pandelone.paths(),
        df = datatree.get(pin.A)
        df[pon.A.B_std] = df[pin.A.B] / df[pin.A.B].std()
    
  • Notice the use of the relocatable-paths marked specifically as input or output.

  • TODO: continue rough example in tutorial…

1.2. Quick-start

The program runs on Python-3.5+ and requires numpy, pandas and (optionally) win32 libraries along with their native backends.

pip install pandalone                 ## Use `--pre` if version-string has a build-suffix.

… but probably you need the following for xleash to work:

pip install pandalone[xlrd]

All “extras” are: test, doc, excel, pandas, xlrd, dev, all

In case you need the very latest from master branch :

pip install git+https://github.com/pandalone/pandalone.git

Or in to install in develop mode, with all dependencies needed for development, and with pre-commit hook for auto-formatting python-code with black, clone locally this project from the remote repo, and run:

pip install -e <pandalone-dr>[dev]
pre-commit install

1.3. Project files and folders

The files and folders of the project are listed below:

+--pandalone/       ## (package) Python-code
+--tests/           ## (package) Test-cases
+--doc/             ## Documentation folder
+--setup.py         ## (script) The entry point for `setuptools`, installing, testing, etc
+--requirements/    ## (txt-files) Various pip and conda dependencies.
+--README.rst
+--CHANGES.rst
+--AUTHORS.rst
+--CONTRIBUTING.rst
+--LICENSE.txt

1.4. Discussion