Case Studies¶

All cases below have been run using conda version 4.3 or above. The cases where the target system is not conda are only recommended for expert users since the system-wide packages may need to be installed and user needs to be aware of what packages to install.

For all case studies, <ip-address> should be replaced with the IP address of the target system and <conda-prefix> should be replaced with the directory under which anaconda or miniconda is installed (eg. /home/joe/miniconda3).

Hello World over network¶

This is a basic use case which demonstrates how you can migrate simple script from one platform to another over a network. The script used for migration is a simple Python script that prints ‘Hello world’ and is located at tests/basic-migration/simple-app/simple_app.py in the git repository.

Two different scenarios, one where the target system is conda and one where the target system is Archlinux were run. The base system used was an Ubuntu 16.04 OS running inside VirtualBox, but it can be any system running conda and bash shell. Miniconda3 was installed on the base system by following the instructions at the Anaconda installation page

Target system with conda¶

First, a conda environment was created on the base system with

<conda-prefix>/bin/conda create -n simple-app python=3
. <conda-prefix>/bin/activate simple-app

The script was created in the home directory with

mkdir -pv simple-app
echo "print('Hello world')" > simple-app/simple_app.py

On the target system, the platform migrator was installed and the server was started with

pip3 install --user git+ssh://git@gitlab.com/mmc691/platform-migrator.git
platform-migrator server --host <IP-address> start

Now, back on the base system, the script was transferred with

curl http://<IP-address>:9001/migrate > script.py
python script.py

When prompted, for the application name, simple-app was entered, and for the directory, simple-app was entered. A zip file was not created by entering n on the prompt and the software was transferred over the network.

Note:: The python script.py command can also be run as python script.py -n simple-app -d simple-app/ to avoid the prompts.

After this, on the target system, a test configuration was created with the below contents and saved as test-config.ini

cat > test-config.ini << EOF
[simple-app]
package_managers = conda
output_dir = /tmp
tests = hello-world

[test_hello-world]
exec = python simple_app.py
EOF

Note:: If you are trying to reproduce the case, you will have to manually create this file in a text editor. It is not generated by platform migrator.

Platform migrator was then executed with the following command

CONDA_HOME=<conda-prefix> platform-migrator migrate simple-app test-config.ini

It was verified that the package is available in /tmp by executing

ls -l /tmp
ls -l /tmp/simple-app

Target system with pacman¶

The initial steps of installing conda on the base system and platform migrator on the target system were completed as described above. The script was also transferred to the base system using curl command and was executed. However, on being prompted for the application name, simple-app-pacman was entered to avoid conflicts with the previous case. On the target system, the previously created test-config.ini was edited and the following content was added to it

[simple-app-pacman]
package_managers = pacman
output_dir = /tmp
tests = hello-world

Platform migrator was then executed with the following command

platform-migrator migrate simple-app-pacman test-config.ini

Since the script only requires Python, all packages other than python were skipped from installation. The tests were run and it was verified that the software was available in /tmp with

ls -l /tmp
ls -l /tmp/simple-app-pacman

This use case is available as a regression test in the git repository as tests/basic-migration/test_simple_app.py and the test config file is available as tests/basic-migration/test-config.ini. The regression test uses two different conda environments on the local machine instead of a VM or a remote machine.

Hello World using zip file¶

This is a basic use case which demonstrates how you can migrate simple script from one platform to another using a zip file. The script used for migration is a simple Python script that prints ‘Hello world’ and is located at tests/basic-migration/simple-app/simple_app.py in the git repository.

Two different scenarios, one where the target system is conda and one where the target system is Archlinux were run. The base system used was an Ubuntu 16.04 OS running inside VirtualBox, but it can be any system running conda and bash shell. Miniconda3 was installed on the base system by following the instructions at the Anaconda installation page

Target system with conda¶

First, a conda environment was created on the base system with

<conda-prefix>/bin/conda create -n simple-app python=3
. <conda-prefix>/bin/activate simple-app

The script was created in the home directory with

mkdir -pv simple-app
echo "print('Hello world')" > simple-app/simple_app.py

Next, platform migrator was installed and the zip file for the software was created with

pip3 install --user git+ssh://git@gitlab.com/mmc691/platform-migrator.git
platform-migrator pack

When prompted, for the application name, simple-app was entered, and for the directory, simple-app was entered. A zip file was created by entering y on the prompt and it was copied manually to the target system.

Note:: The platform-migrator pack command can also be run as platform-migrator pack -n simple-app -d simple-app/ to avoid the prompts.

Now, on the target system, platform migrator was installed and the software was setup for migration using

pip3 install --user git+ssh://git@gitlab.com/mmc691/platform-migrator.git
platform-migrator unpack simple-app.zip

After this, on the target system, a test configuration was created with the below contents and saved as test-config.ini

cat > test-config.ini << EOF
[simple-app]
package_managers = conda
output_dir = /tmp
tests = hello-world

[test_hello-world]
exec = python simple_app.py
EOF

Note:: If you are trying to reproduce the case, you will have to manually create this file in a text editor. It is not generated by platform migrator.

Platform migrator was then executed with the following command

CONDA_HOME=<conda-prefix> platform-migrator migrate simple-app test-config.ini

It was verified that the package is available in /tmp by executing

ls -l /tmp
ls -l /tmp/simple-app

Target system with pacman¶

The initial steps of installing conda and platform migrator on the base system were completed as described above. A zip file was created once again by using the platform-migrator pack command and was copied to the target system. On being prompted for the application name, simple-app-pacman was entered to avoid conflicts with the previous case. Again as described above, platform migrator was installed on the target system and the software was unpacked using platform-migrator unpack.

On the target system, the previously created test-config.ini was edited and the following content was added to it

[simple-app-pacman]
package_managers = pacman
output_dir = /tmp
tests = hello-world

Platform migrator was then executed with the following command

platform-migrator migrate simple-app-pacman test-config.ini

Since the script only requires Python, all packages other than python were skipped from installation. The tests were run and it was verified that the software was available in /tmp with

ls -l /tmp
ls -l /tmp/simple-app-pacman

This use case is available as a regression test in the git repository as tests/basic-migration/test_simple_app.py and the test config file is available as tests/basic-migration/test-config.ini. The regression test uses two different conda environments on the local machine instead of a VM or a remote machine.

scikit-learn and scikit-image over network¶

Similar to the Hello World case, this case was run from an Ubuntu 16.04 VM system, once with conda and once with pacman and pip as the target package managers.

The script used for the software is available as tests/basic-migration/scikit-app/scikit_app.py in the git repo. It was saved on the base system as ~/scikit-app/scikit_app.py.

Target system with conda¶

First, a conda environment was created on the base system with

<conda-prefix>/bin/conda create -n scikit-app python=2 scikit-learn scikit-image
. <conda-prefix>/bin/activate scikit-app

The script was created in the home directory with

git clone https://gitlab.com/mmc691/platform-migrator
cp -Rv platform-migrator/tests/basic-migration/scikit-app .

On the target system, platform migrator was installed and the server was started with

pip3 install --user git+ssh://git@gitlab.com/mmc691/platform-migrator.git
platform-migrator server --host <IP-address> start

Now, back on the base system, the script was transferred using

curl http://<IP-address>:9001/migrate > script.py
python script.py

When prompted, for the application name, scikit-app was entered, and for the directory, scikit-app was entered. A zip file was not created by entering n on the prompt and the software was transferred over the network.

Note:: The python script.py command can also be run as python script.py -n scikit-app -d scikit-app/ to avoid the prompts.

Since pip is listed before pacman, packages are first searched for in pip and only if the user decides not to install from pip, are they searched for in pacman. Also, pip2 was explicitly specified since it is known beforehand that the code works only for Python 2. This requires that pip2 is installed on the target system prior to using platform migrator.

Platform migrator was then executed with the following command

platform-migrator migrate scikit-app-pacman test-config.ini

If everything is successful, the application script will be installed under /tmp.

Target system with aptitude and pip¶

The initial steps of installing conda on the base system and platform migrator on the target system were completed as described above. The script was also transferred to the base system using curl command and was executed. However, on being prompted for the application name, scikit-app-apt was entered to avoid conflicts with the previous case. On the target system, the previously created test-config.ini was edited and the following content was added to it

[simple-app-apt]
package_managers = pip2, aptitude
output_dir = /tmp
tests = scikit-app

Since pip is listed before aptitude, packages are first searched for in pip and only if the user decides not to install from pip, are they searched for in aptitude.

Platform migrator was then executed with the following command

platform-migrator migrate scikit-app-apt test-config.ini

If everything is successful, the application script will be installed under /tmp.

The conda portion of this use case is available as a regression test in the git repository as tests/basic-migration/test_scikit_app.py and the test config file is available as tests/basic-migration/test-config.ini. The regression test uses two different conda environments on the local machine instead of a VM or a remote machine.

scikit-learn and scikit-image using zip file¶

This case study is exactly the same as the previous one, except the software was sent over to the target system as a zip file created using platform-migrator pack. This was also run from an Ubuntu 16.04 VM system, once with conda and once with pacman and pip as the target package managers.

The script used for the software is available as tests/basic-migration/scikit-app/scikit_app.py in the git repo. It was saved on the base system as ~/scikit-app/scikit_app.py.

Target system with conda¶

First, platform migrator was installed on the base system with

pip3 install --user platform-migrator

Next, a conda environment was created on the base system (same as above) with

<conda-prefix>/bin/conda create -n scikit-app python=2 scikit-learn scikit-image
. <conda-prefix>/bin/activate scikit-app

The script was created in the home directory with

git clone https://gitlab.com/mmc691/platform-migrator
cp -Rv platform-migrator/tests/basic-migration/scikit-app .

Then, the a zip file was created using

platform-migrator pack

When prompted, for the application name, scikit-app was entered, and for the directory, ~/scikit-app was entered. Here, a zip file was generated by entering y when prompted to create the file. The zip file was then emailed to the target system.

Note:: The platform-migrator pack command can also be run as platform-migrator pack -n scikit-app -d scikit-app/ to avoid the prompts.

After this, on the target system, a test configuration was created with the below contents and saved as test-config.ini

cat > test-config.ini << EOF
[scikit-app]
package_managers = conda
output_dir = /tmp
tests = scikit-app

[test_scikit-app]
exec = python2 scikit_app.py
EOF

Note:: If you are trying to reproduce the case, you will have to manually create this file in a text editor. It is not generated by platform migrator.

Platform migrator was then executed with the following command

CONDA_HOME=<conda-prefix> platform-migrator migrate scikit-app test-config.ini

If everything is successful, the application script will be installed under /tmp.

Target system with pacman and pip¶

The initial steps of installing conda on the base system and platform migrator on the target system were completed as described above. The script was also transferred to the base system using curl command and was executed. However, on being prompted for the application name, scikit-app-pacman was entered to avoid conflicts with the previous case. On the target system, the previously created test-config.ini was edited and the following content was added to it

[simple-app-pacman]
package_managers = pip2, pacman
output_dir = /tmp
tests = scikit-app

On the target system, platform migrator was installed and the zip file was unpacked with

pip3 install --user git+ssh://git@gitlab.com/mmc691/platform-migrator.git
platform-migrator unpack <zip-file>

The rest of the steps are same as when transferring over a network, but are repeated here for the sake of completeness. After this, on the target system, a test configuration was created with the below contents and saved as test-config.ini

cat > test-config.ini << EOF
[scikit-app]
package_managers = conda
output_dir = /tmp
tests = scikit-app

[test_scikit-app]
exec = python2 scikit_app.py
EOF

Note:: If you are trying to reproduce the case, you will have to manually create this file in a text editor. It is not generated by platform migrator.

Platform migrator was then executed with the following command

CONDA_HOME=<conda-prefix> platform-migrator migrate scikit-app test-config.ini

If everything is successful, the application script will be installed under /tmp.

Target system with pacman and pip¶

The initial steps of installing conda on the base system and platform migrator on the target system were completed as described above. The zip file was created and setup to the target system using pack and unpack commands. On being prompted for the application name, scikit-app-pacman was entered to avoid conflicts with the previous case. On the target system, the previously created test-config.ini was edited and the following content was added to it

[simple-app-pacman]
package_managers = pip2, pacman
output_dir = /tmp
tests = scikit-app

Since pip is listed before pacman, packages are first searched for in pip and only if the user decides not to install from pip, are they searched for in pacman. Also, pip2 was explicitly specified since it is known beforehand that the code works only for Python 2. This requires that pip2 is installed on the target system prior to using platform migrator.

Platform migrator was then executed with the following command

platform-migrator migrate scikit-app-pacman test-config.ini

If everything is successful, the application script will be installed under /tmp.

Target system with aptitude and pip¶

The initial steps of installing conda on the base system and platform migrator on the target system were completed as described above. The zip file was created and setup to the target system using pack and unpack commands. On being prompted for the application name, scikit-app-apt was entered to avoid conflicts with the previous case. On the target system, the previously created test-config.ini was edited and the following content was added to it

[simple-app-apt]
package_managers = pip2, aptitude
output_dir = /tmp
tests = scikit-app

Since pip is listed before aptitude, packages are first searched for in pip and only if the user decides not to install from pip, are they searched for in aptitude.

Platform migrator was then executed with the following command

platform-migrator migrate scikit-app-apt test-config.ini

If everything is successful, the application script will be installed under /tmp.

The conda portion of this use case is available as a regression test in the git repository as tests/basic-migration/test_scikit_app.py and the test config file is available as tests/basic-migration/test-config.ini. The regression test uses two different conda environments on the local machine instead of a VM or a remote machine.

OpenAlea¶

This case was only done for conda to conda migration due to the complex dependencies required to install OpenAlea. The base system used was an Ubuntu 14.04 server with miniconda2 installed. The target system used was Archlinux with miniconda3 installed.

First, on the base system, a list of modules to install was obtained by searching the OpenAlea channel using

conda search -c OpenAlea --override-channels openalea.*
conda search -c OpenAlea --override-channels vplants.*
conda search -c OpenAlea --override-channels alinea.*

A conda environment was created on the base system with the above modules using the following commands

conda create -n openalea
. <conda-prefix>/bin/activate openalea
conda install -c OpenAlea <all-openalea-modules> <all-vplants-modules> <all-alinea-modules>

A simple script was created to see if all packages can be imported. This script was used as the application to migrate.

cd ~
mkdir openalea
cat > openalea/test.py <<< EOF
try:
    import openalea, vplants, alinea
except ImportError:
    print('OpenAlea install failed')
    exit(-1)
else:
    print('OpenAlea installed succesfully')
    exit(0)
EOF

Transferring over network¶

When using the network to transfer the software, the server was started on the target system by executing platform-migrator server --host <ip-address> start

After that, the request was made to the platform-migrator server and the environment and the test script were sent over. The conda environment is still active on the base the system.

curl http://<ip-address>:9001/migrate > script.py
python script.py

The details were entered when prompted and a zip file was not created. After this, the steps from the Installation and Testing section below were executed.

Note:: The python script.py command can also be run as python script.py -n openalea -d openalea/ to avoid the prompts.

Transferring using zip file¶

When using a zip file to transfer the software, platform migrator was installed on the base system and the software was packed with platform-migrator pack -n openalea -d openalea.

The zip file was manually transferred to the target system using scp. Then, it was unpacked on the target system with platform-migrator unpack openalea.zip.

Installation and Testing¶

Now, on the target system, the test-config.ini file was created with

cat > test-config.ini << EOF
[openalea]
package_managers = conda
output_dir = /tmp
tests = import_openalea

[test_import_openalea]
exec = python test.py
EOF

Note:: If you are trying to reproduce the case, you will have to manually create this file in a text editor. It is not generated by platform migrator.

The migration was performed with

platform-migrator migrate openalea test-config.ini

If the conda environment was succesfully created, the test will pass otherwise it will fail.

Configuring apt-get¶

This case study describes how apt-get package manager was configured in the config/package-managers.ini file. This is the typical process that should be followed for configuring a new package manager.

First, a new section called apt-get was created in the config file. Then, the various options were added without any values.

[apt-get]
name =
exec =
install =
install_version =
search =
result_fmt =

The name and exec options were set to apt-get since the executable is called apt-get.

The installation of a new package is done using sudo apt-get -y install <package-name>. So, the install option was set to sudo %e install. The %e wildcard is replaced by the value in exec option and the package name is automatically append to the end of the command by platform migrator since the %p wildcard is not specified.

To install a specific version of the package in apt-get, - is used as a delimiter. So, the install_version option was set to sudo %e apt-get %p-%v and the %p and %v wildcards were used to specify the position of the package name and version.

Now, packages are searched using the apt-cache command. Since platform migrator searched using package name and version only, the -n flag is specified to apt-cache search so that package descriptions are not searched. However, the results from this still contain a small description for the package. So, the description needs to trimmed using other tools typically available on OSes which use apt-get. First, the results are piped into awk with the field delimiter set to ' - '. The spaces around the hyphen make sure that the package name and description are split but the version and build string are part of the name. Only the name is printed out using '{print $1}' command. So, the search command now looks like apt-cache search -n %s | awk -F ' - ' '{print $2}'. Here, the %s wildcard replaced by the search expression.

Since the results are now in a format suitable to use for the install command, each result is treated as a separate package. So, the result_fmt option is set to %p. The final config section looks like

[apt-get]
name = apt-get
exec = apt-get
install = sudo %e -y install
install_version = sudo %e -y install %p-%v
search = apt-cache search -n %s | awk -F ' - ' '{print $2}'
result_fmt = %p

For other default package managers, a similar process was followed.