|mmasson 125338654b||4 months ago|
|config||9 months ago|
|post-launch||4 months ago|
|scripts||4 months ago|
|tools||7 months ago|
|.gitignore||11 months ago|
|.pre-commit-config.yaml||7 months ago|
|CHANGELOG.md||7 months ago|
|Jenkinsfile||11 months ago|
|LICENSE||1 year ago|
|README.md||7 months ago|
|install.sh||8 months ago|
|requirements.txt||7 months ago|
Linux and Windows compliant scripts for the AWS Migration Factory.
These scripts replace and add many features above the scripts given by AWS for the migration execution server.
pwsh 7with your OS package manager.
You need to have git and curl install on the machine.
# install.sh(careful, needs root permissions). You can use
--cronto force scripts installation regularly, making sure VCS and server have the same code;
/etc/migration_factory/endpoints.ymlto add URL API;
/etc/migration_factory/defaults.ymlto give the defaults values by environments;
alias mf_setup_environment="source /usr/local/bin/mf_setup_environment"
A file containing default values will be installed. See installation. When environment in passed as an argument of any script, it will check dynamically that a key corresponding to the given environment exists in the defaults file.
Here are all main environment variable:
MF_USERNAME: The username used to log on the migration factory
MF_PASSWORD: The password used to log on the migration factory
MF_CE_API_TOKEN: The Cloud Endure API token
MF_AWS_ACCESS_KEY_ID: The AWS access key id of the target account
MF_AWS_SECRET_ACCESS_KEY: The AWS secret access key of the target account
MF_AWS_REGION: The AWS region of the target account
MF_ENDPOINT_CONFIG_FILE: The location of endpoint config file
MF_WINDOWS_USERNAME: The Windows username to connect to source host
MF_WINDOWS_PASSWORD: The Windows password to connect to source host
MF_LINUX_USERNAME: The linux username to connect to source host
MF_LINUX_PASSWORD: The linux password to connect to source host
MF_LINUX_PRIVATE_KEY_PASSPHRASE: The linux passphrase for the private key file to connect to source host
MF_LINUX_PRIVATE_KEY_FILE: This linux private key file to connect to source host
Also supported for edge cases:
MF_CONFIG_FILE: Path of the YAML main configuration file
MF_ENDPOINT_CONFIG_FILE: Path of the YAML configuration file containing endpoints configuration
MF_DEFAULTS_CONFIG_FILE: Path of the YAML configuration file containing default values and environments
You can also use the command
source mf_setup_environment to set all these environment variables
See this repository wiki.
This repository follows Semantic Versioning 2.0.0
This repository uses pre-commit hooks.
pre-commit install pre-commit run -a
This repository follows the afcmf standard for it's commit messages.
0-AddProxy-Windows.pyis not working for now.
We would like to get rid of the AWS Migration Factory because of the following reasons:
AWS refused to handle feature updates of the Migration Factory. FX had an official meeting with them and the official Migration Factory project manager said it will not be possible to add any features. E.g. Private IPs addresses or GP3 are not supported and thus these scripts are made unuseful if we want to use these features.
As an alternative, FX could get the Migration Factory code and maintain it themselves. We believe that is too much work to do that, especially since we already decided to support the current scripts. Also, lambdas for Migration Factory are very badly written and would require a lot of rework. For instance, a lot of portion of the lambdas code uses now deprecated python 2.X.
Migration Factory gives very few features above the scripts of this project, mainly two things: handling environments (dev and prod) for security groups and subnets and an interface for cutover/test to cloudendure API.Factory For such little improvement, Migration Factory is a very heavy product that contains: lambdas, CloudFront distribution, API gateway, dynamoDB… Incurring costs, maintenance and complexity in our scripts code. Moreover, we believe such features are not very complex to implement in our current scripts.
Migration Factory brings added complexity when a new feature is released in cloudendure, as it is a middleware. It becomes harder to debug issues, requires time to wait for Migration Factory updates.
As stated in previous point, Migration Factory uses a lot of product from AWS, thus it costs money.
Each of the steps should be its own pull request. These steps are optimized for a quick change first, clean after approach without sacrifice of the code quality in the end.
Optional (but recommended): update all the scripts calling the Migration Factory API directly without using
At the end of this step, every script should use the
This will greatly simplify the jobs to come after.
Add support for a DB storage locally, e.g. sqlite.
Update the installation script to install/configure the chosen DBMS.
requirements.txt to add support for the chosen DBMS
Adapt code to uses local DB storage.
Update the code of
Each call to the AWS Migration Factory must stay for now, but each function should duplicated the create/update/delete operations in the local DB system.
You will probably need to create new library classes to handle inserts/updates/deletes for the DB system.
Import this library in
Do not make any changes in any scripts, except the ones calling migration factory directly (not necessary if you did the first step).
mf.migration_factory.MigrationFactoryRequester.launch_target(). This method will talk to the cloudEndure API directly, using local DB data. Test extensively.
Remove all code making requests to Migration Factory in
Except the term, no requests to Migration Factory should be done by any scripts after this step.
mf.migration_factory lib classes/method to a new namespace named
mf.core_database (or a better name if you think about one).
Update all the scripts to use this new classes instead of the “old” calls to
At the end of this step, the terms “Migration Factory” to refer to AWS Migration Factory should have entirely disappeared from the code.
Update this README to remove these instructions. Remove any trace of MigrationFactory on any client, by running a destroy of the MF CloudFormation stack.