Load user datas
===============

User datas are values setup by user for configuration variables.

There is differents types of user datas for differents sources types.

We can cumulate user datas loader.

For this section, we will use :file:`dict/00-base.yml` a structure file:

.. code-block:: yaml

    %YAML 1.2
    ---
    version: 1.1

    my_variable: my value   # My first variable

    my_boolean_variable: true  # My boolean variable

    my_integer_variable: 1  # My integer variable

    my_secret_variable:
      description: My secret variable
      type: secret

    my_hidden_variable:
      description: My hidden variable
      hidden: true
    ...

Here is the first script which is load this file:

.. code-block:: python
    :caption: the :file:`script.py` file content

    from rougail import Rougail, RougailConfig

    RougailConfig["main_namespace"] = None
    RougailConfig["main_structural_directories"] = ["dist/"]
    rougail = Rougail()
    config = rougail.run()
    print(config.value.get())

Let's execute `script.py`:

.. code-block:: bash

    $ python3 script.py 
    {<TiramisuOption path="my_variable">: 'my value', <TiramisuOption path="my_boolean_variable">: True, <TiramisuOption path="my_integer_variable">: 1, <TiramisuOption path="my_secret_variable">: None}

YAML
----

We want to load this YAML file with value define by user:

.. code-block:: yaml

    ---
    my_variable: a new value

    my_boolean_variable: false

    my_integer_variable: 10

    my_secret_variable: MyVeryStrongPassword

Here is the script which is load user data from the YAML file:

.. code-block:: python
    :caption: the :file:`script.py` file content

    from rougail import Rougail, RougailConfig
    from rougail.user_data_yaml import RougailUserDataYaml

    RougailConfig["main_namespace"] = None
    RougailConfig["main_structural_directories"] = ["dist/"]
    RougailConfig["step.user_data"] = ["yaml"]
    RougailConfig["yaml.filename"] = ["dist.yml"]
    rougail = Rougail()
    config = rougail.run()

    user_datas = RougailUserDataYaml(config).run()
    rougail.user_datas(user_datas)
    print(config.value.get())

Let's execute `script.py`:

.. code-block:: bash

    $ python3 script.py 
    {<TiramisuOption path="my_variable">: 'a new value', <TiramisuOption path="my_boolean_variable">: False, <TiramisuOption path="my_integer_variable">: 10, <TiramisuOption path="my_secret_variable">: 'MyVeryStrongPassword'}

Set a secret in clear text file is not always a good idea.
This is why the `yaml.file_with_secrets` parameter allows you to define whether files define in `yaml.filename` can contain a secret and which one:

- all: all file can contains secret
- first: only the first file can contains secret
- last: only the last file can contains secret
- none: no file can contains secret

.. code-block:: python
    :caption: the :file:`script.py` file content

    from rougail import Rougail, RougailConfig
    from rougail.user_data_yaml import RougailUserDataYaml

    RougailConfig["main_namespace"] = None
    RougailConfig["main_structural_directories"] = ["dist/"]
    RougailConfig["step.user_data"] = ["yaml"]
    RougailConfig["yaml.filename"] = ["dist.yml"]
    RougailConfig["yaml.file_with_secrets"] = "none"
    rougail = Rougail()
    config = rougail.run()

    user_datas = RougailUserDataYaml(
         config,
    ).run()
    rougail.user_datas(user_datas)
    print(config.value.get())

Let's execute `script.py`:

.. code-block:: bash

    $ python3 script.py 
    {<TiramisuOption path="my_variable">: 'a new value', <TiramisuOption path="my_boolean_variable">: False, <TiramisuOption path="my_integer_variable">: 10, <TiramisuOption path="my_secret_variable">: None}

Environment variables
---------------------

We can define use data from environment variables. The environment name is a "prefix" (ROUGAIL by default) with "_" and variable name in uppercase format.

For example:

- `my_variable` has `ROUGAIL_MY_VARIABLE` as a environment variable name
- `my_family.my_variable` has `ROUGAIL_MY_FAMILY.MY_VARIABLE` as a environment variable name

Some shell doesn't allow dot in environment file. In this case use the command "env".

For example: `env ROUGAIL_MY_FAMILY.MY_VARIABLE="value" ./script.py`.

Here is the script:

.. code-block:: python
    :caption: the :file:`script.py` file content

    from rougail import Rougail, RougailConfig
    from rougail.user_data_environment import RougailUserDataEnvironment

    RougailConfig["main_namespace"] = None
    RougailConfig["main_structural_directories"] = ["dist/"]
    RougailConfig["step.user_data"] = ["environment"]
    rougail = Rougail()
    config = rougail.run()

    user_datas = RougailUserDataEnvironment(config).run()
    rougail.user_datas(user_datas)
    print(config.value.get())

Let's execute `script.py`:

.. code-block:: bash

    env ROUGAIL_MY_VARIABLE="a new value" ROUGAIL_MY_BOOLEAN_VARIABLE="False" ROUGAIL_MY_INTEGER_VARIABLE=10 ROUGAIL_MY_SECRET_VARIABLE="MyVeryStrongPassword" python script.py
    {<TiramisuOption path="my_variable">: 'a new value', <TiramisuOption path="my_boolean_variable">: False, <TiramisuOption path="my_integer_variable">: 10, <TiramisuOption path="my_secret_variable">: 'MyVeryStrongPassword'}

We can redefine the prefix with `environment.default_environment_name` (prefix is always uppercase characters):

.. code-block:: python
    :caption: the :file:`script.py` file content

    from rougail import Rougail, RougailConfig
    from rougail.user_data_environment import RougailUserDataEnvironment

    RougailConfig["main_namespace"] = None
    RougailConfig["main_structural_directories"] = ["dist/"]
    RougailConfig["step.user_data"] = ["environment"]
    RougailConfig["environment.default_environment_name"] = "EX"
    rougail = Rougail()
    config = rougail.run()

    user_datas = RougailUserDataEnvironment(config).run()
    rougail.user_datas(user_datas)
    print(config.value.get())

Let's execute `script.py`:

.. code-block:: bash

    env EX_MY_VARIABLE="a new value" EX_MY_BOOLEAN_VARIABLE="False" EX_MY_INTEGER_VARIABLE=10 EX_MY_SECRET_VARIABLE="MyVeryStrongPassword" python script.py
    {<TiramisuOption path="my_variable">: 'a new value', <TiramisuOption path="my_boolean_variable">: False, <TiramisuOption path="my_integer_variable">: 10, <TiramisuOption path="my_secret_variable">: 'MyVeryStrongPassword'}

If you define a `main_namespace` or `extra_namespaces`, the `environment.default_environment_name` is automaticly define with the name of the namespace in uppercase. And the separator is no more "_" but ".":

.. code-block:: python
    :caption: the :file:`script.py` file content

    from rougail import Rougail, RougailConfig
    from rougail.user_data_environment import RougailUserDataEnvironment

    RougailConfig["main_namespace"] = "main"
    RougailConfig["main_structural_directories"] = ["dist/"]
    RougailConfig["step.user_data"] = ["environment"]
    rougail = Rougail()
    config = rougail.run()

    user_datas = RougailUserDataEnvironment(config).run()
    rougail.user_datas(user_datas)
    print(config.value.get())

Let's execute `script.py`:

.. code-block:: bash

    env MAIN.MY_VARIABLE="a new value" MAIN.MY_BOOLEAN_VARIABLE="False" MAIN.MY_INTEGER_VARIABLE=10 MAIN.MY_SECRET_VARIABLE="MyVeryStrongPassword" python script.py
    {<TiramisuOption path="main">: {<TiramisuOption path="main.my_variable">: 'a new value', <TiramisuOption path="main.my_boolean_variable">: False, <TiramisuOption path="main.my_integer_variable">: 10, <TiramisuOption path="main.my_secret_variable">: 'MyVeryStrongPassword'}}

Set a secret in clear variable environment is not always a good idea.
This is why the `environment.with_secrets` parameter allows you to reject secret from environment variable:

.. code-block:: python
    :caption: the :file:`script.py` file content

    from rougail import Rougail, RougailConfig
    from rougail.user_data_environment import RougailUserDataEnvironment

    RougailConfig["main_namespace"] = None
    RougailConfig["main_structural_directories"] = ["dist/"]
    RougailConfig["step.user_data"] = ["environment"]
    RougailConfig["environment.with_secrets"] = False
    rougail = Rougail()
    config = rougail.run()

    user_datas = RougailUserDataEnvironment(config).run()
    rougail.user_datas(user_datas)
    print(config.value.get())

Let's execute `script.py`:

.. code-block:: bash

    env ROUGAIL_MY_VARIABLE="a new value" ROUGAIL_MY_BOOLEAN_VARIABLE="False" ROUGAIL_MY_INTEGER_VARIABLE=10 ROUGAIL_MY_SECRET_VARIABLE="MyVeryStrongPassword" python script.py
    {<TiramisuOption path="my_variable">: 'a new value', <TiramisuOption path="my_boolean_variable">: False, <TiramisuOption path="my_integer_variable">: 10, <TiramisuOption path="my_secret_variable">: None}

Comand line parser user data
----------------------------

Value can be define directly with command line arguments:

.. code-block:: python
    :caption: the :file:`script.py` file content

    from rougail import Rougail, RougailConfig
    from rougail.user_data_commandline import RougailUserDataCommandline

    RougailConfig["main_namespace"] = None
    RougailConfig["main_structural_directories"] = ["dist/"]
    RougailConfig["step.user_data"] = ["commandline"]
    rougail = Rougail()
    config = rougail.run()

    user_datas = RougailUserDataCommandline(
         config,
    ).run()
    rougail.user_datas(user_datas)
    print(config.value.get())

Let's execute `script.py` to display help:

.. code-block:: bash

    $ python script.py -h
    usage: script.py [-h] --my_variable [MY_VARIABLE] --my_boolean_variable --no-my_boolean_variable --my_integer_variable [MY_INTEGER_VARIABLE] --my_secret_variable MY_SECRET_VARIABLE

    options:
      -h, --help            show this help message and exit
      --my_variable [MY_VARIABLE]
                            my_variable (My first variable) (default: my value)
      --my_boolean_variable
                            my_boolean_variable (My boolean variable) (default: True)
      --no-my_boolean_variable
      --my_integer_variable [MY_INTEGER_VARIABLE]
                            my_integer_variable (My integer variable) (default: 1)
      --my_secret_variable MY_SECRET_VARIABLE
                            my_secret_variable (My secret variable)
    {<TiramisuOption path="my_variable">: 'my value', <TiramisuOption path="my_boolean_variable">: True, <TiramisuOption path="my_integer_variable">: 1, <TiramisuOption path="my_secret_variable">: None}


And now with modified value:

.. code-block:: bash

    $ python script.py --my_variable "a new value" --no-my_boolean_variable --my_integer_variable 10 --my_secret_variable MyVeryStrongPassword
    {<TiramisuOption path="my_variable">: 'a new value', <TiramisuOption path="my_boolean_variable">: False, <TiramisuOption path="my_integer_variable">: 10, <TiramisuOption path="my_secret_variable">: 'MyVeryStrongPassword'}

Boolean variable has a special behavour. To set False you need to add --no-VARIABLE, to set True you need to add --VARIABLE parameter.

.. ansible,bitwarden,questionary

Combine user datas
------------------

You can combine user datas, for example if you want to load datas from environment and/or command line argument:

.. code-block:: python
    :caption: the :file:`script.py` file content

    from rougail import Rougail, RougailConfig
    from rougail.user_data_environment import RougailUserDataEnvironment
    from rougail.user_data_commandline import RougailUserDataCommandline

    RougailConfig["main_namespace"] = None
    RougailConfig["main_structural_directories"] = ["dist/"]
    RougailConfig["step.user_data"] = ["environment", "commandline"]
    rougail = Rougail()
    config = rougail.run()

    user_datas = []
    user_datas.extend(RougailUserDataEnvironment(
         config,
    ).run())
    user_datas.extend(RougailUserDataCommandline(
         config,
    ).run())
    rougail.user_datas(user_datas)
    print(config.value.get())

Let's execute `script.py` with environment variable and commandline arguments:

.. code-block:: bash

    $ env ROUGAIL_MY_VARIABLE="a new value" ROUGAIL_MY_BOOLEAN_VARIABLE="False" python script.py --my_integer_variable 10 --my_secret_variable MyVeryStrongPassword
    {<TiramisuOption path="my_variable">: 'a new value', <TiramisuOption path="my_boolean_variable">: False, <TiramisuOption path="my_integer_variable">: 10, <TiramisuOption path="my_secret_variable">: 'MyVeryStrongPassword'}

If the value of a variable is define with an environment variable and commandline argument, the value is the value of the last user data define:

.. code-block:: bash

    $ env ROUGAIL_MY_VARIABLE="not a new" python script.py --my_variable "a new value" --no-my_boolean_variable --my_integer_variable 10 --my_secret_variable MyVeryStrongPassword
   {<TiramisuOption path="my_variable">: 'a new value', <TiramisuOption path="my_boolean_variable">: False, <TiramisuOption path="my_integer_variable">: 10, <TiramisuOption path="my_secret_variable">: 'MyVeryStrongPassword'}

Manage errors and warnings
--------------------------

Recreate a script with environnement variable support which is display the return of user_datas function:

.. code-block:: python
    :caption: the :file:`script.py` file content

    from rougail import Rougail, RougailConfig
    from rougail.user_data_environment import RougailUserDataEnvironment

    RougailConfig["main_namespace"] = None
    RougailConfig["main_structural_directories"] = ["dist/"]
    RougailConfig["step.user_data"] = ["environment"]
    RougailConfig["environment.with_secrets"] = False
    rougail = Rougail()
    config = rougail.run()

    user_datas = RougailUserDataEnvironment(
         config,
    ).run()
    print(rougail.user_datas(user_datas))

Try to load the value an unknown variable:

.. code-block:: bash

    $ env ROUGAIL_UNKNOWN_VARIABLE="a value" python script.py
    {'errors': [], 'warnings': ['variable or family "unknown_variable" does not exist, it will be ignored when loading from environment variable']}

As you can see, a warnings is return.

Try to load the value of an hidden variable:

.. code-block:: bash

   $ env ROUGAIL_MY_HIDDEN_VARIABLE="a value" python script.py
    {'errors': [], 'warnings': ['variable "my_hidden_variable" (My hidden variable) is hidden, it will be ignored when loading from environment variable']}

Finally if a try to change the value of a secret, which is not allowed:

.. code-block:: bash

    $ env ROUGAIL_MY_SECRET_VARIABLE="MyVeryStrongPassword" python script.py 
    {'errors': ['the variable "my_secret_variable" contains secrets and should not be defined in environment variable'], 'warnings': []}

An error is generated.