feat: better errors messages and improvement

Co-authored-by: gwen <gwenaelremond@free.fr>
Co-authored-by: gwen <gwenael.remond@free.fr>
Co-authored-by: Emmanuel Garette <egarette@silique.fr>
Reviewed-on: #24
Co-authored-by: gremond <gwenael.remond@protonmail.com>
Co-committed-by: gremond <gwenael.remond@protonmail.com>

.pre-commit-config.yaml

@@ -0,0 +1,47 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v2.3.0
+    hooks:
+      - id: check-yaml
+      - id: end-of-file-fixer
+      - id: trailing-whitespace
+      - id: check-ast
+      - id: check-added-large-files
+      - id: check-json
+      - id: check-executables-have-shebangs
+      - id: check-symlinks
+
+  - repo: https://github.com/psf/black
+    rev: 22.10.0
+    hooks:
+      - id: black
+
+#  - repo: https://github.com/hhatto/autopep8
+#    rev: v2.0.4
+#    hooks:
+#      - id: autopep8
+
+#  - repo: https://github.com/pre-commit/mirrors-mypy
+#    rev: v1.6.1
+#    hooks:
+#      - id: mypy
+
+#  - repo: https://github.com/PyCQA/pylint
+#    rev: v3.0.2
+#    hooks:
+#      - id: pylint
+
+#  - repo: https://github.com/PyCQA/isort
+#    rev: 5.11.5
+#    hooks:
+#      - id: isort
+
+#  - repo: https://github.com/motet-a/jinjalint
+#    rev: 0.5
+#    hooks:
+#      - id: jinjalint
+
+# - repo: https://github.com/rstcheck/rstcheck
+#   rev: v6.2.0
+#   hooks:
+#     - id: rstcheck

docs/check.rst

@@ -117,7 +117,7 @@ Here is a simple example of validating values:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     my_variable:
       validators:
         - type: jinja
@@ -142,7 +142,7 @@ In the constraint, it is possible to specify the error level and put it as a war
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     my_variable:
       validators:
         - type: jinja
@@ -161,7 +161,7 @@ Verification with parameters
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     my_hidden_variable:
       disabled: true
     my_variable:
@@ -189,7 +189,7 @@ An example with a suffix type parameter:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     varname:
       multi: true
       default:
@@ -221,7 +221,7 @@ An example with an index type parameter:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     family:
       type: leadership
       leader:
@@ -249,7 +249,7 @@ In a first dictionary, let's declare our variable and its verification function:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     my_variable:
       validators:
         - type: jinja
@@ -263,7 +263,7 @@ In a second dictionary it is possible to redefine the calculation:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     my_variable:
       redefine: true
       validators:  
@@ -280,7 +280,7 @@ Here is a third dictionary in which we remove the validation:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     my_variable:
       redefine: true
       validators:

docs/condition.rst

@@ -177,7 +177,7 @@ It is possible to write the condition in Jinja:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     condition:
       default: 'do not hide!'
     my_variable:
@@ -213,7 +213,7 @@ A variable can therefore be calculated via the result of another variable. Pleas
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     condition:
       type: boolean
     my_variable:
@@ -233,7 +233,7 @@ To delete the calculation from a variable, simply do in a new dictionary:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     my_variable:
       redefine: true
       hidden:

docs/developer.rst

@@ -14,3 +14,47 @@ This process describes how to install and run the project locally, e.g. for deve
 *Nota*: command is to be executed through the terminal
 
 `pip install rougail`
+
+Code quality
+---------------
+
+We are using `pre-commit <https://pre-commit.com/>`_, there is a :file:`.pre-commit-config.yaml`
+pre-commit config file in the root's project.
+
+You need to:
+
+- install the pre-commit library::
+
+    pip install pre-commit
+
+- registrer the pre-commit git hooks with this command::
+
+    pre-commit install
+
+- launch the quality code procedure with::
+
+    pre-commit
+
+  or simply just commit your changes, pre-commit will automatically be launched.
+
+.. attention:: If an error is found, the commit will not happen.
+   You must resolve all errors that pre-commit that pre-commit points out to you before.
+
+.. note:: If you need for some reason to disable `pre-commit`, just set
+          the `PRE_COMMIT_ALLOW_NO_CONFIG` environment variable before commiting::
+
+              PRE_COMMIT_ALLOW_NO_CONFIG=1 git commit
+
+Coding standard
+------------------
+
+We use black
+
+.. code-block:: yaml
+
+  - repo: https://github.com/psf/black
+    rev: 22.10.0
+    hooks:
+      - id: black
+
+And some YAML and JSON validators.

docs/family.rst

@@ -15,6 +15,23 @@ It is with this name that we will be able to interact with the family.
 
 It's best to follow the :ref:`convention on variable names`.
 
+Shorthand declaration
+----------------------------
+
+Shorthand declaration is a way to declare a family in a single line. But you can only define family name and description.
+
+To create a family, just add a key with it's name and variables as values. Attention, do not declare any other attributs.
+
+By default, the description of the variable is the family name.
+If you add comment in same line of name, this comment is use as description:
+
+.. code-block:: yaml
+
+    ---
+    version: '1.1'
+    my_family:  # This is a great family
+      variable:
+
 Parameters
 ---------------
 
@@ -92,20 +109,13 @@ Parameters
 Dynamically created family
 -----------------------------
 
-To create a family dynamically, you must create a fictitious family linked to a variable. 
-The family name and description will actually be the prefix of the new name / description. 
-The suffix will come from the value of the bound variable. 
-The name of the families and variables it contains will be preserved, however, the description will be the prefix of the real description.
+To create a family dynamically, you must create a fictitious family linked to a calculation.
+The family name will actually be the prefix of the new name. Alternativly you can specify the suffix in the name, ie `my_{{ suffix }}_name`.
+The suffix will come from the calculation.
 
-Obviously if the content of the linked variable were to evolve, new dynamic families will appear or disappear.
+Obviously if the result of calculation were to evolve, new dynamic families will appear or disappear.
 
-To note that:
-
-- the variable linked to the family must be a multiple variable
-- it is not possible to put a simple family into a dynamic family
-- it is possible to put a leading family into a dynamic family     
-
-Leader or foller variable
+Leader or follower variable
 -----------------------------
 
 A leader family has a typical attribute of “leadership”. The type is required.
@@ -150,12 +160,12 @@ If a leader variable is hidden or disabled, the follower variables will be hidde
 Examples
 ----------
 
-Simple family 
+Simple family:
 
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     my_family:
       type: family
       description: This is a great family
@@ -168,7 +178,7 @@ Dynamically created family
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     varname:
       multi: true
       default:
@@ -176,18 +186,48 @@ Dynamically created family
         - val2
     my_dyn_family_:
       type: dynamic
-      variable: rougail.varname
-      description: 'Describe '
+      dynamic:
+        type: variable
+        variable: rougail.varname
+      description: 'Describe'
       my_dyn_var:
         type: string
-        description: 'Variable description for '
+        description: 'Variable description'
 
 This will dynamically create two families:
 
-- "rougail.my_dyn_family_val1" with the description "Describe val1"
-- "rougail.my_dyn_family_val2" with the description "Describe val2"
+- "rougail.my_dyn_family_val1"
+- "rougail.my_dyn_family_val2"
 
-In the dynamic family "rougail.my_dyn_family_val1" we will find a variable "my_dyn_var" with the description "Variable description for val1".
+In the dynamic family "rougail.my_dyn_family_val1" we will find a variable "my_dyn_var".
+
+Here is a second example:
+
+.. code-block:: yaml
+
+    ---
+    version: '1.1'
+    varname:
+      multi: true
+      default:
+        - val1
+        - val2
+    my_dyn_{{ suffix }}_family:
+      type: dynamic
+      dynamic:
+        type: variable
+        variable: rougail.varname
+      description: 'Describe'
+      my_dyn_var:
+        type: string
+        description: 'Variable description'
+
+This will dynamically create two families:
+
+- "rougail.my_dyn_val1_family"
+- "rougail.my_dyn_val2_family"
+
+In the dynamic family "rougail.my_dyn_val1_family" we will find a variable "my_dyn_var".
 
 Leader or follower variable
 -------------------------------
@@ -200,7 +240,7 @@ Here is an example of defining a leading variable and two following variables:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     family:
       type: leadership
       leader:
@@ -217,7 +257,6 @@ To add a new follower variable, in a new dictionary, simply define one or more n
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     family:
       follower3:
-

docs/fill.rst

@@ -156,6 +156,87 @@ There are two types of parameter:
      - Name of the information whose value we want to retrieve.
      - doc
 
+The variable path
+-----------------
+
+Normal family
+~~~~~~~~~~~~~
+
+The default namespace is defined in RougailConfig["variable_namespace"] with the default value "rougail".
+In addition, there are extras namespaces defined with in RougailConfig["extra_dictionaries"].
+
+Inside those namespaces we can add families and variables.
+
+Here is an hierarchic examples:
+
+.. code-block::
+  rougail
+    ├── variable1
+    ├── family1
+    │     ├── variable2
+    │     └── variable3
+    └── family2
+          └── subfamily1
+                └── variable4
+  extra1
+    └── family3
+          ├── variable5
+          └── variable6
+
+In `calculation` we can use other variables.
+
+Here is all paths:
+
+- rougail.variable1
+- rougail.family1.variable2
+- rougail.family1.variable3
+- rougail.family2.subfamily1.variable4
+- extra1.family3.variable5
+- extra1.family3.variable6
+
+Inside a variable's `calculation` we can use relative path. "_" means that other variable is in same family. "__" means that other variables are in parent family, and so on...
+
+For example, in variable2's `calculation`, we can use relative path:
+
+- __.variable1
+- _.variable3
+- __.family2.subfamily1.variable4
+
+But we cannot access to extra1 variables with relative path.
+
+Dynamic family
+~~~~~~~~~~~~~~~~~~
+
+Hire is a dynamic family "{{ suffix }}":
+
+.. code-block::
+  rougail
+    ├── variable1: ["val1", "val2"]
+    ├── {{ suffix }}
+    │     ├── variable2
+    │     └── variable3
+    └── family
+          └── variable4
+
+For variable2's calculation, we can use:
+
+- rougail.{{ suffix }}.variable3
+- _.variable3
+
+In this case, we get value for "variable3" with the same suffix as "variable2".
+
+For variable4's calculation, we have two possibility:
+
+- retrieves all values with all suffixes:
+
+  - rougail.{{ suffix }}.variable3
+  - __.{{ suffix }}.variable3
+
+- retrieves a value for a specified suffix:
+
+  - rougail.val1.variable3
+  - __.val1.variable3
+
 Examples
 -----------
 
@@ -167,7 +248,7 @@ Let's start with an example from a simple Jinja template:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     my_calculated_variable:
       default:
         type: jinja
@@ -178,7 +259,7 @@ Here is a second example with a boolean variable:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     my_calculated_variable:
       type: boolean
       default:
@@ -190,7 +271,7 @@ And a multiple value of the number type:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     my_calculated_variable:
       type: number
       multi: true
@@ -206,7 +287,7 @@ Let's create a variable whose value is returned by a python function:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     my_calculated_variable:
       default:
         type: jinja
@@ -224,7 +305,7 @@ An example with parameters:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     my_calculated_variable:
       description: my description
       default:
@@ -235,19 +316,19 @@ An example with parameters:
           param1: value
           param2:
             type: variable
-            variable: rougail.unknown_variable
+            variable: _.unknown_variable
             optional: true
           param3:
             type: information
             information: doc
-            variable: rougail.my_calculated_variable
+            variable: _.my_calculated_variable
 
 An example with a `suffix` type parameter:
 
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     varname:
       multi: true
       default:
@@ -255,7 +336,9 @@ An example with a `suffix` type parameter:
         - val2
     my_dyn_family_:
       type: dynamic
-      variable: rougail.varname
+      dynamic:
+        type: variable
+        variable: _.varname
       description: 'Describe '
       my_dyn_var:
         type: string
@@ -275,7 +358,7 @@ An example with an index type parameter:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     family:
       type: leadership
       leader:
@@ -299,7 +382,7 @@ Copy a variable in another:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     my_variable:
       multi: true
       default:
@@ -309,14 +392,32 @@ Copy a variable in another:
       multi: true
       default:
         type: variable
-        variable: rougail.my_variable
+        variable: _.my_variable
+
+Copy the default value from a variable, means copy type, params and multi attribute too if not define in second variable.
+
+.. code-block:: yaml
+
+      ---
+      version: 1.1
+      my_variable:
+        multi: true
+        type: domainname
+        params:
+          allow_ip: true
+      my_calculated_variable:
+        default:
+          type: variable
+          variable: _.var1
+
+Here my_calculated_variable is a domainname variable with parameter allow_ip=True and multi to true.
 
 Copy one variable to another if the source has no `property` problem:
 
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     my_variable:
       default: val1
       disabled: true
@@ -324,7 +425,7 @@ Copy one variable to another if the source has no `property` problem:
       multi: true
       default:
         type: variable
-        variable: rougail.my_variable
+        variable: _.my_variable
         propertyerror: false
 
 Copy two non-multiple variables into a multiple variable:
@@ -332,7 +433,7 @@ Copy two non-multiple variables into a multiple variable:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     my_variable_1:
       default: val1
     my_variable_2:
@@ -341,9 +442,9 @@ Copy two non-multiple variables into a multiple variable:
       multi: true
       default:
         - type: variable
-          variable: rougail.my_variable_1
+          variable: _.my_variable_1
         - type: variable
-          variable: rougail.my_variable_2
+          variable: _.my_variable_2
 
 A variable in a dynamic family can also be used in a calculation.
 
@@ -352,7 +453,7 @@ For example using the variable for a particular suffix:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     varname:
       multi: true
       default:
@@ -360,7 +461,9 @@ For example using the variable for a particular suffix:
         - val2
     my_dyn_family_:
       type: dynamic
-      variable: rougail.varname
+      dynamic:
+        type: variable
+        variable: _.varname
       description: 'Describe '
       my_dyn_var_:
         type: string
@@ -369,7 +472,7 @@ For example using the variable for a particular suffix:
     all_dyn_var:
       default:
         type: variable
-        variable: rougail.my_dyn_family_val1.my_dyn_var_val1
+        variable: _.my_dyn_family_val1.my_dyn_var_val1
 
 In this case, we recover the value `val1`.
 
@@ -378,7 +481,7 @@ Second example using the variable for all suffixes:
 .. code-block:: yaml
 
     ---
-    version: '1.0' 
+    version: '1.1' 
     varname:
       multi: true
       default:
@@ -386,7 +489,9 @@ Second example using the variable for all suffixes:
         - val2
     my_dyn_family_:
       type: dynamic
-      variable: rougail.varname
+      dynamic:
+        type: variable
+        variable: _.varname
       description: 'Describe '
       my_dyn_var_:
         type: string
@@ -396,7 +501,7 @@ Second example using the variable for all suffixes:
       multi: true
       default:
         type: variable
-        variable: rougail.my_dyn_family_.my_dyn_var_
+        variable: _.my_dyn_family_.my_dyn_var_
 
 In this case, we recover the `val1` and `val2` list.
 
@@ -406,7 +511,7 @@ Calculation via a suffix
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     varname:
       multi: true
       default:
@@ -414,7 +519,9 @@ Calculation via a suffix
         - val2
     my_dyn_family_:
       type: dynamic
-      variable: rougail.varname
+      dynamic:
+        type: variable
+        variable: _.varname
       description: 'Describe '
       my_dyn_var_:
         type: string
@@ -427,7 +534,7 @@ Calculation via an index
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     family:
       type: leadership
       leader:
@@ -448,7 +555,7 @@ In a first dictionary, let's declare our variable and our calculation:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     my_calculated_variable:
       default:
         type: jinja
@@ -459,7 +566,7 @@ In a second dictionary, it is possible to redefine the calculation:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     my_calculated_variable:
       redefine: true
       default:
@@ -471,7 +578,7 @@ In a third dictionary, we even can delete the calculation if needed:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     my_calculated_variable:
       redefine: true
       default: null

docs/gettingstarted.rst

@@ -65,7 +65,7 @@ Here is a :term:`dictionary` example:
    :linenos:
 
     ---
-    version: '1.0'
+    version: '1.1'
     proxy:
       description: Configure Proxy Access to the Internet
       type: family
@@ -126,7 +126,7 @@ Here is the tree structure we want to have::
    :caption: The `hello.yaml` file
 
     ---
-    version: '1.0'
+    version: '1.1'
     hello:
       default: world
 
@@ -168,7 +168,7 @@ Let's continuing on our "Hello world" theme and add a :term:`family` container.
    :linenos:
 
     ---
-    version: '1.0'
+    version: '1.1'
     world:
       description: Hello world family container
       name:

docs/library.rst

@@ -26,7 +26,7 @@ Here is a first :file:`dict/00-base.yml` dictionary:
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     my_variable:
       default: my_value
 
@@ -71,7 +71,7 @@ Then let's create an extra :term:`dictionary` :file:`extras/00-base.yml`:
 .. code-block:: yaml
    :caption: the :file:`extras/00-base.yml` file content
     ---
-    version: '1.0'
+    version: '1.1'
     my_variable_extra:
       default: my_value_extra
 
@@ -103,7 +103,7 @@ We create the complementary :term:`dictionary` named :file:`dict/01-function.yml
 .. code-block:: yaml
 
     ---
-    version: '1.0'
+    version: '1.1'
     my_variable_jinja:
       type: "string"
       default:
@@ -141,7 +141,6 @@ Let's execute `script.py`:
 
 The value of the `my_variable_extra` variable is calculated, and it's value comes from the :func:`return_no` function.
 
-
 Create your own type
 ----------------------
 
@@ -180,7 +179,7 @@ Here is a :file:`dict/00-base.yml` dictionary:
 
 .. code-block:: yaml
     ---
-    version: '1.0'
+    version: '1.1'
     var:
       type: lipogram
 
@@ -191,3 +190,44 @@ Here is a :file:`dict/00-base.yml` dictionary:
     >>> config.option('rougail.var').value.set('I just want to add a quality string that has no bad characters')
     [...]
     tiramisu.error.ValueOptionError: "I just want to add a quality string that has no bad characters" is an invalid lipogram for "var", Perec wrote a book without any "e", you could not do it in a simple sentence?
+
+Upgrade dictionnaries to upper version
+----------------------------------------
+
+All dictionnaries has a format version number.
+When a new format version is proposed, it is possible to automatically convert the files to the new version.
+
+We create a term:`dictionary` named :file:`dict/01-upgrade.yml` with version 1.0:
+
+.. code-block:: yaml
+
+    ---
+    version: '1.1'
+    my_variable:
+      multi: true
+    my_dyn_family:
+      type: "dynamic"
+      variable: my_variable
+      a_variable:
+
+
+.. code-block:: python
+    >>> from rougail import RougailUpgrade, RougailConfig
+    >>> RougailConfig['dictionaries_dir'] = ['dict']
+    >>> upgrade = RougailUpgrade()
+    >>> upgrade.load_dictionaries('dict_converted')
+
+The term:`dictionary` named :file:`dict_converted/01-upgrade.yml` is in version 1.1:
+
+.. code-block:: yaml
+
+   version: '1.1'
+   my_variable:
+     multi: true
+   my_dyn_family:
+     type: dynamic
+     a_variable: null
+     dynamic:
+       type: variable
+       variable: my_variable
+       propertyerror: false

docs/tutorial.rst

@@ -35,7 +35,7 @@ Then let's put our first dictionary file in this folder, named :file:`00-proxy.y
    :linenos:
 
     ---
-    version: '1.0'
+    version: '1.1'
     proxy:
       description: Proxy configuration in order to have access to the internet
       type: family
@@ -69,7 +69,7 @@ Let's create a second :file:`dict/01-proxy_mode.yml` file.
    :linenos:
 
     ---
-    version: '1.0'
+    version: '1.1'
     proxy:
       proxy_mode:
         description: Proxy mode
@@ -122,7 +122,7 @@ Let's create the :file:`dict/02-proxy_manual.yml` dictionary:
    :caption: the the :file:`dict/02-proxy_manual.yml` file
 
     ---
-    version: '1.0'
+    version: '1.1'
     proxy:
       manual:
         description: Manual proxy configuration
@@ -170,7 +170,7 @@ Let's create the :file:`dict/03-proxy_manual_http_proxy.yml` dictionary:
    :linenos:
 
     ---
-    version: '1.0'
+    version: '1.1'
     proxy:
       manual:
         http_proxy:
@@ -195,7 +195,7 @@ We then want to offer the user the possibility of providing the same proxy for t
 .. code-block:: yaml
    :caption: the :file:`dict/04-proxy_manual_http_use_for_https.yml` file
 
-    version: '1.0'
+    version: '1.1'
     proxy:
       manual:
         use_for_https:
@@ -216,7 +216,7 @@ Let's create the :file:`dict/05-proxy_manual_ssl_proxy.yml` file:
    :linenos:
 
     ---
-    version: '1.0'
+    version: '1.1'
     proxy:
       manual:
         ssl_proxy:
@@ -360,7 +360,7 @@ Let's create the :file:`dict/06-proxy_manual_socks_proxy.yml` file:
    :caption: the :file:`dict/06-proxy_manual_socks_proxy.yml` file
 
     ---
-    version: '1.0'
+    version: '1.1'
     proxy:
       manual:
         socks_proxy:
@@ -392,7 +392,7 @@ Let's create the :file:`dict/07-proxy_auto.yml` file:
    :caption: the :file:`dict/07-proxy_auto.yml` file
 
     ---
-    version: '1.0'
+    version: '1.1'
     proxy:
       auto:
         type: web_address
@@ -419,7 +419,7 @@ Let's create the :file:`dict/07-proxy_no_proxy.yml` file:
    :linenos:
 
     ---
-    version: '1.0'
+    version: '1.1'
     proxy:
       no_proxy:
         description: Address for which proxy will be desactivated
@@ -510,7 +510,7 @@ Nothing special when creating the authentication request. To do this, let's crea
    :linenos:
 
     ---
-    version: '1.0'
+    version: '1.1'
     proxy:
       prompt_authentication:
         description: Prompt for authentication if password is saved
@@ -535,7 +535,7 @@ Let's create a `dict/09-proxy_proxy_dns_socks5.yml` file:
    :linenos:
 
     ---
-    version: '1.0'
+    version: '1.1'
     proxy:
       proxy_dns_socks5:
         description: Use proxy DNS when using SOCKS v5
@@ -578,7 +578,7 @@ Let's create a `dict/10-proxy_dns_over_https.yml` file:
    :linenos:
 
     ---
-    version: '1.0'
+    version: '1.1'
     proxy:
       dns_over_https:
         description: DNS over HTTPS
@@ -651,7 +651,7 @@ Here is the complete content of the FoxyProxy type proxy configuration
    :linenos:
 
     ---
-    version: '1.0'
+    version: '1.1'
     proxy:
       _type: leadership
       title:
@@ -813,7 +813,7 @@ If you prefer this option, here is a second extra dictionary :file:`foxyproxy/01
    :linenos:
 
     ---
-    version: '1.0'
+    version: '1.1'
     proxy:
       username:
         redefine: true

docs/variable.rst

@@ -23,6 +23,28 @@ Variable's associated symbolic name.
 
 It's best to follow the :ref:`convention on variable names`.
 
+Shorthand declaration
+----------------------------
+
+Shorthand declaration is a way to declare a variable in a single line. But you can only define variable name, description, multi or default value.
+
+To create a variable, just add a key with it's name and default value as value.
+Be careful not to declare any other attributes.
+
+To declare a multi variable just add a list as default value.
+
+By default, the description of the variable is the variable name.
+If you add a comment in the same line of the name, this comment will be used has a description.
+
+.. code-block:: yaml
+
+    ---
+    version: '1.1'
+    my_variable: 1  # This is a great integer variable
+    my_multi_variable:  # This is a great multi string variable
+      - value1
+      - value2
+
 Parameters
 -------------
 

pyproject.toml

@@ -23,13 +23,17 @@ classifiers = [
 
 ]
 dependencies = [
-    "pyyaml ~= 6.0.1",
+    "ruamel.yaml ~= 0.17.40",
     "pydantic ~= 2.5.2",
     "jinja2 ~= 3.1.2",
+    "tiramisu ~= 4.1.0"
 ]
-[project.optional-dependancies]
+
+[project.optional-dependencies]
 dev = [
     "pylint ~= 3.0.3",
+    "pytest ~= 8.2.2",
+    "lxml ~= 5.2.2"
 ]
 
 [tool.commitizen]

src/rougail/__init__.py

@@ -28,19 +28,27 @@ along with this program; if not, write to the Free Software
 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 """
 from tiramisu import Config
-from copy import copy
+from tiramisu.error import PropertiesOptionError
+from warnings import warn
+from typing import List
 
 from .convert import RougailConvert
 from .config import RougailConfig
 from .update import RougailUpgrade
 from .object_model import CONVERT_OPTION
+from .utils import normalize_family
 
 
-def tiramisu_display_name(kls) -> str:
+def tiramisu_display_name(kls, subconfig) -> str:
     """Replace the Tiramisu display_name function to display path + description"""
-    doc = kls.impl_get_information("doc", None)
+    doc = kls._get_information(subconfig, "doc", None)
     comment = f" ({doc})" if doc and doc != kls.impl_getname() else ""
-    return f"{kls.impl_getpath()}{comment}"
+    if "{{ suffix }}" in comment:
+        comment = comment.replace('{{ suffix }}', str(subconfig.suffixes[-1]))
+    path = kls.impl_getpath()
+    if "{{ suffix }}" in path:
+        path = path.replace('{{ suffix }}', normalize_family(str(subconfig.suffixes[-1])))
+    return f"{path}{comment}"
 
 
 class Rougail:
@@ -61,9 +69,10 @@ class Rougail:
         path_prefix: str,
     ) -> None:
         """Add a prefix"""
+        self.converted.load_config()
         self.converted.parse_directories(path_prefix)
 
-    def get_config(self):
+    def run(self):
         """Get Tiramisu Config"""
         if not self.config:
             tiram_obj = self.converted.save(self.rougailconfig["tiramisu_cache"])
@@ -77,5 +86,47 @@ class Rougail:
             self.config.property.read_write()
         return self.config
 
+    def get_config(self):
+        warn("get_config is deprecated, use run instead", DeprecationWarning, stacklevel=2)
+        return self.run()
 
-__ALL__ = ("Rougail", "RougailConfig", "RougailUpgrade")
+    def user_datas(self,
+                   user_datas: List[dict]):
+        values = {}
+        errors = []
+        warnings = []
+        for datas in user_datas:
+            for name, data in datas.get('values', {}).items():
+                values.setdefault(name, {}).update(data)
+            errors.extend(datas.get('errors', []))
+            warnings.extend(datas.get('warnings', []))
+        while values:
+            value_is_set = False
+            for option in self.config:
+                if option.path() in values and option.index() in values[option.path()]:
+                    try:
+                        option.value.set(values[option.path()])
+                        value_is_set = True
+                        values.pop(option.path())
+                    except:
+                        pass
+            if not value_is_set:
+                break
+        for path, data in values.items():
+            for index, value in data.items():
+                try:
+                    print('attention', path, value)
+                    self.config.option(path).value.set(value)
+                    print('pfff')
+                except AttributeError as err:
+                    errors.append(str(err))
+                except ValueError as err:
+                    errors.append(str(err).replace('"', "'"))
+                except PropertiesOptionError as err:
+    #                warnings.append(f'"{err}" but is defined in "{self.filename}"')
+                    warnings.append(str(err))
+        return {'errors': errors,
+                'warnings': warnings,
+                }
+
+__all__ = ("Rougail", "RougailConfig", "RougailUpgrade")

src/rougail/annotator/__init__.py

@@ -45,7 +45,7 @@ def get_annotators(annotators, module_name):
         path = str(pathobj)
         if not path.endswith(".py") or path.endswith("__.py"):
             continue
-        module = load_modules(path)
+        module = load_modules(module_name, path)
         if "Annotator" not in dir(module):
             continue
         annotators[module_name].append(module.Annotator)
@@ -62,21 +62,26 @@ class SpaceAnnotator:  # pylint: disable=R0903
         if ANNOTATORS is None:
             ANNOTATORS = {}
             get_annotators(ANNOTATORS, "rougail.annotator")
-        for extra_annotator in objectspace.rougailconfig["extra_annotators"]:
+        for extra_annotator in objectspace.extra_annotators:
             if extra_annotator in ANNOTATORS:
                 continue
             get_annotators(ANNOTATORS, extra_annotator)
+        for plugin in objectspace.plugins:
+            try:
+                get_annotators(ANNOTATORS, f'rougail.{plugin}.annotator')
+            except ModuleNotFoundError:
+                pass
         annotators = ANNOTATORS["rougail.annotator"].copy()
-        for extra_annotator in objectspace.rougailconfig["extra_annotators"]:
+        for extra_annotator in objectspace.extra_annotators:
             annotators.extend(ANNOTATORS[extra_annotator])
+        for plugin in objectspace.plugins:
+            annotators.extend(ANNOTATORS[f'rougail.{plugin}.annotator'])
         annotators = sorted(annotators, key=get_level)
         functions = {}
-        functions_files = objectspace.rougailconfig["functions_file"]
-        if not isinstance(functions_files, list):
-            functions_files = [functions_files]
+        functions_files = objectspace.functions_files
         for functions_file in functions_files:
             if isfile(functions_file):
-                loaded_modules = load_modules(functions_file)
+                loaded_modules = load_modules('function_file', functions_file)
                 for function in dir(loaded_modules):
                     if function.startswith("_"):
                         continue

src/rougail/annotator/family.py

@@ -31,6 +31,7 @@ from typing import Optional
 from rougail.i18n import _
 from rougail.error import DictConsistencyError
 from rougail.annotator.variable import Walk
+from rougail.object_model import VariableCalculation
 
 
 class Mode:  # pylint: disable=R0903
@@ -63,16 +64,30 @@ class Annotator(Walk):
         self.objectspace = objectspace
         if not self.objectspace.paths:
             return
-        self.modes = {
-            name: Mode(idx)
-            for idx, name in enumerate(self.objectspace.rougailconfig["modes_level"])
-        }
+        self.check_leadership()
         self.remove_empty_families()
         self.family_names()
-        self.change_modes()
-        self.dynamic_families()
+        if self.objectspace.modes_level:
+            self.modes = {
+                name: Mode(idx)
+                for idx, name in enumerate(self.objectspace.modes_level)
+            }
+            self.default_variable_mode = self.objectspace.default_variable_mode
+            self.default_family_mode = self.objectspace.default_family_mode
+            self.change_modes()
         self.convert_help()
 
+    def check_leadership(self) -> None:
+        """No subfamily in a leadership"""
+        for family in self.get_families():
+            if family.type != "leadership":
+                continue
+            for variable_path in self.objectspace.parents[family.path]:
+                variable = self.objectspace.paths[variable_path]
+                if variable.type in self.objectspace.family_types:
+                    msg = f'the leadership "{family.path}" cannot have the { variable.type } "{ variable.path}"'
+                    raise DictConsistencyError(msg, 24, variable.xmlfiles)
+
     def remove_empty_families(self) -> None:
         """Remove all families without any variable"""
         removed_families = []
@@ -80,7 +95,7 @@ class Annotator(Walk):
             if isinstance(family, self.objectspace.family) and not self._has_variable(
                 family.path
             ):
-                if "." in family.path:
+                if self.objectspace.paths.default_namespace is None or "." in family.path:
                     removed_families.append(family.path)
         removed_families.reverse()
         for family in removed_families:
@@ -104,20 +119,17 @@ class Annotator(Walk):
             if not family.description:
                 family.description = family.name
 
-    #            family.doc = family.description
-    #            del family.description
-
     def change_modes(self):
         """change the mode of variables"""
-        modes_level = self.objectspace.rougailconfig["modes_level"]
-        default_variable_mode = self.objectspace.rougailconfig["default_variable_mode"]
+        modes_level = self.objectspace.modes_level
+        default_variable_mode = self.default_variable_mode
         if default_variable_mode not in modes_level:
             msg = _(
                 f'default variable mode "{default_variable_mode}" is not a valid mode, '
                 f"valid modes are {modes_level}"
             )
             raise DictConsistencyError(msg, 72, None)
-        default_family_mode = self.objectspace.rougailconfig["default_family_mode"]
+        default_family_mode = self.default_family_mode
         if default_family_mode not in modes_level:
             msg = _(
                 f'default family mode "{default_family_mode}" is not a valid mode, '
@@ -131,12 +143,21 @@ class Annotator(Walk):
         families.reverse()
         for family in families:
             self._change_family_mode(family)
+        if self.objectspace.paths.default_namespace is None:
+            for variable_path in self.objectspace.parents['.']:
+                variable = self.objectspace.paths[variable_path]
+                if variable.type == "symlink" or variable_path in self.objectspace.families:
+                    continue
+                self._set_default_mode_variable(variable,
+                                                self.default_variable_mode,
+                                                check_level=False,
+                                                )
 
     def valid_mode(
         self,
         obj,
     ) -> None:
-        modes_level = self.objectspace.rougailconfig["modes_level"]
+        modes_level = self.objectspace.modes_level
         if self._has_mode(obj) and obj.mode not in modes_level:
             msg = _(
                 f'mode "{obj.mode}" for "{obj.name}" is not a valid mode, '
@@ -162,6 +183,7 @@ class Annotator(Walk):
                 continue
             if leader is None and family.type == "leadership":
                 leader = variable
+                leader_mode = leader.mode
             if variable_path in self.objectspace.families:
                 # set default mode a subfamily
                 if family_mode and not self._has_mode(variable):
@@ -172,9 +194,9 @@ class Annotator(Walk):
                 if leader:
                     self._set_default_mode_leader(leader, variable)
                 self._set_default_mode_variable(variable, family_mode)
-        if leader:
+        if leader and leader_mode is not None:
             # here because follower can change leader mode
-            self._set_auto_mode(family, leader.mode)
+            self._set_auto_mode(family, leader_mode)
 
     def _has_mode(self, obj) -> bool:
         return obj.mode and not obj.path in self.mode_auto
@@ -183,11 +205,12 @@ class Annotator(Walk):
         self,
         variable: "self.objectspace.variable",
         family_mode: Optional[str],
+        check_level: bool=True,
     ) -> None:
         # auto_save variable is set to 'basic' mode
         # if its mode is not defined by the user
         if not self._has_mode(variable) and variable.auto_save is True:
-            variable.mode = self.objectspace.rougailconfig["modes_level"][0]
+            variable.mode = self.objectspace.modes_level[0]
         # mandatory variable without value is a basic variable
         elif (
             not self._has_mode(variable)
@@ -195,8 +218,8 @@ class Annotator(Walk):
             and variable.default is None
             and variable.path not in self.objectspace.default_multi
         ):
-            variable_mode = self.objectspace.rougailconfig["modes_level"][0]
-            if family_mode and self.modes[variable_mode] < self.modes[family_mode]:
+            variable_mode = self.objectspace.modes_level[0]
+            if check_level and family_mode and self.modes[variable_mode] < self.modes[family_mode]:
                 msg = _(
                     f'the variable "{variable.name}" is mandatory so in "{variable_mode}" mode '
                     f'but family has the higher family mode "{family_mode}"'
@@ -220,20 +243,17 @@ class Annotator(Walk):
         leader: "self.objectspace.variable",
         follower: "self.objectspace.variable",
     ) -> None:
-        if follower.auto_save is True:
-            msg = _(f'leader/followers "{follower.name}" could not be auto_save')
-            raise DictConsistencyError(msg, 29, follower.xmlfiles)
         if leader == follower:
             # it's a leader
             if not leader.mode:
                 self._set_auto_mode(
-                    leader, self.objectspace.rougailconfig["default_variable_mode"]
+                    leader, self.default_variable_mode
                 )
             return
         if self._has_mode(follower):
             follower_mode = follower.mode
         else:
-            follower_mode = self.objectspace.rougailconfig["default_variable_mode"]
+            follower_mode = self.default_variable_mode
         if self.modes[leader.mode] > self.modes[follower_mode]:
             if self._has_mode(follower) and not self._has_mode(leader):
                 # if follower has mode but not the leader
@@ -255,20 +275,18 @@ class Annotator(Walk):
         if family.mode:
             family_mode = family.mode
         else:
-            family_mode = self.objectspace.rougailconfig["default_family_mode"]
-        min_variable_mode = self.objectspace.rougailconfig["modes_level"][-1]
+            family_mode = self.default_family_mode
+        min_variable_mode = self.objectspace.modes_level[-1]
         # change variable mode, but not if variables are not in a family
         is_leadership = family.type == "leadership"
         if family.path in self.objectspace.parents:
-            for idx, variable_path in enumerate(self.objectspace.parents[family.path]):
+            for variable_path in self.objectspace.parents[family.path]:
                 variable = self.objectspace.paths[variable_path]
                 if variable.type == "symlink":
                     continue
                 if variable_path in self.objectspace.families:
                     if not variable.mode:
-                        variable.mode = self.objectspace.rougailconfig[
-                            "default_family_mode"
-                        ]
+                        variable.mode = self.default_family_mode
                 else:
                     self._change_variable_mode(variable, family_mode, is_leadership)
                 if self.modes[min_variable_mode] > self.modes[variable.mode]:
@@ -276,6 +294,10 @@ class Annotator(Walk):
         if not family.mode:
             # set the lower variable mode to family
             self._set_auto_mode(family, min_variable_mode)
+        if self.modes[family.mode] < self.modes[min_variable_mode]:
+            msg = _(f'the family "{family.name}" is in "{family.mode}" mode but variables and '
+                    f'families inside have the higher modes "{min_variable_mode}"')
+            raise DictConsistencyError(msg, 62, family.xmlfiles)
 
     def _change_variable_mode(
         self,
@@ -286,7 +308,7 @@ class Annotator(Walk):
         if variable.mode:
             variable_mode = variable.mode
         else:
-            variable_mode = self.objectspace.rougailconfig["default_variable_mode"]
+            variable_mode = self.default_variable_mode
         # none basic variable in high level family has to be in high level
         if not is_follower and self.modes[variable_mode] < self.modes[family_mode]:
             if self._has_mode(variable):
@@ -299,33 +321,6 @@ class Annotator(Walk):
         if not variable.mode:
             variable.mode = variable_mode
 
-    def dynamic_families(self):
-        """link dynamic families to object"""
-        for family in self.get_families():
-            if family.type != "dynamic":
-                continue
-            try:
-                family.variable = self.objectspace.paths[family.variable]
-            except AttributeError as err:
-                raise Exception(
-                    f'cannot load the dynamic family "{family.path}", cannot find variable "{family.variable}"'
-                )
-            if not family.variable.multi:
-                msg = _(
-                    f'dynamic family "{family.name}" must be linked '
-                    f"to multi variable"
-                )
-                raise DictConsistencyError(msg, 16, family.xmlfiles)
-            for variable in self.objectspace.parents[family.path]:
-                if (
-                    isinstance(variable, self.objectspace.family)
-                    and not variable.leadership
-                ):
-                    msg = _(
-                        f'dynamic family "{family.name}" cannot contains another family'
-                    )
-                    raise DictConsistencyError(msg, 22, family.xmlfiles)
-
     def convert_help(self):
         """Convert variable help"""
         for family in self.get_families():

src/rougail/annotator/property.py

@@ -112,37 +112,37 @@ class Annotator(Walk):
         self._convert_property(variable)
         if variable.hidden:
             if variable.hidden is True:
-                self.frozen[variable.path] = True
-            elif self.frozen.get(variable.path) is not True:
-                self.frozen.setdefault(variable.path, []).append(variable.hidden)
-        if variable.path in self.frozen:
-            frozen = self.frozen[variable.path]
+                self.frozen[path] = True
+            elif self.frozen.get(path) is not True:
+                self.frozen.setdefault(path, []).append(variable.hidden)
+        if path in self.frozen:
+            frozen = self.frozen[path]
             if frozen is True:
                 value = True
             else:
                 value = []
                 for calculation in frozen:
-                    calculation_object = calculation.__class__
-                    calculation_dict = calculation.model_dump().copy()
-                    calculation_dict["attribute_name"] = "frozen"
-                    calculation_dict["path"] = variable.path
-                    value.append(calculation_object(**calculation_dict))
+                    calculation_copy = calculation.copy()
+                    calculation_copy.attribute_name = 'frozen'
+                    calculation_copy.ori_path = calculation_copy.path
+                    calculation_copy.path = path
+                    value.append(calculation_copy)
                 if len(value) == 1:
                     value = value[0]
             self.objectspace.properties.add(path, "frozen", value)
             if not variable.auto_save:
                 # if auto_save, save calculated value
                 self.objectspace.properties.add(path, "force_default_on_freeze", True)
-        if variable.mandatory and variable.multi:
+        if not variable.empty and self.objectspace.multis.get(variable.path, False):
             # a multi could not have "None" has value
-            # to permit it, just add mandatory="False"
+            # to permit it, just add empty="false"
             self.objectspace.properties.add(path, "notempty", True)
         if variable.unique:
             self.objectspace.properties.add(path, "unique", True)
         if variable.unique is False:
             self.objectspace.properties.add(path, "notunique", True)
         if variable.auto_save:
-            self.objectspace.properties.add(variable.path, "force_store_value", True)
+            self.objectspace.properties.add(path, "force_store_value", True)
 
     def _convert_property(
         self,

src/rougail/annotator/value.py

@@ -48,13 +48,15 @@ class Annotator(Walk):  # pylint: disable=R0903
             return
         self.objectspace = objectspace
         self.convert_value()
-        self.add_choice_nil()
+        self.valid_choices()
 
     def convert_value(self) -> None:
         """convert value"""
         for variable in self.get_variables():
             if variable.type == "symlink":
                 continue
+            if variable.version != '1.0' and variable.type == 'port':
+                self._convert_port(variable)
             self._convert_value(variable)
 
     def _convert_value(
@@ -66,58 +68,62 @@ class Annotator(Walk):  # pylint: disable=R0903
         if variable.type == "boolean" and multi is False and variable.default is None:
             variable.default = True
 
-        if variable.default is None:
+        if variable.default is None or isinstance(variable.default, Calculation):
             return
-        has_value = False
-        if isinstance(variable.default, Calculation):
-            pass
-        #            variable.default = variable.default.to_function(self.functions)
-        elif isinstance(variable.default, list):
-            if not multi:
-                raise Exception(
-                    f'The variable "{variable.path}" with a list has default value must have "multi" attribute'
-                )
-            if variable.path in self.objectspace.followers:
-                if multi != "submulti" and len(variable.default) != 1:
-                    msg = _(
-                        f'the follower "{variable.name}" without multi attribute can only have one value'
-                    )
-                    raise DictConsistencyError(msg, 87, variable.xmlfiles)
-            #            else:
-            #                variable.default = [value.name for value in variable.default]
-            if variable.path not in self.objectspace.leaders:
-                if multi == "submulti":
-                    self.objectspace.default_multi[
-                        variable.path
-                    ] = variable.default  # [value.name for value in variable.value]
-                    variable.default = None
-                else:
-                    self.objectspace.default_multi[variable.path] = variable.default[
-                        0
-                    ]  # .name
-            has_value = True
-        elif variable.multi:
-            # msg = _(f'the none multi variable "{variable.name}" cannot have '
-            #        'more than one value')
-            # raise DictConsistencyError(msg, 68, variable.xmlfiles)
-            raise Exception("pfff")
-        else:
-            if variable.path in self.objectspace.followers:
-                self.objectspace.default_multi[variable.path] = variable.default
-                variable.default = None
-            has_value = True
 
-    def add_choice_nil(self) -> None:
+        if isinstance(variable.default, list):
+            if not multi:
+                msg = f'The variable "{variable.path}" with a list as default value must have "multi" attribute'
+                raise DictConsistencyError(msg, 68, variable.xmlfiles)
+            if variable.path in self.objectspace.followers and multi != "submulti":
+                msg = _(
+                    f'the follower "{variable.name}" without multi attribute can only have one value'
+                )
+                raise DictConsistencyError(msg, 87, variable.xmlfiles)
+            if not variable.default:
+                variable.default = None
+            else:
+                if variable.path not in self.objectspace.leaders:
+                    if multi == "submulti":
+                        self.objectspace.default_multi[
+                            variable.path
+                        ] = variable.default
+                        variable.default = None
+                    else:
+                        self.objectspace.default_multi[variable.path] = variable.default[
+                            0
+                        ]
+        elif variable.multi:
+            msg = _(f'the variable "{variable.name}" is multi but has a non list default value')
+            raise DictConsistencyError(msg, 12, variable.xmlfiles)
+        elif variable.path in self.objectspace.followers:
+            self.objectspace.default_multi[variable.path] = variable.default
+            variable.default = None
+
+    def _convert_port(self, variable) -> None:
+        if variable.multi is False and isinstance(variable.default, int):
+            variable.default = str(variable.default)
+        elif variable.multi is True and isinstance(variable.default, list):
+            for idx, value in enumerate(variable.default):
+                if isinstance(value, int):
+                    variable.default[idx] = str(value)
+
+    def valid_choices(self) -> None:
         """A variable with type "Choice" that is not mandatory must has "nil" value"""
         for variable in self.get_variables():
             if variable.type != "choice":
                 continue
-            is_none = False
             if isinstance(variable.choices, Calculation):
                 continue
-            for choice in variable.choices:
-                if choice is None:
-                    is_none = True
-                    break
-            if not variable.mandatory and not is_none:
-                variable.choices.append(None)
+            if variable.choices is None:
+                msg = f'the variable "{variable.path}" is a "choice" variable but don\'t have any choice'
+                raise DictConsistencyError(msg, 19, variable.xmlfiles)
+            if not variable.mandatory and not variable.multi:
+                self.add_choice_nil(variable)
+
+    def add_choice_nil(self, variable) -> None:
+        """A variable with type "Choice" that is not mandatory must has "nil" value"""
+        for choice in variable.choices:
+            if choice is None:
+                return
+        variable.choices.append(None)

src/rougail/annotator/variable.py

@@ -30,7 +30,8 @@ Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 
 from rougail.i18n import _
 from rougail.error import DictConsistencyError
-from rougail.object_model import Calculation
+from rougail.object_model import Calculation, VariableCalculation
+from tiramisu.error import display_list
 
 
 class Walk:
@@ -64,23 +65,92 @@ class Annotator(Walk):  # pylint: disable=R0903
         if not objectspace.paths:
             return
         self.objectspace = objectspace
-        self.forbidden_name = [
-            "services",
-            self.objectspace.rougailconfig["variable_namespace"],
-        ]
-        for extra in self.objectspace.rougailconfig["extra_dictionaries"]:
-            self.forbidden_name.append(extra)
+        if self.objectspace.main_namespace:
+            self.forbidden_name = [
+                self.objectspace.main_namespace
+            ]
+            for extra in self.objectspace.extra_dictionaries:
+                self.forbidden_name.append(extra)
+        else:
+            self.forbidden_name = []
+        # default type inference from a default value with :term:`basic types`
+        self.basic_types = {str: "string", int: "number", bool: "boolean", float: "float"}
         self.convert_variable()
         self.convert_test()
+        self.convert_examples()
         self.convert_help()
+        self.verify_choices()
 
     def convert_variable(self):
         """convert variable"""
+        for variable in self.get_variables():
+            if variable.version != "1.0":
+                if variable.type == "symlink":
+                    continue
+                self._convert_variable_inference(variable)
         for variable in self.get_variables():
             if variable.type == "symlink":
                 continue
+            if variable.version != "1.0":
+                self._default_variable_copy_informations(variable)
+            if variable.multi is None:
+                variable.multi = False
+            if variable.type is None:
+                variable.type = "string"
+            self.objectspace.informations.add(
+                variable.path, "type", variable.type
+            )
             self._convert_variable(variable)
 
+    def _convert_variable_inference(
+            self,
+            variable,
+    ) -> None:
+        # variable has no type
+        if variable.type is None:
+            # choice type inference from the `choices` attribute
+            if variable.choices is not None:
+                variable.type = "choice"
+            elif variable.regexp is not None:
+                variable.type = "regexp"
+            elif variable.default not in [None, []]:
+                if isinstance(variable.default, list):
+                    tested_value = variable.default[0]
+                else:
+                    tested_value = variable.default
+                variable.type = self.basic_types.get(type(tested_value), None)
+        # variable has no multi attribute
+        if variable.multi is None and not (variable.type is None and isinstance(variable.default, VariableCalculation)):
+            if variable.path in self.objectspace.leaders:
+                variable.multi = True
+            else:
+                variable.multi = isinstance(variable.default, list)
+
+    def _default_variable_copy_informations(
+            self,
+            variable,
+    ) -> None:
+        # if a variable has a variable as default value, that means the type/params or multi should has same value
+        if variable.type is not None or not isinstance(variable.default, VariableCalculation):
+            return
+        # copy type and params
+        calculated_variable_path = variable.default.variable
+        calculated_variable, suffix = self.objectspace.paths.get_with_dynamic(
+            calculated_variable_path, variable.default.path_prefix, variable.path, variable.version, variable.namespace, variable.xmlfiles
+        )
+        if calculated_variable is None:
+            return
+        variable.type = calculated_variable.type
+        if variable.params is None and calculated_variable.params is not None:
+            variable.params = calculated_variable.params
+        # copy multi attribut
+        if variable.multi is None:
+            calculated_path = calculated_variable.path
+            if calculated_path in self.objectspace.leaders and variable.path in self.objectspace.followers and calculated_path.rsplit('.')[0] == variable.path.rsplit('.')[0]:
+                variable.multi = False
+            else:
+                variable.multi = calculated_variable.multi
+
     def _convert_variable(
         self,
         variable: dict,
@@ -97,25 +167,42 @@ class Annotator(Walk):  # pylint: disable=R0903
             self.objectspace.multis[variable.path] = True
         if variable.path in self.objectspace.leaders:
             if not self.objectspace.multis.get(variable.path, False):
-                msg = _(f'the variable "{variable.path}" in a leadership must be multi')
-                raise DictConsistencyError(msg, 32, variable.xmlfiles)
+                variable.multi = self.objectspace.multis[variable.path] = True
             family = self.objectspace.paths[variable.path.rsplit(".", 1)[0]]
             if variable.hidden:
                 family.hidden = variable.hidden
             elif family.hidden:
                 variable.hidden = family.hidden
             variable.hidden = None
+        if variable.choices is not None and variable.type != 'choice':
+            msg = _(f'the variable "{variable.path}" has choices attribut but has not the "choice" type')
+            raise DictConsistencyError(msg, 11, variable.xmlfiles)
+        if variable.regexp is not None and variable.type != 'regexp':
+            msg = _(f'the variable "{variable.path}" has regexp attribut but has not the "regexp" type')
+            raise DictConsistencyError(msg, 37, variable.xmlfiles)
 
     def convert_test(self):
         """Convert variable tests value"""
         for variable in self.get_variables():
+            if variable.type == "symlink":
+                continue
             if variable.test is None:
-                # with we want remove test, we set "" has test value
                 continue
             self.objectspace.informations.add(
                 variable.path, "test", tuple(variable.test)
             )
 
+    def convert_examples(self):
+        """Convert variable tests value"""
+        for variable in self.get_variables():
+            if variable.type == "symlink":
+                continue
+            if variable.examples is None:
+                continue
+            self.objectspace.informations.add(
+                variable.path, "examples", tuple(variable.examples)
+            )
+
     def convert_help(self):
         """Convert variable help"""
         for variable in self.get_variables():
@@ -123,3 +210,28 @@ class Annotator(Walk):  # pylint: disable=R0903
                 continue
             self.objectspace.informations.add(variable.path, "help", variable.help)
             del variable.help
+
+    def verify_choices(self):
+        for variable in self.get_variables():
+            if variable.type != 'choice' or variable.default is None:
+                continue
+            if not isinstance(variable.choices, list):
+                continue
+            choices = variable.choices
+            has_calculation = False
+            for choice in choices:
+                if isinstance(choice, Calculation):
+                    has_calculation = True
+                    break
+            if has_calculation:
+                continue
+
+            default = variable.default
+            if not isinstance(default, list):
+                default = [default]
+            for value in default:
+                if isinstance(value, Calculation):
+                    continue
+                if value not in choices:
+                    msg = _(f'the variable "{variable.path}" has an unvalid default value "{value}" should be in {display_list(choices, separator="or", add_quote=True)}')
+                    raise DictConsistencyError(msg, 26, variable.xmlfiles)

src/rougail/config.py

@@ -28,54 +28,427 @@ You should have received a copy of the GNU General Public License
 along with this program; if not, write to the Free Software
 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 """
-from os.path import join, abspath, dirname
+from pathlib import Path
+from tiramisu import Config
+from ruamel.yaml import YAML
+from .utils import _, load_modules, normalize_family
+from .convert import RougailConvert
 
 
-ROUGAILROOT = "/srv/rougail"
-DTDDIR = join(dirname(abspath(__file__)), "data")
+RENAMED = {'dictionaries_dir': 'main_dictionaries',
+           'variable_namespace': 'main_namespace',
+           'functions_file': 'functions_files',
+           }
+NOT_IN_TIRAMISU = {'custom_types': {},
+                   }
+SUBMODULES = None
 
 
-RougailConfig = {
-    "dictionaries_dir": [join(ROUGAILROOT, "dictionaries")],
-    "extra_dictionaries": {},
-    "services_dir": [join(ROUGAILROOT, "services")],
-    "patches_dir": join(ROUGAILROOT, "patches"),
-    "templates_dir": join(ROUGAILROOT, "templates"),
-    "destinations_dir": join(ROUGAILROOT, "destinations"),
-    "tmp_dir": join(ROUGAILROOT, "tmp"),
-    "dtdfilename": join(DTDDIR, "rougail.dtd"),
-    "yamlschema_filename": join(DTDDIR, "rougail.yml"),
-    "functions_file": join(ROUGAILROOT, "functions.py"),
-    "system_service_directory": "/usr/lib/systemd/system",
-    "systemd_service_destination_directory": "/usr/local/lib",
-    "systemd_service_directory": "/systemd",
-    "systemd_service_file": "rougail.conf",
-    "systemd_service_ip_file": "rougail_ip.conf",
-    "systemd_tmpfile_factory_dir": "/usr/local/lib",
-    "systemd_tmpfile_directory": "/tmpfiles.d",
-    "systemd_tmpfile_file": "0rougail.conf",
-    "systemd_tmpfile_delete_before_create": False,
-    "variable_namespace": "rougail",
-    "variable_namespace_description": "Rougail",
-    "auto_freeze_variable": "server_deployed",
-    "internal_functions": [],
-    "multi_functions": [],
-    "extra_annotators": [],
-    "modes_level": ["basic", "standard", "advanced"],
-    "default_family_mode": "basic",
-    "default_variable_mode": "standard",
-    "default_files_engine": "jinja",
-    "default_files_mode": 644,
-    "default_files_owner": "root",
-    "default_files_group": "root",
-    "default_files_included": "no",
-    "default_overrides_engine": "jinja",
-    "default_service_names_engine": "none",
-    "default_certificate_domain": "rougail.server_name",
-    "base_option_name": "baseoption",
-    "export_with_import": True,
-    "force_convert_dyn_option_description": False,
-    "suffix": "",
-    "tiramisu_cache": None,
-    "custom_types": {},
-}
+def get_sub_modules():
+    global SUBMODULES
+    if SUBMODULES is None:
+        SUBMODULES = {}
+        for submodule in Path(__file__).parent.iterdir():
+            if submodule.name.startswith('_') or not submodule.is_dir():
+                continue
+            config_file = submodule / 'config.py'
+            if config_file.is_file():
+                SUBMODULES[submodule.name] = load_modules('rougail.' + submodule.name + '.config', str(config_file))
+    return SUBMODULES
+                    
+
+def get_level(module):
+    return module['level']
+
+
+class _RougailConfig:
+    def __init__(self,
+                 backward_compatibility: bool,
+                 root,
+                 extra_vars: dict
+                 ):
+        self.backward_compatibility = backward_compatibility
+        self.root = root
+        self.config = Config(
+                    self.root,
+                )
+        self.config.property.read_only()
+        self.extra_vars = extra_vars
+        self.not_in_tiramisu = NOT_IN_TIRAMISU | extra_vars
+        for variable, default_value in self.not_in_tiramisu.items():
+            if not isinstance(default_value, str):
+                default_value = default_value.copy()
+            setattr(self, variable, default_value)
+
+    def copy(self):
+        rougailconfig = _RougailConfig(self.backward_compatibility, self.root, self.extra_vars)
+        rougailconfig.config.value.importation(self.config.value.exportation())
+        rougailconfig.config.property.importation(self.config.property.exportation())
+        rougailconfig.config.property.read_only()
+        for variable in self.not_in_tiramisu:
+            value = getattr(self, variable)
+            if not isinstance(value, str):
+                value = value.copy()
+            setattr(rougailconfig, variable, value)
+        return rougailconfig
+
+    def __setitem__(self,
+                    key,
+                    value,
+                    ) -> None:
+        if key in self.not_in_tiramisu:
+            setattr(self, key, value)
+        else:
+            self.config.property.read_write()
+            if key == 'export_with_import':
+                key = 'not_export_with_import'
+            key = RENAMED.get(key, key)
+            option = self.config.option(key)
+            if option.isoptiondescription() and option.isleadership():
+                leader = list(value)
+                option.leader().value.reset()
+                option.leader().value.set(leader)
+                follower = option.followers()[0]
+                for idx, val in enumerate(value.values()):
+                    self.config.option(follower.path(), idx).value.set(val)
+            elif key == 'not_export_with_import':
+                option.value.set(not value)
+            else:
+                option.value.set(value)
+            self.config.property.read_only()
+
+    def __getitem__(self,
+                    key,
+                    ) -> None:
+        if key in self.not_in_tiramisu:
+            return getattr(self, key)
+        if key == 'export_with_import':
+            key = 'not_export_with_import'
+        option = self.config.option(key)
+        if option.isoptiondescription() and option.isleadership():
+            return self.get_leadership(option)
+        ret = self.config.option(key).value.get()
+        if key == 'not_export_with_import':
+            return not ret
+        return ret
+
+    def get_leadership(self,
+                       option
+                       ) -> dict:
+        leader = None
+        followers = []
+        for opt, value in option.value.get().items():
+            if opt.issymlinkoption():
+                continue
+            if leader is None:
+                leader = value
+            else:
+                followers.append(value)
+        return dict(zip(leader, followers))
+
+    def parse(self, config) -> str:
+        for option in config:
+            if option.isoptiondescription():
+                yield from self.parse(option)
+            elif not option.issymlinkoption():
+                yield f'{option.path()}: {option.value.get()}'
+
+    def __repr__(self):
+        self.config.property.read_write()
+        try:
+            values = "\n".join(self.parse(self.config))
+        except Exception as err:
+            values = str(err)
+        self.config.property.read_only()
+        return values
+
+
+class FakeRougailConvert(RougailConvert):
+    def __init__(self,
+                 add_extra_options: bool,
+                 ) -> None:
+        self.add_extra_options = add_extra_options
+        super().__init__({})
+
+    def load_config(self) -> None:
+        self.sort_dictionaries_all = False
+        self.main_namespace = None
+        self.suffix = ''
+        self.custom_types = {}
+        self.functions_files = []
+        self.modes_level = []
+        self.extra_annotators = []
+        self.base_option_name = "baseoption"
+        self.export_with_import = True
+        self.internal_functions = []
+        self.plugins = ['structural_commandline']
+        self.add_extra_options = self.add_extra_options
+
+
+def get_rougail_config(*,
+                       backward_compatibility: bool=True,
+                       add_extra_options: bool=True,
+                       ) -> _RougailConfig:
+    if backward_compatibility:
+        main_namespace_default = 'rougail'
+    else:
+        main_namespace_default = 'null'
+    rougail_options = """default_dictionary_format_version:
+  description: Dictionary format version by default, if not specified in dictionary file
+  alternative_name: v
+  choices:
+    - '1.0'
+    - '1.1'
+  mandatory: false
+
+main_dictionaries:
+  description: 'Directories where dictionary files are placed'
+  type: unix_filename
+  alternative_name: m
+  params:
+     allow_relative: True
+     test_existence: True
+     types:
+       - directory
+  multi: true
+
+sort_dictionaries_all:
+  description: Sort dictionaries from differents directories
+  negative_description: Sort dictionaries directory by directory
+  default: false
+
+main_namespace:
+  description: Main namespace name
+  default: MAIN_MAMESPACE_DEFAULT
+  alternative_name: s
+  mandatory: false
+
+extra_dictionaries:
+  description: Extra namespaces
+  type: leadership
+  disabled:
+    variable: main_namespace
+    when: null
+
+  names:
+    description: 'Extra namespace name'
+    alternative_name: xn
+    multi: true
+    mandatory: false
+
+  directories:
+    description: Directories where extra dictionary files are placed
+    alternative_name: xd
+    type: unix_filename
+    params:
+      allow_relative: true
+      test_existence: true
+      types:
+        - directory
+    multi: true
+
+upgrade:
+  description: Update dictionaries to newest Rougail format version
+  negative_description: Do not update dictionaries to newest Rougail format version
+  default: false
+
+upgrade_options:
+  description: Update informations
+  disabled:
+    variable: upgrade
+    when: false
+
+  main_dictionaries:
+    description: 'Directories where dictionary files will be placed'
+    default:
+      variable: __.main_dictionaries
+
+  extra_dictionary:
+    description: 'Directories where extra files will be placed'
+    type: unix_filename
+    params:
+      allow_relative: true
+      test_existence: true
+      types:
+        - directory
+    disabled:
+      variable: __.main_namespace
+      when: null
+
+functions_files:
+  description: File with functions
+  alternative_name: c
+  type: unix_filename
+  params:
+    allow_relative: true
+    test_existence: true
+    types:
+      - file
+  multi: true
+  mandatory: false
+
+modes_level:
+  description: All modes level available
+  default:
+    - basic
+    - standard
+    - advanced
+  commandline: false
+
+default_family_mode:
+  description: Default mode for a family
+  default:
+    type: jinja
+    jinja: |
+      {{ modes_level[0] }}
+  validators:
+    - type: jinja
+      jinja: |
+        {% if default_family_mode not in modes_level %}
+        not in modes_level ({modes_level})
+        {% endif %}
+  commandline: false
+
+default_variable_mode:
+  description: Default mode for a variable
+  default:
+    type: jinja
+    jinja: |
+      {{ modes_level[1] }}
+  validators:
+    - type: jinja
+      jinja: |
+        {% if default_variable_mode not in modes_level %}
+        not in modes_level ({modes_level})
+        {% endif %}
+  commandline: false
+
+base_option_name:
+  description: Option name for the base option
+  default: baseoption
+  commandline: false
+
+not_export_with_import:
+  description: In cache file, do not importation of Tiramisu and other dependencies
+  default: false
+  commandline: false
+
+tiramisu_cache:
+  description: Tiramisu cache filename
+  alternative_name: t
+  type: unix_filename
+  mandatory: false
+  params:
+    allow_relative: true
+
+internal_functions:
+  description: Name of internal functions that we can use as a function
+  multi: true
+  mandatory: false
+  commandline: false
+
+extra_annotators:
+  description: Name of extra annotators
+  multi: true
+  mandatory: false
+  commandline: false
+
+plugins:
+  description: Name of Rougail plugins
+  multi: true
+  mandatory: false
+  commandline: false
+
+suffix:
+  description: Suffix add to generated option name
+  default: ''
+  mandatory: false
+  commandline: false
+""".replace('MAIN_MAMESPACE_DEFAULT', main_namespace_default)
+    processes = {'structural': [],
+                 'output': [],
+                 'user data': [],
+                 }
+    for module in get_sub_modules().values():
+        data = module.get_rougail_config()
+        processes[data['process']].append(data)
+    # reorder
+    for process in processes:
+        processes[process] = list(sorted(processes[process], key=get_level))
+    rougail_process = """step:  # Load and exporter steps
+  disabled:
+    variable: upgrade"""
+    for process in processes:
+        if processes[process]:
+            objects = processes[process]
+            rougail_process += """
+  {NAME}:
+    description: Select for {NAME}
+    alternative_name: {NAME[0]}
+    choices:
+""".format(NAME=normalize_family(process),
+           )
+            for obj in objects:
+                rougail_process += f"      - {obj['name']}\n"
+            if process == 'structural':
+                rougail_process += "    commandline: false"
+            elif process == 'user data':
+                rougail_process += """    multi: true
+    mandatory: false
+"""
+                hidden_outputs = [process['name'] for process in processes['output'] if not process.get('allow_user_data', True)]
+                if hidden_outputs:
+                    rougail_process += """    hidden:
+      type: jinja
+      jinja: |
+"""
+                    for hidden_output in hidden_outputs:
+                        rougail_process += """        {% if _.output == 'NAME' %}
+        Cannot load user data for NAME output
+        {% endif %}
+""".replace('NAME', hidden_output)
+            else:
+                rougail_process += '    default: {DEFAULT}'.format(DEFAULT=objects[0]['name'])
+        else:
+            rougail_process += """
+  {NAME}:
+    description: Select for {NAME}
+    hidden: true
+    mandatory: false
+    multi: true
+""".format(NAME=normalize_family(process),
+           )
+    rougail_options += rougail_process
+    convert = FakeRougailConvert(add_extra_options)
+    convert._init()
+    convert.namespace = None
+    convert.parse_root_file(
+        'rougail.config',
+        '',
+        '1.1',
+        YAML().load(rougail_options),
+    )
+    extra_vars = {}
+    for process in processes:
+        for obj in processes[process]:
+            if 'extra_vars' in obj:
+                extra_vars |= obj['extra_vars']
+            if not 'options' in obj:
+                continue
+            convert.parse_root_file(
+                f'rougail.config.{obj["name"]}',
+                '',
+                '1.1',
+                YAML().load(obj['options']),
+            )
+            
+    tiram_obj = convert.save(None)
+    optiondescription = {}
+    exec(tiram_obj, {}, optiondescription)  # pylint: disable=W0122
+    return _RougailConfig(backward_compatibility,
+                          optiondescription["option_0"],
+                          extra_vars=extra_vars,
+                          )
+
+
+RougailConfig = get_rougail_config()

src/rougail/error.py

@@ -73,3 +73,14 @@ class DictConsistencyError(Exception):
 
 class UpgradeError(Exception):
     """Error during XML upgrade"""
+
+## ---- generic exceptions ----
+
+class NotFoundError(Exception):
+    "not found error"
+    pass
+    
+## ---- specific exceptions ----
+
+class VariableCalculationDependencyError(Exception):
+    pass

src/rougail/object_model.py

@@ -29,10 +29,12 @@ from pydantic import (
     StrictStr,
     ConfigDict,
 )
+from tiramisu import undefined
 from .utils import get_jinja_variable_to_param, get_realpath
-
+from .error import DictConsistencyError, VariableCalculationDependencyError
 
 BASETYPE = Union[StrictBool, StrictInt, StrictFloat, StrictStr, None]
+PROPERTY_ATTRIBUTE = ["frozen", "hidden", "disabled", "mandatory"]
 
 
 def convert_boolean(value: str) -> bool:
@@ -44,44 +46,63 @@ def convert_boolean(value: str) -> bool:
         return True
     elif value == "false":
         return False
-    raise Exception(f"unknown boolean value {value}")
+    elif value in ["", None]:
+        return None
+    raise Exception(f'unknown boolean value "{value}"')
 
 
 CONVERT_OPTION = {
-    "string": dict(opttype="StrOption"),
-    "number": dict(opttype="IntOption", func=int),
-    "float": dict(opttype="FloatOption", func=float),
+    "string": dict(opttype="StrOption", example="example"),
+    "number": dict(opttype="IntOption", func=int, example=42),
+    "float": dict(opttype="FloatOption", func=float, example=1.42),
     "boolean": dict(opttype="BoolOption", func=convert_boolean),
-    "secret": dict(opttype="PasswordOption"),
-    "mail": dict(opttype="EmailOption"),
-    "unix_filename": dict(opttype="FilenameOption"),
-    "date": dict(opttype="DateOption"),
-    "unix_user": dict(opttype="UsernameOption"),
-    "ip": dict(opttype="IPOption", initkwargs={"allow_reserved": True}),
-    "cidr": dict(opttype="IPOption", initkwargs={"cidr": True}),
-    "netmask": dict(opttype="NetmaskOption"),
-    "network": dict(opttype="NetworkOption"),
-    "network_cidr": dict(opttype="NetworkOption", initkwargs={"cidr": True}),
-    "broadcast": dict(opttype="BroadcastOption"),
+    "secret": dict(opttype="PasswordOption", example="secrets"),
+    "mail": dict(opttype="EmailOption", example="user@example.net"),
+    "unix_filename": dict(opttype="FilenameOption", example="/tmp/myfile.txt"),
+    "date": dict(opttype="DateOption", example="2000-01-01"),
+    "unix_user": dict(opttype="UsernameOption", example="username"),
+    "ip": dict(
+        opttype="IPOption", initkwargs={"allow_reserved": True}, example="1.1.1.1"
+    ),
+    "cidr": dict(opttype="IPOption", initkwargs={"cidr": True}, example="1.1.1.0/24"),
+    "netmask": dict(opttype="NetmaskOption", example="255.255.255.0"),
+    "network": dict(opttype="NetworkOption", example="1.1.1.0"),
+    "network_cidr": dict(
+        opttype="NetworkOption", initkwargs={"cidr": True}, example="1.1.1.0/24"
+    ),
+    "broadcast": dict(opttype="BroadcastOption", example="1.1.1.255"),
     "netbios": dict(
         opttype="DomainnameOption",
         initkwargs={"type": "netbios", "warnings_only": True},
+        example="example",
     ),
     "domainname": dict(
-        opttype="DomainnameOption", initkwargs={"type": "domainname", "allow_ip": False}
+        opttype="DomainnameOption",
+        initkwargs={"type": "domainname", "allow_ip": False},
+        example="example.net",
     ),
     "hostname": dict(
-        opttype="DomainnameOption", initkwargs={"type": "hostname", "allow_ip": False}
+        opttype="DomainnameOption",
+        initkwargs={"type": "hostname", "allow_ip": False},
+        example="example",
     ),
     "web_address": dict(
-        opttype="URLOption", initkwargs={"allow_ip": False, "allow_without_dot": True}
+        opttype="URLOption",
+        initkwargs={"allow_ip": False, "allow_without_dot": True},
+        example="https://example.net",
     ),
-    "port": dict(opttype="PortOption", initkwargs={"allow_private": True}),
-    "mac": dict(opttype="MACOption"),
+    "port": dict(
+        opttype="PortOption", initkwargs={"allow_private": True}, example="111"
+    ),
+    "mac": dict(opttype="MACOption", example="00:00:00:00:00"),
     "unix_permissions": dict(
-        opttype="PermissionsOption", initkwargs={"warnings_only": True}, func=int
+        opttype="PermissionsOption",
+        initkwargs={"warnings_only": True},
+        func=int,
+        example="644",
     ),
-    "choice": dict(opttype="ChoiceOption"),
+    "choice": dict(opttype="ChoiceOption", example="a_choice"),
+    "regexp": dict(opttype="RegexpOption"),
     #
     "symlink": dict(opttype="SymLinkOption"),
 }
@@ -89,24 +110,45 @@ CONVERT_OPTION = {
 
 class Param(BaseModel):
     key: str
-
     model_config = ConfigDict(extra="forbid")
 
+    def __init__(
+        self,
+        path,
+        attribute,
+        family_is_dynamic,
+        is_follower,
+        xmlfiles,
+        **kwargs,
+    ) -> None:
+        super().__init__(**kwargs)
+
 
 class AnyParam(Param):
     type: str
-    value: BASETYPE
+    value: Union[BASETYPE, List[BASETYPE]]
 
 
 class VariableParam(Param):
     type: str
     variable: str
     propertyerror: bool = True
+    whole: bool = False
     optional: bool = False
 
 
 class SuffixParam(Param):
     type: str
+    suffix: Optional[int] = None
+
+    def __init__(
+        self,
+        **kwargs,
+    ) -> None:
+        if not kwargs["family_is_dynamic"]:
+            msg = f'suffix parameter for "{kwargs["attribute"]}" in "{kwargs["path"]}" cannot be set none dynamic family'
+            raise DictConsistencyError(msg, 10, kwargs["xmlfiles"])
+        super().__init__(**kwargs)
 
 
 class InformationParam(Param):
@@ -118,6 +160,16 @@ class InformationParam(Param):
 class IndexParam(Param):
     type: str
 
+    def __init__(
+        self,
+        **kwargs,
+    ) -> None:
+
+        if not kwargs["is_follower"]:
+            msg = f'the variable "{kwargs["path"]}" is not a follower, so cannot have index type for param in "{kwargs["attribute"]}"'
+            raise DictConsistencyError(msg, 25, kwargs["xmlfiles"])
+        super().__init__(**kwargs)
+
 
 PARAM_TYPES = {
     "any": AnyParam,
@@ -132,6 +184,11 @@ class Calculation(BaseModel):
     path_prefix: Optional[str]
     path: str
     inside_list: bool
+    version: str
+    ori_path: Optional[str] = None
+    default_values: Any = None
+    namespace: Optional[str]
+    xmlfiles: List[str]
 
     model_config = ConfigDict(extra="forbid")
 
@@ -148,24 +205,49 @@ class Calculation(BaseModel):
         for param_obj in self.params:
             param = param_obj.model_dump()
             if param.get("type") == "variable":
-                variable_path = self.get_realpath(param["variable"])
-                variable, suffix, dynamic = objectspace.paths.get_with_dynamic(variable_path)
+                if self.ori_path is None:
+                    path = self.path
+                else:
+                    path = self.ori_path
+                variable, suffix = objectspace.paths.get_with_dynamic(
+                    param["variable"],
+                    self.path_prefix,
+                    path,
+                    self.version,
+                    self.namespace,
+                    self.xmlfiles,
+                )
                 if not variable:
                     if not param.get("optional"):
-                        raise Exception(f"cannot find {variable_path}")
+                        msg = f'cannot find variable "{param["variable"]}" defined attribute in "{self.attribute_name}" for "{self.path}"'
+                        raise DictConsistencyError(msg, 22, self.xmlfiles)
                     continue
                 if not isinstance(variable, objectspace.variable):
                     raise Exception("pfff it's a family")
                 param["variable"] = variable
                 if suffix:
                     param["suffix"] = suffix
-                    param["dynamic"] = dynamic
             if param.get("type") == "information":
                 if param["variable"]:
-                    variable_path = self.get_realpath(param["variable"])
-                    param["variable"] = objectspace.paths[variable_path]
-                    if not param["variable"]:
-                        raise Exception("pffff")
+                    if self.ori_path is None:
+                        path = self.path
+                    else:
+                        path = self.ori_path
+                    variable, suffix = objectspace.paths.get_with_dynamic(
+                        param["variable"],
+                        self.path_prefix,
+                        path,
+                        self.version,
+                        self.namespace,
+                        self.xmlfiles,
+                    )
+                    if not variable:
+                        msg = f'cannot find variable "{param["variable"]}" defined in "{self.attribute_name}" for "{self.path}"'
+                        raise DictConsistencyError(msg, 14, self.xmlfiles)
+                    param["variable"] = variable
+                    if suffix:
+                        msg = f'variable "{param["variable"]}" defined in "{self.attribute_name}" for "{self.path}" is a dynamic variable'
+                        raise DictConsistencyError(msg, 15, self.xmlfiles)
                 else:
                     del param["variable"]
             params[param.pop("key")] = param
@@ -174,11 +256,20 @@ class Calculation(BaseModel):
 
 class JinjaCalculation(Calculation):
     attribute_name: Literal[
-        "frozen", "hidden", "mandatory", "disabled", "default", "validators", "choices"
+        "frozen",
+        "hidden",
+        "mandatory",
+        "empty",
+        "disabled",
+        "default",
+        "validators",
+        "choices",
+        "dynamic",
     ]
     jinja: StrictStr
     params: Optional[List[Param]] = None
     return_type: BASETYPE = None
+    description: Optional[StrictStr] = None
 
     def _jinja_to_function(
         self,
@@ -203,29 +294,47 @@ class JinjaCalculation(Calculation):
                 "__internal_jinja": jinja_path,
                 "__internal_type": return_type,
                 "__internal_multi": multi,
+                "__internal_files": self.xmlfiles,
+                "__internal_attribute": self.attribute_name,
+                "__internal_variable": self.path,
             },
         }
+        if self.default_values:
+            default["params"]["__default_value"] = self.default_values
         if add_help:
             default["help"] = function + "_help"
         if self.params:
             default["params"] |= self.get_params(objectspace)
         if params:
             default["params"] |= params
-        for sub_variable, suffix, true_path, dynamic in get_jinja_variable_to_param(
+        if self.ori_path is None:
+            path = self.path
+        else:
+            path = self.ori_path
+        for sub_variable, suffix, true_path in get_jinja_variable_to_param(
+            path,
             self.jinja,
             objectspace,
             variable.xmlfiles,
             objectspace.functions,
             self.path_prefix,
+            self.version,
+            self.namespace,
         ):
-            if sub_variable.path in objectspace.variables:
+            if true_path in default["params"]:
+                continue
+            if isinstance(sub_variable, dict):
+                default["params"][true_path] = {
+                    "type": "value",
+                    "value": sub_variable,
+                }
+            else:
                 default["params"][true_path] = {
                     "type": "variable",
                     "variable": sub_variable,
                 }
                 if suffix:
                     default["params"][true_path]["suffix"] = suffix
-                    default["params"][true_path]["dynamic"] = dynamic
         return default
 
     def to_function(
@@ -258,7 +367,7 @@ class JinjaCalculation(Calculation):
                 False,
                 objectspace,
             )
-        elif self.attribute_name in ["frozen", "hidden", "disabled", "mandatory"]:
+        elif self.attribute_name in PROPERTY_ATTRIBUTE:
             if self.return_type:
                 raise Exception("return_type not allowed!")
             return self._jinja_to_function(
@@ -267,7 +376,7 @@ class JinjaCalculation(Calculation):
                 False,
                 objectspace,
                 add_help=True,
-                params={None: [self.attribute_name]},
+                params={None: [self.attribute_name], "when": True, "inverse": False},
             )
         elif self.attribute_name == "choices":
             return_type = self.return_type
@@ -279,26 +388,52 @@ class JinjaCalculation(Calculation):
                 not self.inside_list,
                 objectspace,
             )
+        elif self.attribute_name == "dynamic":
+            return self._jinja_to_function(
+                "jinja_to_function",
+                "string",
+                True,
+                objectspace,
+            )
         raise Exception("hu?")
 
 
-class VariableCalculation(Calculation):
-    attribute_name: Literal[
-        "frozen", "hidden", "mandatory", "disabled", "default", "choices"
-    ]
+class _VariableCalculation(Calculation):
     variable: StrictStr
     propertyerror: bool = True
+    allow_none: bool = False
 
-    def to_function(
+    def get_variable(self,
+                     objectspace,
+                     ) -> "Variable":
+        if self.ori_path is None:
+            path = self.path
+        else:
+            path = self.ori_path
+        variable, suffix = objectspace.paths.get_with_dynamic(
+            self.variable,
+            self.path_prefix,
+            path,
+            self.version,
+            self.namespace,
+            self.xmlfiles,
+        )
+        if variable and not isinstance(variable, objectspace.variable):
+            # FIXME remove the pfff
+            raise Exception("pfff it's a family")
+        return variable, suffix
+
+    def get_params(
         self,
         objectspace,
-    ) -> dict:
-        variable_path = self.get_realpath(self.variable)
-        variable, suffix, dynamic = objectspace.paths.get_with_dynamic(variable_path)
+        variable: "Variable",
+        suffix: Optional[str],
+        *,
+        needs_multi: Optional[bool] = None,
+    ):
         if not variable:
-            raise Exception(f"pffff {variable_path}")
-        if not isinstance(variable, objectspace.variable):
-            raise Exception("pfff it's a family")
+            msg = f'Variable not found "{self.variable}" for attribut "{self.attribute_name}" for variable "{self.path}"'
+            raise DictConsistencyError(msg, 88, self.xmlfiles)
         param = {
             "type": "variable",
             "variable": variable,
@@ -306,36 +441,124 @@ class VariableCalculation(Calculation):
         }
         if suffix:
             param["suffix"] = suffix
-            param["dynamic"] = dynamic
         params = {None: [param]}
-        function = "calc_value"
-        help_function = None
-        if self.attribute_name in ["frozen", "hidden", "disabled", "mandatory"]:
-            function = "variable_to_property"
-            help_function = "variable_to_property"
-            if variable.type != "boolean":
-                raise Exception("only boolean!")
-            params[None].insert(0, self.attribute_name)
-        if not self.inside_list and self.path in objectspace.multis:
-            if (
-                not objectspace.paths.is_dynamic(variable_path)
-                and variable_path not in objectspace.multis
-            ):
-                params["multi"] = True
+        if self.default_values:
+            params["__default_value"] = self.default_values
+        if self.allow_none:
             params["allow_none"] = True
-        if self.inside_list and variable.path in objectspace.multis:
-            raise Exception("pfff")
-        ret = {
-            "function": function,
+        if needs_multi is None:
+            if self.attribute_name != "default":
+                needs_multi = True
+            else:
+                needs_multi = self.path in objectspace.multis
+        calc_variable_is_multi = variable.path in objectspace.multis
+        if not calc_variable_is_multi:
+            if variable.path in objectspace.paths._dynamics and (
+                suffix is None or suffix[-1] is None
+            ):
+                self_dyn_path = objectspace.paths._dynamics.get(self.path)
+                if self_dyn_path is not None:
+                    var_dyn_path = objectspace.paths._dynamics[variable.path]
+                    if self_dyn_path != var_dyn_path and not self_dyn_path.startswith(
+                        f"{var_dyn_path}."
+                    ):
+                        calc_variable_is_multi = True
+                else:
+                    calc_variable_is_multi = True
+            elif suffix and '{{ suffix }}' in suffix:
+                calc_variable_is_multi = True
+        if needs_multi:
+            if calc_variable_is_multi:
+                if self.inside_list:
+                    msg = f'the variable "{self.path}" has an invalid attribute "{self.attribute_name}", the variable "{variable.path}" is multi but is inside a list'
+                    raise DictConsistencyError(msg, 18, self.xmlfiles)
+            elif not self.inside_list:
+                msg = f'the variable "{self.path}" has an invalid attribute "{self.attribute_name}", the variable "{variable.path}" is not multi but is not inside a list'
+                raise DictConsistencyError(msg, 20, self.xmlfiles)
+        elif self.inside_list:
+            msg = f'the variable "{self.path}" has an invalid attribute "{self.attribute_name}", it\'s a list'
+            raise DictConsistencyError(msg, 23, self.xmlfiles)
+        elif calc_variable_is_multi:
+            if variable.multi or variable.path.rsplit('.', 1)[0] != self.path.rsplit('.', 1)[0]:
+                # it's not a follower or not in same leadership
+                msg = f'the variable "{self.path}" has an invalid attribute "{self.attribute_name}", the variable "{variable.path}" is a multi'
+                raise DictConsistencyError(msg, 21, self.xmlfiles)
+            else:
+                params[None][0]['index'] = {'index': {'type': 'index'}}
+        return params
+
+
+class VariableCalculation(_VariableCalculation):
+    attribute_name: Literal["default", "choices", "dynamic"]
+    optional: bool = False
+
+    def to_function(
+        self,
+        objectspace,
+    ) -> dict:
+        if self.attribute_name != "default" and self.optional:
+            msg = f'"{self.attribute_name}" variable shall not have an "optional" attribute for variable "{self.variable}"'
+            raise DictConsistencyError(msg, 33, self.xmlfiles)
+        variable, suffix = self.get_variable(objectspace)
+        if not variable and self.optional:
+            raise VariableCalculationDependencyError()
+        params = self.get_params(objectspace,
+                                 variable,
+                                 suffix,
+                                 )
+        return {
+            "function": "calc_value",
             "params": params,
         }
-        if help_function:
-            ret["help"] = help_function
-        return ret
+
+
+class VariablePropertyCalculation(_VariableCalculation):
+    attribute_name: Literal[*PROPERTY_ATTRIBUTE]
+    when: Any = undefined
+    when_not: Any = undefined
+
+    def to_function(
+        self,
+        objectspace,
+    ) -> dict:
+        variable, suffix = self.get_variable(objectspace)
+        params = self.get_params(objectspace,
+                                 variable,
+                                 suffix,
+                                 needs_multi=False,)
+        variable = params[None][0]["variable"]
+        if self.when is not undefined:
+            if self.version == "1.0":
+                msg = f'when is not allowed in format version 1.0 for attribute "{self.attribute_name}" for variable "{self.path}"'
+                raise DictConsistencyError(msg, 103, variable.xmlfiles)
+            if self.when_not is not undefined:
+                msg = f'the variable "{self.path}" has an invalid attribute "{self.attribute_name}", when and when_not cannot set together'
+                raise DictConsistencyError(msg, 31, variable.xmlfiles)
+            when = self.when
+            inverse = False
+        elif self.when_not is not undefined:
+            if self.version == "1.0":
+                msg = f'when_not is not allowed in format version 1.0 for attribute "{self.attribute_name}" for variable "{self.path}"'
+                raise DictConsistencyError(msg, 104, variable.xmlfiles)
+            when = self.when_not
+            inverse = True
+        else:
+            if variable.type != "boolean":
+                raise Exception("only boolean!")
+            when = True
+            inverse = False
+        params[None].insert(0, self.attribute_name)
+        params["when"] = when
+        params["inverse"] = inverse
+        return {
+            "function": "variable_to_property",
+            "params": params,
+            "help": "variable_to_property",
+        }
 
 
 class InformationCalculation(Calculation):
-    attribute_name: Literal["default"]
+    attribute_name: Literal["default", "choice", "dynamic"]
     information: StrictStr
     variable: Optional[StrictStr]
 
@@ -343,42 +566,109 @@ class InformationCalculation(Calculation):
         self,
         objectspace,
     ) -> dict:
-        param = {
-            "type": "information",
-            "information": self.information,
+        params = {
+            None: [
+                {
+                    "type": "information",
+                    "information": self.information,
+                }
+            ]
         }
         if self.variable:
-            variable_path = self.get_realpath(self.variable)
-            variable = objectspace.paths[variable_path]
-            if variable is None:
+            if self.ori_path is None:
+                path = self.path
+            else:
+                path = self.ori_path
+            variable, suffix = objectspace.paths.get_with_dynamic(
+                self.variable,
+                self.path_prefix,
+                path,
+                self.version,
+                self.namespace,
+                self.xmlfiles,
+            )
+            if variable is None or suffix is not None:
                 raise Exception("pfff")
-            param["variable"] = variable
+            params[None][0]["variable"] = variable
+        if self.default_values:
+            params["__default_value"] = self.default_values
         return {
             "function": "calc_value",
-            "params": {None: [param]},
+            "params": params,
         }
 
 
-class SuffixCalculation(Calculation):
-    attribute_name: Literal["default"]
+class _SuffixCalculation(Calculation):
+    suffix: Optional[int] = None
+
+    def get_suffix(self) -> dict:
+        suffix = {"type": "suffix"}
+        if self.suffix is not None:
+            suffix["suffix"] = self.suffix
+        return suffix
+
+
+class SuffixCalculation(_SuffixCalculation):
+    attribute_name: Literal["default", "choice", "dynamic"]
 
     def to_function(
         self,
         objectspace,
     ) -> dict:
+        suffix = {"type": "suffix"}
+        if self.suffix is not None:
+            suffix["suffix"] = self.suffix
         return {
             "function": "calc_value",
-            "params": {None: [{"type": "suffix"}]},
+            "params": {None: [self.get_suffix()]},
+        }
+
+
+class SuffixPropertyCalculation(_SuffixCalculation):
+    attribute_name: Literal[*PROPERTY_ATTRIBUTE]
+    when: Any = undefined
+    when_not: Any = undefined
+
+    def to_function(
+        self,
+        objectspace,
+    ) -> dict:
+        if self.version == "1.0":
+            msg = f'when is not allowed in format version 1.0 for attribute "{self.attribute_name}"'
+            raise DictConsistencyError(msg, 105, variable.xmlfiles)
+        if self.when is not undefined:
+            if self.when_not is not undefined:
+                msg = f'the suffix has an invalid attribute "{self.attribute_name}", when and when_not cannot set together'
+                raise DictConsistencyError(msg, 35, variable.xmlfiles)
+            when = self.when
+            inverse = False
+        elif self.when_not is not undefined:
+            when = self.when_not
+            inverse = True
+        else:
+            msg = f'the suffix has an invalid attribute "{self.attribute_name}", when and when_not cannot set together'
+            raise DictConsistencyError
+        params = {None: [self.attribute_name, self.get_suffix()],
+                  "when": when,
+                  "inverse": inverse,
+                  }
+        return {
+            "function": "variable_to_property",
+            "params": params,
+            "help": "variable_to_property",
         }
 
 
 class IndexCalculation(Calculation):
-    attribute_name: Literal["default"]
+    attribute_name: Literal["default", "choice", "dynamic"]
 
     def to_function(
         self,
         objectspace,
     ) -> dict:
+        if self.path not in objectspace.followers:
+            msg = f'the variable "{self.path}" is not a follower, so cannot have index type for "{self.attribute_name}"'
+            raise DictConsistencyError(msg, 60, self.xmlfiles)
         return {
             "function": "calc_value",
             "params": {None: [{"type": "index"}]},
@@ -392,58 +682,76 @@ CALCULATION_TYPES = {
     "suffix": SuffixCalculation,
     "index": IndexCalculation,
 }
-BASETYPE_CALC = Union[StrictBool, StrictInt, StrictFloat, StrictStr, None, Calculation]
+CALCULATION_PROPERTY_TYPES = {
+    "jinja": JinjaCalculation,
+    "variable": VariablePropertyCalculation,
+    "information": InformationCalculation,
+    "suffix": SuffixPropertyCalculation,
+    "index": IndexCalculation,
+}
+BASETYPE_CALC = Union[StrictBool, StrictInt, StrictFloat, StrictStr, Calculation, None]
 
 
 class Family(BaseModel):
     name: str
     description: Optional[str] = None
     type: Literal["family", "leadership", "dynamic"] = "family"
+    path: str
     help: Optional[str] = None
     mode: Optional[str] = None
     hidden: Union[bool, Calculation] = False
     disabled: Union[bool, Calculation] = False
+    namespace: Optional[str]
+    version: str
     xmlfiles: List[str] = []
-    path: str
 
     model_config = ConfigDict(extra="forbid", arbitrary_types_allowed=True)
 
 
 class Dynamic(Family):
-    variable: str
+    # None only for format 1.0
+    variable: str = None
+    dynamic: Union[List[Union[StrictStr, Calculation]], Calculation]
 
 
-class _Variable(BaseModel):
+class Variable(BaseModel):
+    # type will be set dynamically in `annotator/value.py`, default is None
+    type: str = None
     name: str
     description: Optional[str] = None
     default: Union[List[BASETYPE_CALC], BASETYPE_CALC] = None
+    choices: Optional[Union[List[BASETYPE_CALC], Calculation]] = None
+    regexp: Optional[str] = None
     params: Optional[List[Param]] = None
     validators: Optional[List[Calculation]] = None
-    multi: bool = False
+    multi: Optional[bool] = None
     unique: Optional[bool] = None
     help: Optional[str] = None
     hidden: Union[bool, Calculation] = False
     disabled: Union[bool, Calculation] = False
     mandatory: Union[None, bool, Calculation] = True
+    empty: Union[None, bool, Calculation] = True
     auto_save: bool = False
     mode: Optional[str] = None
     test: Optional[list] = None
-    xmlfiles: List[str] = []
+    examples: Optional[list] = None
     path: str
+    namespace: Optional[str]
+    version: str
+    path_prefix: Optional[str]
+    xmlfiles: List[str] = []
 
     model_config = ConfigDict(extra="forbid", arbitrary_types_allowed=True)
 
 
-class Choice(_Variable):
-    type: Literal["choice"] = "choice"
-    choices: Union[List[BASETYPE_CALC], Calculation]
-
-
 class SymLink(BaseModel):
-    name: str
     type: Literal["symlink"] = "symlink"
-    opt: _Variable
-    xmlfiles: List[str] = []
+    name: str
     path: str
+    opt: Variable
+    namespace: Optional[str]
+    version: str
+    path_prefix: Optional[str]
+    xmlfiles: List[str] = []
 
     model_config = ConfigDict(extra="forbid")

src/rougail/path.py

@@ -1,527 +0,0 @@
-"""Manage path to find objects
-
-Created by:
-EOLE (http://eole.orion.education.fr)
-Copyright (C) 2005-2018
-
-Forked by:
-Cadoles (http://www.cadoles.com)
-Copyright (C) 2019-2021
-
-Silique (https://www.silique.fr)
-Copyright (C) 2022-2024
-
-distribued with GPL-2 or later license
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; either version 2 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
-"""
-from typing import List
-from .i18n import _
-from .error import DictConsistencyError
-from .utils import normalize_family
-
-
-class Path:
-    """Helper class to handle the `path` attribute.
-
-    sample: path="creole.general.condition"
-    """
-
-    def __init__(
-        self,
-        rougailconfig: "RougailConfig",
-    ) -> None:
-        self.variables = {}
-        self.families = {}
-        # self.names = {}
-        self.full_paths_families = {}
-        self.full_paths_variables = {}
-        self.full_dyn_paths_families = {}
-        self.valid_enums = {}
-        self.variable_namespace = rougailconfig["variable_namespace"]
-        self.providers = {}
-        self.suppliers = {}
-        self.list_conditions = {}
-        self.suffix = rougailconfig["suffix"]
-        self.index = 0
-
-    def set_path_prefix(self, prefix: str) -> None:
-        self._path_prefix = prefix
-        if prefix:
-            if None in self.full_paths_families:
-                raise DictConsistencyError(
-                    _(f'prefix "{prefix}" cannot be set if a prefix "None" exists'),
-                    39,
-                    None,
-                )
-        else:
-            for old_prefix in self.full_paths_families:
-                if old_prefix != None:
-                    raise DictConsistencyError(
-                        _(f"no prefix cannot be set if a prefix exists"), 84, None
-                    )
-        if prefix in self.full_paths_families:
-            raise DictConsistencyError(_(f'prefix "{prefix}" already exists'), 83, None)
-        self.full_paths_families[prefix] = {}
-        self.full_paths_variables[prefix] = {}
-        self.valid_enums[prefix] = {}
-        self.providers[prefix] = {}
-        self.suppliers[prefix] = {}
-        self.list_conditions[prefix] = {}
-
-    def has_path_prefix(self) -> bool:
-        return None not in self.full_paths_families
-
-    def get_path_prefixes(self) -> list:
-        return list(self.full_paths_families)
-
-    def get_path_prefix(self) -> str:
-        return self._path_prefix
-
-    # Family
-    def add_family(
-        self,
-        namespace: str,
-        subpath: str,
-        variableobj: str,
-        is_dynamic: str,
-        force_path_prefix: str = None,
-    ) -> str:  # pylint: disable=C0111
-        """Add a new family"""
-        if force_path_prefix is None:
-            force_path_prefix = self._path_prefix
-        path = subpath + "." + variableobj.name
-        if namespace == self.variable_namespace:
-            if variableobj.name in self.full_paths_families[force_path_prefix]:
-                msg = _(f'Duplicate family name "{variableobj.name}"')
-                raise DictConsistencyError(msg, 55, variableobj.xmlfiles)
-            self.full_paths_families[force_path_prefix][variableobj.name] = path
-        if is_dynamic:
-            if subpath in self.full_dyn_paths_families:
-                dyn_subpath = self.full_dyn_paths_families[subpath]
-            else:
-                dyn_subpath = subpath
-            self.full_dyn_paths_families[
-                path
-            ] = f"{dyn_subpath}.{variableobj.name}{{suffix}}"
-        if path in self.families:
-            msg = _(f'Duplicate family name "{path}"')
-            raise DictConsistencyError(msg, 37, variableobj.xmlfiles)
-        if path in self.variables:
-            msg = _(f'A variable and a family has the same path "{path}"')
-            raise DictConsistencyError(msg, 56, variableobj.xmlfiles)
-        self.families[path] = dict(
-            name=path,
-            namespace=namespace,
-            variableobj=variableobj,
-        )
-        self.set_name(variableobj, "optiondescription_")
-        variableobj.path = path
-        variableobj.path_prefix = force_path_prefix
-
-    def get_family(
-        self,
-        path: str,
-        current_namespace: str,
-        path_prefix: str,
-        allow_variable_namespace: bool = False,
-    ) -> "Family":  # pylint: disable=C0111
-        """Get a family"""
-        if (
-            current_namespace == self.variable_namespace or allow_variable_namespace
-        ) and path in self.full_paths_families[path_prefix]:
-            path = self.full_paths_families[path_prefix][path]
-        elif allow_variable_namespace and path_prefix:
-            path = f"{path_prefix}.{path}"
-        if path not in self.families:
-            raise DictConsistencyError(_(f'unknown option "{path}"'), 42, [])
-        dico = self.families[path]
-        if current_namespace != dico["namespace"] and (
-            not allow_variable_namespace or current_namespace != self.variable_namespace
-        ):
-            msg = _(
-                f'A family located in the "{dico["namespace"]}" namespace '
-                f'shall not be used in the "{current_namespace}" namespace'
-            )
-            raise DictConsistencyError(msg, 38, [])
-        return dico["variableobj"]
-
-    def _get_dyn_path(
-        self,
-        subpath: str,
-        name: bool,
-    ) -> str:
-        if subpath in self.full_dyn_paths_families:
-            subpath = self.full_dyn_paths_families[subpath]
-            path = f"{subpath}.{name}{{suffix}}"
-        else:
-            path = f"{subpath}.{name}"
-        return path
-
-    def set_provider(
-        self,
-        variableobj,
-        name,
-        family,
-    ):
-        if not hasattr(variableobj, "provider"):
-            return
-        p_name = "provider:" + variableobj.provider
-        if "." in name:
-            msg = f'provider "{p_name}" not allowed in extra'
-            raise DictConsistencyError(msg, 82, variableobj.xmlfiles)
-        if p_name in self.providers[variableobj.path_prefix]:
-            msg = f'provider "{p_name}" declare multiple time'
-            raise DictConsistencyError(msg, 79, variableobj.xmlfiles)
-        self.providers[variableobj.path_prefix][p_name] = {
-            "path": self._get_dyn_path(
-                family,
-                name,
-            ),
-            "option": variableobj,
-        }
-
-    def get_provider(
-        self,
-        name: str,
-        path_prefix: str = None,
-    ) -> "self.objectspace.variable":
-        return self.providers[path_prefix][name]["option"]
-
-    def get_providers_path(self, path_prefix=None):
-        if path_prefix:
-            return {
-                name: option["path"].split(".", 1)[-1]
-                for name, option in self.providers[path_prefix].items()
-            }
-        return {
-            name: option["path"] for name, option in self.providers[path_prefix].items()
-        }
-
-    def set_supplier(
-        self,
-        variableobj,
-        name,
-        family,
-    ):
-        if not hasattr(variableobj, "supplier"):
-            return
-        s_name = "supplier:" + variableobj.supplier
-        if "." in name:
-            msg = f'supplier "{s_name}" not allowed in extra'
-            raise DictConsistencyError(msg, 82, variableobj.xmlfiles)
-        if s_name in self.suppliers[variableobj.path_prefix]:
-            msg = f'supplier "{s_name}" declare multiple time'
-            raise DictConsistencyError(msg, 79, variableobj.xmlfiles)
-        self.suppliers[variableobj.path_prefix][s_name] = {
-            "path": self._get_dyn_path(family, name),
-            "option": variableobj,
-        }
-
-    def get_supplier(
-        self,
-        name: str,
-        path_prefix: str = None,
-    ) -> "self.objectspace.variable":
-        return self.suppliers[path_prefix][name]["option"]
-
-    def get_suppliers_path(self, path_prefix=None):
-        if path_prefix:
-            return {
-                name: option["path"].split(".", 1)[-1]
-                for name, option in self.suppliers[path_prefix].items()
-            }
-        return {
-            name: option["path"] for name, option in self.suppliers[path_prefix].items()
-        }
-
-    # Variable
-    def add_variable(
-        self,  # pylint: disable=R0913
-        namespace: str,
-        subpath: str,
-        variableobj: "self.objectspace.variable",
-        is_dynamic: bool = False,
-        is_leader: bool = False,
-        force_path_prefix: str = None,
-    ) -> str:  # pylint: disable=C0111
-        """Add a new variable (with path)"""
-        if force_path_prefix is None:
-            force_path_prefix = self._path_prefix
-        path = subpath + "." + variableobj.name
-        if namespace == self.variable_namespace:
-            self.full_paths_variables[force_path_prefix][variableobj.name] = path
-        if path in self.families:
-            msg = _(f'A family and a variable has the same path "{path}"')
-            raise DictConsistencyError(msg, 57, variableobj.xmlfiles)
-        if is_leader:
-            leader = subpath
-        else:
-            leader = None
-        self.variables[path] = dict(
-            name=path,
-            family=subpath,
-            leader=leader,
-            is_dynamic=is_dynamic,
-            variableobj=variableobj,
-        )
-        variableobj.path = path
-        variableobj.path_prefix = force_path_prefix
-        self.set_name(variableobj, "option_")
-
-    def set_name(
-        self,
-        variableobj,
-        option_prefix,
-    ):
-        self.index += 1
-        variableobj.reflector_name = f"{option_prefix}{self.index}{self.suffix}"
-
-    def get_variable(
-        self,
-        name: str,
-        namespace: str,
-        xmlfiles: List[str] = [],
-        allow_variable_namespace: bool = False,
-        force_path_prefix: str = None,
-        add_path_prefix: bool = False,
-    ) -> "Variable":  # pylint: disable=C0111
-        """Get variable object from a path"""
-        if force_path_prefix is None:
-            force_path_prefix = self._path_prefix
-        try:
-            variable, suffix = self._get_variable(
-                name,
-                namespace,
-                with_suffix=True,
-                xmlfiles=xmlfiles,
-                path_prefix=force_path_prefix,
-                add_path_prefix=add_path_prefix,
-            )
-        except DictConsistencyError as err:
-            if (
-                not allow_variable_namespace
-                or err.errno != 42
-                or namespace == self.variable_namespace
-            ):
-                raise err from err
-            variable, suffix = self._get_variable(
-                name,
-                self.variable_namespace,
-                with_suffix=True,
-                xmlfiles=xmlfiles,
-                path_prefix=force_path_prefix,
-            )
-        if suffix:
-            raise DictConsistencyError(_(f"{name} is a dynamic variable"), 36, [])
-        return variable["variableobj"]
-
-    def get_variable_family_path(
-        self,
-        name: str,
-        namespace: str,
-        xmlfiles: List[str] = False,
-        force_path_prefix: str = None,
-    ) -> str:  # pylint: disable=C0111
-        """Get the full path of a family"""
-        if force_path_prefix is None:
-            force_path_prefix = self._path_prefix
-        return self._get_variable(
-            name,
-            namespace,
-            xmlfiles=xmlfiles,
-            path_prefix=force_path_prefix,
-        )["family"]
-
-    def get_variable_with_suffix(
-        self,
-        name: str,
-        current_namespace: str,
-        xmlfiles: List[str],
-        path_prefix: str,
-    ) -> str:  # pylint: disable=C0111
-        """get full path of a variable"""
-        try:
-            dico, suffix = self._get_variable(
-                name,
-                current_namespace,
-                with_suffix=True,
-                xmlfiles=xmlfiles,
-                path_prefix=path_prefix,
-                add_path_prefix=True,
-            )
-        except DictConsistencyError as err:
-            if err.errno != 42 or current_namespace == self.variable_namespace:
-                raise err from err
-            dico, suffix = self._get_variable(
-                name,
-                self.variable_namespace,
-                with_suffix=True,
-                xmlfiles=xmlfiles,
-                path_prefix=path_prefix,
-                add_path_prefix=True,
-            )
-        namespace = dico["variableobj"].namespace
-        if (
-            namespace not in [self.variable_namespace, "services"]
-            and current_namespace != "services"
-            and current_namespace != namespace
-        ):
-            msg = _(
-                f'A variable located in the "{namespace}" namespace shall not be used '
-                f'in the "{current_namespace}" namespace'
-            )
-            raise DictConsistencyError(msg, 41, xmlfiles)
-        return dico["variableobj"], suffix
-
-    def path_is_defined(
-        self,
-        path: str,
-        namespace: str,
-        force_path_prefix: str = None,
-    ) -> str:  # pylint: disable=C0111
-        """The path is a valid path"""
-        if namespace == self.variable_namespace:
-            if force_path_prefix is None:
-                force_path_prefix = self._path_prefix
-            return path in self.full_paths_variables[force_path_prefix]
-        return path in self.variables
-
-    def get_path(
-        self,
-        path: str,
-        namespace: str,
-    ) -> str:
-        if namespace == self.variable_namespace:
-            if path not in self.full_paths_variables[self._path_prefix]:
-                return None
-            path = self.full_paths_variables[self._path_prefix][path]
-        else:
-            path = f"{self._path_prefix}.{path}"
-        return path
-
-    def is_dynamic(self, variableobj) -> bool:
-        """This variable is in dynamic family"""
-        return self._get_variable(
-            variableobj.path,
-            variableobj.namespace,
-            path_prefix=variableobj.path_prefix,
-        )["is_dynamic"]
-
-    def is_leader(self, variableobj):  # pylint: disable=C0111
-        """Is the variable is a leader"""
-        path = variableobj.path
-        variable = self._get_variable(
-            path,
-            variableobj.namespace,
-            path_prefix=variableobj.path_prefix,
-        )
-        if not variable["leader"]:
-            return False
-        leadership = self.get_family(
-            variable["leader"],
-            variableobj.namespace,
-            path_prefix=variableobj.path_prefix,
-        )
-        return next(iter(leadership.variable.values())).path == path
-
-    def is_follower(self, variableobj) -> bool:
-        """Is the variable is a follower"""
-        variable = self._get_variable(
-            variableobj.path,
-            variableobj.namespace,
-            path_prefix=variableobj.path_prefix,
-        )
-        if not variable["leader"]:
-            return False
-        leadership = self.get_family(
-            variable["leader"],
-            variableobj.namespace,
-            path_prefix=variableobj.path_prefix,
-        )
-        return next(iter(leadership.variable.values())).path != variableobj.path
-
-    def get_leader(self, variableobj) -> str:
-        variable = self._get_variable(
-            variableobj.path,
-            variableobj.namespace,
-            path_prefix=variableobj.path_prefix,
-        )
-        if not variable["leader"]:
-            raise Exception(f"cannot find leader for {variableobj.path}")
-        leadership = self.get_family(
-            variable["leader"],
-            variableobj.namespace,
-            path_prefix=variableobj.path_prefix,
-        )
-        return next(iter(leadership.variable.values()))
-
-    def _get_variable(
-        self,
-        path: str,
-        namespace: str,
-        with_suffix: bool = False,
-        xmlfiles: List[str] = [],
-        path_prefix: str = None,
-        add_path_prefix: bool = False,
-    ) -> str:
-        if namespace == self.variable_namespace:
-            if path in self.full_paths_variables[path_prefix]:
-                path = self.full_paths_variables[path_prefix][path]
-            else:
-                if with_suffix:
-                    for var_name, full_path in self.full_paths_variables[
-                        path_prefix
-                    ].items():
-                        if not path.startswith(var_name):
-                            continue
-                        variable = self._get_variable(
-                            full_path, namespace, path_prefix=path_prefix
-                        )
-                        if not variable["is_dynamic"]:
-                            continue
-                        return variable, path[len(var_name) :]
-                if path_prefix and add_path_prefix:
-                    path = f"{path_prefix}.{path}"
-        elif path_prefix and add_path_prefix:
-            path = f"{path_prefix}.{path}"
-        # FIXME with_suffix and variable in extra?
-        if path not in self.variables:
-            raise DictConsistencyError(_(f'unknown option "{path}"'), 42, xmlfiles)
-        if with_suffix:
-            return self.variables[path], None
-        return self.variables[path]
-
-    def set_valid_enums(
-        self,
-        path,
-        values,
-        path_prefix,
-    ):
-        self.valid_enums[path_prefix][path] = values
-
-    def has_valid_enums(
-        self,
-        path: str,
-        path_prefix: str,
-    ) -> bool:
-        return path in self.valid_enums[path_prefix]
-
-    def get_valid_enums(
-        self,
-        path: str,
-        path_prefix: str,
-    ):
-        return self.valid_enums[path_prefix][path]

src/rougail/reflector.py

@@ -1,220 +0,0 @@
-"""load XML and YAML file from directory
-
-Created by:
-EOLE (http://eole.orion.education.fr)
-Copyright (C) 2005-2018
-
-Forked by:
-Cadoles (http://www.cadoles.com)
-Copyright (C) 2019-2021
-
-Silique (https://www.silique.fr)
-Copyright (C) 2022-2024
-
-distribued with GPL-2 or later license
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; either version 2 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
-"""
-from typing import List
-from os.path import join, isfile
-from os import listdir
-
-from lxml.etree import DTD, parse, XMLSyntaxError  # pylint: disable=E0611
-from pykwalify.compat import yml
-from pykwalify.core import Core
-from pykwalify.errors import SchemaError
-
-
-from .i18n import _
-from .error import DictConsistencyError
-
-
-FORCE_SUBYAML = ["override"]
-SCHEMA_DATA = {}
-
-
-class Reflector:
-    """Helper class for loading the Creole XML file,
-    parsing it, validating against the Creole DTD
-    """
-
-    def __init__(
-        self,
-        rougailconfig: "RougailConfig",
-    ) -> None:
-        """Loads the Creole DTD
-
-        :raises IOError: if the DTD is not found
-
-        :param dtdfilename: the full filename of the Creole DTD
-        """
-        dtdfilename = rougailconfig["dtdfilename"]
-        yamlschema_filename = rougailconfig["yamlschema_filename"]
-        if not isfile(dtdfilename):
-            raise IOError(_(f"no such DTD file: {dtdfilename}"))
-        with open(dtdfilename, "r") as dtdfd:
-            self.dtd = DTD(dtdfd)
-        if not isfile(yamlschema_filename):
-            raise IOError(_(f"no such YAML Schema file: {yamlschema_filename}"))
-        self.yamlschema_filename = yamlschema_filename
-        self.schema_data = None
-
-    def load_dictionaries_from_folders(
-        self,
-        folders: List[str],
-        just_doc: bool,
-    ):
-        """Loads all the dictionary files located in the folders' list
-
-        :param folders: list of full folder's name
-        """
-        filenames = {}
-        for folder in folders:
-            for filename in listdir(folder):
-                if filename.endswith(".xml"):
-                    ext = "xml"
-                    full_filename = join(folder, filename)
-                elif filename.endswith(".yml"):
-                    ext = "yml"
-                    full_filename = join(folder, filename)
-                else:
-                    continue
-                if filename in filenames:
-                    raise DictConsistencyError(
-                        _(f"duplicate dictionary file name {filename}"),
-                        78,
-                        [filenames[filename][1], full_filename],
-                    )
-                filenames[filename] = (ext, full_filename)
-        if not filenames and not just_doc:
-            raise DictConsistencyError(_("there is no dictionary file"), 77, folders)
-        file_names = list(filenames.keys())
-        file_names.sort()
-        for filename in file_names:
-            ext, filename = filenames[filename]
-            if ext == "xml":
-                yield self.load_xml_file(filename)
-            else:
-                yield self.load_yml_file(filename)
-
-    def load_xml_file(
-        self,
-        filename: str,
-    ):
-        try:
-            document = parse(filename)
-        except XMLSyntaxError as err:
-            raise DictConsistencyError(
-                _(f"not a XML file: {err}"), 52, [filename]
-            ) from err
-        if not self.dtd.validate(document):
-            dtd_error = self.dtd.error_log.filter_from_errors()[0]
-            msg = _(f"not a valid XML file: {dtd_error}")
-            raise DictConsistencyError(msg, 43, [filename])
-        return filename, document.getroot()
-
-    def load_yml_file(
-        self,
-        filename: str,
-    ):
-        global SCHEMA_DATA
-        if self.yamlschema_filename not in SCHEMA_DATA:
-            with open(self.yamlschema_filename, "r") as fh:
-                SCHEMA_DATA[self.yamlschema_filename] = yml.load(fh)
-        try:
-            document = Core(
-                source_file=filename,
-                schema_data=SCHEMA_DATA[self.yamlschema_filename],
-            )
-        except XMLSyntaxError as err:
-            raise DictConsistencyError(
-                _(f"not a XML file: {err}"), 52, [filename]
-            ) from err
-        try:
-            return filename, YParser(document.validate(raise_exception=True))
-        except SchemaError as yaml_error:
-            msg = _(f"not a valid YAML file: {yaml_error}")
-            raise DictConsistencyError(msg, 43, [filename])
-
-
-class SubYAML:
-    def __init__(self, key, value):
-        if value is None:
-            value = {}
-        self.tag = key
-        self.dico = value
-        if "text" in value:
-            self.text = value["text"]
-        else:
-            self.text = None
-        if isinstance(value, list):
-            self.attrib = {}
-        else:
-            self.attrib = {
-                k: v
-                for k, v in value.items()
-                if not isinstance(v, list) and k not in FORCE_SUBYAML
-            }
-
-    def __str__(self):
-        return f"<SubYAML {self.tag} at {id(self)}>"
-
-    def __iter__(self):
-        if isinstance(self.dico, list):
-            lists = []
-            for dico in self.dico:
-                for key, value in dico.items():
-                    if not isinstance(value, list):
-                        value = [value]
-                    lists.append((key, value))
-        else:
-            lists = []
-            for key, values in self.dico.items():
-                if key == "variables":
-                    for v in values:
-                        if "variable" in v:
-                            lists.append(("variable", v["variable"]))
-                        if "family" in v:
-                            lists.append(("family", v["family"]))
-                else:
-                    lists.append((key, values))
-        for key, values in lists:
-            if key not in FORCE_SUBYAML and not isinstance(values, list):
-                continue
-            if values is None:
-                values = [None]
-            for value in values:
-                yield SubYAML(key, value)
-
-    def __len__(self):
-        length = 0
-        for _ in self.__iter__():
-            length += 1
-        return length
-
-
-class YParser:
-    def __init__(self, dico):
-        self.dico = dico
-
-    def __iter__(self):
-        for key, values in self.dico.items():
-            if not isinstance(values, list):
-                continue
-            if key == "variables":
-                yield SubYAML(key, values)
-            else:
-                for val in values:
-                    yield SubYAML(key, val)

src/rougail/structural_commandline/annotator.py

@@ -0,0 +1,89 @@
+"""Annotate to add specify attribute for tiramisu-cmdline
+
+Silique (https://www.silique.fr)
+Copyright (C) 2024
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+from rougail.annotator.variable import Walk
+from rougail.utils import _
+from rougail.error import DictConsistencyError
+
+class Annotator(Walk):
+    """Annotate value"""
+    level = 80
+
+    def __init__(self, objectspace, *args) -> None:
+        if not objectspace.paths:
+            return
+        self.alternative_names = {}
+        self.objectspace = objectspace
+        not_for_commandlines = []
+        for family in self.get_families():
+            if family.commandline:
+                continue
+            self.not_for_commandline(family)
+            not_for_commandlines.append(family.path + '.')
+        for variable in self.get_variables():
+            if variable.type == 'symlink':
+                continue
+            variable_path = variable.path
+            for family_path in not_for_commandlines:
+                if variable_path.startswith(family_path):
+                    break
+            else:
+                if not variable.commandline:
+                    self.not_for_commandline(variable)
+                else:
+                    self.manage_alternative_name(variable)
+                    self.manage_negative_description(variable)
+
+    def not_for_commandline(self, variable) -> None:
+        self.objectspace.properties.add(variable.path, 'not_for_commandline', True)
+
+    def manage_alternative_name(self, variable) -> None:
+        if not variable.alternative_name:
+            return
+        alternative_name = variable.alternative_name
+        variable_path = variable.path
+        all_letters = ''
+        for letter in alternative_name:
+            all_letters += letter
+            if all_letters == 'h':
+                msg = _(f'alternative_name "{alternative_name}" conflict with "--help"')
+                raise DictConsistencyError(msg, 202, variable.xmlfiles)
+            if all_letters in self.alternative_names:
+                msg = _(f'conflict alternative_name "{alternative_name}": "{variable_path}" and "{self.alternative_names[all_letters]}"')
+                raise DictConsistencyError(msg, 202, variable.xmlfiles)
+
+        self.alternative_names[alternative_name] = variable_path
+        if '.' not in variable_path:
+            path = alternative_name
+        else:
+            path = variable_path.rsplit('.', 1)[0] + '.' + alternative_name
+        self.objectspace.add_variable(alternative_name, {'type': 'symlink', 'path': path, 'opt': variable}, variable.xmlfiles, False, False, variable.version)
+
+    def manage_negative_description(self, variable) -> None:
+        if not variable.negative_description:
+            if variable.type == 'boolean' and not self.objectspace.add_extra_options:
+                raise DictConsistencyError(_(f'negative_description is mandatory for boolean variable, but "{variable.path}" hasn\'t'), 200, variable.xmlfiles)
+            return
+        if variable.type != 'boolean':
+            raise DictConsistencyError(_(f'negative_description is only available for boolean variable, but "{variable.path}" is "{variable.type}"'), 201, variable.xmlfiles)
+        self.objectspace.informations.add(
+            variable.path, "negative_description", variable.negative_description
+        )

src/rougail/structural_commandline/config.py

@@ -0,0 +1,42 @@
+"""
+Config file for Rougail-structural_commandline
+
+Silique (https://www.silique.fr)
+Copyright (C) 2024
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+def get_rougail_config(*,
+                       backward_compatibility=True,
+                       ) -> dict:
+    options = """
+structural_commandline:
+  description: Configuration rougail-structural_commandline
+  commandline: false
+  add_extra_options:
+    description: Add extra options to tiramisu-cmdline-parser
+    default: true
+"""
+    return {'name': 'exporter',
+            'process': 'structural',
+            'options': options,
+            'level': 20,
+            }
+
+
+__all__ = ('get_rougail_config')
+

src/rougail/structural_commandline/object_model.py

@@ -0,0 +1,36 @@
+"""Annotate to add specify attribute for tiramisu-cmdline
+
+Silique (https://www.silique.fr)
+Copyright (C) 2024
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+from typing import Optional
+from pydantic import BaseModel
+
+
+class Variable(BaseModel):
+    alternative_name: Optional[str]=None
+    commandline: bool=True
+    negative_description: Optional[str]=None
+
+
+class Family(BaseModel):
+    commandline: bool=True
+
+
+__all__ = ('Variable', 'Family')

src/rougail/tiramisu.py

@@ -28,12 +28,115 @@ along with this program; if not, write to the Free Software
 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 """
 try:
-    from tiramisu4 import DynOptionDescription
+    from tiramisu5 import DynOptionDescription, calc_value
 except ModuleNotFoundError:
-    from tiramisu import DynOptionDescription
+    from tiramisu import DynOptionDescription, calc_value
+from importlib.machinery import SourceFileLoader as _SourceFileLoader
+from importlib.util import spec_from_loader as _spec_from_loader, module_from_spec as _module_from_spec
+from jinja2 import StrictUndefined, DictLoader
+from jinja2.sandbox import SandboxedEnvironment
+from rougail.object_model import CONVERT_OPTION
+from rougail.error import display_xmlfiles
+from tiramisu.error import ValueWarning, ConfigError
 from .utils import normalize_family
 
 
+global func
+dict_env = {}
+ENV = SandboxedEnvironment(loader=DictLoader(dict_env), undefined=StrictUndefined)
+func = ENV.filters
+ENV.compile_templates('jinja_caches', zip=None)
+
+
+def load_functions(path):
+    global _SourceFileLoader, _spec_from_loader, _module_from_spec, func
+    loader = _SourceFileLoader('func', path)
+    spec = _spec_from_loader(loader.name, loader)
+    func_ = _module_from_spec(spec)
+    loader.exec_module(func_)
+    for function in dir(func_):
+        if function.startswith('_'):
+            continue
+        func[function] = getattr(func_, function)
+
+
+def rougail_calc_value(*args, __default_value=None, **kwargs):
+    values = calc_value(*args, **kwargs)
+    if __default_value is not None and values in [None, []]:
+        return __default_value
+    return values
+
+
+def jinja_to_function(__internal_variable, __internal_attribute, __internal_jinja, __internal_type, __internal_multi, __internal_files, __default_value=None, **kwargs):
+    global ENV, CONVERT_OPTION
+    kw = {}
+    for key, value in kwargs.items():
+        if '.' in key:
+            c_kw = kw
+            path, var = key.rsplit('.', 1)
+            for subkey in path.split('.'):
+                c_kw = c_kw.setdefault(subkey, {})
+            c_kw[var] = value
+        else:
+            if key in kw:
+                raise ConfigError(f'internal error, multi key for "{key}" in jinja_to_function')
+            kw[key] = value
+    try:
+        values = ENV.get_template(__internal_jinja).render(kw, **func).strip()
+    except Exception as err:
+        raise ConfigError(f'cannot calculating "{__internal_attribute}" attribute for variable "{__internal_variable}" in {display_xmlfiles(__internal_files)}: {err}') from err
+    convert = CONVERT_OPTION[__internal_type].get('func', str)
+    if __internal_multi:
+        values = [convert(val) for val in values.split('\n') if val != ""]
+        if not values and __default_value is not None:
+            return __default_value
+        return values
+    try:
+        values = convert(values)
+    except Exception as err:
+        raise ConfigError(f'cannot converting "{__internal_attribute}" attribute for variable "{__internal_variable}" in {display_xmlfiles(__internal_files)}: {err}') from err
+    values = values if values != '' and values != 'None' else None
+    if values is None and __default_value is not None:
+        return __default_value
+    return values
+
+
+def variable_to_property(prop, value, when, inverse):
+    if inverse:
+        is_match = value != when
+    else:
+        is_match = value == when
+    return prop if is_match else None
+
+
+def jinja_to_property(prop, when, inverse, **kwargs):
+    value = func['jinja_to_function'](**kwargs)
+    return func['variable_to_property'](prop, value is not None, when, inverse)
+
+
+def jinja_to_property_help(prop, **kwargs):
+    value = func['jinja_to_function'](**kwargs)
+    return (prop, f'\"{prop}\" ({value})')
+
+
+def valid_with_jinja(warnings_only=False, **kwargs):
+    global ValueWarning
+    value = func['jinja_to_function'](**kwargs)
+    if value:
+       if warnings_only:
+           raise ValueWarning(value)
+       else:
+           raise ValueError(value)
+
+
+func['calc_value'] = rougail_calc_value
+func['jinja_to_function'] = jinja_to_function
+func['jinja_to_property'] = jinja_to_property
+func['jinja_to_property_help'] = jinja_to_property_help
+func['variable_to_property'] = variable_to_property
+func['valid_with_jinja'] = valid_with_jinja
+
+
 class ConvertDynOptionDescription(DynOptionDescription):
     """Suffix could be an integer, we should convert it in str
     Suffix could also contain invalid character, so we should "normalize" it
@@ -58,3 +161,12 @@ class ConvertDynOptionDescription(DynOptionDescription):
         if "{{ suffix }}" in name:
             return name.replace("{{ suffix }}", path_suffix)
         return name + path_suffix
+
+    def impl_get_display_name(
+        self,
+        subconfig,
+        ) -> str:
+        display = super().impl_get_display_name(subconfig)
+        if "{{ suffix }}" in display:
+            return display.replace("{{ suffix }}", self.convert_suffix_to_path(self.get_suffixes(subconfig)[-1]))
+        return display

src/rougail/tiramisureflector.py

@@ -28,12 +28,12 @@ You should have received a copy of the GNU General Public License
 along with this program; if not, write to the Free Software
 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 """
-from typing import Optional
+from typing import Optional, Union
 from json import dumps
 from os.path import isfile, basename
 
 from .i18n import _
-from .error import DictConsistencyError
+from .error import DictConsistencyError, VariableCalculationDependencyError
 from .utils import normalize_family
 from .object_model import Calculation, CONVERT_OPTION
 
@@ -59,120 +59,47 @@ class TiramisuReflector:
         objectspace,
         funcs_paths,
     ):
-        self.rougailconfig = objectspace.rougailconfig
-        self.jinja_added = False
+        self.informations_idx = -1
         self.reflector_objects = {}
         self.text = {
             "header": [],
             "option": [],
         }
-        if self.rougailconfig["export_with_import"]:
-            if self.rougailconfig["internal_functions"]:
-                for func in self.rougailconfig["internal_functions"]:
+        self.objectspace = objectspace
+        if self.objectspace.export_with_import:
+            if self.objectspace.internal_functions:
+                for func in self.objectspace.internal_functions:
                     self.text["header"].append(f"func[func] = func")
             self.text["header"].extend(
                 [
                     "from tiramisu import *",
                     "from tiramisu.setting import ALLOWED_LEADER_PROPERTIES",
+                    "from re import compile as re_compile",
+                ]
+            )
+        if self.objectspace.export_with_import:
+            self.text["header"].extend(
+                [
+                    "from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription"
                 ]
             )
-            for mode in self.rougailconfig["modes_level"]:
-                self.text["header"].append(f'ALLOWED_LEADER_PROPERTIES.add("{mode}")')
         if funcs_paths:
-            if self.rougailconfig["export_with_import"]:
-                self.text["header"].extend(
-                    [
-                        "from importlib.machinery import SourceFileLoader as _SourceFileLoader",
-                        "from importlib.util import spec_from_loader as _spec_from_loader, module_from_spec as _module_from_spec",
-                        "global func",
-                        "func = {'calc_value': calc_value}",
-                        "",
-                        "def _load_functions(path):",
-                        "    global _SourceFileLoader, _spec_from_loader, _module_from_spec, func",
-                        "    loader = _SourceFileLoader('func', path)",
-                        "    spec = _spec_from_loader(loader.name, loader)",
-                        "    func_ = _module_from_spec(spec)",
-                        "    loader.exec_module(func_)",
-                        "    for function in dir(func_):",
-                        "        if function.startswith('_'):",
-                        "            continue",
-                        "        func[function] = getattr(func_, function)",
-                    ]
-                )
             for funcs_path in sorted(funcs_paths, key=sorted_func_name):
                 if not isfile(funcs_path):
                     continue
-                self.text["header"].append(f"_load_functions('{funcs_path}')")
-        self.objectspace = objectspace
+                self.text["header"].append(f"load_functions('{funcs_path}')")
+        if self.objectspace.export_with_import:
+            for mode in self.objectspace.modes_level:
+                self.text["header"].append(f'ALLOWED_LEADER_PROPERTIES.add("{mode}")')
         self.make_tiramisu_objects()
-        if self.rougailconfig["export_with_import"] and (
-            self.rougailconfig["force_convert_dyn_option_description"]
-            or self.objectspace.has_dyn_option is True
-        ):
-            self.text["header"].append(
-                "from rougail.tiramisu import ConvertDynOptionDescription"
-            )
         for key, value in self.objectspace.jinja.items():
             self.add_jinja_to_function(key, value)
 
-    def add_jinja_support(self):
-        if not self.jinja_added:
-            self.text["header"].extend(
-                [
-                    "from jinja2 import StrictUndefined, DictLoader",
-                    "from jinja2.sandbox import SandboxedEnvironment",
-                    "from rougail import CONVERT_OPTION",
-                    "from tiramisu.error import ValueWarning",
-                    "def jinja_to_function(__internal_jinja, __internal_type, __internal_multi, **kwargs):",
-                    "    global ENV, CONVERT_OPTION",
-                    "    kw = {}",
-                    "    for key, value in kwargs.items():",
-                    "        if '.' in key:",
-                    "            c_kw = kw",
-                    "            path, var = key.rsplit('.', 1)",
-                    "            for subkey in path.split('.'):",
-                    "                c_kw = c_kw.setdefault(subkey, {})",
-                    "            c_kw[var] = value",
-                    "        else:",
-                    "            kw[key] = value",
-                    "    values = ENV.get_template(__internal_jinja).render(kw, **func).strip()",
-                    "    convert = CONVERT_OPTION[__internal_type].get('func', str)",
-                    "    if __internal_multi:",
-                    "        return [convert(val) for val in values.split()]",
-                    "    values = convert(values)",
-                    "    return values if values != '' and values != 'None' else None",
-                    "def variable_to_property(prop, value):",
-                    "    return prop if value else None",
-                    "def jinja_to_property(prop, **kwargs):",
-                    "    value = func['jinja_to_function'](**kwargs)",
-                    "    return func['variable_to_property'](prop, value is not None)",
-                    "def jinja_to_property_help(prop, **kwargs):",
-                    "    value = func['jinja_to_function'](**kwargs)",
-                    "    return (prop, f'\"{prop}\" ({value})')",
-                    "def valid_with_jinja(warnings_only=False, **kwargs):",
-                    "    global ValueWarning",
-                    "    value = func['jinja_to_function'](**kwargs)",
-                    "    if value:",
-                    "       if warnings_only:",
-                    "           raise ValueWarning(value)",
-                    "       else:",
-                    "           raise ValueError(value)",
-                    "func['jinja_to_function'] = jinja_to_function",
-                    "func['jinja_to_property'] = jinja_to_property",
-                    "func['jinja_to_property_help'] = jinja_to_property_help",
-                    "func['variable_to_property'] = variable_to_property",
-                    "func['valid_with_jinja'] = valid_with_jinja",
-                    "dict_env = {}",
-                ]
-            )
-            self.jinja_added = True
-
     def add_jinja_to_function(
         self,
         variable_name: str,
         jinja: str,
     ) -> None:
-        self.add_jinja_support()
         jinja_text = dumps(jinja, ensure_ascii=False)
         self.text["header"].append(f"dict_env['{variable_name}'] = {jinja_text}")
 
@@ -181,54 +108,51 @@ class TiramisuReflector:
         baseelt = BaseElt()
         self.objectspace.reflector_names[
             baseelt.path
-        ] = f'option_0{self.rougailconfig["suffix"]}'
+        ] = f"option_0{self.objectspace.suffix}"
         basefamily = Family(
             baseelt,
             self,
         )
-        # FIXMEif not self.objectspace.paths.has_path_prefix():
-        if 1:
-            #            for elt in self.reorder_family(self.objectspace.space):
-            for elt in self.objectspace.paths.get():
-                if elt.path in self.objectspace.families:
-                    Family(
-                        elt,
-                        self,
-                    )
-                else:
-                    Variable(
-                        elt,
-                        self,
-                    )
-        else:
-            path_prefixes = self.objectspace.paths.get_path_prefixes()
-            for path_prefix in path_prefixes:
-                space = self.objectspace.space.variables[path_prefix]
-                self.set_name(space)
-                baseprefix = Family(
-                    space,
+        for elt in self.objectspace.paths.get():
+            if elt.path in self.objectspace.families:
+                Family(
+                    elt,
                     self,
                 )
-                basefamily.add(baseprefix)
-                for elt in self.reorder_family(space):
-                    self.populate_family(
-                        baseprefix,
-                        elt,
-                    )
-                if not hasattr(baseprefix.elt, "information"):
-                    baseprefix.elt.information = self.objectspace.information(
-                        baseprefix.elt.xmlfiles
-                    )
-                for key, value in self.objectspace.paths.get_providers_path(
-                    path_prefix
-                ).items():
-                    setattr(baseprefix.elt.information, key, value)
-                for key, value in self.objectspace.paths.get_suppliers_path(
-                    path_prefix
-                ).items():
-                    setattr(baseprefix.elt.information, key, value)
-        baseelt.name = normalize_family(self.rougailconfig["base_option_name"])
-        baseelt.description = self.rougailconfig["base_option_name"]
+            else:
+                Variable(
+                    elt,
+                    self,
+                )
+        #        else:
+        #            path_prefixes = self.objectspace.paths.get_path_prefixes()
+        #            for path_prefix in path_prefixes:
+        #                space = self.objectspace.space.variables[path_prefix]
+        #                self.set_name(space)
+        #                baseprefix = Family(
+        #                    space,
+        #                    self,
+        #                )
+        #                basefamily.add(baseprefix)
+        #                for elt in self.reorder_family(space):
+        #                    self.populate_family(
+        #                        baseprefix,
+        #                        elt,
+        #                    )
+        #                if not hasattr(baseprefix.elt, "information"):
+        #                    baseprefix.elt.information = self.objectspace.information(
+        #                        baseprefix.elt.xmlfiles
+        #                    )
+        #                for key, value in self.objectspace.paths.get_providers_path(
+        #                    path_prefix
+        #                ).items():
+        #                    setattr(baseprefix.elt.information, key, value)
+        #                for key, value in self.objectspace.paths.get_suppliers_path(
+        #                    path_prefix
+        #                ).items():
+        #                    setattr(baseprefix.elt.information, key, value)
+        baseelt.name = normalize_family(self.objectspace.base_option_name)
+        baseelt.description = self.objectspace.base_option_name
         self.reflector_objects[baseelt.path].get(
             [], baseelt.description
         )  # pylint: disable=E1101
@@ -242,16 +166,12 @@ class TiramisuReflector:
             self.objectspace.set_name(elt, "optiondescription_")
         return self.objectspace.reflector_names[elt.path]
 
+    def get_information_name(self):
+        self.informations_idx += 1
+        return f"information_{self.informations_idx}"
+
     def get_text(self):
         """Get text"""
-        if self.jinja_added:
-            self.text["header"].extend(
-                [
-                    "ENV = SandboxedEnvironment(loader=DictLoader(dict_env), undefined=StrictUndefined)",
-                    "ENV.filters = func",
-                    "ENV.compile_templates('jinja_caches', zip=None)",
-                ]
-            )
         return "\n".join(self.text["header"] + self.text["option"])
 
 
@@ -269,19 +189,24 @@ class Common:
         self.tiramisu = tiramisu
         tiramisu.reflector_objects[elt.path] = self
         self.object_type = None
+        self.informations = []
 
     def get(self, calls, parent_name):
         """Get tiramisu's object"""
-        self_calls = calls.copy()
-        if self.elt.path in self_calls:
+        if self.elt.path in calls:
             msg = f'"{self.elt.path}" will make an infinite loop'
             raise DictConsistencyError(msg, 80, self.elt.xmlfiles)
+        self_calls = calls.copy()
         self_calls.append(self.elt.path)
         self.calls = self_calls
         if self.option_name is None:
             self.option_name = self.objectspace.reflector_names[self.elt.path]
             self.populate_attrib()
-            self.populate_informations()
+            if self.informations:
+                for information in self.informations:
+                    self.tiramisu.text["option"].append(
+                        f"{information}.set_option({self.option_name})"
+                    )
         return self.option_name
 
     def populate_attrib(self):
@@ -294,6 +219,7 @@ class Common:
             keys["properties"] = self.properties_to_string(
                 self.objectspace.properties[self.elt.path]
             )
+        self.populate_informations(keys)
         attrib = ", ".join([f"{key}={value}" for key, value in keys.items()])
         self.tiramisu.text["option"].append(
             f"{self.option_name} = {self.object_type}({attrib})"
@@ -322,12 +248,11 @@ class Common:
         for property_, value in values.items():
             if value is True:
                 properties.append(self.convert_str(property_))
+            elif isinstance(value, list):
+                for val in value:
+                    calc_properties.append(self.calculation_value(val))
             else:
-                if isinstance(value, list):
-                    for val in value:
-                        calc_properties.append(self.calculation_value(val))
-                else:
-                    calc_properties.append(self.calculation_value(value))
+                calc_properties.append(self.calculation_value(value))
         return "frozenset({" + ", ".join(sorted(properties) + calc_properties) + "})"
 
     def calc_properties(
@@ -350,17 +275,12 @@ class Common:
             f"kwargs={{{kwargs}}}), func['calc_value_property_help'])"
         )
 
-    def populate_informations(self):
+    def populate_informations(self, keys):
         """Populate Tiramisu's informations"""
         informations = self.objectspace.informations.get(self.elt.path)
         if not informations:
             return
-        for key, value in informations.items():
-            if isinstance(value, str):
-                value = self.convert_str(value)
-            self.tiramisu.text["option"].append(
-                f"{self.option_name}.impl_set_information('{key}', {value})"
-            )
+        keys["informations"] = informations
 
     def populate_param(
         self,
@@ -373,7 +293,10 @@ class Common:
             else:
                 value = param
             return f"ParamValue({value})"
+        if param["type"] == "value":
+            return f"ParamValue({param['value']})"
         if param["type"] == "information":
+            # default? really?
             if self.elt.multi:
                 default = []
             else:
@@ -381,9 +304,27 @@ class Common:
             if "variable" in param:
                 if param["variable"].path == self.elt.path:
                     return f'ParamSelfInformation("{param["information"]}", {default})'
-                return f'ParamInformation("{param["information"]}", {default}, option={self.tiramisu.reflector_objects[param["variable"].path].get(self.calls, self.elt.path)})'
+                information_variable_path = param["variable"].path
+                information_variable = self.tiramisu.reflector_objects[
+                    information_variable_path
+                ]
+                if information_variable_path not in self.calls:
+                    option_name = information_variable.get(self.calls, self.elt.path)
+                    return f'ParamInformation("{param["information"]}", {default}, option={option_name})'
+                else:
+                    information = (
+                        f'ParamInformation("{param["information"]}", {default})'
+                    )
+                    information_name = self.tiramisu.get_information_name()
+                    self.tiramisu.text["option"].append(
+                        f"{information_name} = {information}"
+                    )
+                    information_variable.informations.append(information_name)
+                    return information_name
             return f'ParamInformation("{param["information"]}", {default})'
         if param["type"] == "suffix":
+            if "suffix" in param and param["suffix"] != None:
+                return f"ParamSuffix(suffix_index={param['suffix']})"
             return "ParamSuffix()"
         if param["type"] == "index":
             return "ParamIndex()"
@@ -393,6 +334,7 @@ class Common:
                 param.get("propertyerror", True),
                 param.get("suffix"),
                 param.get("dynamic"),
+                param.get('whole', False),
             )
         if param["type"] == "any":
             if isinstance(param["value"], str):
@@ -404,24 +346,25 @@ class Common:
 
     def build_option_param(
         self,
-        param,
+        variable,
         propertyerror,
         suffix: Optional[str],
         dynamic,
+        whole: bool,
     ) -> str:
         """build variable parameters"""
-        if param.path == self.elt.path:
-            return "ParamSelfOption(whole=False)"
-        option_name = self.tiramisu.reflector_objects[param.path].get(
+        if variable.path == self.elt.path:
+            return f"ParamSelfOption(whole={whole})"
+        if whole:
+            msg = f'variable param "{variable.path}" has whole attribute but it\'s not allowed for external variable'
+            raise DictConsistencyError(msg, 34, self.elt.xmlfiles)
+        option_name = self.tiramisu.reflector_objects[variable.path].get(
             self.calls, self.elt.path
         )
         params = [f"{option_name}"]
         if suffix is not None:
             param_type = "ParamDynOption"
-            family = self.tiramisu.reflector_objects[dynamic.path].get(
-                self.calls, self.elt.path
-            )
-            params.extend([f"'{suffix}'", f"{family}"])
+            params.append(self.convert_str(suffix))
         else:
             param_type = "ParamOption"
         if not propertyerror:
@@ -433,7 +376,6 @@ class Common:
         function,
     ) -> str:
         """Generate calculated value"""
-        self.tiramisu.add_jinja_support()
         child = function.to_function(self.objectspace)
         new_args = []
         kwargs = []
@@ -460,6 +402,41 @@ class Common:
         ret = ret + ")"
         return ret
 
+    def populate_calculation(
+        self,
+        datas: Union[Calculation, str, list],
+        return_a_tuple: bool = False,
+    ) -> str:
+        if isinstance(datas, str):
+            return self.convert_str(datas)
+        if isinstance(datas, Calculation):
+            return self.calculation_value(datas)
+        if not isinstance(datas, list):
+            return datas
+        params = []
+        for idx, data in enumerate(datas):
+            if isinstance(data, Calculation):
+                try:
+                    params.append(self.calculation_value(data))
+                except VariableCalculationDependencyError:
+                    pass
+            elif isinstance(data, str):
+                params.append(self.convert_str(data))
+            else:
+                params.append(str(data))
+        if return_a_tuple:
+            ret = "("
+        else:
+            ret = "["
+        ret += ", ".join(params)
+        if return_a_tuple:
+            if len(params) <= 1:
+                ret += ","
+            ret += ")"
+        else:
+            ret += "]"
+        return ret
+
 
 class Variable(Common):
     """Manage variable"""
@@ -470,8 +447,8 @@ class Variable(Common):
         tiramisu,
     ):
         super().__init__(elt, tiramisu)
-        if elt.type in self.tiramisu.objectspace.rougailconfig['custom_types']:
-            self.object_type = self.tiramisu.objectspace.rougailconfig['custom_types'][elt.type].__name__
+        if elt.type in self.tiramisu.objectspace.custom_types:
+            self.object_type = self.tiramisu.objectspace.custom_types[elt.type].__name__
         else:
             self.object_type = CONVERT_OPTION[elt.type]["opttype"]
 
@@ -483,58 +460,39 @@ class Variable(Common):
             keys["opt"] = self.tiramisu.reflector_objects[self.elt.opt.path].get(
                 self.calls, self.elt.path
             )
+            return
         if self.elt.type == "choice":
-            choices = self.elt.choices
-            if isinstance(choices, Calculation):
-                keys["values"] = self.calculation_value(choices)
-            else:
-                new_values = []
-                for value in choices:
-                    if isinstance(value, Calculation):
-                        new_values.append(self.calculation_value(value))
-                    elif isinstance(value, str):
-                        new_values.append(self.convert_str(value))
-                    else:
-                        new_values.append(str(value))
-                keys["values"] = "(" + ", ".join(new_values)
-                if len(new_values) <= 1:
-                    keys["values"] += ","
-                keys["values"] += ")"
+            keys["values"] = self.populate_calculation(
+                self.elt.choices, return_a_tuple=True
+            )
+        if self.elt.type == 'regexp':
+            self.object_type = 'Regexp_' + self.option_name
+            self.tiramisu.text['header'].append(f'''class {self.object_type}(RegexpOption):
+    __slots__ = tuple()
+    _type = 'value'
+{self.object_type}._regexp = re_compile(r"{self.elt.regexp}")
+''')
         if self.elt.path in self.objectspace.multis:
             keys["multi"] = self.objectspace.multis[self.elt.path]
         if hasattr(self.elt, "default") and self.elt.default is not None:
-            value = self.elt.default
-            if isinstance(value, str):
-                value = self.convert_str(value)
-            elif isinstance(value, Calculation):
-                value = self.calculation_value(value)
-            elif isinstance(value, list):
-                value = value.copy()
-                for idx, val in enumerate(value):
-                    if isinstance(val, Calculation):
-                        value[idx] = self.calculation_value(val)
-                    else:
-                        value[idx] = self.convert_str(val)
-                value = "[" + ", ".join(value) + "]"
-            keys["default"] = value
+            try:
+                keys["default"] = self.populate_calculation(self.elt.default)
+            except VariableCalculationDependencyError:
+                pass
         if self.elt.path in self.objectspace.default_multi:
-            value = self.objectspace.default_multi[self.elt.path]
+            try:
+                keys["default_multi"] = self.populate_calculation(
+                    self.objectspace.default_multi[self.elt.path]
+                )
+            except VariableCalculationDependencyError:
+                pass
+        if self.elt.validators:
+            keys["validators"] = self.populate_calculation(self.elt.validators)
+        for key, value in (
+            CONVERT_OPTION.get(self.elt.type, {}).get("initkwargs", {}).items()
+        ):
             if isinstance(value, str):
                 value = self.convert_str(value)
-            elif isinstance(value, Calculation):
-                value = self.calculation_value(value)
-            keys["default_multi"] = value
-        if self.elt.validators:
-            validators = []
-            for val in self.elt.validators:
-                if isinstance(val, Calculation):
-                    validators.append(self.calculation_value(val))
-                else:
-                    validators.append(val)
-            keys["validators"] = "[" + ", ".join(validators) + "]"
-        for key, value in CONVERT_OPTION.get(self.elt.type, {}).get("initkwargs", {}).items():
-            if isinstance(value, str):
-                value = f"'{value}'"
             keys[key] = value
         if self.elt.params:
             for param in self.elt.params:
@@ -571,12 +529,7 @@ class Family(Common):
         keys: list,
     ) -> None:
         if self.elt.type == "dynamic":
-            dyn = self.tiramisu.reflector_objects[self.elt.variable.path].get(
-                self.calls, self.elt.path
-            )
-            keys[
-                "suffixes"
-            ] = f"Calculation(func['calc_value'], Params((ParamOption({dyn}, notraisepropertyerror=True))))"
+            keys["suffixes"] = self.populate_calculation(self.elt.dynamic)
         children = []
         for path in self.objectspace.parents[self.elt.path]:
             children.append(self.objectspace.paths[path])

src/rougail/update/__init__.py

@@ -0,0 +1,21 @@
+"""
+Silique (https://www.silique.fr)
+Copyright (C) 2024
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+from .update import RougailUpgrade

src/rougail/update/update.py

@@ -1,4 +1,4 @@
-"""Update Rougail XML file to new version
+"""Update Rougail structure file to new version
 
 Cadoles (http://www.cadoles.com)
 Copyright (C) 2021
@@ -23,38 +23,28 @@ along with this program; if not, write to the Free Software
 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 """
 
-from typing import List, Any, Optional, Tuple
-from os.path import join, isfile, isdir, basename
-from os import listdir, makedirs
+from os import listdir
+from os.path import basename, isdir, isfile, join
+from typing import Any, List, Optional, Tuple
+
 try:
-    from lxml.etree import parse, XMLParser, XMLSyntaxError  # pylint: disable=E0611
-    from lxml.etree import Element, SubElement, tostring
+    from lxml.etree import SubElement  # pylint: disable=E0611
+    from lxml.etree import Element, XMLParser, XMLSyntaxError, parse, tostring
 except ModuleNotFoundError as err:
     parse = None
 
 # from ast import parse as ast_parse
 from json import dumps
-from yaml import safe_load, dump, SafeDumper
+from ruamel.yaml import YAML
 from pathlib import Path
 
-from .i18n import _
-from .error import UpgradeError
+from ..config import RougailConfig
+from ..error import UpgradeError
+from ..i18n import _
+from ..object_model import CONVERT_OPTION
+from ..utils import normalize_family
 
-from .utils import normalize_family
-from .config import RougailConfig
-from .object_model import CONVERT_OPTION
-
-
-VERSIONS = ["0.10", "1.0"]
-
-FIXME_PRINT_FILENAME = True
-FIXME_PRINT_FILENAME = False
-FIXME_PRINT_FILE = True
-FIXME_PRINT_FILE = False
-FIXME_PRINT_UNKNOWN_VAR = True
-FIXME_PRINT_UNKNOWN_VAR = False
-FIXME_PRINT_REMOVE = True
-FIXME_PRINT_REMOVE = False
+VERSIONS = ["0.10", "1.0", "1.1"]
 
 
 def get_function_name(version):
@@ -65,24 +55,15 @@ def get_function_name(version):
 FUNCTION_VERSIONS = [(version, get_function_name(version)) for version in VERSIONS]
 
 
-class NoAliasDumper(SafeDumper):
-    def ignore_aliases(self, data):
-        return True
-
-
-class upgrade_010_to_100:
+class upgrade_010_to_10:
     def __init__(
         self,
         dico: dict,
         namespace: str,
         xmlsrc: str,
     ) -> None:
-        if FIXME_PRINT_FILE:
-            from pprint import pprint
-
-            pprint(dico)
         self.xmlsrc = xmlsrc
-        self.paths = {"family": {}, "variable": {}}
+        self.paths = {"family": {}, "variable": {}, 'dynamic': {}}
         self.lists = {
             "service": {},
             "ip": {},
@@ -90,26 +71,29 @@ class upgrade_010_to_100:
             "file": {},
         }
         self.flatten_paths = {"family": {}, "variable": {}}
-        self.variables = self.parse_variables(dico, namespace)
+        self.variables = self.parse_variables(dico, namespace, namespace, root=True)
         self.parse_variables_with_path()
         self.parse_services(dico)
         self.parse_constraints(dico)
-        if FIXME_PRINT_FILE:
-            print("==")
-            pprint(self.variables)
-            pprint(self.services)
 
     def parse_variables(
         self,
         family: dict,
         sub_path: str,
+        true_sub_path: str,
+        *,
+        root: bool=False,
+        is_dynamic: bool=False,
     ) -> dict:
         new_families = {}
+        if root:
+            new_families['version'] = None
         if "variables" in family:
             for subelt in family["variables"]:
                 for typ, obj in subelt.items():
                     for subobj in obj:
-                        getattr(self, f"convert_{typ}")(subobj, new_families, sub_path)
+                        local_is_dynamic = is_dynamic or subobj.get('dynamic') is not None
+                        getattr(self, f"convert_{typ}")(subobj, new_families, sub_path, true_sub_path, local_is_dynamic)
             family.pop("variables")
         return new_families
 
@@ -118,9 +102,17 @@ class upgrade_010_to_100:
         family: dict,
         new_families: dict,
         sub_path: str,
+        true_sub_path: str,
+        is_dynamic: bool,
     ) -> None:
         # name is the key, do not let it in values
         name = family.pop("name")
+        if true_sub_path:
+            true_sub_path = true_sub_path + "." + name
+        else:
+            true_sub_path = name
+        if is_dynamic:
+            name += '{{ suffix }}'
         if sub_path:
             sub_path = sub_path + "." + name
         else:
@@ -134,7 +126,7 @@ class upgrade_010_to_100:
                 if typ == "dynamic":
                     family["variable"] = self.get_variable_path(value)
         # add sub families and sub variables
-        sub_families = self.parse_variables(family, sub_path)
+        sub_families = self.parse_variables(family, sub_path, true_sub_path, is_dynamic=is_dynamic)
         for sub_name, sub_family in sub_families.copy().items():
             if sub_name not in family:
                 continue
@@ -150,22 +142,34 @@ class upgrade_010_to_100:
         variable: dict,
         new_families: dict,
         sub_path: str,
+        true_sub_path: str,
+        is_dynamic: bool,
     ) -> dict:
         name = variable.pop("name")
+        if is_dynamic:
+            if true_sub_path:
+                true_sub_path = true_sub_path + "." + name
+            else:
+                true_sub_path = name
+            self.flatten_paths["variable"][name] = true_sub_path
+            name += '{{ suffix }}'
         if sub_path:
             sub_path = sub_path + "." + name
         else:
             sub_path = name
+        if is_dynamic:
+            self.paths['dynamic'][true_sub_path] = sub_path
         new_families[name] = variable
         self.flatten_paths["variable"][name] = sub_path
         self.paths["variable"][sub_path] = variable
-        if "redefine" not in variable and "value" not in variable and "mandatory" not in variable and ("type" not in variable or variable["type"] != "boolean"):
+        if (
+            "redefine" not in variable
+            and "value" not in variable
+            and "mandatory" not in variable
+            and ("type" not in variable or variable["type"] != "boolean")
+        ):
             variable["mandatory"] = False
         if "remove_condition" in variable and variable.pop("remove_condition"):
-            if FIXME_PRINT_REMOVE:
-                print(
-                    f"variable {name} in file {self.xmlsrc} has remove_condition, all properties (hidden, disabled and mandatory) are set to False"
-                )
             for prop in ["hidden", "disabled", "mandatory"]:
                 if prop not in variable:
                     variable[prop] = False
@@ -194,24 +198,32 @@ class upgrade_010_to_100:
                         )(test)
                     )
             variable["test"] = tests
+        if variable.get('mandatory', False):
+            del variable["mandatory"]
+        CONVERT_TYPE = {'filename': 'unix_filename',
+                        'password': 'secret',
+                        }
+        if variable.get('type') in CONVERT_TYPE:
+            variable['type'] = CONVERT_TYPE[variable['type']]
 
     def parse_variables_with_path(self):
         for variable in self.paths["variable"].values():
+            multi = variable.get('multi', False)
             if "value" in variable:
                 default = variable.pop("value")
                 if default is not None:
-                    if not variable.get("multi", False) and len(default) == 1:
-                        variable["default"] = self.get_value(default[0])
+                    if not multi and len(default) == 1:
+                        variable["default"] = self.get_value(default[0], multi)
                     else:
                         variable["default"] = [
-                            self.get_value(value) for value in default
+                            self.get_value(value, multi) for value in default
                         ]
             if "choice" in variable:
                 if not variable["choice"]:
                     variable["choices"] = variable.pop("choice")
                 else:
                     variable["choices"] = [
-                        self.get_value(choice) for choice in variable.pop("choice")
+                        self.get_value(choice, multi) for choice in variable.pop("choice")
                     ]
 
     def parse_services(
@@ -327,34 +339,43 @@ class upgrade_010_to_100:
             apply_on_fallback = False
         source = self.get_variable_path(condition["source"])
         if not source:
-            source = f'__{condition["source"]}'
+            source = condition["source"]
         name = condition.pop("name")
         prop = name.split("_", 1)[0]
-        if apply_on_fallback:
-            condition_value = True
-        else:
-            condition_value = self.params_condition_to_jinja(
-                source, condition["param"], name.endswith("if_in")
-            )
+        multi = False
+        for target in condition["target"]:
+            typ = target.get("type", "variable")
+            if typ == "variable":
+                variable_path = self.get_variable_path(target["text"])
+                if variable_path is None:
+                    continue
+                variable = self.paths["variable"][variable_path]
+                if variable.get('multi', False):
+                    multi = True
+        if apply_on_fallback:
+            condition_value = True
+        else:
+            if "{{ suffix }}" in source:
+                force_param = {'__var': source}
+                source = '__var'
+            else:
+                force_param = None
+            condition_value = self.params_condition_to_jinja(
+                prop, source, condition["param"], name.endswith("if_in"), multi
+            )
+            if force_param:
+                condition_value.setdefault('params', {}).update(force_param)
         for target in condition["target"]:
             typ = target.get("type", "variable")
             if typ == "variable":
                 variable_path = self.get_variable_path(target["text"])
                 if variable_path is None:
-                    if FIXME_PRINT_UNKNOWN_VAR and not target.get("optional", False):
-                        print(
-                            f'pffff la target {target["text"]} de la condition n\'est pas trouvable'
-                        )
                     continue
                 variable = self.paths["variable"][variable_path]
                 variable[prop] = condition_value
             elif typ == "family":
                 family_path = self.get_family_path(target["text"])
                 if family_path is None:
-                    if FIXME_PRINT_UNKNOWN_VAR and not target.get("optional", False):
-                        print(
-                            f'pffff la target {target["text"]} de la condition n\'est pas trouvable'
-                        )
                     continue
                 family = self.paths["family"][family_path]
                 family[prop] = condition_value
@@ -386,10 +407,6 @@ class upgrade_010_to_100:
         for target in check["target"]:
             variable_path = self.get_variable_path(target["text"])
             if variable_path is None:
-                if FIXME_PRINT_UNKNOWN_VAR and not target.get("optional", False):
-                    print(
-                        f'pffff la target {target["text"]} dans le check n\'est pas trouvable'
-                    )
                 continue
             variable = self.paths["variable"][variable_path]
             if "validators" in variable and variable["validators"] is None:
@@ -400,7 +417,7 @@ class upgrade_010_to_100:
                 check["param"] = [
                     {"text": variable_path, "type": "variable"}
                 ] + check.get("param", [])
-                check_value = self.convert_param_function(check)
+                check_value = self.convert_param_function(check, variable.get('multi', False))
             variable.setdefault("validators", []).append(check_value)
 
     def parse_fill(
@@ -410,27 +427,30 @@ class upgrade_010_to_100:
         for target in fill.pop("target"):
             params = []
             variable_path = self.get_variable_path(target["text"])
-            if variable_path is None:
-                if FIXME_PRINT_UNKNOWN_VAR and not target.get("optional", False):
-                    print(
-                        f'pffff la target {target["text"]} dans le fill n\'est pas trouvable'
-                    )
-                continue
-            variable = self.paths["variable"][variable_path]
-            if fill.get("type") == "jinja":
-                fill_value = {
-                    "type": "jinja",
-                    "jinja": fill["name"],
-                }
+            if variable_path in self.paths["dynamic"]:
+                variable_path = self.paths["dynamic"][variable_path]
+            if variable_path in self.paths["variable"]:
+                variable = self.paths["variable"][variable_path]
+                if fill.get("type") == "jinja":
+                    fill_value = {
+                        "type": "jinja",
+                        "jinja": fill["name"],
+                    }
+                else:
+                    fill_value = self.convert_param_function(fill, variable.get('multi', False))
+                variable["default"] = fill_value
+                if variable.get('mandatory') is False:
+                    del variable['mandatory']
             else:
-                fill_value = self.convert_param_function(fill)
-            variable["default"] = fill_value
+                raise Exception(f'cannot set fill to unknown variable "{variable_path}"')
 
     def params_condition_to_jinja(
         self,
+        prop: str,
         path: str,
         params: List[dict],
         if_in: bool,
+        multi: bool,
     ) -> str:
         new_params = {}
         jinja = "{% if "
@@ -438,15 +458,15 @@ class upgrade_010_to_100:
             if idx:
                 jinja += " or "
 
-            new_param, value = self.get_jinja_param_and_value(param)
+            new_param, value = self.get_jinja_param_and_value(param, multi)
             if value:
                 jinja += path + " == " + value
             if new_param:
                 new_params |= new_param
         if if_in:
-            jinja += " %}true{% else %}false{% endif %}"
+            jinja += " %}" + prop + "{% endif %}"
         else:
-            jinja += " %}false{% else %}true{% endif %}"
+            jinja += " %}{% else %}" + prop + "{% endif %}"
         ret = {
             "type": "jinja",
             "jinja": jinja,
@@ -458,12 +478,11 @@ class upgrade_010_to_100:
     def get_value(
         self,
         param: dict,
+        multi: bool,
     ) -> Any:
-        # <!ATTLIST type (string|number|nil|space|boolean|variable|function|information|suffix|index) "string">
         typ = param.get("type", "string")
         if typ == "string":
-            value = param["text"]
-        #            value = dumps(value, ensure_ascii=False)
+            value = param.get("text")
         elif typ == "number":
             value = int(param["text"])
         elif typ == "nil":
@@ -485,7 +504,7 @@ class upgrade_010_to_100:
             if "propertyerror" in param:
                 value["propertyerror"] = param["propertyerror"]
         elif typ == "function":
-            value = self.convert_param_function(param)
+            value = self.convert_param_function(param, multi)
         elif typ == "information":
             value = {
                 "type": "information",
@@ -503,10 +522,11 @@ class upgrade_010_to_100:
     def get_jinja_param_and_value(
         self,
         param,
+        multi: bool,
     ) -> Tuple[list, Any]:
         new_param = None
         typ = param.get("type", "string")
-        value = self.get_value(param)
+        value = self.get_value(param, multi)
         if isinstance(value, dict):
             if typ == "information":
                 key = normalize_family(value["information"])
@@ -514,17 +534,25 @@ class upgrade_010_to_100:
                     attr_name = f'{value["variable"]}.{key}'
                 else:
                     attr_name = key
-                attr_name = f"__information.{attr_name}"
+                attr_name = f"__information_{attr_name}"
                 new_param = {attr_name: value}
                 value = attr_name
             elif typ in ["index", "suffix"]:
-                attr_name = f"__{typ}"
-                new_param = {attr_name: value}
+                if 'name' in value:
+                    attr_name = value['name']
+                else:
+                    attr_name = f"__{typ}"
+                new_param = {attr_name: {"type": typ}}
                 value = attr_name
             elif "propertyerror" in param or "optional" in param:
                 attr_name = value["variable"]
                 new_param = {attr_name: value}
                 value = value[typ]
+            elif "{{ suffix }}" in value[typ]:
+                path = value[typ]
+                attr_name = path.split('.')[-1][:-12]  # remove {{ suffix }}
+                new_param = {attr_name: value}
+                value = attr_name
             else:
                 value = value[typ]
             if not value:
@@ -536,23 +564,35 @@ class upgrade_010_to_100:
     def convert_param_function(
         self,
         param: dict,
+        multi: bool,
     ) -> str:
         text = param["name"]
         params = {}
+#        multi = False
         if "param" in param and param["param"]:
+            if text == 'calc_value' and len(param["param"]) == 1 and isinstance(param["param"][0], dict) and param["param"][0].get('type') == 'variable' and param["param"][0].get("text"):
+                value = param["param"][0]["text"]
+                path = self.get_variable_path(value)
+                if not path:
+                    path = value
+                ret = {"type": "variable", "variable": path}
+                if 'optional' in param["param"][0]:
+                    ret['optional'] = param["param"][0]["optional"]
+                return ret
             first, *others = param["param"]
-            new_param, first = self.get_jinja_param_and_value(first)
+            new_param, first = self.get_jinja_param_and_value(first, multi)
             text = f"{first} | {text}"
             if new_param:
                 params |= new_param
             if others:
                 values = []
                 for param in others:
-                    new_param, value = self.get_jinja_param_and_value(param)
+                    new_param, value = self.get_jinja_param_and_value(param, multi)
                     if new_param:
                         params |= new_param
-                    #                    if param.get('type') != 'variable' or value is not None:
                     if "name" in param:
+                        if param["name"] == "multi" and value == "true":
+                            multi = True
                         values.append(f'{param["name"]}={value}')
                     else:
                         values.append(value)
@@ -561,7 +601,12 @@ class upgrade_010_to_100:
                 text += ")"
         else:
             text += "()"
-        text = "{{ " + text + " }}"
+        if not multi:
+            text = "{{ " + text + " }}"
+        else:
+            text = """{% for __variable in """ + text + """ %}
+{{ __variable }}
+{% endfor %}"""
         ret = {"type": "jinja", "jinja": text}
         if params:
             ret["params"] = params
@@ -576,10 +621,10 @@ class upgrade_010_to_100:
             and path in self.flatten_paths["variable"]
         ):
             path = self.flatten_paths["variable"][path]
+        if path in self.paths["dynamic"]:
+            path = self.paths["dynamic"][path]
         if path not in self.paths["variable"]:
-            if FIXME_PRINT_UNKNOWN_VAR:
-                print("pffff impossible de trouver la variable", path)
-            return
+            return path
         return path
 
     def get_family_path(
@@ -589,8 +634,6 @@ class upgrade_010_to_100:
         if path not in self.paths["family"] and path in self.flatten_paths["family"]:
             path = self.flatten_paths["family"][path]
         if path not in self.paths["family"]:
-            if FIXME_PRINT_UNKNOWN_VAR:
-                print("pffff impossible de trouver la famille", path)
             return
         return path
 
@@ -613,42 +656,38 @@ class RougailUpgrade:
             rougailconfig = RougailConfig
         self.rougailconfig = rougailconfig
 
-    def load_dictionaries(
+    def run(
         self,
-        #                          srcfolder: str,
-        dstfolder: str,
-        services_dstfolder: Optional[str],
-        extra_dstfolder: Optional[str] = None,
-        #                          namespace: str,
-        #                          display: bool=True,
     ):
-        if extra_dstfolder is None:
-            extra_dstfolder = dstfolder
-        self._load_dictionaries(
-            self.rougailconfig["dictionaries_dir"],
-            dstfolder,
-            services_dstfolder,
-            self.rougailconfig["variable_namespace"],
-        )
-        for namespace, extra_dirs in self.rougailconfig["extra_dictionaries"].items():
-            extra_dstsubfolder = join(extra_dstfolder, namespace)
-            if not isdir(extra_dstsubfolder):
-                makedirs(extra_dstsubfolder)
-            for extra_dir in extra_dirs:
-                self._load_dictionaries(
-                    extra_dir,
-                    extra_dstsubfolder,
-                    None,
-                    namespace,
-                )
+        for dict_dir, dest_dir in zip(self.rougailconfig["main_dictionaries"], self.rougailconfig["upgrade_options.main_dictionaries"]):
+            self._load_dictionaries(
+                dict_dir,
+                dest_dir,
+                normalize_family(self.rougailconfig["main_namespace"]),
+            )
+        if self.rougailconfig['main_namespace']:
+            if self.rougailconfig["extra_dictionaries"]:
+                dst_extra_dir = self.rougailconfig["upgrade_options.extra_dictionary"]
+                for namespace, extra_dirs in self.rougailconfig["extra_dictionaries"].items():
+                    extra_dstsubfolder = Path(dst_extra_dir) / namespace
+                    if not extra_dstsubfolder.is_dir():
+                        extra_dstsubfolder.mkdir()
+                    for extra_dir in extra_dirs:
+                        self._load_dictionaries(
+                            str(extra_dir),
+                            str(extra_dstsubfolder),
+                            normalize_family(namespace),
+                    )
 
     def _load_dictionaries(
         self,
         srcfolder: str,
-        dstfolder: str,
-        services_dstfolder: Optional[str],
+        dstfolder: Optional[str],
         namespace: str,
     ) -> None:
+        if dstfolder is None:
+            dstfolder = srcfolder
+        Path(dstfolder).mkdir(parents=True, exist_ok=True)
         filenames = [
             filename
             for filename in listdir(srcfolder)
@@ -657,21 +696,11 @@ class RougailUpgrade:
         filenames.sort()
         for filename in filenames:
             xmlsrc = Path(srcfolder) / Path(filename)
-            ymlfile = filename[:-3] + "yml"
-            xmldst = Path(dstfolder) / Path(ymlfile)
-            if xmldst.is_file():
-                raise Exception(
-                    f'cannot update "{xmlsrc}" destination file "{xmldst}" already exists'
-                )
-            if services_dstfolder:
-                ymldst_services = Path(services_dstfolder) / ymlfile
-                if ymldst_services.is_file():
-                    raise Exception(
-                        f'cannot update "{xmlsrc}" destination file "{ymldst_services}" already exists'
-                    )
+
+            ymldst = Path(dstfolder) / (Path(filename).stem + '.yml')
             if filename.endswith(".xml"):
                 if parse is None:
-                    raise Exception('XML module is not installed')
+                    raise Exception("XML module is not installed")
                 try:
                     parser = XMLParser(remove_blank_text=True)
                     document = parse(xmlsrc, parser)
@@ -683,61 +712,43 @@ class RougailUpgrade:
                 )
                 ext = "xml"
             else:
-                with xmlsrc.open() as xml_fh:
-                    root = safe_load(xml_fh)
-                search_function_name = get_function_name(root["version"])
+                with xmlsrc.open() as file_fh:
+                    root = YAML(typ="safe").load(file_fh)
+                search_function_name = get_function_name(str(root["version"]))
                 ext = "yml"
             function_found = False
-            if FIXME_PRINT_FILENAME:
-                print(
-                    "========================================================================"
-                )
-                print(xmlsrc)
-                print(
-                    "========================================================================"
-                )
+            root_services = None
             for version, function_version in FUNCTION_VERSIONS:
                 if function_found and hasattr(self, function_version):
-                    #                    if display:
-                    #                        print(f'  - convert {filename} to version {version}')
                     upgrade_help = self.upgrade_help.get(function_version, {}).get(
                         filename, {}
                     )
                     if upgrade_help.get("remove") is True:
                         continue
-                    root, root_services, new_type = getattr(self, function_version)(
+                    root, root_services_, new_type = getattr(self, function_version)(
                         root, upgrade_help, namespace, xmlsrc, ext
                     )
+                    if root_services_ is not None:
+                        root_services = root_services_
                 if function_version == search_function_name:
                     function_found = True
-            if root:
-                root["version"] = version
-                xmldst.parent.mkdir(parents=True, exist_ok=True)
-                with xmldst.open("w") as ymlfh:
-                    dump(
+            if root != {'version': None}:
+                root["version"] = float(version)
+                with ymldst.open("w") as ymlfh:
+                    yaml = YAML()
+                    yaml.dump(
                         root,
                         ymlfh,
-                        allow_unicode=True,
-                        sort_keys=False,
-                        Dumper=NoAliasDumper,
                     )
-            if root_services and services_dstfolder:
-                root_services["version"] = version
-                ymldst_services.parent.mkdir(parents=True, exist_ok=True)
-                with ymldst_services.open("w") as ymlfh:
-                    dump(
-                        root_services,
-                        ymlfh,
-                        allow_unicode=True,
-                        sort_keys=False,
-                        Dumper=NoAliasDumper,
-                    )
-
-    #            if not self.dtd.validate(document):
-    #                dtd_error = self.dtd.error_log.filter_from_errors()[0]
-    #                msg = _(f'not a valid XML file: {dtd_error}')
-    #                raise DictConsistencyError(msg, 43, [xmlfile])
-    #            yield xmlfile, document.getroot()
+            # if root_services and services_dstfolder:
+            #     root_services["version"] = version
+            #     ymldst_services.parent.mkdir(parents=True, exist_ok=True)
+            #     with ymldst_services.open("w") as ymlfh:
+            #         yaml = YAML()
+            #         yaml.dump(
+            #             root_services,
+            #             ymlfh,
+            #         )
 
     def _attribut_to_bool(self, variable):
         for prop in [
@@ -766,10 +777,11 @@ class RougailUpgrade:
     def _attribut_to_int(self, variable):
         for prop in ["mode"]:
             if prop in variable:
-                if variable[prop] in ['expert', 'normal']:
-                    variable[prop] = {'expert': 'advanced',
-                                      'normal': 'standard',
-                                      }.get(variable[prop])
+                if variable[prop] in ["expert", "normal"]:
+                    variable[prop] = {
+                        "expert": "advanced",
+                        "normal": "standard",
+                    }.get(variable[prop])
                     continue
                 try:
                     variable[prop] = int(variable[prop])
@@ -891,6 +903,60 @@ class RougailUpgrade:
             dico = {obj_name: dico}
         return dico
 
+    def _update_1_1(self, root):
+        new_root = {}
+        update_root = False
+        for key, value in root.items():
+            new_root[key] = value
+            if not isinstance(value, dict):
+                continue
+            # migrate dynamic family
+            if (
+                ("variable" in value and isinstance(value["variable"], str))
+                or ("_variable" in value and isinstance(value["_variable"], str))
+            ) and (
+                ("_type" in value and value["_type"] == "dynamic")
+                or ("type" in value and value["type"] == "dynamic")
+            ):
+                value["dynamic"] = {
+                    "type": "variable",
+                    "variable": value.pop("variable"),
+                    "propertyerror": False,
+                }
+                if '{{ suffix }}' not in key:
+                    new_root[key + '{{ suffix }}'] = new_root.pop(key)
+                    update_root = True
+            self._update_1_1(value)
+            for typ, obj in {'boolean': bool,
+                             'number': int,
+                             'string': str,
+                             'float': float,
+                             }.items():
+                if value.get('type') == typ:
+                    default = value.get('default')
+                    if default is None or default == []:
+                        continue
+                    if isinstance(default, obj):
+                        del value['type']
+                    elif isinstance(default, list) and isinstance(default[0], obj):
+                        del value['type']
+            if value.get('multi') and isinstance(value.get('default'), list):
+                del value['multi']
+        if update_root:
+            root.clear()
+            root.update(new_root)
+
+    def update_1_1(
+        self,
+        root,
+        upgrade_help: dict,
+        namespace: str,
+        xmlsrc: str,
+        ext: str,
+    ):
+        self._update_1_1(root)
+        return root, None, "yml"
+
     def update_1_0(
         self,
         root: "Element",
@@ -906,22 +972,12 @@ class RougailUpgrade:
                 objects = root.find(typ)
                 if objects is None:
                     objects = []
-                new_objects = self._xml_to_yaml(objects, typ, variables, "")
+                new_objects = self._xml_to_yaml(objects, typ, variables, namespace)
                 if new_objects[typ]:
                     new_root.update(new_objects)
-        #            services = root.find('services')
-        #            if services is None:
-        #                services = []
-        #            new_services = self._xml_to_yaml_service(services)
-        #            if new_services:
-        #                new_root['services'] = new_services
-        #        paths = self._get_path_variables(variables,
-        #                                         namespace == 'configuration',
-        #                                         namespace,
-        #                                         )
         else:
             new_root = root
-        variables, services = upgrade_010_to_100(new_root, namespace, xmlsrc).get()
+        variables, services = upgrade_010_to_10(new_root, namespace, xmlsrc).get()
         return variables, services, "yml"
 
     def update_0_10(
@@ -966,7 +1022,6 @@ class RougailUpgrade:
                 if not has_value:
                     value = SubElement(variable, "value")
                     value.text = choices[0]
-                variable.attrib["mandatory"] = "True"
 
         # convert group to leadership
         groups = []

src/rougail/utils.py

@@ -30,6 +30,7 @@ Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 from typing import List, Union
 from unicodedata import normalize, combining
 import re
+from itertools import chain
 
 from importlib.machinery import SourceFileLoader
 from importlib.util import spec_from_loader, module_from_spec
@@ -37,7 +38,9 @@ from importlib.util import spec_from_loader, module_from_spec
 from jinja2 import DictLoader, TemplateSyntaxError
 from jinja2.sandbox import SandboxedEnvironment
 from jinja2.parser import Parser
-from jinja2.nodes import Getattr
+from jinja2.nodes import Name, Getattr
+
+from tiramisu.config import get_common_path
 
 from .i18n import _
 from .error import DictConsistencyError
@@ -62,15 +65,16 @@ def normalize_family(family_name: str) -> str:
     """replace space, accent, uppercase, ... by valid character"""
     if not family_name:
         return
+    family_name = family_name.lower()
     family_name = family_name.replace("-", "_").replace(" ", "_").replace(".", "_")
     nfkd_form = normalize("NFKD", family_name)
     family_name = "".join([c for c in nfkd_form if not combining(c)])
     return family_name.lower()
 
 
-def load_modules(eosfunc_file) -> List[str]:
-    """list all functions in eosfunc"""
-    loader = SourceFileLoader("eosfunc", eosfunc_file)
+def load_modules(name, module) -> List[str]:
+    """list all functions in a module"""
+    loader = SourceFileLoader(name, module)
     spec = spec_from_loader(loader.name, loader)
     eosfunc = module_from_spec(spec)
     loader.exec_module(eosfunc)
@@ -87,11 +91,14 @@ def get_realpath(
 
 
 def get_jinja_variable_to_param(
+    current_path: str,
     jinja_text,
     objectspace,
     xmlfiles,
     functions,
     path_prefix,
+    version,
+    namespace,
 ):
     try:
         env = SandboxedEnvironment(loader=DictLoader({"tmpl": jinja_text}))
@@ -104,16 +111,58 @@ def get_jinja_variable_to_param(
             return g.node.name + "." + g.attr
 
         variables = set()
+        if objectspace.namespace is None:
+            for n in parsed_content.find_all(Name):
+                variables.add(n.name)
         for g in parsed_content.find_all(Getattr):
             variables.add(recurse_getattr(g))
     except TemplateSyntaxError as err:
-        msg = _(f'error in jinja "{jinja_text}": {err}')
-        raise Exception(msg) from err
+        msg = _(f'error in jinja "{jinja_text}" for the variable "{ current_path }": {err}')
+        raise DictConsistencyError(msg, 39, xmlfiles) from err
     variables = list(variables)
-    variables.sort()
+    variables.sort(reverse=True)
+    founded_variables = {}
+    unknown_variables = []
     for variable_path in variables:
-        variable, suffix, dynamic = objectspace.paths.get_with_dynamic(
-            get_realpath(variable_path, path_prefix)
+        variable, suffix = objectspace.paths.get_with_dynamic(
+            variable_path,
+            path_prefix,
+            current_path,
+            version,
+            namespace,
+            xmlfiles,
         )
         if variable and variable.path in objectspace.variables:
-            yield variable, suffix, variable_path, dynamic
+            founded_variables[variable_path] = (suffix, variable)
+        else:
+            sub_family = variable_path + '.'
+            for founded_variable in chain(founded_variables, unknown_variables):
+                if founded_variable.startswith(sub_family):
+                    break
+            else:
+                unknown_variables.append(variable_path)
+
+    for variable_path in unknown_variables:
+        for v in founded_variables:
+            if get_common_path(v, variable_path) == v:
+                break
+        else:
+            root_path = None
+            vpath = variable_path
+            while '.' in vpath:
+                vpath = vpath.rsplit('.', 1)[0]
+                variable, suffix = objectspace.paths.get_with_dynamic(
+                    vpath,
+                    path_prefix,
+                    current_path,
+                    version,
+                    namespace,
+                    xmlfiles,
+                )
+                if variable and variable.path in objectspace.families:
+                    root_path = vpath
+                    break
+            if root_path:
+                yield {}, None, root_path
+    for variable_path, data in founded_variables.items():
+        yield data[1], data[0], variable_path

tests/data/dict1/dict.yml

@@ -0,0 +1,5 @@
+# dict with a correct version declared
+#version: "1.1"
+hello:
+  type: string
+  default: world

tests/data/dict2/dict.yml

@@ -0,0 +1,5 @@
+# dict with a correct version declared
+version: "1.0"
+hello:
+  type: string
+  default: world

tests/dictionaries/00_0empty/tiramisu/base.py

@@ -0,0 +1,9 @@
+from tiramisu import *
+from tiramisu.setting import ALLOWED_LEADER_PROPERTIES
+from re import compile as re_compile
+from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription
+load_functions('tests/dictionaries/../eosfunc/test.py')
+ALLOWED_LEADER_PROPERTIES.add("basic")
+ALLOWED_LEADER_PROPERTIES.add("standard")
+ALLOWED_LEADER_PROPERTIES.add("advanced")
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[])

tests/dictionaries/00_0empty/tiramisu/multi.py

@@ -0,0 +1,11 @@
+from tiramisu import *
+from tiramisu.setting import ALLOWED_LEADER_PROPERTIES
+from re import compile as re_compile
+from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription
+load_functions('tests/dictionaries/../eosfunc/test.py')
+ALLOWED_LEADER_PROPERTIES.add("basic")
+ALLOWED_LEADER_PROPERTIES.add("standard")
+ALLOWED_LEADER_PROPERTIES.add("advanced")
+optiondescription_1 = OptionDescription(name="1", doc="1", children=[], properties=frozenset({"advanced"}))
+optiondescription_2 = OptionDescription(name="2", doc="2", children=[], properties=frozenset({"advanced"}))
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[optiondescription_1, optiondescription_2])

tests/dictionaries/00_0empty/tiramisu/no_namespace.py

@@ -0,0 +1,9 @@
+from tiramisu import *
+from tiramisu.setting import ALLOWED_LEADER_PROPERTIES
+from re import compile as re_compile
+from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription
+load_functions('tests/dictionaries/../eosfunc/test.py')
+ALLOWED_LEADER_PROPERTIES.add("basic")
+ALLOWED_LEADER_PROPERTIES.add("standard")
+ALLOWED_LEADER_PROPERTIES.add("advanced")
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[])

tests/dictionaries/00_0version_underscore/dictionaries/rougail/00-base.yml

@@ -0,0 +1,3 @@
+---
+_version: '1.1'
+version:  # a variable

tests/dictionaries/00_0version_underscore/makedict/after.json

@@ -0,0 +1,6 @@
+{
+    "rougail.version": {
+        "owner": "default",
+        "value": null
+    }
+}

tests/dictionaries/00_0version_underscore/makedict/base.json

@@ -0,0 +1,3 @@
+{
+    "rougail.version": null
+}

tests/dictionaries/00_0version_underscore/makedict/before.json

@@ -0,0 +1,6 @@
+{
+    "rougail.version": {
+        "owner": "default",
+        "value": null
+    }
+}

tests/dictionaries/00_0version_underscore/makedict/mandatory.json

@@ -0,0 +1 @@
+["rougail.version"]

tests/dictionaries/00_0version_underscore/tiramisu/base.py

@@ -0,0 +1,11 @@
+from tiramisu import *
+from tiramisu.setting import ALLOWED_LEADER_PROPERTIES
+from re import compile as re_compile
+from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription
+load_functions('tests/dictionaries/../eosfunc/test.py')
+ALLOWED_LEADER_PROPERTIES.add("basic")
+ALLOWED_LEADER_PROPERTIES.add("standard")
+ALLOWED_LEADER_PROPERTIES.add("advanced")
+option_2 = StrOption(name="version", doc="a variable", properties=frozenset({"basic", "mandatory"}), informations={'type': 'string'})
+optiondescription_1 = OptionDescription(name="rougail", doc="Rougail", children=[option_2], properties=frozenset({"basic"}))
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[optiondescription_1])

tests/dictionaries/00_0version_underscore/tiramisu/multi.py

@@ -0,0 +1,15 @@
+from tiramisu import *
+from tiramisu.setting import ALLOWED_LEADER_PROPERTIES
+from re import compile as re_compile
+from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription
+load_functions('tests/dictionaries/../eosfunc/test.py')
+ALLOWED_LEADER_PROPERTIES.add("basic")
+ALLOWED_LEADER_PROPERTIES.add("standard")
+ALLOWED_LEADER_PROPERTIES.add("advanced")
+option_3 = StrOption(name="version", doc="a variable", properties=frozenset({"basic", "mandatory"}), informations={'type': 'string'})
+optiondescription_2 = OptionDescription(name="rougail", doc="Rougail", children=[option_3], properties=frozenset({"basic"}))
+optiondescription_1 = OptionDescription(name="1", doc="1", children=[optiondescription_2], properties=frozenset({"basic"}))
+option_6 = StrOption(name="version", doc="a variable", properties=frozenset({"basic", "mandatory"}), informations={'type': 'string'})
+optiondescription_5 = OptionDescription(name="rougail", doc="Rougail", children=[option_6], properties=frozenset({"basic"}))
+optiondescription_4 = OptionDescription(name="2", doc="2", children=[optiondescription_5], properties=frozenset({"basic"}))
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[optiondescription_1, optiondescription_4])

tests/dictionaries/00_0version_underscore/tiramisu/no_namespace.py

@@ -0,0 +1,10 @@
+from tiramisu import *
+from tiramisu.setting import ALLOWED_LEADER_PROPERTIES
+from re import compile as re_compile
+from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription
+load_functions('tests/dictionaries/../eosfunc/test.py')
+ALLOWED_LEADER_PROPERTIES.add("basic")
+ALLOWED_LEADER_PROPERTIES.add("standard")
+ALLOWED_LEADER_PROPERTIES.add("advanced")
+option_1 = StrOption(name="version", doc="a variable", properties=frozenset({"basic", "mandatory"}), informations={'type': 'string'})
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[option_1])

tests/dictionaries/00_1empty_variable/makedict/base.json

@@ -0,0 +1,3 @@
+{
+    "rougail.empty": null
+}

tests/dictionaries/00_1empty_variable/tiramisu/base.py

@@ -0,0 +1,11 @@
+from tiramisu import *
+from tiramisu.setting import ALLOWED_LEADER_PROPERTIES
+from re import compile as re_compile
+from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription
+load_functions('tests/dictionaries/../eosfunc/test.py')
+ALLOWED_LEADER_PROPERTIES.add("basic")
+ALLOWED_LEADER_PROPERTIES.add("standard")
+ALLOWED_LEADER_PROPERTIES.add("advanced")
+option_2 = StrOption(name="empty", doc="empty", properties=frozenset({"basic", "mandatory"}), informations={'type': 'string'})
+optiondescription_1 = OptionDescription(name="rougail", doc="Rougail", children=[option_2], properties=frozenset({"basic"}))
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[optiondescription_1])

tests/dictionaries/00_1empty_variable/tiramisu/multi.py

@@ -0,0 +1,15 @@
+from tiramisu import *
+from tiramisu.setting import ALLOWED_LEADER_PROPERTIES
+from re import compile as re_compile
+from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription
+load_functions('tests/dictionaries/../eosfunc/test.py')
+ALLOWED_LEADER_PROPERTIES.add("basic")
+ALLOWED_LEADER_PROPERTIES.add("standard")
+ALLOWED_LEADER_PROPERTIES.add("advanced")
+option_3 = StrOption(name="empty", doc="empty", properties=frozenset({"basic", "mandatory"}), informations={'type': 'string'})
+optiondescription_2 = OptionDescription(name="rougail", doc="Rougail", children=[option_3], properties=frozenset({"basic"}))
+optiondescription_1 = OptionDescription(name="1", doc="1", children=[optiondescription_2], properties=frozenset({"basic"}))
+option_6 = StrOption(name="empty", doc="empty", properties=frozenset({"basic", "mandatory"}), informations={'type': 'string'})
+optiondescription_5 = OptionDescription(name="rougail", doc="Rougail", children=[option_6], properties=frozenset({"basic"}))
+optiondescription_4 = OptionDescription(name="2", doc="2", children=[optiondescription_5], properties=frozenset({"basic"}))
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[optiondescription_1, optiondescription_4])

tests/dictionaries/00_1empty_variable/tiramisu/no_namespace.py

@@ -0,0 +1,10 @@
+from tiramisu import *
+from tiramisu.setting import ALLOWED_LEADER_PROPERTIES
+from re import compile as re_compile
+from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription
+load_functions('tests/dictionaries/../eosfunc/test.py')
+ALLOWED_LEADER_PROPERTIES.add("basic")
+ALLOWED_LEADER_PROPERTIES.add("standard")
+ALLOWED_LEADER_PROPERTIES.add("advanced")
+option_1 = StrOption(name="empty", doc="empty", properties=frozenset({"basic", "mandatory"}), informations={'type': 'string'})
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[option_1])

tests/dictionaries/00_2default_calculated/dictionaries/rougail/00-base.yml

@@ -0,0 +1,10 @@
+---
+version: 1.1
+var1: "no"  # a first variable
+var2:
+  description: a second variable
+  multi: true
+  default:
+    jinja: |
+      {{ _.var1 }}
+    description: the value of var1

tests/dictionaries/00_2default_calculated/tiramisu/base.py

@@ -0,0 +1,13 @@
+from tiramisu import *
+from tiramisu.setting import ALLOWED_LEADER_PROPERTIES
+from re import compile as re_compile
+from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription
+load_functions('tests/dictionaries/../eosfunc/test.py')
+ALLOWED_LEADER_PROPERTIES.add("basic")
+ALLOWED_LEADER_PROPERTIES.add("standard")
+ALLOWED_LEADER_PROPERTIES.add("advanced")
+dict_env['default_rougail.var2'] = "{{ _.var1 }}\n"
+option_2 = StrOption(name="var1", doc="a first variable", default="no", properties=frozenset({"mandatory", "standard"}), informations={'type': 'string'})
+option_3 = StrOption(name="var2", doc="a second variable", multi=True, default=Calculation(func['jinja_to_function'], Params((), kwargs={'__internal_jinja': ParamValue("default_rougail.var2"), '__internal_type': ParamValue("string"), '__internal_multi': ParamValue(True), '__internal_files': ParamValue(['tests/dictionaries/00_2default_calculated/dictionaries/rougail/00-base.yml']), '__internal_attribute': ParamValue("default"), '__internal_variable': ParamValue("rougail.var2"), '_.var1': ParamOption(option_2)})), properties=frozenset({"mandatory", "standard"}), informations={'type': 'string'})
+optiondescription_1 = OptionDescription(name="rougail", doc="Rougail", children=[option_2, option_3], properties=frozenset({"standard"}))
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[optiondescription_1])

tests/dictionaries/00_2default_calculated/tiramisu/multi.py

@@ -0,0 +1,19 @@
+from tiramisu import *
+from tiramisu.setting import ALLOWED_LEADER_PROPERTIES
+from re import compile as re_compile
+from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription
+load_functions('tests/dictionaries/../eosfunc/test.py')
+ALLOWED_LEADER_PROPERTIES.add("basic")
+ALLOWED_LEADER_PROPERTIES.add("standard")
+ALLOWED_LEADER_PROPERTIES.add("advanced")
+dict_env['default_1.rougail.var2'] = "{{ _.var1 }}\n"
+dict_env['default_2.rougail.var2'] = "{{ _.var1 }}\n"
+option_3 = StrOption(name="var1", doc="a first variable", default="no", properties=frozenset({"mandatory", "standard"}), informations={'type': 'string'})
+option_4 = StrOption(name="var2", doc="a second variable", multi=True, default=Calculation(func['jinja_to_function'], Params((), kwargs={'__internal_jinja': ParamValue("default_1.rougail.var2"), '__internal_type': ParamValue("string"), '__internal_multi': ParamValue(True), '__internal_files': ParamValue(['tests/dictionaries/00_2default_calculated/dictionaries/rougail/00-base.yml']), '__internal_attribute': ParamValue("default"), '__internal_variable': ParamValue("1.rougail.var2"), '_.var1': ParamOption(option_3)})), properties=frozenset({"mandatory", "standard"}), informations={'type': 'string'})
+optiondescription_2 = OptionDescription(name="rougail", doc="Rougail", children=[option_3, option_4], properties=frozenset({"standard"}))
+optiondescription_1 = OptionDescription(name="1", doc="1", children=[optiondescription_2], properties=frozenset({"standard"}))
+option_7 = StrOption(name="var1", doc="a first variable", default="no", properties=frozenset({"mandatory", "standard"}), informations={'type': 'string'})
+option_8 = StrOption(name="var2", doc="a second variable", multi=True, default=Calculation(func['jinja_to_function'], Params((), kwargs={'__internal_jinja': ParamValue("default_2.rougail.var2"), '__internal_type': ParamValue("string"), '__internal_multi': ParamValue(True), '__internal_files': ParamValue(['tests/dictionaries/00_2default_calculated/dictionaries/rougail/00-base.yml']), '__internal_attribute': ParamValue("default"), '__internal_variable': ParamValue("2.rougail.var2"), '_.var1': ParamOption(option_7)})), properties=frozenset({"mandatory", "standard"}), informations={'type': 'string'})
+optiondescription_6 = OptionDescription(name="rougail", doc="Rougail", children=[option_7, option_8], properties=frozenset({"standard"}))
+optiondescription_5 = OptionDescription(name="2", doc="2", children=[optiondescription_6], properties=frozenset({"standard"}))
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[optiondescription_1, optiondescription_5])

tests/dictionaries/00_2default_calculated/tiramisu/no_namespace.py

@@ -0,0 +1,12 @@
+from tiramisu import *
+from tiramisu.setting import ALLOWED_LEADER_PROPERTIES
+from re import compile as re_compile
+from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription
+load_functions('tests/dictionaries/../eosfunc/test.py')
+ALLOWED_LEADER_PROPERTIES.add("basic")
+ALLOWED_LEADER_PROPERTIES.add("standard")
+ALLOWED_LEADER_PROPERTIES.add("advanced")
+dict_env['default_var2'] = "{{ _.var1 }}\n"
+option_1 = StrOption(name="var1", doc="a first variable", default="no", properties=frozenset({"mandatory", "standard"}), informations={'type': 'string'})
+option_2 = StrOption(name="var2", doc="a second variable", multi=True, default=Calculation(func['jinja_to_function'], Params((), kwargs={'__internal_jinja': ParamValue("default_var2"), '__internal_type': ParamValue("string"), '__internal_multi': ParamValue(True), '__internal_files': ParamValue(['tests/dictionaries/00_2default_calculated/dictionaries/rougail/00-base.yml']), '__internal_attribute': ParamValue("default"), '__internal_variable': ParamValue("var2"), '_.var1': ParamOption(option_1)})), properties=frozenset({"mandatory", "standard"}), informations={'type': 'string'})
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[option_1, option_2])

tests/dictionaries/00_2default_calculated_multi/dictionaries/rougail/00-base.yml

@@ -0,0 +1,15 @@
+---
+version: 1.1
+var1:  # a first variable
+  - 'no'
+  - 'yes'
+  - maybe
+var2:
+  description: a second variable
+  multi: true
+  default:
+    jinja: |
+      {% for val in _.var1 %}
+      {{ val }}
+      {% endfor %}
+    description: the value of _.var1

tests/dictionaries/00_2default_calculated_multi/tiramisu/base.py

@@ -0,0 +1,13 @@
+from tiramisu import *
+from tiramisu.setting import ALLOWED_LEADER_PROPERTIES
+from re import compile as re_compile
+from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription
+load_functions('tests/dictionaries/../eosfunc/test.py')
+ALLOWED_LEADER_PROPERTIES.add("basic")
+ALLOWED_LEADER_PROPERTIES.add("standard")
+ALLOWED_LEADER_PROPERTIES.add("advanced")
+dict_env['default_rougail.var2'] = "{% for val in _.var1 %}\n{{ val }}\n{% endfor %}\n"
+option_2 = StrOption(name="var1", doc="a first variable", multi=True, default=["no", "yes", "maybe"], default_multi="no", properties=frozenset({"mandatory", "standard"}), informations={'type': 'string'})
+option_3 = StrOption(name="var2", doc="a second variable", multi=True, default=Calculation(func['jinja_to_function'], Params((), kwargs={'__internal_jinja': ParamValue("default_rougail.var2"), '__internal_type': ParamValue("string"), '__internal_multi': ParamValue(True), '__internal_files': ParamValue(['tests/dictionaries/00_2default_calculated_multi/dictionaries/rougail/00-base.yml']), '__internal_attribute': ParamValue("default"), '__internal_variable': ParamValue("rougail.var2"), '_.var1': ParamOption(option_2)})), properties=frozenset({"mandatory", "standard"}), informations={'type': 'string'})
+optiondescription_1 = OptionDescription(name="rougail", doc="Rougail", children=[option_2, option_3], properties=frozenset({"standard"}))
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[optiondescription_1])

tests/dictionaries/00_2default_calculated_multi/tiramisu/multi.py

@@ -0,0 +1,19 @@
+from tiramisu import *
+from tiramisu.setting import ALLOWED_LEADER_PROPERTIES
+from re import compile as re_compile
+from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription
+load_functions('tests/dictionaries/../eosfunc/test.py')
+ALLOWED_LEADER_PROPERTIES.add("basic")
+ALLOWED_LEADER_PROPERTIES.add("standard")
+ALLOWED_LEADER_PROPERTIES.add("advanced")
+dict_env['default_1.rougail.var2'] = "{% for val in _.var1 %}\n{{ val }}\n{% endfor %}\n"
+dict_env['default_2.rougail.var2'] = "{% for val in _.var1 %}\n{{ val }}\n{% endfor %}\n"
+option_3 = StrOption(name="var1", doc="a first variable", multi=True, default=["no", "yes", "maybe"], default_multi="no", properties=frozenset({"mandatory", "standard"}), informations={'type': 'string'})
+option_4 = StrOption(name="var2", doc="a second variable", multi=True, default=Calculation(func['jinja_to_function'], Params((), kwargs={'__internal_jinja': ParamValue("default_1.rougail.var2"), '__internal_type': ParamValue("string"), '__internal_multi': ParamValue(True), '__internal_files': ParamValue(['tests/dictionaries/00_2default_calculated_multi/dictionaries/rougail/00-base.yml']), '__internal_attribute': ParamValue("default"), '__internal_variable': ParamValue("1.rougail.var2"), '_.var1': ParamOption(option_3)})), properties=frozenset({"mandatory", "standard"}), informations={'type': 'string'})
+optiondescription_2 = OptionDescription(name="rougail", doc="Rougail", children=[option_3, option_4], properties=frozenset({"standard"}))
+optiondescription_1 = OptionDescription(name="1", doc="1", children=[optiondescription_2], properties=frozenset({"standard"}))
+option_7 = StrOption(name="var1", doc="a first variable", multi=True, default=["no", "yes", "maybe"], default_multi="no", properties=frozenset({"mandatory", "standard"}), informations={'type': 'string'})
+option_8 = StrOption(name="var2", doc="a second variable", multi=True, default=Calculation(func['jinja_to_function'], Params((), kwargs={'__internal_jinja': ParamValue("default_2.rougail.var2"), '__internal_type': ParamValue("string"), '__internal_multi': ParamValue(True), '__internal_files': ParamValue(['tests/dictionaries/00_2default_calculated_multi/dictionaries/rougail/00-base.yml']), '__internal_attribute': ParamValue("default"), '__internal_variable': ParamValue("2.rougail.var2"), '_.var1': ParamOption(option_7)})), properties=frozenset({"mandatory", "standard"}), informations={'type': 'string'})
+optiondescription_6 = OptionDescription(name="rougail", doc="Rougail", children=[option_7, option_8], properties=frozenset({"standard"}))
+optiondescription_5 = OptionDescription(name="2", doc="2", children=[optiondescription_6], properties=frozenset({"standard"}))
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[optiondescription_1, optiondescription_5])

tests/dictionaries/00_2default_calculated_multi/tiramisu/no_namespace.py

@@ -0,0 +1,12 @@
+from tiramisu import *
+from tiramisu.setting import ALLOWED_LEADER_PROPERTIES
+from re import compile as re_compile
+from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription
+load_functions('tests/dictionaries/../eosfunc/test.py')
+ALLOWED_LEADER_PROPERTIES.add("basic")
+ALLOWED_LEADER_PROPERTIES.add("standard")
+ALLOWED_LEADER_PROPERTIES.add("advanced")
+dict_env['default_var2'] = "{% for val in _.var1 %}\n{{ val }}\n{% endfor %}\n"
+option_1 = StrOption(name="var1", doc="a first variable", multi=True, default=["no", "yes", "maybe"], default_multi="no", properties=frozenset({"mandatory", "standard"}), informations={'type': 'string'})
+option_2 = StrOption(name="var2", doc="a second variable", multi=True, default=Calculation(func['jinja_to_function'], Params((), kwargs={'__internal_jinja': ParamValue("default_var2"), '__internal_type': ParamValue("string"), '__internal_multi': ParamValue(True), '__internal_files': ParamValue(['tests/dictionaries/00_2default_calculated_multi/dictionaries/rougail/00-base.yml']), '__internal_attribute': ParamValue("default"), '__internal_variable': ParamValue("var2"), '_.var1': ParamOption(option_1)})), properties=frozenset({"mandatory", "standard"}), informations={'type': 'string'})
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[option_1, option_2])

tests/dictionaries/00_2default_calculated_variable_transitive/dictionaries/rougail/00-base.yml

@@ -0,0 +1,13 @@
+---
+version: 1.1
+var1:
+  description: a first variable
+  multi: true
+  type: domainname
+  params:
+    allow_ip: true
+var2:
+  description: a second variable
+  default:
+    type: variable
+    variable: _.var1

tests/dictionaries/00_2default_calculated_variable_transitive/makedict/after.json

@@ -0,0 +1,10 @@
+{
+    "rougail.var1": {
+        "owner": "default",
+        "value": []
+    },
+    "rougail.var2": {
+        "owner": "default",
+        "value": []
+    }
+}

tests/dictionaries/00_2default_calculated_variable_transitive/makedict/base.json

@@ -0,0 +1,4 @@
+{
+    "rougail.var1": [],
+    "rougail.var2": []
+}

tests/dictionaries/00_2default_calculated_variable_transitive/makedict/before.json

@@ -0,0 +1,10 @@
+{
+    "rougail.var1": {
+        "owner": "default",
+        "value": []
+    },
+    "rougail.var2": {
+        "owner": "default",
+        "value": []
+    }
+}

tests/dictionaries/00_2default_calculated_variable_transitive/makedict/mandatory.json

@@ -0,0 +1 @@
+["rougail.var1", "rougail.var2"]

tests/dictionaries/00_2default_calculated_variable_transitive/tiramisu/base.py

@@ -0,0 +1,12 @@
+from tiramisu import *
+from tiramisu.setting import ALLOWED_LEADER_PROPERTIES
+from re import compile as re_compile
+from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription
+load_functions('tests/dictionaries/../eosfunc/test.py')
+ALLOWED_LEADER_PROPERTIES.add("basic")
+ALLOWED_LEADER_PROPERTIES.add("standard")
+ALLOWED_LEADER_PROPERTIES.add("advanced")
+option_2 = DomainnameOption(name="var1", doc="a first variable", multi=True, type="domainname", allow_ip=True, properties=frozenset({"basic", "mandatory"}), informations={'type': 'domainname'})
+option_3 = DomainnameOption(name="var2", doc="a second variable", multi=True, default=Calculation(func['calc_value'], Params((ParamOption(option_2)))), type="domainname", allow_ip=True, properties=frozenset({"mandatory", "standard"}), informations={'type': 'domainname'})
+optiondescription_1 = OptionDescription(name="rougail", doc="Rougail", children=[option_2, option_3], properties=frozenset({"basic"}))
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[optiondescription_1])

tests/dictionaries/00_2default_calculated_variable_transitive/tiramisu/multi.py

@@ -0,0 +1,17 @@
+from tiramisu import *
+from tiramisu.setting import ALLOWED_LEADER_PROPERTIES
+from re import compile as re_compile
+from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription
+load_functions('tests/dictionaries/../eosfunc/test.py')
+ALLOWED_LEADER_PROPERTIES.add("basic")
+ALLOWED_LEADER_PROPERTIES.add("standard")
+ALLOWED_LEADER_PROPERTIES.add("advanced")
+option_3 = DomainnameOption(name="var1", doc="a first variable", multi=True, type="domainname", allow_ip=True, properties=frozenset({"basic", "mandatory"}), informations={'type': 'domainname'})
+option_4 = DomainnameOption(name="var2", doc="a second variable", multi=True, default=Calculation(func['calc_value'], Params((ParamOption(option_3)))), type="domainname", allow_ip=True, properties=frozenset({"mandatory", "standard"}), informations={'type': 'domainname'})
+optiondescription_2 = OptionDescription(name="rougail", doc="Rougail", children=[option_3, option_4], properties=frozenset({"basic"}))
+optiondescription_1 = OptionDescription(name="1", doc="1", children=[optiondescription_2], properties=frozenset({"basic"}))
+option_7 = DomainnameOption(name="var1", doc="a first variable", multi=True, type="domainname", allow_ip=True, properties=frozenset({"basic", "mandatory"}), informations={'type': 'domainname'})
+option_8 = DomainnameOption(name="var2", doc="a second variable", multi=True, default=Calculation(func['calc_value'], Params((ParamOption(option_7)))), type="domainname", allow_ip=True, properties=frozenset({"mandatory", "standard"}), informations={'type': 'domainname'})
+optiondescription_6 = OptionDescription(name="rougail", doc="Rougail", children=[option_7, option_8], properties=frozenset({"basic"}))
+optiondescription_5 = OptionDescription(name="2", doc="2", children=[optiondescription_6], properties=frozenset({"basic"}))
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[optiondescription_1, optiondescription_5])

tests/dictionaries/00_2default_calculated_variable_transitive/tiramisu/no_namespace.py

@@ -0,0 +1,11 @@
+from tiramisu import *
+from tiramisu.setting import ALLOWED_LEADER_PROPERTIES
+from re import compile as re_compile
+from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription
+load_functions('tests/dictionaries/../eosfunc/test.py')
+ALLOWED_LEADER_PROPERTIES.add("basic")
+ALLOWED_LEADER_PROPERTIES.add("standard")
+ALLOWED_LEADER_PROPERTIES.add("advanced")
+option_1 = DomainnameOption(name="var1", doc="a first variable", multi=True, type="domainname", allow_ip=True, properties=frozenset({"basic", "mandatory"}), informations={'type': 'domainname'})
+option_2 = DomainnameOption(name="var2", doc="a second variable", multi=True, default=Calculation(func['calc_value'], Params((ParamOption(option_1)))), type="domainname", allow_ip=True, properties=frozenset({"mandatory", "standard"}), informations={'type': 'domainname'})
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[option_1, option_2])

tests/dictionaries/00_4load_subfolder/dictionaries/rougail/99-base.yml

@@ -0,0 +1,4 @@
+---
+version: '1.0'
+var1:
+  description: a variable

tests/dictionaries/00_4load_subfolder/dictionaries/rougail2/00-base.yml

@@ -0,0 +1,4 @@
+---
+version: "1.0"
+var2:
+  description: a variable

tests/dictionaries/00_4load_subfolder/makedict/after.json

@@ -0,0 +1,10 @@
+{
+    "rougail.var1": {
+        "owner": "default",
+        "value": null
+    },
+    "rougail.var2": {
+        "owner": "default",
+        "value": null
+    }
+}

tests/dictionaries/00_4load_subfolder/makedict/base.json

@@ -0,0 +1,4 @@
+{
+    "rougail.var1": null,
+    "rougail.var2": null
+}

tests/dictionaries/00_4load_subfolder/makedict/before.json

@@ -0,0 +1,10 @@
+{
+    "rougail.var1": {
+        "owner": "default",
+        "value": null
+    },
+    "rougail.var2": {
+        "owner": "default",
+        "value": null
+    }
+}

tests/dictionaries/00_4load_subfolder/makedict/mandatory.json

@@ -0,0 +1 @@
+["rougail.var1", "rougail.var2"]

tests/dictionaries/00_4load_subfolder/tiramisu/base.py

@@ -0,0 +1,12 @@
+from tiramisu import *
+from tiramisu.setting import ALLOWED_LEADER_PROPERTIES
+from re import compile as re_compile
+from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription
+load_functions('tests/dictionaries/../eosfunc/test.py')
+ALLOWED_LEADER_PROPERTIES.add("basic")
+ALLOWED_LEADER_PROPERTIES.add("standard")
+ALLOWED_LEADER_PROPERTIES.add("advanced")
+option_2 = StrOption(name="var1", doc="a variable", properties=frozenset({"basic", "mandatory"}), informations={'type': 'string'})
+option_3 = StrOption(name="var2", doc="a variable", properties=frozenset({"basic", "mandatory"}), informations={'type': 'string'})
+optiondescription_1 = OptionDescription(name="rougail", doc="Rougail", children=[option_2, option_3], properties=frozenset({"basic"}))
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[optiondescription_1])

tests/dictionaries/00_4load_subfolder/tiramisu/multi.py

@@ -0,0 +1,17 @@
+from tiramisu import *
+from tiramisu.setting import ALLOWED_LEADER_PROPERTIES
+from re import compile as re_compile
+from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription
+load_functions('tests/dictionaries/../eosfunc/test.py')
+ALLOWED_LEADER_PROPERTIES.add("basic")
+ALLOWED_LEADER_PROPERTIES.add("standard")
+ALLOWED_LEADER_PROPERTIES.add("advanced")
+option_3 = StrOption(name="var1", doc="a variable", properties=frozenset({"basic", "mandatory"}), informations={'type': 'string'})
+option_4 = StrOption(name="var2", doc="a variable", properties=frozenset({"basic", "mandatory"}), informations={'type': 'string'})
+optiondescription_2 = OptionDescription(name="rougail", doc="Rougail", children=[option_3, option_4], properties=frozenset({"basic"}))
+optiondescription_1 = OptionDescription(name="1", doc="1", children=[optiondescription_2], properties=frozenset({"basic"}))
+option_7 = StrOption(name="var1", doc="a variable", properties=frozenset({"basic", "mandatory"}), informations={'type': 'string'})
+option_8 = StrOption(name="var2", doc="a variable", properties=frozenset({"basic", "mandatory"}), informations={'type': 'string'})
+optiondescription_6 = OptionDescription(name="rougail", doc="Rougail", children=[option_7, option_8], properties=frozenset({"basic"}))
+optiondescription_5 = OptionDescription(name="2", doc="2", children=[optiondescription_6], properties=frozenset({"basic"}))
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[optiondescription_1, optiondescription_5])

tests/dictionaries/00_4load_subfolder/tiramisu/no_namespace.py

@@ -0,0 +1,11 @@
+from tiramisu import *
+from tiramisu.setting import ALLOWED_LEADER_PROPERTIES
+from re import compile as re_compile
+from rougail.tiramisu import func, dict_env, load_functions, ConvertDynOptionDescription
+load_functions('tests/dictionaries/../eosfunc/test.py')
+ALLOWED_LEADER_PROPERTIES.add("basic")
+ALLOWED_LEADER_PROPERTIES.add("standard")
+ALLOWED_LEADER_PROPERTIES.add("advanced")
+option_1 = StrOption(name="var1", doc="a variable", properties=frozenset({"basic", "mandatory"}), informations={'type': 'string'})
+option_2 = StrOption(name="var2", doc="a variable", properties=frozenset({"basic", "mandatory"}), informations={'type': 'string'})
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[option_1, option_2])

tests/dictionaries/00_5load_notype/dictionaries/rougail/00-base.yml

@@ -0,0 +1,5 @@
+---
+version: '1.0'
+without_type:
+  description: a variable
+  default: non

tests/dictionaries/00_5load_notype/makedict/after.json

@@ -0,0 +1,6 @@
+{
+    "rougail.without_type": {
+        "owner": "default",
+        "value": "non"
+    }
+}

tests/dictionaries/00_5load_notype/makedict/base.json

@@ -0,0 +1,3 @@
+{
+    "rougail.without_type": "non"
+}