diff --git a/docs/.gitignore b/docs/.gitignore new file mode 100644 index 000000000..cb3900cd2 --- /dev/null +++ b/docs/.gitignore @@ -0,0 +1,2 @@ +.venv/ +build/ diff --git a/docs/Makefile b/docs/Makefile new file mode 100644 index 000000000..ed8809902 --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,20 @@ +# Minimal makefile for Sphinx documentation +# + +# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXOPTS ?= +SPHINXBUILD ?= sphinx-build +SOURCEDIR = . +BUILDDIR = build + +# Put it first so that "make" without argument is like "make help". +help: + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +.PHONY: help Makefile + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs/_static/css/custom.css b/docs/_static/css/custom.css index 0d59fcadf..c78c5ca75 100644 --- a/docs/_static/css/custom.css +++ b/docs/_static/css/custom.css @@ -1,4 +1,7 @@ .wy-table-responsive table td { white-space: normal; } +.underline { + text-decoration: underline; +} diff --git a/docs/_static/terminal.css b/docs/_static/terminal.css new file mode 100644 index 000000000..401d3b7e0 --- /dev/null +++ b/docs/_static/terminal.css @@ -0,0 +1,73 @@ +.terminal { + background-color: #000000; /* Fond noir */ + color: #00ff00; /* Texte vert, typique des anciens terminaux */ + font-family: 'Courier New', Courier, monospace; /* Police à chasse fixe */ + padding: 5px; /* Espace réduit autour du texte */ + border-radius: 5px; + border: 1px solid #00ff00; /* Bordure verte */ + white-space: pre-wrap; + overflow-x: auto; /* Défilement horizontal si nécessaire */ + box-shadow: 0 0 10px rgba(0, 255, 0, 0.5); /* Ombre verte pour un effet rétro */ + display: flex; /* Active Flexbox */ + line-height: 1.2; /* Espacement entre les lignes */ + display: inline-block; /* Pour que le fond s'adapte au contenu */ + width: 100%; /* Largeur maximale */ +} + +.terminal, +.terminal * { + border: none !important; /* Supprime toutes les bordures à l'intérieur du terminal */ +} + +.terminal .highlight { + margin: 0; /* Supprime les marges */ + padding: 0; /* Supprime les paddings */ + background-color: transparent; /* Fond transparent pour éviter les conflits */ +} + +.terminal .highlight pre { + margin: 0; /* Supprime les marges */ + padding: 0; /* Supprime les paddings */ + background-color: transparent; /* Fond transparent */ + color: inherit; /* Hérite la couleur du texte du parent */ + font-family: inherit; /* Hérite la police du parent */ + line-height: inherit; /* Hérite l'espacement des lignes du parent */ +} + +.terminal .highlight pre span { + display: inline; /* Évite les espaces inutiles causés par inline-block */ + margin: 0; /* Supprime les marges */ + padding: 0; /* Supprime les paddings */ +} + +.terminal .go { + color: #00ff00; /* Couleur pour le texte de sortie du terminal */ +} + +/* raw html output css */ + +/* Styles communs pour les deux classes */ +.error-box, .output { + padding: 10px; /* Espace intérieur */ + font-size: 0.8em; /* Taille de police plus petite */ + width: fit-content; /* Ajuste la largeur au contenu */ + border-radius: 5px; /* Coins arrondis */ + display: inline-block; /* Pour que la boîte s'ajuste au contenu */ + margin-bottom: 20px; +} + +.error-box pre, .output pre { + margin: 0; /* Supprime la marge par défaut du
 */
+}
+
+/* Styles spécifiques à la classe error-box */
+.error-box {
+    border: 2px solid #ff0000; /* Bordure rouge */
+    background-color: #ffe6e6; /* Fond légèrement rouge */
+}
+
+/* Styles spécifiques à la classe output */
+.output {
+    border: 2px solid #00ff00; /* Bordure verte */
+    background-color: #e6ffe6; /* Fond légèrement vert */
+}
diff --git a/docs/abstract.rst b/docs/abstract.rst
new file mode 100644
index 000000000..410389dba
--- /dev/null
+++ b/docs/abstract.rst
@@ -0,0 +1,345 @@
+Abstract presentation
+=========================
+
+Rougail is a robust and powerful, free/open-source configuration manager.
+
+Rougail is:
+
+- a CLI (command line interface) utility
+- a `Python `_ library
+- a YAML based description language
+
+The language combines declaration, data validation, and templating in a single, declarative syntax.
+
+It will be useful to:
+
+- define variables (the structure) easily and their constraints
+- loads those variables
+- generate variables documentation
+- loads variables' values
+- validate the variables and the overall consistency handling system
+- generate custom table views
+- export to different common format (YAML, Ansible, ...)
+- ...
+
+Why another configuration manager?
+-------------------------------------
+
+Using Rougail tansforms end user consumer defined consistency rules into highly consistent business objects.
+It's a configuration language designed to simplify and unify the way you define,
+validate and generate configuration data.
+
+In other word, making it easier to manage complex configurations across multiple environments.
+
+You might tell me that other configuration management tools do the same thing. And that's partly true.
+
+But Rougail adds interesting features in variable management that other projects don't have.
+We are of course referring to the documentation management included directly in the configuration manager.
+
+Your configuration is therefore consistent, easily accessible, and modifiable.
+
+Not to mention the always up-to-date documentation and the information you provide to your users regarding changes to variables.
+
+What kind of configuration manager?
+---------------------------------------------
+
+At the time of the design of Rougail, there were structuring choices
+that defined the functioning of the tool.
+
+The steps in Rougail can be explained as follows:
+
+- loading the variable's :term:`structured data`
+- loading the :term:`user data`
+- reading, validating, exporting, documenting (and so on...) the data
+
+Structured data
+~~~~~~~~~~~~~~~~~
+
+Structured data is also called the Rougail format.
+
+.. glossary::
+
+   structured data
+
+       A variable-first :term:`DSL` (Domain-Specific Language) designed for describing variables, consistency
+       and describe the relationships between variables in a declarative style.
+
+       The language is a mix of YAML and Jinja Templating.
+
+       It goes beyond traditional schema languages (like `JSON Schema `_,
+       `OpenAPI `_, `Cuelang `_)
+       by combinig type systems, powerful validation, consistency of the configuration and documentation.
+
+Structured data are commonly placed in :term:`structure files `.
+
+User data
+~~~~~~~~~~
+
+.. glossary::
+
+    user data
+
+        User data, as opposed to structured data, are data that only concern the assignment of values
+        and not the consistency of the variables between them.
+
+        The variable's values are also called **user values**.
+
+        The consistency field is outside of the user data scope.
+        The consistency is handled in the :term:`structured data `\ 's scope.
+
+Here a some user data examples:
+
+- configuration files
+- environment variables
+- external sources
+- command-line options
+- a form
+
+Output
+~~~~~~~
+
+.. glossary::
+
+   output
+
+        Structured and user data form a coherent configuration useful for different purposes.
+
+        Output is here to define what kind of data we want.
+
+Here are some output examples:
+
+- :term:`Tiramisu` object
+- JSON extraction
+- Ansible inventory extraction
+- documentation
+- custom table views
+
+.. list-table::
+   :header-rows: 1
+
+   * - **Step**
+
+   * - **Structured data**
+
+   * - **User data**
+
+   * - **Output**
+
+.. _configuration:
+
+Configuration
+~~~~~~~~~~~~~
+
+We've been talking about `configuration` for a while now. It's time to define what a configuration is.
+
+.. glossary::
+
+    configuration
+
+        The configuration represents the set of variables and their values loaded in a tree-like fashion into memory from structural data, user data and any modifications during its lifecycle.
+
+        In practice, it's :term:`Tiramisu` who manages this part.
+
+What kind of actor?
+---------------------
+
+It's clear that Rougail can be used in many contexts.
+
+Here we'll define actor names. Obviously, these aren't the only possible actors.
+
+We're just defining the actors within the Rougail context.
+Choice your own actor name if you wish to personnalize in your specific context.
+
+.. glossary::
+
+    integrator
+
+        An integrator in the Rougail field is the person who writes the :term:`structured `\ .
+        He has the responsibility of the integration process, that is,
+        he defines the variables and the relationship between them, the variables that are allowed
+        (or not) to be set, and so on. His responsabilites are the **structuration** and the **consistency**
+        of the organisation of the variables between them.
+
+.. glossary::
+
+    operator
+
+        An operator in the Rougail field is the person who assigns :term:`value`\ s to the pre-defined variables,
+        his responsabilities are to set variable values correctly.
+
+        The user :term:`value`\ s, that is the values that have been set by the :term:`operator`, are of course type validated.
+        The type validation is driven by the definitions in the :term:`structure file `.
+
+Here is a reminder of the different steps:
+
+- the :term:`integrator` defines the structure
+- the :term:`operator` sets the value
+- the :term:`integrator` and/or the :term:`operator` use the output
+
+.. list-table::
+   :header-rows: 1
+
+   * - **Step**
+     - Actor
+
+   * - **Structured data**
+     - Integrator
+
+   * - **User data**
+     - Operator
+
+   * - **Output**
+     - - Operator
+       - Integrator
+
+Variable lifecyle
+----------------------
+
+Rougail's a configuration language and data validation tool is designed 
+to simplify defining, validating, and generating structured configuration and data.
+
+Rougail aims at defining variables.
+
+Here we are talking about the variable lifecyle.
+
+The variable’s lifecyle is the period between its creation and its destruction.
+
+The lifecycle of a variable includes the generic stages (for example, like in the C language):
+
+- Creation: variables are assigned a name and a type
+- Initialization: they are assigned their first value (we call it :term:`default value`)
+- Assignment: the variable's value is modified
+- Reading: the variable's value is used
+- Destruction: the variable terminates upon the destruction of the object
+
+.. note:: No other concept is included in the lifecycle (ni particular the access control, with properties describing access constraints).
+
+.. list-table::
+   :header-rows: 1
+
+   * - **Step**
+     - Actor
+     - Lifecyle
+
+   * - **Structured data**
+     - Integrator
+     - - Creation
+       - Initialization
+
+   * - **User data**
+     - Operator
+     - - Reading
+       - Assignment
+       - Access control
+
+
+   * - **Output**
+     - - Operator
+       - Integrator
+     - - Reading
+       - Access control
+
+.. _variable_mutability:
+
+Variable mutability
+---------------------
+
+.. _structured_data:
+
+Structured data
+~~~~~~~~~~~~~~~~
+
+When the :term:`integrator` defines the structure, variables are mutable.
+
+Even if the default behavior is inconsistent, like when there is a conflict between multiple declarations for the same variable,
+
+it's possible to explicitly allow to unify (combine) multiple variables declarations.
+
+User data
+~~~~~~~~~~
+
+At the user data level, it's no more possible to modify any variable definition.
+Here we are just talking about variable's values.
+
+Output
+~~~~~~~
+
+Variable definition settings are immutable.
+
+.. list-table::
+   :header-rows: 1
+
+   * - **Step**
+     - Actor
+     - Lifecyle
+     - Mutability
+
+   * - **Structured data**
+     - Integrator
+     - - Creation
+       - Initialization
+     - Mutable
+
+   * - **User data**
+     - Operator
+     - - Reading
+       - Assignment
+       - Access control
+     - Immutable
+
+   * - **Output**
+     - - Operator
+       - Integrator
+     - - Reading
+       - Access control
+     - Immutable
+
+Value access
+-------------
+
+In the :term:`structured data` step, there is no value, strictly speaking. 
+The values set are default values.
+Like other parameters, the default value is explicitly mutable.
+
+In the `user data` step, we can modify the values. That's precisely the purpose of this step.
+The configuration is said to be in :term:`read write mode`.
+
+But at the `output` step, it is obviously no longer possible to modify the value.
+The configuration is said to be in :term:`read only mode`.
+
+.. attention::
+   It is important not to confuse `value` and `calculated value`.
+   The result of a calculation can change over time.
+   In this case, the value does indeed correspond to the result of the calculation.
+
+.. list-table::
+   :header-rows: 1
+
+   * - **Step**
+     - Actor
+     - Lifecyle
+     - Mutability
+     - Value
+
+   * - **Structured data**
+     - Integrator
+     - - Creation
+       - Initialization
+     - Mutable
+     - Mutable default value
+
+   * - **User data**
+     - Operator
+     - - Reading
+       - Assignment
+       - Access control
+     - Immutable
+     - Read write
+
+   * - **Output**
+     - - Operator
+       - Integrator
+     - - Reading
+       - Access control
+     - Immutable
+     - Read only
diff --git a/docs/bribes.txt b/docs/bribes.txt
deleted file mode 100644
index 2f1e0205b..000000000
--- a/docs/bribes.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-variable
- `_
diff --git a/docs/check.rst b/docs/check.rst
deleted file mode 100644
index 82ccd7f3f..000000000
--- a/docs/check.rst
+++ /dev/null
@@ -1,286 +0,0 @@
-Verification function
-==========================
-
-Synopsis
--------------
-
-A verification is a complementary validation to the type which allows the content of a variable to be validated more precisely.
-
-A :term:`validator` is necessarily a Jinja type calculation.
-
-Parameters
---------------
-
-Depending on the types of calculation, the parameters will be different:
-
-.. list-table:: 
-   :widths: 15 25 20 15
-   :header-rows: 1
-   
-   * - **Calculation type**
-     - **Parameter**
-     - **Comments**
-     - **Sample**
-     
-   * - 
-     - **type** 
-     
-       `string`
-       
-       `mandatory`
-     - Type of calculation, the only possible value is: jinja    
-     - jinja
-   * - **jinja** 
-     
-       `string`
-       
-       `mandatory`
-     - Jinja template
-     - {% if rougail.variable == 'not_allowed' %}not allowed!{% endif %}
-     - 
-   * - **params**
-       
-       `list`
-     - Additional parameters passed to the Jinja template
-     - 
-     - 
-
-There are two types of parameter:
-
-- the standard parameters (string, boolean, integer, null), in this case just do: "key: value"
-- advanced settings:
-  
-  - parameter via a variable
-  - parameter via an information
-  - parameter via a suffix: in the case of a variable in a dynamic family
-  - parameter via an index: in the case of a :term:`follower` variable
-  
-.. list-table:: 
-   :widths: 15 25 20 15
-   :header-rows: 1
-   
-   * - **Parameter's type**
-     - **Parameter**
-     - **Comments**
-     - **Sample**
-   * - 
-     - **name** 
-     
-       `string`
-       
-       `mandatory`
-     - Parameter's name 
-     - my_param
-   * - 
-     - **type** 
-     
-       `string`
-       
-       `mandatory`
-     - Type of parameter, possible values are: variable, information, suffix or index
-     - suffix 
-   * - Variable
-     - **variable** 
-     
-       `string`
-       
-       `mandatory`
-     - variable's name
-     - rougail.variable
-   * - Variable (`mandatory`) information
-     - **propertyerror**
-     
-       `boolean`
-     - If access to the variable is not possible due to a property 
-       (for example `disabled`) by default an error is returned. 
-       If the attribute is `False`, the parameter is not passed to the Jinja template.
-       
-       **Default value**: `True`
-     - True  
-   * - Information
-     - **information** 
-     
-       `string`
-       
-       `mandatory`
-     - Name of the information whose value we want to retrieve.
-     - doc
-
-Samples
---------------
-
-Strict verification of values
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Here is a simple example of validating values:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_variable:
-      validators:
-        - type: jinja
-          jinja: |
-            {% if rougail.my_variable and not rougail.my_variable.islower() %}
-            {{ rougail.my_variable }} is not lowercase string
-            {% endif %}        
-     
-
-A verification function must take into account 2 important aspects:
-
-- the value may not be entered (even if the variable is mandatory), the None value must be taken into account
-- if the value is invalid, a sentence must be returned with an explicit message.
-
-From now on only `None` and lowercase values will be allowed.       
-
-Checking values with warning
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-In the constraint, it is possible to specify the error level and put it as a warning:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_variable:
-      validators:
-        - type: jinja
-          jinja: |+
-            {% if rougail.my_variable and not rougail.my_variable.islower() %}
-            {{ rougail.my_variable }} is not lowercase string
-            {% endif %}        
-          params:
-            warnings_only: true
-
-In this case a value with a capital letter will be accepted, but a warning message will appear.
-
-Verification with parameters
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_hidden_variable:
-      disabled: true
-    my_variable:
-      validators:
-        - type: jinja
-          jinja: |
-            {% if param1 is defined and rougail.my_variable == param1 %}
-            has same value as rougail.unknown_variable
-            {% endif %}
-            {% if param2 is defined and rougail.my_variable == param2 %}
-            has same value as rougail.my_hidden_variable
-            {% endif %}        
-          params:
-            param1:
-              type: variable
-              variable: rougail.unknown_variable
-              optional: true
-            param2:
-              type: variable
-              variable: rougail.my_hidden_variable
-              propertyerror: false
-
-An example with a suffix type parameter:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    varname:
-      multi: true
-      default:
-        - val1
-        - val2
-    my_dyn_family_:
-      type: dynamic
-      variable: rougail.varname
-      description: 'Describe '
-      my_dyn_var:
-        type: string
-        validators:
-          - type: jinja
-            jinja: |
-              {% if rougail.my_dyn_family_.my_dyn_var == param1 %}
-              forbidden!
-              {% endif %}          
-            params:
-              param1:
-                type: suffix
-
-In this example, we see a dynamic family. Two families will be created: `rougail.my_dyn_family_val1.my_dyn_var` and `rougail.my_dyn_family_val2.my_dyn_var`.
-
-The value of the variable within this family cannot be equal to the value 
-of the suffix (`val1` and `val2` respectively).
-
-An example with an index type parameter:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    family:
-      type: leadership
-      leader:
-        multi: true
-        default:
-          - val1
-          - val2
-      follower1:
-        type: number
-        validators:
-          - type: jinja
-            jinja: |
-              {% if rougail.family.follower1 == param1 %}
-              forbidden!
-              {% endif %}          
-            params:
-              param1:
-                type: index
-
-Redefinition
----------------
-
-In a first dictionary, let's declare our variable and its verification function:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_variable:
-      validators:
-        - type: jinja
-          jinja: |
-            {% if rougail.my_variable and not rougail.my_variable.islower() %}
-            {{ rougail.my_variable }} is not lowercase string
-            {% endif %}        
-
-In a second dictionary it is possible to redefine the calculation:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_variable:
-      redefine: true
-      validators:  
-        - type: jinja
-          jinja: |
-            {% if rougail.my_variable and ' ' in rougail.my_variable %}
-            {{ rougail.my_variable }} has a space
-            {% endif %}        
-
-In this case only this validator will be executed.
-
-Here is a third dictionary in which we remove the validation:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_variable:
-      redefine: true
-      validators:
diff --git a/docs/cli.rst b/docs/cli.rst
new file mode 100644
index 000000000..d1abe6d22
--- /dev/null
+++ b/docs/cli.rst
@@ -0,0 +1,207 @@
+The Rougail Command Line Interface
+========================================
+
+help
+------------
+
+::
+
+    rougail -u yaml -h
+
+Standard usage 
+-----------------
+
+::
+
+    rougail -m firefox/ -u yaml -yf config/02/config.yml
+
+
+
+::
+
+    rougail --cli.versions 
+
+    tiramisu: 5.2.0a9
+    tiramisu-cmdline-parser: 0.7.0a1
+    rougail: 1.2.0a29
+    rougail-cli: 0.2.0a19
+    rougail-user-data-environment: 0.1.0a9
+    rougail-user-data-yaml: 0.2.0a11
+    rougail-output-console: 0.2.0a11
+    rougail-output-json: 0.2.0a8
+
+::
+
+    env ROUGAIL_MANUAL.USE_FOR_HTTPS=true rougail -m structfile/proxy2.yml -u yaml  environment --yaml.filename userdata/proxy.yml -o json
+
+::
+
+    env ROUGAIL_MANUAL.USE_FOR_HTTPS=true rougail -m structfile/proxy2.yml -u yaml --yaml.filename userdata/proxy.yml -o json --json.get manual.https_proxy --json.read_write
+    {
+      "address": "toto.fr",
+      "port": "8888"
+    }
+
+CLI root
+-------------
+
+::
+
+    (.venv) ubuntu@monessai:~/workplace/rougail-tutorials$ env ROUGAIL_MANUAL.USE_FOR_HTTPS=true rougail -m firefox/ --cli.root manual.https_proxy -u yaml environment -yf ^Cnfig/01/config.yml --cli.read_write
+    (.venv) ubuntu@monessai:~/workplace/rougail-tutorials$ cat config/01/config.yml 
+    ---
+    proxy_mode: Manual proxy configuration
+    manual:
+      http_proxy:
+        address: http.proxy.net
+        port: 3128
+      use_for_https: false
+      https_proxy:
+        address: https.proxy.net
+    (.venv) ubuntu@monessai:~/workplace/rougail-tutorials$ rougail -m firefox/ --cli.root manual.https_proxy  -u yaml -yf config/01/config.yml
+    [WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg.
+    [DEPRECATION WARNING]: Importing 'to_bytes' from 'ansible.module_utils._text' is deprecated. This feature will be removed from ansible-core version 2.24. Use ansible.module_utils.common.text.converters instead.
+    ╭────────────── Caption ───────────────╮
+    │ Variable Default value               │
+    │          Modified value              │
+    │          (⏳ Original default value) │
+    ╰──────────────────────────────────────╯
+    Variables:
+    ┣━━ 📓 HTTPS address: https.proxy.net ◀ loaded from the YAML file "config/01/config.yml" (⏳ https.proxy.net ◀ loaded from the YAML file "config/01/config.yml" ⏳ http.proxy.net)
+    ┗━━ 📓 HTTPS port: 3128
+
+
+.. FIXME: sortie normale: 
+
+.. FIXME: sortie pour n'afficher qu'une family: 
+
+::
+
+    rougail -m firefox/ -u yaml -yf config/01/config
+
+on a:
+
+::
+
+    Variables:
+    ┣━━ 📓 Configure Proxy Access to the Internet: Manual proxy configuration ◀ loaded from the YAML file "config/01/config.yml" (⏳ No proxy)
+    ┗━━ 📂 Manual proxy configuration
+        ┣━━ 📂 HTTP Proxy
+        ┃   ┣━━ 📓 HTTP address: http.proxy.net ◀ loaded from the YAML file "config/01/config.yml"
+        ┃   ┗━━ 📓 HTTP Port: 3128 ◀ loaded from the YAML file "config/01/config.yml" (⏳ 8080)
+        ┣━━ 📓 Also use this proxy for HTTPS: false ◀ loaded from the YAML file "config/01/config.yml" (⏳ true)
+        ┣━━ 📂 HTTPS Proxy
+        ┃   ┣━━ 📓 HTTPS address: https.proxy.net ◀ loaded from the YAML file "config/01/config.yml" (⏳ http.proxy.net)
+        ┃   ┗━━ 📓 HTTPS port: 3128
+        ┗━━ 📂 SOCKS Proxy
+            ┣━━ 📓 SOCKS address: http.proxy.net
+            ┣━━ 📓 SOCKS port: 3128
+            ┗━━ 📓 SOCKS host version used by proxy: v5
+
+
+for a more precise output: 
+
+::
+
+    rougail -m firefox/ -u yaml -yf config/01/config.yml --cli.root manual.https_proxy
+
+we have
+
+::
+
+    Variables:
+    ┣━━ 📓 HTTPS address: https.proxy.net ◀ loaded from the YAML file "config/01/config.yml" (⏳ https.proxy.net ◀ loaded from the YAML file "config/01/config.yml" ⏳ http.proxy.net)
+    ┗━━ 📓 HTTPS port: 3128
+
+
+Invalid user data error
+-----------------------------
+
+::
+
+    rougail -m firefox/ -u yaml -yf config/02/config.yml --cli.invalid_user_data_error
+
+Unknown user data error 
+-----------------------------
+
+The option "`--cli.unknown_user_data_error`" looks like "`--cli.invalid_user_data_error`" but here it concerns unknown variables (or here disabled or hidden but it amounts to the same thing) loaded into user data.
+
+
+.. note:: The `--cli.unknown_user_data_error` option changes the behaviour of the Rougail CLI's standard output: 
+          when an unknown (or disabled or hidden) variable is declared in the :term:`user data file ` 
+          then it appears in the output as an error instead of a warning.
+
+The output doc 
+----------------
+
+::
+
+    rougail -m structure.yml -o doc --doc.output_format github > README.md 
+
+::
+
+      doc:
+      Generate documentation from structural files
+
+      -do [{console,asciidoc,html,github,gitlab,json}], --doc.output_format [{console,asciidoc,html,github,gitlab,json}]
+                            The output format of the generated documentation (default: console)
+      -dm [{two_columns,three_columns,four_columns,five_columns,six_columns}], --doc.tabular_template [{two_columns,three_columns,four_columns,five_columns,six_columns}]
+                            Generate document with this tabular model (default: two_columns)
+      -dc {variables,example,changelog} [{variables,example,changelog} ...], --doc.contents {variables,example,changelog} [{variables,example,changelog} ...]
+                            Generated content (default: variables)
+      -dt [TITLE_LEVEL], --doc.title_level [TITLE_LEVEL]
+                            Starting title level (default: 1)
+      --doc.default_values  Modify values to document all variables (default: True)
+      --doc.no-default_values
+      --doc.document_a_type
+                            Documentation a structural type (default: False)
+      --doc.no-document_a_type
+
+
+The formatter
+----------------
+
+::
+
+    rougail -o formatter -m structure.yml > a.yml; mv a.yml structure.yml
+
+The namespaces
+-----------------
+
+::
+
+    rougail -m structure.yml -u yaml -yf userdata.yml -s intranet -xn hosts -xd 0 hosts.yml
+
+
+::
+
+    rougail -m structure.yml -u yaml -yf userdata.yml -s intranet -xn hosts orem srem prom -xd 0 hosts.yml -xd 1 packages/OREM -xd 2 packages/SREM -xd 3 packages/SREP
+
+Commandline user data
+----------------------------
+
+Commandline user data can be used:
+
+::
+
+    rougail -m mon_fichier.yml -u commandline --family.variable pouet
+
+Ansible export 
+-------------------
+
+rougail -m structure.yml -u yaml -yf userdata.yml -s intranet -xn hosts -xd 0 hosts.yml -o ansible > ansible.json
+
+:: 
+
+    ansible-inventory -i $(which rougail) --list
+
+Ansible lists your inventory
+
+::
+
+    ansible-inventory -i $(which rougail) --host tutu.fr 
+
+::
+
+    ansible-playbook -i $(which rougail) install.yml
+
diff --git a/docs/condition.rst b/docs/condition.rst
deleted file mode 100644
index 3d1895639..000000000
--- a/docs/condition.rst
+++ /dev/null
@@ -1,240 +0,0 @@
-Calculated properties
-==========================
-
-Synopsis
-------------
-
-Calculated properties allow you to add or remove properties to a :term:`variable`
-or a :term:`family` depending on the context.
-
-Here is the list of editable properties:
-
-.. list-table:: 
-   :widths: 15 15 25
-   :header-rows: 1
-   
-   * - **Attribute applicable on**
-     - **Property's name**
-     - Comment
-     
-   * - Variable
-   
-       Family 
-     - hidden
-     - Hides a variable or a family, in this case it is not accessible in `read-write` mode, 
-       but remains accessible in a calculation or in `read-only` mode
-   * - Variable
-   
-       Family 
-     - disabled
-     - Deactivates a variable or family, in this case it is never accessible
-   * - Variable
-     - mandatory
-     - The variable expects a value other than `None` or `[]` for multiple variables
-
-A property can be calculated. In this case we have two possibilities:
-
-- calculation via Jinja
-- calculation via a variable          
-
-Parameters
----------------
-
-.. list-table:: 
-   :widths: 15 25 20 15
-   :header-rows: 1
-   
-   * - **Calculation type**
-     - **Parameter**
-     - **Comment**
-     - **Sample**
-   * - 
-     - **type**
-     
-       `string`
-       
-       `mandatory`
-     - Calculation type, possible values are: jinja, variable, information, suffix or index  
-     - jinja
-   * - Jinja 
-     - **jinja**
-     
-       `string`
-       
-       `mandatory`
-     - Jinja template . For a multiple variable, each line represents a value.
-     - {% if rougail.variable %}
-
-       {{ rougail.variable }}
-
-       {% endif %}
-   * - Jinja
-     - **params**
-     
-       `list`
-     - Additional parameters passed to the Jinja template
-     - 
-   * - Variable
-     - **variable**
-     
-       `string`
-       
-       `mandatory`
-     - Name of the associated variable.
-     
-       .. attention:: The variable must be of `boolean` type.
-     - rougail.variable 
-   * - Variable
-     - **propertyerror**
-     
-       `boolean` 
-     - If access to the variable is not possible due to a property 
-       (for example `disabled`) by default an error is returned. 
-       If the attribute is `False`, the calculated value is False.
-
-       **Default value**: `True`
-     - False
-
-In the case of a Jinja type calculation, it is possible to have parameters.
-
-There are two types of parameter:
-
-- the standard parameters (string, boolean, integer, null), in this case just do: "key: value"
-
-- advanced settings:
-
-   - parameter via a variable
-   - parameter via information
-   - parameter via a suffix: in the case of a variable in a dynamic family
-   - parameter via an index: in the case of a follower variable       
-   
-.. list-table:: 
-   :widths: 15 25 20 15
-   :header-rows: 1
-   
-   * - **Parameter's type**
-     - **Parameter**
-     - **Comments**
-     - **Sample**
-   * - 
-     - **name**
-     
-       `string`
-       
-       `mandatory`
-     - parameter's name 
-     - my_param
-   * - 
-     - **type**
-     
-       `string`
-       
-       `mandatory`
-     - Parameter's type, possible values are: variable, information, suffix or index
-     - suffix
-   * - Variable 
-     - **variable**
-     
-       `string`
-       
-       `mandatory`
-     - variable's name 
-     - rougail.variable
-   * - Variable (`mandatory`) Information   
-     - **propertyerror**
-     
-       `boolean`
-     - If access to the variable is not possible due to a property (for example `disabled`) by default an error is returned. If the attribute is False, the parameter is not passed to the Jinja template.
-
-       **Default value**: `True`
-     - False
-   * - Variable
-     - **optional**
-     
-       `boolean`
-     - The variable may not exist depending on YAML file imports. If the optional parameter is True, the parameter will simply be deleted if the variable does not exist.
-       **Default value**: `False`
-     - True
-     
-   * - information
-     - **information**
-     
-       `string`
-       
-       `mandatory`
-     - Name of the information whose value we want to retrieve.
-     - doc
-
-Samples
-------------
-
-A Jinja-type calculated property
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-It is possible to write the condition in Jinja:
-
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    condition:
-      default: 'do not hide!'
-    my_variable:
-      hidden:
-        type: jinja
-        jinja: |
-          {% if rougail.condition and rougail.condition == "hide!" %}
-          this rougail.condition value is 'hide!'
-          {% endif %}      
-          
-           
-In this case the variable is hidden if the value of the variable "rougail.condition" is `hide!` and it did not hide for any other value. Be careful, always take into consideration that "rougail.condition" can be equal to `None`.
-
-The message returned by the function is visible in the error message in the event of an access problem:                  
-
-.. code-block:: python
-
-    >>> from rougail import Rougail, RougailConfig
-    >>> RougailConfig['dictionaries_dir'] = ['dict']
-    >>> rougail = Rougail()
-    >>> config = rougail.get_config()
-    >>> config.property.read_write()
-    [..]
-    tiramisu.error.PropertiesOptionError: cannot access to option "my_variable" because has property "hidden" (this rougail.condition value is 'hide!')
-
-It is possible to use parameters when calculating properties as for calculating the `default` attribute.
-
-A calculated property of variable type 
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-A variable can therefore be calculated via the result of another variable. Please note, this other variable must be of `boolean` type:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    condition:
-      type: boolean
-    my_variable:
-      hidden:
-        type: variable
-        variable: rougail.condition
-
-If the value of the variable "rougail.condition" is `True` then the variable "rougail.my_variable" is hidden.
-
-Redefintion
-~~~~~~~~~~~~~~~~~
-
-It may be that in a dictionary we decide to define a condition.
-
-To delete the calculation from a variable, simply do in a new dictionary:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_variable:
-      redefine: true
-      hidden:
-
diff --git a/docs/conf.py b/docs/conf.py
index 8b2a7e831..32a502e00 100755
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -1,24 +1,20 @@
 # Configuration file for the Sphinx documentation builder.
 #
-# This file does only contain a selection of the most common options. For a
-# full list see the documentation:
-# http://www.sphinx-doc.org/en/master/config
-
 # -- Path setup --------------------------------------------------------------
 
-# If extensions (or modules to document with autodoc) are in another directory,
-# add these directories to sys.path here. If the directory is relative to the
-# documentation root, use os.path.abspath to make it absolute, like shown here.
-#
-# import os
-# import sys
+import sys, os
 # sys.path.insert(0, os.path.abspath('.'))
+#sys.path.append(os.path.abspath('ext'))
+sys.path.append('.')
 
+#---- debug mode ----
+# shows/hides the todos
+todo_include_todos = False
 
 # -- Project information -----------------------------------------------------
 
 project = 'Rougail'
-copyright = '2019-2023, Silique'
+copyright = '2019-2026, Silique'
 author = 'gwen'
 
 # The short X.Y version
@@ -33,55 +29,64 @@ release = '1.0'
 #
 # needs_sphinx = '1.0'
 
-# Add any Sphinx extension module names here, as strings. They can be
-# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
-# ones.
-
 extensions = [
-        'sphinx.ext.extlinks', 'sphinx_lesson',
-        #'myst_parser', 'sphinx.ext.extlinks'
+        'sphinx.ext.extlinks', 'sphinx_lesson', 'sphinx.ext.todo',
+        'ext.xref', 'ext.extinclude', 'sphinxnotes.strike',
 ]
-#
-#myst_enable_extensions = [
-#    "amsmath",
-#    "attrs_inline",
-#    "colon_fence",
-#    "deflist",
-#    "dollarmath",
-#    "fieldlist",
-#    "html_admonition",
-#    "html_image",
-##    "linkify",
-#    "replacements",
-#    "smartquotes",
-#    "strikethrough",
-#    "substitution",
-#    "tasklist",
-#]
 
+#---- disable highlight warnings with yaml new version ----
+# Configuration pour les blocs de code
+highlight_language = 'yaml'
 
+# Options spécifiques pour YAML
+highlight_options = {
+    'yaml': {
+        'startinline': True
+    }
+}
+
+suppress_warnings = [
+    'misc.highlighting_failure'
+]
+
+#---- xref links ----
+#import the xref.py extension
+xref_links = {"link_name" : ("user text", "url")}
+
+#link_name = "Sphinx External Links"
+#user_text = "modified External Links Extension"
+#url = "http://www.sphinx-doc.org/en/stable/ext/extlinks.html"
+#enables syntax like:
+" :xref:`tiramisu` "
+links = {
+        'tiramisu': ('Tiramisu', 'https://tiramisu.readthedocs.io/en/latest/'),
+        'tiramisu library': ('Tiramisu library homepage', 'https://forge.cloud.silique.fr/stove/tiramisu'),
+        }
+xref_links.update(links)
+
+#---- ext links ----
 # **extlinks** 'sphinx.ext.extlinks',
-# enables syntax like :proxy:`my source ` in the src files
-extlinks = {'proxy': ('/proxy/%s.html',
-                      'external link: ')}
+# enables syntax like
+" :source:`v1.1_010/firefox/00-proxy.yml` "
+extlinks = {'source': ('https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/%s',
+                       'source: %s'),
+            'tiramisu': ('https://tiramisu.readthedocs.io/en/latest/%s', 'tiramisu: %s'),
+            'tutorial': ('https://forge.cloud.silique.fr/stove/rougail-tutorials/%s', 'tutorial %s'),
+            }
 
+#---- options for HTML output ----
 default_role = "code"
-
 html_theme = "sphinx_rtd_theme"
-
 pygments_style = 'sphinx'
-
 html_short_title = "Rougail"
-html_title = "Rougail documenation"
-
-# If true, links to the reST sources are added to the pages.
+html_title = "Rougail documentation"
 html_show_sourcelink = False
-
-# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
 html_show_sphinx = False
-
-# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
 html_show_copyright = True
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = None
+html_static_path = ['_static']
+html_css_files = ['terminal.css', 'custom.css']
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
@@ -109,41 +114,21 @@ language = 'en'
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This pattern also affects html_static_path and html_extra_path.
-exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+exclude_patterns = ['.venv', 'build', '_build', 'Thumbs.db', '.DS_Store']
 
-# The name of the Pygments (syntax highlighting) style to use.
-pygments_style = None
+# ------------------------------------------------------------------------------
+## underline role
+## sample: Voici du texte :underline:`souligné`.
 
+from docutils import nodes
+from docutils.parsers.rst import roles
 
-# -- Options for HTML output -------------------------------------------------
-
-# The theme to use for HTML and HTML Help pages.  See the documentation for
-# a list of builtin themes.
-#
-#html_theme = 'alabaster'
-# **themes**
-#html_theme = 'bizstyle'
-
-# Theme options are theme-specific and customize the look and feel of a theme
-# further.  For a list of options available for each theme, see the
-# documentation.
-#
-# html_theme_options = {}
-
-# Add any paths that contain custom static files (such as style sheets) here,
-# relative to this directory. They are copied after the builtin static files,
-# so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
-
-# Custom sidebar templates, must be a dictionary that maps document names
-# to template names.
-#
-# The default sidebars (for documents that don't match any pattern) are
-# defined by theme itself.  Builtin themes are using these templates by
-# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
-# 'searchbox.html']``.
-#
-# html_sidebars = {}
+def underline_role(name, rawtext, text, lineno, inliner, options={}, content=[]):
+    node = nodes.inline(rawtext, text, **options)
+    node['classes'].append('underline')
+    return [node], []
 
 def setup(app):
-   app.add_css_file('css/custom.css')
+    app.add_role('underline', underline_role)
+    app.add_css_file('css/custom.css')
+    
diff --git a/docs/configuration.rst b/docs/configuration.rst
index 5f067482f..cc96b7a45 100644
--- a/docs/configuration.rst
+++ b/docs/configuration.rst
@@ -40,7 +40,7 @@ The functions file
 
 The file which contains the custom functions is managed in the `functions_file` key and has the default value `/srv/rougail/functions.py`. This key can contain a list if there are several files.
 
-.. important:: Functions must return a value, even if the variable being calculated is a :term:`multiple` variable. If the function can return a multiple value (a list), you must put the name of the function in the `multi_functions` key.
+.. important:: Functions must return a value, even if the variable being calculated is a :term:`multiple ` variable. If the function can return a multiple value (a list), you must put the name of the function in the `multi_functions` key.
 
 The `auto_freeze` variable
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -50,15 +50,11 @@ The `auto_freeze` property is only applied until a specific variable becomes `Tr
 Modes
 ~~~~~~~~
 
-.. glossary::
-
-   mode
+A mode is a solution that allows a variable or a family to be categorized. It can be used, for example, to define a level of expertise or a particular context. Have a look at the :term:`mode definition ` for further details.
    
-       modes are views on variables. 
-   
-       Modes are customizable in Rougail. By default the modes are `basic`, `standard` and `advanced`. It is possible to change this list via the `modes_level` key.
+Modes are customizable in Rougail.
 
-If you change these values, consider changing the default modes of families and variables in your dictionaries. 
+Often, we categorize by level of expertise to access a variable: `basic`, `standard` and `advanced`.
 
 Default mode for a family 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
diff --git a/docs/developer.rst b/docs/developer.rst
index 8fafd92fb..225fd519a 100644
--- a/docs/developer.rst
+++ b/docs/developer.rst
@@ -58,3 +58,285 @@ We use black
       - id: black
 
 And some YAML and JSON validators.
+
+
+----
+
+The pre commit tool
+----------------------
+
+**pre-commit** is a fantastic tool for automating code quality checks
+before you commit your changes. It’s especially useful in Python
+projects for running linters, formatters, and other checks
+automatically.
+
+Here’s a **practical, minimal example** to get you started with a small
+Python project.
+
+
+1. Install pre-commit
+---------------------
+
+First, install the ``pre-commit`` package:
+
+.. code:: bash
+
+   pip install pre-commit
+
+
+2. Create a ``.pre-commit-config.yaml`` File
+--------------------------------------------
+
+In the root of your Python project, create a file named
+``.pre-commit-config.yaml`` with the following content:
+
+.. code:: yaml
+
+   repos:
+     - repo: https://github.com/pre-commit/pre-commit-hooks
+       rev: v4.4.0
+       hooks:
+         - id: trailing-whitespace
+         - id: end-of-file-fixer
+         - id: check-yaml
+         - id: check-added-large-files
+
+     - repo: https://github.com/psf/black
+       rev: 23.12.1
+       hooks:
+         - id: black
+
+     - repo: https://github.com/pycqa/flake8
+       rev: 6.1.0
+       hooks:
+         - id: flake8
+
+This config: - Uses popular pre-commit hooks for basic checks - Adds
+`Black `__ for code formatting - Adds
+`Flake8 `__ for linting
+
+
+3. Install the Git Hook
+-----------------------
+
+Run this command in your project root:
+
+.. code:: bash
+
+   pre-commit install
+
+This sets up a Git hook that runs the checks before each commit.
+
+
+4. Try It Out
+-------------
+
+Now, when you try to commit:
+
+.. code:: bash
+
+   git add .
+   git commit -m "My commit message"
+
+pre-commit will run the hooks. If any check fails, you’ll see output
+like this:
+
+::
+
+   Trim Trailing Whitespace............................................Passed
+   Fix End of Files....................................................Passed
+   Check Yaml..........................................................Passed
+   Check for added large files..........................................Passed
+   Black..............................................................Failed
+   - hook id: black
+   - exit code: 1
+
+   Reformatted /path/to/your/file.py
+   All done! ✨ 🍰 ✨
+   1 file reformatted.
+
+If any hook fails, fix the issues and try committing again.
+
+
+5. (Optional) Run pre-commit Manually
+-------------------------------------
+
+You can run all hooks against all files at any time:
+
+.. code:: bash
+
+   pre-commit run --all-files
+
+
+6. Update Hooks
+---------------
+
+To update your hooks to the latest versions:
+
+.. code:: bash
+
+   pre-commit autoupdate
+
+**Summary Table**
+~~~~~~~~~~~~~~~~~
+
+==== ===================== ==============================
+Step Action                Command
+==== ===================== ==============================
+1    Install pre-commit    ``pip install pre-commit``
+2    Create config file    ``.pre-commit-config.yaml``
+3    Install Git hook      ``pre-commit install``
+4    Commit and see checks ``git commit -m "..."``
+5    Run manually          ``pre-commit run --all-files``
+6    Update hooks          ``pre-commit autoupdate``
+==== ===================== ==============================
+
+more python linters
+-------------------
+
+Customizing your ``.pre-commit-config.yaml`` to include **pylint**,
+**ruff**, and additional hooks is a great way to maintain code quality.
+Here’s how you can structure your config for a robust setup:
+
+**1. Basic Setup with pylint and ruff**
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Here’s a starting point for your ``.pre-commit-config.yaml``:
+
+.. code:: yaml
+
+   repos:
+     - repo: https://github.com/pre-commit/pre-commit-hooks
+       rev: v4.4.0
+       hooks:
+         - id: trailing-whitespace
+         - id: end-of-file-fixer
+         - id: check-yaml
+         - id: check-added-large-files
+
+     - repo: https://github.com/astral-sh/ruff-pre-commit
+       rev: v0.1.6
+       hooks:
+         - id: ruff
+           args: [--fix, --exit-non-zero-on-fix]
+
+     - repo: https://github.com/pycqa/pylint
+       rev: v3.0.3
+       hooks:
+         - id: pylint
+           args: [--rcfile=.pylintrc]  # Optional: specify a pylint config file
+
+**2. Adding More Hooks**
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Here are some popular hooks you might want to add:
+
+**a. Black (Code Formatter)**
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code:: yaml
+
+     - repo: https://github.com/psf/black
+       rev: 23.12.1
+       hooks:
+         - id: black
+
+**b. Mypy (Static Type Checker)**
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code:: yaml
+
+     - repo: https://github.com/pre-commit/mirrors-mypy
+       rev: v1.8.0
+       hooks:
+         - id: mypy
+
+**c. isort (Import Sorter)**
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code:: yaml
+
+     - repo: https://github.com/pycqa/isort
+       rev: 5.13.2
+       hooks:
+         - id: isort
+
+**d. Bandit (Security Linter)**
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code:: yaml
+
+     - repo: https://github.com/pycqa/bandit
+       rev: 1.7.7
+       hooks:
+         - id: bandit
+
+**3. Full Example Config**
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Here’s a full example with all the above hooks:
+
+.. code:: yaml
+
+   repos:
+     - repo: https://github.com/pre-commit/pre-commit-hooks
+       rev: v4.4.0
+       hooks:
+         - id: trailing-whitespace
+         - id: end-of-file-fixer
+         - id: check-yaml
+         - id: check-added-large-files
+
+     - repo: https://github.com/astral-sh/ruff-pre-commit
+       rev: v0.1.6
+       hooks:
+         - id: ruff
+           args: [--fix, --exit-non-zero-on-fix]
+
+     - repo: https://github.com/pycqa/pylint
+       rev: v3.0.3
+       hooks:
+         - id: pylint
+           args: [--rcfile=.pylintrc]
+
+     - repo: https://github.com/psf/black
+       rev: 23.12.1
+       hooks:
+         - id: black
+
+     - repo: https://github.com/pre-commit/mirrors-mypy
+       rev: v1.8.0
+       hooks:
+         - id: mypy
+
+     - repo: https://github.com/pycqa/isort
+       rev: 5.13.2
+       hooks:
+         - id: isort
+
+     - repo: https://github.com/pycqa/bandit
+       rev: 1.7.7
+       hooks:
+         - id: bandit
+
+**4. Customizing Hooks**
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  **pylint**: You can specify a config file (``.pylintrc``) or pass
+   arguments directly.
+-  **ruff**: You can customize rules in ``pyproject.toml`` or
+   ``.ruff.toml``.
+-  **Black, isort, mypy**: Each can be configured via their respective
+   config files.
+
+**5. Installing and Running**
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Save the config to ``.pre-commit-config.yaml``.
+
+2. Run:
+
+   .. code:: bash
+
+      pre-commit install
+      pre-commit run --all-files
diff --git a/docs/dict_convention.rst b/docs/dict_convention.rst
deleted file mode 100644
index 2cef856a5..000000000
--- a/docs/dict_convention.rst
+++ /dev/null
@@ -1,19 +0,0 @@
-Dictionary conventions
-=========================
-
-Dictionary file naming convention
-------------------------------------
-
-The order of dictionaries is important for the order in which variables and families are created.
-
-The files must therefore be started with two numbers followed by a hyphen.
-
-For example: `00-base.xml`
-
-Naming convention for families and variables
------------------------------------------------
-
-The only restriction on the name of families and variables is that the name must not start with the `"_"` (undescore) character. 
-
-However, it is preferable to only use lowercase ASCII letters, numbers and the `"_"` (undescore) character. 
-The snake case typographic convention is therefore used.
diff --git a/docs/dictionary.rst b/docs/dictionary.rst
deleted file mode 100644
index e8f483295..000000000
--- a/docs/dictionary.rst
+++ /dev/null
@@ -1,36 +0,0 @@
-The dictionaries
-=====================
-
-What do you mean by :term:`dictionary`?
--------------------------------------------
-
-A :term:`dictionary` is a YAML file whose structure is described in this documentation page.
-
-A dictionary contains a set of variables loaded into :term:`Tiramisu`, usable at any time, especially in a :term:`templates`.
-
-:term:`Families` and :term:`variables` can be defined in several dictionaries. These dictionaries are then aggregated.
-
-Dictionaries are loaded in the directory order defined by the `dictionaries_dir` configuration parameter. 
-Each directory is loaded one after the other. 
-Inside these directories the YAML files will be classified in alphabetical order.
-
-There is no alphabetical ordering of all YAML files in all directories.
-
-It is also possible to :term:`redefine` elements to change the behavior of a family or a variable. 
-
-The default namespace
--------------------------
-
-The families and variables contained in these dictionaries are ordered, by default, in the `rougail` namespace. It is possible to change the name of this namespace :doc:`with the `variable_namespace` parameter of the configuration `.
-
-This namespace is a bit special, it can access variables in another namespace.
-
-The extra dictionaries
----------------------------
-
-An extra is a different namespace. The idea is to be able to classify the variables by theme.
-
-Extra namespaces must be declared :doc:`when configuring Rougail `.
-
-In this namespace we cannot access variables from another `extra` namespace. 
-On the other hand, it is possible to access the variable of the default namespace.
diff --git a/docs/documentation.rst b/docs/documentation.rst
new file mode 100644
index 000000000..2e518e1da
--- /dev/null
+++ b/docs/documentation.rst
@@ -0,0 +1,222 @@
+Documentation Writing Guide
+===========================
+
+This project uses **Sphinx** to generate documentation from reStructuredText (``.rst``) files.
+Following a few consistent conventions will keep our docs clear, maintainable, and correctly rendered.
+
+Section title levels
+--------------------
+
+Use the following underline-only characters for headings:
+
+- **Level 1 titles** (page titles): underline with ``======`` (equal signs)
+
+Example:
+
+.. code-block:: rst
+
+    My Page Title
+    =============
+
+- **Level 2 headings** (major sections): underline with ``-------`` (dashes)
+
+Example:
+
+.. code-block:: rst
+
+    My Second-level Heading
+    -----------------------
+
+- **Level 3 headings** (sub-sections): underline with ``~~~~~~~~~~~`` (tildes)
+
+Example:
+
+.. code-block:: rst
+
+    My Third-level Heading
+    ~~~~~~~~~~~~~~~~~~~~~~
+
+Do **not** skip levels (e.g. going from a level 1 title directly to a level 3 heading).
+
+Handling TODOs and FIXMEs
+------------------------------
+
+We use two complementary mechanisms to track pending work:
+
+- **FIXME** – keyword for things that need **correction** (bugs, typos, broken links, wrong logic)
+- **``.. todo::`` directive** – for things that need to be **done** (missing sections, improvements, new features)
+
+Example:
+
+.. code-block:: rst
+
+    FIXME: The example below uses a deprecated function.
+
+    .. todo::
+        Add a section about configuration file formats.
+
+To show or hide todos in the rendered output, set in your ``conf.py``:
+
+.. code-block:: python
+
+    todo_include_todos = True   # shows todos in the HTML output
+    # todo_include_todos = False  # hides them
+
+When ``True``, all ``.. todo::`` directives appear inline. Use this for local reviews; keep it ``False`` for public releases unless you want todos visible.
+
+Extensions
+----------
+
+Our Sphinx configuration uses the following extensions:
+
+.. code-block:: python
+
+    extensions = [
+        'sphinx.ext.extlinks',
+        'sphinx_lesson',
+        'sphinx.ext.todo',
+    ]
+
+- **``sphinx.ext.todo``** – enables the ``.. todo::`` directive and the ``todo_include_todos`` setting.
+- **``sphinx_lesson``** – provides lesson‑specific markup (see its documentation for details).
+- **``sphinx.ext.extlinks``** – creates short aliases for long, repetitive URLs.
+
+External links (extlinks)
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``extlinks`` extension defines convenient shortcuts. For each alias you write ``::``, and Sphinx expands it into a full URL.
+
+Our configuration:
+
+.. code-block:: python
+
+    extlinks = {
+        'source': ('https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/%s',
+                   'source: %s'),
+        'tiramisu': ('https://tiramisu.readthedocs.io/en/latest/%s', 'tiramisu: %s'),
+        'tutorial': ('https://forge.cloud.silique.fr/stove/rougail-tutorials/%s', 'tutorial %s'),
+    }
+
+Usage examples:
+
+- ``:source:`v1.1_010/firefox/00-proxy.yml``` → links to the raw file at that tag, displayed as ``source: v1.1_010/firefox/00-proxy.yml``
+- ``:tiramisu:`configuration.html``` → links to the Tiramisu documentation, displayed as ``tiramisu: configuration.html``
+- ``:tutorial:`README.md``` → links to the rougail-tutorials file, displayed as ``tutorial README.md``
+
+The first string in each tuple is the URL pattern (the ``%s`` is replaced by your custom text). The second string is the visible link title (where ``%s`` is again replaced by your custom text).
+
+General writing tips
+--------------------
+
+- **Keep lines wrapped at 80–90 characters** for readability in plain text.
+- **Use blank lines** before and after headings, lists, code blocks, and tables.
+- **Prefer explicit cross-references** with Sphinx roles like ``:ref:`` or ``:doc:`` over raw URLs.
+- **Write in present tense** and imperative mood for instructions (e.g. “Define the function”, not “The function should be defined”).
+- **Use ``.. code-block::``** for syntax highlighting. Always specify the language (``python``, ``bash``, ``json``, etc.).
+- **Review the rendered output** with ``make html`` before committing.
+
+File organisation
+-----------------
+
+- Place ``.rst`` files in the ``source/`` directory (or as defined in your Sphinx ``conf.py``).
+- One page per logical topic – split long documents into sub-pages and link them with ``toctree``.
+- Name files with lowercase letters and hyphens instead of underscores or spaces (e.g. ``api-reference.rst``).
+
+Example – a typical page
+------------------------
+
+.. code-block:: rst
+
+    Using the Logger
+    ================
+
+    Basic setup
+    -----------
+
+    Add the following to your configuration:
+
+    .. code-block:: python
+
+        import logging
+        logging.basicConfig(level=logging.INFO)
+
+    Advanced filtering
+    ~~~~~~~~~~~~~~~~~~
+
+    For more complex filtering, see :ref:`custom-filters`.
+
+    FIXME: The example above should mention log levels.
+
+    .. todo::
+        Add a section about rotating log files.
+
+For any questions about Sphinx directives or reStructuredText syntax, check the `official Sphinx documentation `_.
+
+Displaying YAML files
+-------------------------
+
+We host our data files on a remote forge. To include and display YAML files directly from that remote repository, we use a custom directive called **``extinclude``**.
+
+This directive works like the standard ``.. include::`` but fetches content from a **remote URL** instead of a local file.
+
+YAML files coming from a data repository
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Use the following syntax to include a remote YAML file with syntax highlighting:
+
+.. code-block:: rst
+
+    .. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_170/firefox/60-dns_over_https.yml
+       :language: yaml
+       :caption: The :file:`firefox/60-dns_over_https.yml` with the jinja validator
+
+Explanation of options:
+
+- **``extinclude:: ``** – the full URL to the raw YAML file on the remote forge
+- **``:language: yaml``** – enables YAML syntax highlighting in the rendered output
+- **``:caption: ``** – adds a descriptive caption below the code block
+
+Displaying terminal commands
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To include and display a text file containing bash commands or terminal output, use the ``raw`` directive with the ``terminal`` class:
+
+.. code-block:: rst
+
+    .. raw:: html
+       :class: terminal
+       :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_033/config/02/cmd_ro.txt
+
+This fetches the remote text file and renders it with terminal-style formatting (typically a dark background, monospaced font, and command-line appearance).
+
+Displaying HTML output
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To include and display an HTML file from the remote forge, use the ``raw`` directive with the ``output`` class:
+
+.. code-block:: rst
+
+    .. raw:: html
+       :class: output
+       :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_033/config/02/output_ro.html
+
+This fetches the remote HTML file and embeds it directly into the generated documentation page.
+
+Summary
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
++----------------------------+----------------------+---------------------------+
+| Content type               | Directive            | Class                     |
++============================+======================+===========================+
+| YAML files                 | ``.. extinclude::``  | ``:language: yaml``       |
++----------------------------+----------------------+---------------------------+
+| Terminal commands / text   | ``.. raw:: html``    | ``:class: terminal``      |
++----------------------------+----------------------+---------------------------+
+| HTML output                | ``.. raw:: html``    | ``:class: output``        |
++----------------------------+----------------------+---------------------------+
+
+
+
+All three directives fetch content from our remote forge at build time, ensuring the documentation always displays the latest version of the referenced files.
+
diff --git a/docs/ext/extinclude.py b/docs/ext/extinclude.py
new file mode 100644
index 000000000..f5f1915f0
--- /dev/null
+++ b/docs/ext/extinclude.py
@@ -0,0 +1,95 @@
+from __future__ import annotations
+
+from docutils import nodes
+from sphinx.application import Sphinx
+from sphinx.util.docutils import SphinxDirective, SphinxRole
+from sphinx.util.typing import ExtensionMetadata
+from sphinx.directives.code import LiteralInclude, container_wrapper
+
+import requests
+from requests.exceptions import RequestException
+from docutils.parsers.rst import directives
+
+class ExtInclude(LiteralInclude):
+    """A directive to include code that comes from an url
+
+    Sample use::
+
+        .. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_010/firefox/00-proxy.yml
+           :linenos:
+           :language: yaml
+           :caption: this is a interesting code
+
+    - parameter required
+    - linenos, language and caption are optionnal.
+
+    :default language: yaml
+    :default caption: extinclude parameter (url)
+
+    """
+
+    def run(self) -> list[nodes.Node]:
+        url = self.arguments[0]
+        
+        try:
+            headers = {
+                'accept': 'application/text',
+                'Content-Type': 'application/text',
+            }
+            response = requests.get(url, headers=headers)
+            response.raise_for_status()  # This will raise an exception for 4xx/5xx status codes
+            
+        except requests.exceptions.HTTPError as e:
+            if response.status_code == 404:
+                error_msg = f"extinclude: URL not found (404): {url}"
+            else:
+                error_msg = f"extinclude: HTTP error {response.status_code}: {url}"
+            
+            # Create an error node that will be displayed in the documentation
+            error_node = nodes.error()
+            para = nodes.paragraph()
+            para += nodes.Text(error_msg)
+            error_node += para
+            self.state.document.reporter.warning(error_msg, line=self.lineno)
+            return [error_node]
+            
+        except requests.exceptions.RequestException as e:
+            error_msg = f"extinclude: Failed to fetch URL {url}: {str(e)}"
+            
+            # Create an error node that will be displayed in the documentation
+            error_node = nodes.error()
+            para = nodes.paragraph()
+            para += nodes.Text(error_msg)
+            error_node += para
+            self.state.document.reporter.warning(error_msg, line=self.lineno)
+            return [error_node]
+
+        code = response.text
+
+        literal = nodes.literal_block(code, code)
+        if 'language' in self.options:
+            literal['language'] = self.options['language']
+        else:
+            literal['language'] = 'yaml'
+        literal['linenos'] = 'linenos' in self.options
+        if 'caption' in self.options:
+            caption = self.options.get('caption')
+        else:
+            caption = url
+        literal['caption'] = caption
+        if 'name' in self.options:
+            literal['name'] = self.options.get('name')
+        literal = container_wrapper(self, literal, caption)
+        self.add_name(literal)
+
+        return [literal]
+
+
+def setup(app: Sphinx) -> ExtensionMetadata:
+    app.add_directive('extinclude', ExtInclude)
+
+    return {
+        'version': '0.1',
+        'parallel_read_safe': True,
+        'parallel_write_safe': True,
+    }
diff --git a/docs/ext/xref.py b/docs/ext/xref.py
new file mode 100644
index 000000000..71948a3fa
--- /dev/null
+++ b/docs/ext/xref.py
@@ -0,0 +1,67 @@
+"""adds link url in the global scope
+
+sample use:
+
+:xref:`Tiramisu `
+
+You must declare in the `conf.py`
+
+::
+
+    #---- xref links ----
+    #import the xref.py extension
+            xref_links = {"link_name" : ("user text", "url")}
+    #link_name = "Sphinx External Links"
+    #user_text = "modified External Links Extension"
+    #url = "http://www.sphinx-doc.org/en/stable/ext/extlinks.html"
+    #enables syntax like:
+            " :xref:`tiramisu` "
+            links = {
+                    'tiramisu': ('Tiramisu', 'https://tiramisu.readthedocs.io/en/latest/'),
+                    'tiramisu library': ('Tiramisu library homepage', 'https://forge.cloud.silique.fr/stove/tiramisu'),
+                    }
+            xref_links.update(links)
+
+
+
+"""
+from docutils import nodes
+
+from sphinx.util import caption_ref_re
+
+def xref( typ, rawtext, text, lineno, inliner, options={}, content=[] ):
+
+    title = target = text
+    titleistarget = True
+    # look if explicit title and target are given with `foo ` syntax
+    brace = text.find('<')
+    if brace != -1:
+        titleistarget = False
+        m = caption_ref_re.match(text)
+        if m:
+            target = m.group(2)
+            title = m.group(1)
+        else:
+            # fallback: everything after '<' is the target
+            target = text[brace+1:]
+            title = text[:brace]
+
+    link = xref.links[target]
+
+    if brace != -1:
+        pnode = nodes.reference(target, title, refuri=link[1])
+    else:
+        pnode = nodes.reference(target, link[0], refuri=link[1])
+
+    return [pnode], []
+
+def get_refs(app):
+
+    xref.links = app.config.xref_links
+
+def setup(app):
+
+    app.add_config_value('xref_links', {}, True)
+    app.add_role('xref', xref)
+    app.connect("builder-inited", get_refs)
+
diff --git a/docs/family.rst b/docs/family.rst
deleted file mode 100644
index 964a4006e..000000000
--- a/docs/family.rst
+++ /dev/null
@@ -1,262 +0,0 @@
-A family
-============
-
-Synopsis
----------
-
-A family is a container of variables and subfamily.
-
-.. attention:: A family without a subfamily or subvariable will be automatically deleted.
-
-Name
--------------
-
-It is with this name that we will be able to interact with the family.
-
-It's best to follow the :ref:`convention on variable names`.
-
-Shorthand declaration
-----------------------------
-
-Shorthand declaration is a way to declare a family in a single line. But you can only define family name and description.
-
-To create a family, just add a key with it's name and variables as values. Attention, do not declare any other attributs.
-
-By default, the description of the variable is the family name.
-If you add comment in same line of name, this comment is use as description:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_family:  # This is a great family
-      variable:
-
-Parameters
----------------
-
-.. FIXME: faire une page sur la "convention on variable names"
-
-.. list-table::
-   :widths: 15 45
-   :header-rows: 1
-
-   * - Parameter
-     - Comments
-
-   * - type, _type
-
-       `string`
-
-     - possile values:
-
-       - `family` (**default value**)
-       - `leadership`
-       - `dynamic`
-
-       .. note:: If a subfamily or a subvariable already has the name `"type"`, it is possible to use the `"_type"` attribute.
-
-   * - description, _description
-
-       `string`
-     - Description of the family.
-
-       User information to understand the usefulness of the family.
-
-       ..note:: If a subfamily or subvariable already has the name "description" it is possible to use the "_description" attribute.
-
-   * - help, _help
-
-       `string`
-     - Additional help associated with the family.
-
-       .. note:: If a subfamily or a subvariable already has the name "help" it is possible to use the "_help" attribute.
-
-   * - mode, _mode
-
-       `string`
-     - Family mode.
-
-       The default mode of a family is the smallest mode of the parent families, child variables, or child families that are contained in that family.
-
-       This mode also allows you to define the default mode for variables or families included in this family.
-
-       .. note:: If a subfamily or a subvariable already has the name "mode" it is possible to add the "_mode" attribute.
-
-   * - hidden, _hidden
-
-       `string`
-     - Invisible family.
-
-       Allows you to hide a family as well as the variables or families included in this family.
-
-       This means that the family will no longer be visible in `read-write` mode, but only for calculations or in `read-only` mode.
-
-       .. note:: If a subfamily or a subvariable already has the name "hidden" it is possible to add the "_hidden" attribute.
-
-   * - disabled, _disabled
-
-       `string`
-
-     - Disabled family.
-
-       Allows you to deactivate a family as well as the variables or families included in this family.
-
-       This means that the family will no longer be visible to the user but also to a :term:`calculation`.
-
-       .. note:: If a subfamily or a subvariable already has the name "disabled" it is possible to use the "_disabled" attribute.
-
-Dynamically created family
------------------------------
-
-To create a family dynamically, you must create a fictitious family linked to a calculation.
-The family name will actually be the prefix of the new name. Alternativly you can specify the suffix in the name, ie `my_{{ suffix }}_name`.
-The suffix will come from the calculation.
-
-Obviously if the result of calculation were to evolve, new dynamic families will appear or disappear.
-
-Leader or follower variable
------------------------------
-
-A leader family has a typical attribute of “leadership”. The type is required.
-
-A leader family
-----------------
-
-The leader and follower variables are placed in a leader family.
-
-A leader family cannot contain other families.
-
-The default mode of the leader family is the mode of the leader variable.
-
-Leader variable
-----------------
-
-A leader variable is a variable that will guide the length of other variables (called follower variables).
-
-A leader variable is a :doc:`variable` that must have the `multiple` type.
-
-A leader variable may be mandatory.
-
-The default mode corresponds to the smallest mode defined for the follower variables.
-
-Follower variable
---------------------
-
-A follower variable is a variable whose length is not determined by itself, but is identical to that of the leader variable on which it depends.
-
-A follower variable is a variable placed just behind a leader variable or another follower variable.
-
-The order in which the tracking variables are defined is important.
-
-This variable can be of multiple type. In this case, for a determined index of the leading variable, it is possible to put several values to the same variable.
-
-A follower variable may be required. This means that when a leader variable is entered, the follower variable must also be a value at the index considered. If no value is defined for the leader variable, no value is specified for the follower variable.
-
-The default mode of a follower variable corresponds to the mode of the leader variable.
-
-If a leader variable is hidden or disabled, the follower variables will be hidden or disabled as well.
-
-Examples
-----------
-
-Simple family:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_family:
-      type: family
-      description: This is a great family
-      help: This is the help of a great family
-      mode: expert
-
-Dynamically created family
-----------------------------
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    varname:
-      multi: true
-      default:
-        - val1
-        - val2
-    my_dyn_family_:
-      type: dynamic
-      dynamic:
-        type: variable
-        variable: rougail.varname
-      description: 'Describe'
-      my_dyn_var:
-        type: string
-        description: 'Variable description'
-
-This will dynamically create two families:
-
-- "rougail.my_dyn_family_val1"
-- "rougail.my_dyn_family_val2"
-
-In the dynamic family "rougail.my_dyn_family_val1" we will find a variable "my_dyn_var".
-
-Here is a second example:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    varname:
-      multi: true
-      default:
-        - val1
-        - val2
-    my_dyn_{{ suffix }}_family:
-      type: dynamic
-      dynamic:
-        type: variable
-        variable: rougail.varname
-      description: 'Describe'
-      my_dyn_var:
-        type: string
-        description: 'Variable description'
-
-This will dynamically create two families:
-
-- "rougail.my_dyn_val1_family"
-- "rougail.my_dyn_val2_family"
-
-In the dynamic family "rougail.my_dyn_val1_family" we will find a variable "my_dyn_var".
-
-Leader or follower variable
--------------------------------
-
-Definition of leader and follower variables
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Here is an example of defining a leading variable and two following variables:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    family:
-      type: leadership
-      leader:
-        multi: true
-      follower1:
-      follower2:
-        multi: true
-
-Adding a new follower variable
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-To add a new follower variable, in a new dictionary, simply define one or more new variables in the leader family:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    family:
-      follower3:
diff --git a/docs/fill.rst b/docs/fill.rst
deleted file mode 100644
index d796bff25..000000000
--- a/docs/fill.rst
+++ /dev/null
@@ -1,585 +0,0 @@
-Calculated default values
-==============================
-
-Synopsis
------------
-
-A value can be calculated. In this case we have four possibilities:
-
-- calculation via Jinja
-- calculation via a variable
-- calculation via information
-- calculation via a suffix: in the case of a variable in a dynamic family
-- calculation via an index: in the case of a follower variable
-
-If the user modifies the value of the variable, the default value is no longer used, so the calculation is no longer carried out. This is also the case if the variable has the `auto_save` attribute.
-
-On the other hand, if the variable is hidden (with the `hidden` parameter), it is the default value that is used and not the value customized by the user.
-
-.. note:: A follower variable cannot be calculated automatically.
-
-Parameters
---------------
-
-Depending on the types of calculation, the parameters will be different:
-
-.. list-table:: 
-   :widths: 15 25 20 15
-   :header-rows: 1
-   
-   * - Calculation type 
-     - Parameter
-     - Comments
-     - Sample
-
-   * - 
-     - **type** 
-     
-       `string`
-       
-       `mandatory`
-       
-     - Type of calculation, possible values are: jinja, variable, information, suffix or index
-     - jinja
-   * - Jinja 
-     - **jinja**
-     
-       `string`
-       
-       `mandatory`
-     - Template Jinja. For a multiple variable, each line represents a value.
-     - `{% if rougail.variable %}
-
-       {{ rougail.variable }}
-
-       {% endif %}`
-   * - Jinja 
-     - **params** 
-     
-       `list` 
-     - Additional parameters passed to the Jinja template
-     -  
-   * - Variable (`mandatory`)
-
-       Information
-     - **variable** 
-
-       `string`
-     - Name of associated variable
-     - rougail.variable 
-   * - Variable
-     - **propertyerror** 
-     
-       `boolean`
-     - If access to the variable is not possible due to a property (for example `disabled`) by default an error is returned. If the attribute is `false`, the calculated value is empty.
-
-       **Default value:** `true`
-     - false 
-     
-   * - Information
-     - **information**
-            
-       `string`
-       
-       `mandatory`
-     - Name of the information whose value we want to retrieve.
-     - doc
-
-In the case of a Jinja type calculation, it is possible to have parameters.
-
-There are two types of parameter:
-
--  the standard parameters (string, boolean, integer, null), in this case just do: "key: value"
-
--  the advanced settings:
-
-    - parameter via a variable
-    - parameter via an information
-    - parameter via a suffix: in the case of a variable in a dynamic family
-    - parameter via an index: in the case of a follower variable
-
-.. list-table:: 
-   :widths: 15 25 20 15
-   :header-rows: 1
-   
-   * - Parameter type 
-     - Parameter
-     - Comments
-     - Sample
-
-   * - 
-     - **name**
-                 
-       `string`
-       
-       `mandatory`
-     - parameter's name
-     - my_param  
-   * - 
-     - **type**
-                 
-       `string`
-       
-       `mandatory`
-     - parameter's type, possible values are: variable, information, suffix or index
-     - suffix
-   * - Variable
-     - **variable**
-     
-       `string`
-       
-       `mandatory`
- 
-     - Variable's name 
-     - rougail.variable
-   * - Variable (`mandatory`) information
-     - **propertyerror** 
-     
-       `boolean`
-     - If access to the variable is not possible due to a property (for example `disabled`) by default an error is returned. If the attribute is `False`, the parameter is not passed to the Jinja template.
-     - **Default value**: `True`
-   * - Variable 
-     - **optional**
-             
-       `boolean`
-     - The variable may not exist depending on YAML file imports. 
-       If the optional parameter is `True`, the parameter will simply be deleted if the variable does not exist.
-
-       Default value : `False`  
-     - True
-   * - Information
-     - **information**
-     
-       `string`
-       
-       `mandatory`
-     - Name of the information whose value we want to retrieve.
-     - doc
-
-The variable path
------------------
-
-Normal family
-~~~~~~~~~~~~~
-
-The default namespace is defined in RougailConfig["variable_namespace"] with the default value "rougail".
-In addition, there are extras namespaces defined with in RougailConfig["extra_dictionaries"].
-
-Inside those namespaces we can add families and variables.
-
-Here is an hierarchic examples:
-
-.. code-block::
-  rougail
-    ├── variable1
-    ├── family1
-    │     ├── variable2
-    │     └── variable3
-    └── family2
-          └── subfamily1
-                └── variable4
-  extra1
-    └── family3
-          ├── variable5
-          └── variable6
-
-In `calculation` we can use other variables.
-
-Here is all paths:
-
-- rougail.variable1
-- rougail.family1.variable2
-- rougail.family1.variable3
-- rougail.family2.subfamily1.variable4
-- extra1.family3.variable5
-- extra1.family3.variable6
-
-Inside a variable's `calculation` we can use relative path. "_" means that other variable is in same family. "__" means that other variables are in parent family, and so on...
-
-For example, in variable2's `calculation`, we can use relative path:
-
-- __.variable1
-- _.variable3
-- __.family2.subfamily1.variable4
-
-But we cannot access to extra1 variables with relative path.
-
-Dynamic family
-~~~~~~~~~~~~~~~~~~
-
-Hire is a dynamic family "{{ suffix }}":
-
-.. code-block::
-  rougail
-    ├── variable1: ["val1", "val2"]
-    ├── {{ suffix }}
-    │     ├── variable2
-    │     └── variable3
-    └── family
-          └── variable4
-
-For variable2's calculation, we can use:
-
-- rougail.{{ suffix }}.variable3
-- _.variable3
-
-In this case, we get value for "variable3" with the same suffix as "variable2".
-
-For variable4's calculation, we have two possibility:
-
-- retrieves all values with all suffixes:
-
-  - rougail.{{ suffix }}.variable3
-  - __.{{ suffix }}.variable3
-
-- retrieves a value for a specified suffix:
-
-  - rougail.val1.variable3
-  - __.val1.variable3
-
-Examples
------------
-
-Calculation via a Jinja template
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Let's start with an example from a simple Jinja template:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_calculated_variable:
-      default:
-        type: jinja
-        jinja: 'no'
-
-Here is a second example with a boolean variable:        
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_calculated_variable:
-      type: boolean
-      default:
-        type: jinja
-        jinja: 'false'
-
-And a multiple value of the number type:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_calculated_variable:
-      type: number
-      multi: true
-      default:
-        type: jinja
-        jinja: |
-          1
-          2
-          3      
-
-Let's create a variable whose value is returned by a python function:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_calculated_variable:
-      default:
-        type: jinja
-        jinja: '{{ return_no() }}'
-
-Then let's create the `return_no` function:
-
-.. code-block:: python 
-
-    def return_no():
-        return 'no'
-
-An example with parameters:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_calculated_variable:
-      description: my description
-      default:
-        type: jinja
-        jinja: |
-                {{ param1 }}{% if param2 is defined %}_{{ param2 }}{% endif %}_{{ param3 }}
-        params:
-          param1: value
-          param2:
-            type: variable
-            variable: _.unknown_variable
-            optional: true
-          param3:
-            type: information
-            information: doc
-            variable: _.my_calculated_variable
-
-An example with a `suffix` type parameter:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    varname:
-      multi: true
-      default:
-        - val1
-        - val2
-    my_dyn_family_:
-      type: dynamic
-      dynamic:
-        type: variable
-        variable: _.varname
-      description: 'Describe '
-      my_dyn_var:
-        type: string
-        default:
-          type: jinja
-          jinja: 'the suffix is: {{ param1 }}'
-          params:
-            param1:
-              type: suffix
-
-In this example, we see a dynamic family. Two families will be created: `rougail.my_dyn_family_val1.my_dyn_var` and `rougail.my_dyn_family_val2.my_dyn_var`.
-
-The value of the variable inside this family 'this suffix is: ' + the value of the suffix (`val1` and `val2` respectively).
-
-An example with an index type parameter:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    family:
-      type: leadership
-      leader:
-        multi: true
-        default:
-          - val1
-          - val2
-      follower1:
-        default:
-          type: jinja
-          jinja: 'the index is: {{ param1 }}'
-          params:
-            param1:
-              type: index
-
-Calculation via a variable
------------------------------
-
-Copy a variable in another: 
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_variable:
-      multi: true
-      default:
-        - val1
-        - val2
-    my_calculated_variable:
-      multi: true
-      default:
-        type: variable
-        variable: _.my_variable
-
-Copy the default value from a variable, means copy type, params and multi attribute too if not define in second variable.
-
-.. code-block:: yaml
-
-      ---
-      version: 1.1
-      my_variable:
-        multi: true
-        type: domainname
-        params:
-          allow_ip: true
-      my_calculated_variable:
-        default:
-          type: variable
-          variable: _.var1
-
-Here my_calculated_variable is a domainname variable with parameter allow_ip=True and multi to true.
-
-Copy one variable to another if the source has no `property` problem:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_variable:
-      default: val1
-      disabled: true
-    my_calculated_variable:
-      multi: true
-      default:
-        type: variable
-        variable: _.my_variable
-        propertyerror: false
-
-Copy two non-multiple variables into a multiple variable:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_variable_1:
-      default: val1
-    my_variable_2:
-      default: val2
-    my_calculated_variable:
-      multi: true
-      default:
-        - type: variable
-          variable: _.my_variable_1
-        - type: variable
-          variable: _.my_variable_2
-
-A variable in a dynamic family can also be used in a calculation.
-
-For example using the variable for a particular suffix:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    varname:
-      multi: true
-      default:
-        - val1
-        - val2
-    my_dyn_family_:
-      type: dynamic
-      dynamic:
-        type: variable
-        variable: _.varname
-      description: 'Describe '
-      my_dyn_var_:
-        type: string
-        default:
-          type: suffix
-    all_dyn_var:
-      default:
-        type: variable
-        variable: _.my_dyn_family_val1.my_dyn_var_val1
-
-In this case, we recover the value `val1`.
-
-Second example using the variable for all suffixes:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1' 
-    varname:
-      multi: true
-      default:
-        - val1
-        - val2
-    my_dyn_family_:
-      type: dynamic
-      dynamic:
-        type: variable
-        variable: _.varname
-      description: 'Describe '
-      my_dyn_var_:
-        type: string
-        default:
-          type: suffix
-    all_dyn_var:
-      multi: true
-      default:
-        type: variable
-        variable: _.my_dyn_family_.my_dyn_var_
-
-In this case, we recover the `val1` and `val2` list.
-
-Calculation via a suffix
----------------------------
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    varname:
-      multi: true
-      default:
-        - val1
-        - val2
-    my_dyn_family_:
-      type: dynamic
-      dynamic:
-        type: variable
-        variable: _.varname
-      description: 'Describe '
-      my_dyn_var_:
-        type: string
-        default:
-          type: suffix
-
-Calculation via an index
---------------------------
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    family:
-      type: leadership
-      leader:
-        multi: true
-        default:
-          - val1
-          - val2
-      follower1:
-        type: number
-        default:
-          type: index
-
-Redefinition
-----------------
-
-In a first dictionary, let's declare our variable and our calculation:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_calculated_variable:
-      default:
-        type: jinja
-        jinja: 'the value is calculated'
-
-In a second dictionary, it is possible to redefine the calculation:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_calculated_variable:
-      redefine: true
-      default:
-        type: jinja
-        jinja: 'the value is redefined'
-
-In a third dictionary, we even can delete the calculation if needed:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_calculated_variable:
-      redefine: true
-      default: null
-
diff --git a/docs/format_content/calculation.rst b/docs/format_content/calculation.rst
new file mode 100644
index 000000000..dd1e66f80
--- /dev/null
+++ b/docs/format_content/calculation.rst
@@ -0,0 +1,563 @@
+.. _calculated_variable:
+
+Calculated default values
+==============================
+
+Synopsis
+-----------
+
+A value can be calculated. In this case we have four possibilities:
+
+- calculation via Jinja template
+- calculation via a variable
+- calculation via an information
+- calculation via a identifier: in a :term:`dynamically built family` returns the current identifier
+- calculation via an index: in the case of a follower variable returns the current index
+- calculation via :ref:`the current namespace name `
+
+If the user modifies the value of the variable, the default value is no longer used, so the calculation is no longer carried out. This is also the case if the variable has the `auto_save` attribute.
+
+On the other hand, if the variable is hidden (with the `hidden` parameter), it is the default value that is used and not the value customized by the user.
+
+.. note:: A follower variable cannot be calculated automatically.
+
+.. seealso:: The tutorial with a real world sample :ref:`calculation `
+
+Parameters
+--------------
+
+Jinja template
+~~~~~~~~~~~~~~
+
+.. list-table::
+   :header-rows: 1
+
+   * - **Calculation type**
+     - **Comment**
+
+   * - **type**
+
+       `string`
+     - The value is `jinja`
+
+   * - **jinja**
+
+       `string`
+
+       `mandatory`
+     - Please set a Jinja template.
+
+   * - **params**
+
+       `dict`
+     - Additional parameters passed to the Jinja template (see below).
+
+   * - **description**
+
+       `string`
+     - Additional information for the :ref:`documentation ` and the error message in case of return_type is a boolean.
+
+Variable
+~~~~~~~~
+
+.. list-table::
+   :header-rows: 1
+
+   * - Parameter
+     - Comment
+     - Sample
+
+   * - type
+
+       `string`
+     - Calculation type.
+
+       This parameter is optional. It is deduced from the presence of the parameter "variable" in a calculation.
+     - variable
+
+   * - variable
+
+       `string`
+
+       `mandatory`
+     - Name of the associated variable.
+
+       It's better to use :term:`relative path` notation.
+     - _.my_variable
+
+   * - propertyerror
+
+       `boolean`
+     - If access to the variable is not possible due to a property
+       (for example `disabled`) by default an error is returned.
+       The overall coherence is not guaranteed.
+
+       Default value: `true`
+     - false
+
+   * - optional
+
+       `boolean`
+     - The variable in calculation may not exist. This generates an error. Overall coherence is not guaranteed.
+       To avoid this set the optional parameter to `true`. In this case, if variable in calculation is not exists, the variable is always accessible.
+     - true
+
+   * - default
+
+       `boolean`
+     - If optional parameter is set to `true` and the variable in calculation is not exist, the variable is accessible.
+       If you do not want this behavior, set this parameter to `false`.
+     - false
+
+   * - description
+
+       `string`
+     - Rougail engine create a description like "depends on a calculation".
+       You can personnalize here the calculation description
+     -
+
+Information
+~~~~~~~~~~~
+
+.. list-table::
+   :header-rows: 1
+
+   * - Parameter
+     - Comments
+     - Sample
+
+   * - information
+
+       `string`
+
+       `mandatory`
+     - Name of the information whose value we want to retrieve.
+     - doc
+
+   * - variable
+
+       `string`
+     - Variable's name. If not specified, the information is retrieve in :term:`Tiramisu` Config.
+
+       It's better to use :term:`relative path` notation.
+     - _.my_variable
+
+Params
+~~~~~~
+
+In the case of a Jinja type calculation, it is possible to have parameters.
+
+There are two types of params:
+
+- the standard parameters (string, boolean, integer, float or null), in this case just do: "key: value"
+- advanced settings (with something like "key: {}":
+
+  - parameter via a variable
+  - parameter via an information
+  - parameter via an identifier: in a :term:`dynamically built family` returns the current identifier
+  - parameter via an index: in the case of a follower variable returns the current index
+  - parameter via :ref:`the current namespace name `
+
+Variable params
+"""""""""""""""
+
+.. list-table::
+   :header-rows: 1
+
+   * - **Parameter**
+     - **Comments**
+     - **Sample**
+
+   * - **type**
+
+       `string`
+     - Type of parameter, which is variable.
+     - variable
+
+   * - **variable**
+
+       `string`
+
+       `mandatory`
+     - Variable's path.
+     - my_variable
+
+   * - **propertyerror**
+
+       `boolean`
+     - If access to the variable is not possible due to a property
+       (for example `disabled`) by default an error is returned.
+       If the attribute is `false`, the parameter is not passed to the Jinja template.
+
+       **Default value**: `false`
+     - `true`
+
+   * - **optional**
+
+       `boolean`
+     - If the variable is not defined the value is always `null`.
+
+       **Default value**: `false`
+     - `true`
+
+   * - **whole**
+
+       `boolean`
+     - In :term:`dynamically built family` only the value of the current index is retrieve. If this parameter is set to `true` it returns all
+       If this parameter is set to `true`, it returns the set of values for all indices.
+
+       **Default value**: `false`
+     - `true`
+
+Information params
+~~~~~~~~~~~~~~~~~~
+
+.. list-table::
+   :header-rows: 1
+
+   * - **Parameter**
+     - **Comments**
+     - **Sample**
+
+   * - **type**
+
+       `string`
+     - Type of parameter, which is information
+     - information
+
+   * - **information**
+
+       `string`
+
+       `mandatory`
+     - Name of the information whose value we want to retrieve.
+     - doc
+
+   * - **variable**
+
+       `string`
+     - By default, the information is a context information. It is possible to get a variable information. In this case just specify a path.
+     - my_variable
+
+Sample
+-----------
+
+Calculation via a Jinja template
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Let's start with an example from a simple Jinja template:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_calculated_variable:
+      default:
+        jinja: 'no'
+    ...
+
+Here is a second example with a boolean variable:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_calculated_variable:
+      type: boolean
+      default:
+        jinja: 'false'
+    ...
+
+And a multiple value of the integer type:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_calculated_variable:
+      type: integer
+      multi: true
+      default:
+        jinja: |
+          1
+          2
+          3
+    ...
+
+Let's create a variable whose value is returned by a python function:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_calculated_variable:
+      default:
+        jinja: '{{ return_no() }}'
+    ...
+
+Then let's create the `return_no` function:
+
+.. code-block:: python
+
+    def return_no():
+        return 'no'
+
+An example with parameters:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_calculated_variable:
+      description: my description
+      default:
+        jinja: |
+                {{ param1 }}{% if param2 is defined %}_{{ param2 }}{% endif %}_{{ param3 }}
+        params:
+          param1: value
+          param2:
+            type: variable
+            variable: _.unknown_variable
+            optional: true
+          param3:
+            type: information
+            information: doc
+            variable: _.my_calculated_variable
+    ...
+
+An example with a `identifier` type parameter:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_dyn_family_{{ identifier }}:
+      dynamic:
+        - val1
+        - val2
+      description: 'Describe {{ identifier }}'
+
+      my_dyn_var:
+        default:
+          jinja: 'the identifier is: {{ param1 }}'
+          params:
+            param1:
+              type: identifier
+
+In this example, we see a :term:`dynamically built family`. Two families will be created: `my_dyn_family_val1.my_dyn_var` and `my_dyn_family_val2.my_dyn_var`.
+
+The value of the variable inside this family 'this identifier is: ' + the value of the identifier (`val1` and `val2` respectively).
+
+An example with an index type parameter:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    family:
+      type: sequence
+
+      leader:
+        - val1
+        - val2
+
+      follower1:
+        default:
+          type: jinja
+          jinja: 'the index is: {{ param1 }}'
+          params:
+            param1:
+              type: index
+    ...
+
+Calculation via a variable
+-----------------------------
+
+Copy a variable in another:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_variable:
+      default:
+        - val1
+        - val2
+
+    my_calculated_variable:
+      default:
+        type: variable
+        variable: _.my_variable
+    ...
+
+Copy the default value from a variable, means copy type, params and multi attribute too if not define in second variable.
+
+.. code-block:: yaml
+
+      %YAML 1.2
+      ---
+      version: 1.1
+
+      my_variable:
+        multi: true
+        type: domainname
+        params:
+          allow_ip: true
+
+      my_calculated_variable:
+        default:
+          type: variable
+          variable: _.var1
+
+Here my_calculated_variable is a domainname variable with parameter allow_ip=True and multi to true.
+
+Copy one variable to another if the source has no `property` problem:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_variable:
+      default: val1
+      disabled: true
+
+    my_calculated_variable:
+      multi: true
+      default:
+        variable: _.my_variable
+        propertyerror: false
+    ...
+
+Copy two non-multiple variables into a multiple variable:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_variable_1: val1
+
+    my_variable_2: val2
+
+    my_calculated_variable:
+      default:
+        - variable: _.my_variable_1
+        - variable: _.my_variable_2
+
+A variable in a :term:`dynamically built family` can also be used in a calculation.
+
+For example using the variable for a particular identifier:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    varname:
+      - val1
+      - val2
+
+    my_dyn_family_{{ identifier }}:
+      dynamic:
+        - val1
+        - val2
+      description: 'Describe {{ identifier }}'
+      my_dyn_var_{{ identifier }}:
+        default:
+          type: identifier
+
+    all_dyn_var:
+      default:
+        variable: _.my_dyn_family_val1.my_dyn_var_val1
+
+In this case, we recover the value `val1`.
+
+Second example using the variable for all identifieres:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    varname:
+    my_dyn_family_{{ identifier }}:
+      dynamic:
+        - val1
+        - val2
+      description: 'Describe {{ identifier }}'
+
+      my_dyn_var_{{ identifier }}:
+        default:
+          type: identifier
+
+    all_dyn_var:
+      multi: true
+      default:
+        variable: _.my_dyn_family_.my_dyn_var_
+    ...
+
+In this case, we recover the `val1` and `val2` list.
+
+Calculation via a identifier
+----------------------------
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_dyn_family_{{ identifier }}:
+      dynamic:
+        - val1
+        - val2
+      description: 'Describe {{ identifier }}'
+
+      my_dyn_var_{{ identifier }}:
+        type: string
+        default:
+          type: identifier
+    ...
+
+Calculation via an index
+--------------------------
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    family:
+      type: sequence
+
+      leader:
+        - val1
+        - val2
+
+      follower1:
+        type: integer
+        default:
+          type: index
+    ...
diff --git a/docs/format_content/condition.rst b/docs/format_content/condition.rst
new file mode 100644
index 000000000..a64257a4d
--- /dev/null
+++ b/docs/format_content/condition.rst
@@ -0,0 +1,276 @@
+Calculated properties
+==========================
+
+Synopsis
+------------
+
+Calculated properties allow you to add or remove properties to a :term:`variable` or a :term:`family` depending on the context.
+
+Here is the list of editable properties:
+
+.. list-table::
+   :widths: 15 15 25
+   :header-rows: 1
+
+   * - Attribute applying to
+     - Property's name
+     - Comment
+
+   * - Variable
+
+       Family
+     - hidden
+     - Hides a variable or a family, in this case it is not accessible in `read-write` mode,
+       but remains accessible in a calculation or in `read-only` mode.
+
+       .. seealso:: :ref:`hidden access control `.
+
+   * - Variable
+
+       Family
+     - disabled
+     - Deactivates a variable or family, in this case it is never accessible.
+
+       .. seealso:: :ref:`disabled access control `.
+
+   * - Variable
+     - frozen
+     - A frozen variable is a variable that the :term:`operator` cannot change the value.
+
+   * - Variable
+     - mandatory
+     - The variable expects a value other than `null` (or `None` in Python) or `[]` for multiple variables.
+
+       .. seealso:: for non multiple variables we call it a :ref:`nullable variable `.
+
+   * - Variable
+     - empty
+     - The multiple variable allow the value `null` (or `None` in Python)
+
+       .. seealso:: we call it a :ref:`nullable variable `.
+
+A property can be calculated. In this case we have two possibilities:
+
+- calculation via Jinja (see also the tutorial with a real world sample :doc:`variable jinja <../tutorial/jinja>`)
+- calculation via a variable (see also the tutorial with a real world sample :doc:`variable calculation <../tutorial/properties>`)
+
+Parameters
+---------------
+
+Variable parameters
+~~~~~~~~~~~~~~~~~~~
+
+.. list-table::
+   :header-rows: 1
+
+   * - Parameter
+     - Comment
+     - Sample
+
+   * - type
+
+       `string`
+     - Calculation type.
+
+       This parameter is optional. It is deduced from the presence of the parameter "variable" in a calculation.
+     - variable
+
+   * - variable
+
+       `string`
+
+       `mandatory`
+     - Name of the associated variable.
+
+       It's better to use :term:`relative path` notation.
+     - _.my_variable
+
+   * - when/when_not
+     - Compare the value of the variable define in `variable` attribute.
+
+       If the related variable is the boolean typed variable, the default value is `true`
+     - See the tutorial with a real world sample :doc:`when_not <../tutorial/properties>`)
+
+   * - propertyerror
+
+       `boolean` or `transitive`
+     - If access to the variable is not possible due to a property
+       (for example `disabled`) by default an error is returned.
+       The overall coherence is not guaranteed.
+       If the parameter is `false`, the calculated value is `false` in this condition.
+       If the parameter is `transitive`, the calculated is `true` if the variable raise propertyerror
+       or `false` otherwise.
+
+       Default value: `true`
+     - transitive
+
+       false
+
+   * - optional
+
+       `boolean`
+     - The variable in calculation may not exist. This generates an error. Overall coherence is not guaranteed.
+       To avoid this set the optional parameter to `true`. In this case, if variable in calculation is not exists, the variable is always accessible.
+     - true
+
+   * - default
+
+       `boolean`
+     - If optional parameter is set to `true` and the variable in calculation is not exist, the variable is accessible.
+       If you do not want this behavior, set this parameter to `false`.
+     - false
+
+   * - description
+
+       `string`
+     - Rougail engine create a description like "when the variable "xxxx" has the value "yyyy".
+       You can personnalize here the calculation description
+     -
+
+Jinja parameters
+~~~~~~~~~~~~~~~~
+
+.. list-table::
+   :header-rows: 1
+
+   * - Parameter
+     - Comment
+     - Sample
+
+   * - type
+
+       `string`
+     - The calculation type.
+
+       This parameter is optional. It is deduced from the presence of the parameter "jinja" in calculation.
+     - jinja
+
+   * - jinja
+
+       `string`
+
+       `mandatory`
+     - Jinja template. For a multiple variable, each line represents a value.
+
+       It's better to use :term:`relative path` notation in Jinja template.
+     - .. code-block:: jinja
+
+         {% if _.my_variable %}
+           {{ _.my_variable }}
+         {% endif %}
+
+   * - params
+
+       `dict`
+     - Additional parameters passed to the Jinja template
+
+       See below to have params possibility.
+     - See the tutorial with a real world sample :doc:`params <../tutorial/jinja>`)
+
+   * - return_type
+
+       `string`
+     - Jinja template could return two type of value:
+
+       - `string` (default): that means the Jinja template return an error message that explain why we don't have access to this variable
+       - `boolean`: the Jinja template return "true" or "false". The value true means that the properties is present.
+
+     - See the tutorial with a real world sample :doc:`return_type <../tutorial/jinja>`)
+
+   * - description
+
+       `string`
+       `recommanded`
+     - Rougail engine cannot deduce what you want to do in Jinja template. You have to describe it for documentation purpose.
+       If return_type parameter is `boolean`, this description is used in error message.
+     - See the tutorial with a real world sample :doc:`description <../tutorial/jinja>`)
+
+In the case of a Jinja type calculation, it is possible to have parameters (`params`).
+
+There are two types of parameter:
+
+- the standard parameters (string, boolean, integer, null), in this case just do: "key: value"
+
+- advanced settings:
+
+   - parameter via a variable
+   - parameter via information
+   - parameter via an identifier: in the case of a variable in a :term:`dynamically built family`
+   - parameter via an index: in the case of a variable in a :term:`sequence`
+   - parameter via the current :term:`namespace`
+
+Parameters for variable type `param`
+''''''''''''''''''''''''''''''''''''
+
+.. list-table::
+   :header-rows: 1
+
+   * - Parameter
+     - Comments
+     - Sample
+
+   * - type
+
+       `string`
+     - Parameter's type
+     - variable
+
+   * - variable
+
+       `string`
+
+       `mandatory`
+     - Variable's name
+
+       It's better to use :term:`relative path` notation.
+     - _.my_variable
+
+   * - propertyerror
+
+       `boolean`
+     - If access to the variable is not possible due to a property (for example `disabled`) by default an error is returned. If the parameter is false, the parameter is not passed to the Jinja template.
+
+       Default value: `true`
+     - false
+
+   * - optional
+
+       `boolean`
+     - The variable may not exist depending on YAML file imports. If the optional parameter is true, the parameter will simply be deleted if the variable does not exist.
+       Default value: `false`
+     - true
+
+   * - whole
+
+       `boolean`
+     - In :term:`sequence` family, only the value of the variable in the current element is propose.
+       This parameter allow you to have all values for all elements.
+
+       Default value: `false`.
+     - true
+
+Parameters for information type `param`
+'''''''''''''''''''''''''''''''''''''''
+
+.. list-table::
+   :header-rows: 1
+
+   * - Parameter
+     - Comments
+     - Sample
+
+   * - information
+
+       `string`
+
+       `mandatory`
+     - Name of the information whose value we want to retrieve.
+     - doc
+
+   * - variable
+
+       `string`
+     - Variable's name. If not specified, the information is retrieve in :term:`Tiramisu` Config.
+
+       It's better to use :term:`relative path` notation.
+     - _.my_variable
diff --git a/docs/format_content/family.rst b/docs/format_content/family.rst
new file mode 100644
index 000000000..d187a178f
--- /dev/null
+++ b/docs/format_content/family.rst
@@ -0,0 +1,285 @@
+.. _family:
+
+The families
+=============
+
+Synopsis
+---------
+
+.. glossary::
+
+   family
+   subfamily
+   
+       A family (or a subfamily) is simply a collection of variables that refer to
+       the same business model category. It's just a variables container.
+       Think of it as a container as well as a namespace.
+
+.. attention:: A family without a subfamily or subvariable will be automatically deleted.
+
+Naming
+---------
+
+It is with its name that we will be able to interact with the family.
+
+.. seealso:: 
+
+   A family is a variable. So have a look at the :ref:`variable naming convention `.
+
+Parameters
+---------------
+
+.. list-table::
+   :widths: 15 45
+   :header-rows: 1
+
+   * - Parameter
+     - Comments
+
+   * - **description**, **_description**
+
+       `string`
+     - Description of the family.
+
+       User information to understand the usefulness of the family.
+
+       .. seealso:: tutorial with a real world sample :doc:`description parameter <../tutorial/family>`
+
+   * - **help**, **_help**
+
+       `string`
+     - Additional help associated with the family.
+
+       .. seealso:: tutorial with a real world sample :doc:`help parameter <../tutorial/document>` (the tutorial focuses on variable, but the principle is the same for a family)
+
+   * - **mode**, **_mode**
+
+       `string`
+     - Family mode.
+
+       The mode is a way of classifying the importance, the level of expertise required, or the access rights to a family.
+
+       This parameter allows you to define the family mode, and also the default mode for variables or families included in this family.
+
+       .. attention:: Mode is not configured by default. You have to define mode level before use this parameter.
+
+       **Default value**: The default mode of a family is the smallest mode of the parent families, child variables, or child families that are contained in that family.
+
+       .. seealso:: tutorial with a real world sample :doc:`mode parameter <../tutorial/modes>` (the tutorial focuses on variable, but the principle is the same for a family)
+
+   * - **type**, **_type**
+
+       `string`
+     - possible values:
+
+       - `family` (**default value**)
+       - `dynamic`
+       - `sequence`
+
+   * - **hidden**, **_hidden**
+
+       `boolean` or :term:`calculation`
+     - Invisible family.
+
+       Enables us to *hide* a family as well as the variables or families included in this family.
+
+       This means that the family will no longer be visible in `read-write` mode, but only for calculations or in `read-only` mode.
+
+       **Default value**: `false`
+
+       .. seealso:: tutorial with a real world sample :doc:`hidden parameter <../tutorial/properties>`
+
+   * - **disabled**, **_disabled**
+
+       `boolean` or :term:`calculation`
+     - Disabled family.
+
+       Allows us to *deactivate* a family as well as the variables or families included in this family.
+
+       This means that the family will no longer be visible to the user but also to a :term:`calculation`.
+
+       **Default value**: `false`
+
+       .. seealso:: tutorial with a real world sample :doc:`disabled parameter <../tutorial/properties>`
+
+   * - **dynamic**, **_dynamic**
+
+       :term:`calculation` or a list of :term:`calculation` or `string`
+     - Dynamic identifiers.
+
+       .. important:: This parameter is only available for dynamically built family. See the explanations about this below.
+
+   * - **redefine**
+
+       `boolean`
+     - It is possible to define a family in one :term:`structure file` and change its behavior in a second :term:`structure file`. In this case you must explicitly redefine the family.
+
+       **Default value**: `false`
+
+   * - **exists**
+
+       `boolean`
+     - This parameter does two things:
+
+         - creates a family if it does not exist in another :term:`structure file` (otherwise do nothing), in this case the value of the parameter must be `true`
+         - in conjunction with the `redefine` parameter set to `true`, only modifies the behavior if it is pre-existing, in which case the parameter's value must be `false`.
+
+       **Default value**: `null`
+
+
+.. note::
+
+   If a subfamily or a subvariable already has the name of a parameter it is possible to use the "_" name.
+   You can have a look at the tutorial with a real world sample :doc:`of parameter with "_" <../tutorial/underscore_parameter>`.
+
+Short-hand declaration
+----------------------------
+
+Short-hand declaration is a way to declare a family in a single line.
+
+To create a family, just add a key with it's name and variables as values:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+    my_family:
+
+      my_variable:
+    ...
+
+By default, the description of the family is the family name.
+It's a best practice to description a family. Just add comment in same line of name, this comment is use as description:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+    my_family:  # This is a great family
+
+      my_variable:
+    ...
+
+But in short-hand notation, you can only define family name and description.
+
+.. attention:: Any other parameters will be considered as a variable. Do not use short-hand in this case.
+
+A family
+---------
+
+Here is a simple example:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+    my_family:
+      description: This is a great family
+      help: This family is great because we have great variables inside
+
+      my_variable:
+    ...
+
+.. seealso::
+
+   You can have a look at the tutorial with a real world sample:
+
+   - :doc:`family <../tutorial/family>`
+   - :doc:`help parameter <../tutorial/document>` (the tutorial focuses on variable, but the principle is the same for a family)
+
+A dynamically built family
+-----------------------------
+
+To create a :term:`dynamically built family`, you must create a fictitious family linked to a list of uniq identifiers.
+This list could be hard coded or resulting from a calculation.
+
+The name of this family is particular. You have to add `{{ identifier }}` string inside. It will be replace by each identifiers.
+
+Here is an simple example:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+    
+    my_{{ identifier }}_family:
+      description: 'A family for {{ identifier }}'
+      dynamic:
+        - one thing
+        - another
+
+      variable:
+        description: 'A variable for {{ identifier }}'
+    ...
+
+This will dynamically create two families:
+
+- "my_one_thing_family"
+- "my_another_family"
+
+.. seealso::
+
+   You can have a look at the tutorial with a real world sample :doc:`dynamically built family <../tutorial/dynamic>`
+
+.. _sequence_family:
+
+Sequence of homogeneous elements
+---------------------------------------
+
+A family with `type` set to `sequence` is a sequence of homogeneous elements.
+
+Let's start the explanation with a concrete example:
+
+.. code-block:: yaml
+
+   %YAML 1.2
+   ---
+   version: 1.1
+
+   accounts:
+     description: All accounts
+     type: sequence
+
+     login:
+       description: Account login
+       type: unix_user
+
+     secret:
+       description: Account secret
+       type: secret
+   ...
+
+What is expected, it's something like:
+
+.. code-block:: yaml
+
+    ---
+    accounts:
+      - login: foo
+        secret: 0hM%G0dW4a7ASecr3t
+      - login: bar
+        secret: NotGoodSecret
+
+It's what we call a Sequence of homogeneous elements.
+
+.. attention:: A sequence cannot contain other families.
+
+.. seealso::
+
+   You can have a look at the tutorial with a real world sample :doc:`Sequence of homogeneous elements <../tutorial/sequence>`
+
+A custom type family
+----------------------
+
+In short, a custom type is nothing more than a kind of a template for a family.
+Define a custom type family is much like defining families.
+
+This custom type family can be used as many times as desired and customized as you wish.
+
+.. seealso::
+
+   You can have a look at the tutorial with a real world sample :doc:`family custom type <../tutorial/family>`
diff --git a/docs/format_content/index.rst b/docs/format_content/index.rst
new file mode 100644
index 000000000..1c43fb9fd
--- /dev/null
+++ b/docs/format_content/index.rst
@@ -0,0 +1,18 @@
+The Rougail format
+=====================
+
+Now that we understand what can be described in a structured data file, let's see how to describe variables.
+
+The variable is described in Rougail format.
+
+.. toctree::
+   :titlesonly:
+   :caption: Rougail format content
+
+   variable
+   family
+   path
+   mutability
+   calculation
+   condition
+   validation
diff --git a/docs/format_content/mutability.rst b/docs/format_content/mutability.rst
new file mode 100644
index 000000000..593a4aa6e
--- /dev/null
+++ b/docs/format_content/mutability.rst
@@ -0,0 +1,129 @@
+.. _redefine:
+
+Redefine a variable
+===================
+
+We have already introduced the concept of :ref:`variable mutability `.
+To summarize, an :term:`integrator` allows the :ref:`structured data ` to evolve at any time.
+
+.. glossary::
+
+   redefine
+
+      A redefine is a redefinition of all or part of the parameters of a variable or a family as it was previously defined.
+
+.. note::
+
+   Adding a variable to a family is not considered redefining the family. This is implicitly permitted. Redefining a family requires modifying one or more of its parameters.
+
+.. seealso::
+
+   See the tutorial with a real world sample :ref:`redefine `.
+
+Explicit redefine
+~~~~~~~~~~~~~~~~~
+
+The redefinition is explicit.
+This means that if an :term:`integrator` defines the same tree twice for two variables, Rougail will consider that there is a naming conflict.
+
+To illustrate, we could create two :term:`structure files `:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_variable:  # A description
+    ...
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_variable:  # A redefined description
+    ...
+
+And the name conflict is detected:
+
+.. code-block:: bash
+   :class: terminal 
+
+   $ rougail -m variable.yml redefine.yml
+   ERROR: variable "my_variable" define multiple time in "variable.yml" and "redefine.yml"
+
+So we have to explicit redefine a variable like this:
+
+.. code-block:: yaml
+
+   %YAML 1.2
+   ---
+   version: 1.1
+   
+   my_variable:
+     redefine: true
+     description: A redefined description
+   ...
+
+Now we can see that the redefinition is correct:
+
+.. code-block:: bash
+   :class: terminal 
+
+    $rougail -m variable.yml redefine.yml -o doc
+    ┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+    ┃ Variable            ┃ Description              ┃
+    ┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
+    │ my_variable         │ A redefined description. │
+    │  string   mandatory │                          │
+    └─────────────────────┴──────────────────────────┘
+
+The new description has been taken into account.
+
+Unify
+~~~~~
+
+When redefining a variable, we define the list of parameters that we wish to redefine.
+
+In other words the variable is unify (combine) from multiple variables declarations.
+
+The old parameters that were present before the redefinition are still present.
+
+If we modify the `redefine.yml` :term:`structure file`:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+    
+    my_variable:
+      redefine: true
+      type: integer
+    ...
+
+.. code-block:: bash
+   :class: terminal 
+
+   $ rougail -m variable.yml redefine.yml -o doc
+   ┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
+   ┃ Variable             ┃ Description    ┃
+   ┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
+   │ my_variable          │ A description. │
+   │  integer   mandatory │                │
+   └──────────────────────┴────────────────┘
+
+Here we can see that the variable has the original description, but the new type.
+
+Exception
+~~~~~~~~~
+
+We cannot change a parameter for a variable or a family implicitly.
+
+This is always true, unless you want to change the default value of a child of a custom type in a family.
+
+.. seealso::
+
+   See the tutorial with a real world sample :ref:`custom type `.
diff --git a/docs/format_content/path.rst b/docs/format_content/path.rst
new file mode 100644
index 000000000..0ecc81cd8
--- /dev/null
+++ b/docs/format_content/path.rst
@@ -0,0 +1,69 @@
+
+.. index:: path, variable's relative path, variable's full path
+
+.. _variablepath:
+
+Full or relative path
+=====================
+
+To access a variable, it is necessary to know which family or subfamily it belongs to.
+
+Let's create some variables to illustrate this point.
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_family:  # A family
+
+      my_sub_family:  # A family inside an other family
+
+        my_first_variable:  # My first variable
+
+        my_second_variable:  # My second variable
+
+      my_third_variable:  # My third variable
+
+    my_fourth_variable:  # My forth variable
+    ...
+
+So we have three variables.
+
+Here the full paths are:
+
+- `my_family.my_sub_family.my_first_variable`
+- `my_family.my_sub_family.my_second_variable`
+- `my_family.my_third_variable`
+- `my_fourth_variable`
+
+But in calculation it's often better to use :term:`relative path`.
+
+.. glossary::
+
+   relative path
+
+        In a calculation definition, a relative path defines the location of a variable relative to the family of the variable where the parameter is calculated.
+        Instead of starting from the root, it uses references like `_. (current family)`, `__. (parent family)`, `___. (sub parent family)` and so on.
+
+        Relative paths are shorter and portable across custom type.
+
+Now we can define the :term:`relative path` from the `my_first_variable` variable:
+
+- `_.my_second_variable`
+- `__.my_third_variable`
+- `___.my_fourth_variable`
+
+From the `my_third_variable`:
+
+- `_.my_sub_family.my_first_variable`
+- `_.my_sub_family.my_second_variable`
+- `__.my_fourth_variable`
+
+Finally from the `my_fourth_variable`:
+
+- `_.my_family.my_sub_family.my_first_variable`
+- `_.my_family.my_sub_family.my_second_variable`
+- `_.my_family.my_third_variable`
+
diff --git a/docs/format_content/validation.rst b/docs/format_content/validation.rst
new file mode 100644
index 000000000..d068bb914
--- /dev/null
+++ b/docs/format_content/validation.rst
@@ -0,0 +1,303 @@
+Value validations
+=================
+
+Synopsis
+-------------
+
+A verification is a complementary validation to the type which allows the content of a variable to be validated more precisely.
+
+The purpose of validation is to verify that the variable is of :ref:`good quality ` and/or that the variable is :ref:`consistent `.
+
+.. glossary::
+
+   validator
+
+       A validator is a Jinja code that parses a variable's value and checks
+       that this value corresponds to some stated criteria.
+       This is a `validator` parameter of our variable, and another `jinja` sub-parameter
+       contains validation code for the variable's value.
+
+.. seealso:: The tutorial with a real world sample :ref:`validators `
+
+Parameters
+--------------
+
+Depending on the types of calculation, the parameters will be different:
+
+.. list-table::
+   :header-rows: 1
+
+   * - **Calculation type**
+     - **Comment**
+
+   * - **type**
+
+       `string`
+     - The value is `jinja`
+
+   * - **jinja**
+
+       `string`
+
+       `mandatory`
+     - Please set a Jinja template.
+
+   * - **params**
+
+       `dict`
+     - Additional parameters passed to the Jinja template (see below).
+
+   * - **return_type**
+
+       `string`
+     - A validation could returns the error message directly, so return_type is `string` (the default behavior).
+       Or a `boolean`, in this case if it's return `true` this means that the value is not valid.
+
+   * - **description**
+
+       `string`
+     - Additional information for the :ref:`documentation ` and the error message in case of return_type is a boolean.
+
+   * - **warnings**
+
+       `boolean`
+     - If `true` the validation did not raise, it only display a warning.
+
+
+Params
+~~~~~~
+
+There are two types of params:
+
+- the standard parameters (string, boolean, integer, float or null), in this case just do: "key: value"
+- advanced settings (with something like "key: {}":
+
+  - parameter via a variable
+  - parameter via an information
+  - parameter via an identifier: in a :ref:`dynamically built family ` returns the current identifier
+  - parameter via an index: in the case of a follower variable returns the current index
+  - parameter via :ref:`the current namespace name `
+
+Variable params
+"""""""""""""""
+
+.. list-table::
+   :header-rows: 1
+
+   * - **Parameter**
+     - **Comments**
+     - **Sample**
+
+   * - **type**
+
+       `string`
+     - Type of parameter, which is variable.
+     - variable
+
+   * - **variable**
+
+       `string`
+
+       `mandatory`
+     - Variable's path.
+     - rougail.variable
+
+   * - **propertyerror**
+
+       `boolean`
+     - If access to the variable is not possible due to a property
+       (for example `disabled`) by default an error is returned.
+       If the attribute is `false`, the parameter is not passed to the Jinja template.
+
+       **Default value**: `false`
+     - `true`
+
+   * - **optional**
+
+       `boolean`
+     - If the variable is not defined the value is always `null`.
+
+       **Default value**: `false`
+     - `true`
+
+   * - **whole**
+
+       `boolean`
+     - In :term:`dynamically built family` only the value of the current index is retrieve. If this parameter is set to `true` it returns all
+       If this parameter is set to `true`, it returns the set of values for all indices.
+
+       **Default value**: `false`
+     - `true`
+
+Information params
+~~~~~~~~~~~~~~~~~~
+
+.. list-table::
+   :header-rows: 1
+
+   * - **Parameter**
+     - **Comments**
+     - **Sample**
+
+   * - **type**
+
+       `string`
+     - Type of parameter, which is information
+     - information
+
+   * - **information**
+
+       `string`
+
+       `mandatory`
+     - Name of the information whose value we want to retrieve.
+     - doc
+
+   * - **variable**
+
+       `string`
+     - By default, the information is a context information. It is possible to get a variable information. In this case just specify a path.
+     - rougail.variable
+
+Samples
+--------------
+
+Strict verification of values
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Here is a simple example of validating values:
+
+.. code-block:: yaml
+
+   %YAML 1.2
+   ---
+   version: 1.1
+
+   my_variable:
+     validators:
+       - jinja: |-
+           {% if my_variable is not none and not my_variable.islower() %}
+             {{ my_variable }} is not lowercase string
+           {% endif %}
+   ...
+
+A verification function must take into account 2 important aspects:
+
+- the value may not be entered (even if the variable is mandatory), the None value must be taken into account
+- if return_type is `string` and the value is invalid, a sentence must be returned with an explicit message.
+
+From now on only `null` and lowercase values will be allowed.
+
+Checking values with warning
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In the constraint, it is possible to specify the error level and put it as a warning:
+
+.. code-block:: yaml
+
+   %YAML 1.2
+   ---
+   version: 1.1
+
+   my_variable:
+     validators:
+       - jinja: |-
+           {% if my_variable is not none and not my_variable.islower() %}
+             {{ my_variable }} is not lowercase string
+           {% endif %}
+         params:
+           warnings: true
+   ...
+
+In this case a value with a capital letter will be accepted, but a warning message will appear.
+
+Verification with parameters
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: yaml
+
+   %YAML 1.2
+   ---
+   version: 1.1
+
+   my_hidden_variable:
+     disabled: true
+
+   my_variable:
+     validators:
+       - type: jinja
+         jinja: |-
+           {% if param1 is defined and my_variable == param1 %}
+             has same value as unknown_variable
+           {% endif %}
+           {% if param2 is defined and my_variable == param2 %}
+             has same value as my_hidden_variable
+           {% endif %}
+         params:
+           param1:
+             variable: unknown_variable
+             optional: true
+           param2:
+             variable: my_hidden_variable
+             propertyerror: false
+   ...
+
+An example with an identifier type parameter:
+
+.. code-block:: yaml
+
+   %YAML 1.2
+   ---
+   version: 1.1
+
+   my_dyn_family_{{ identifier }}:
+     type: dynamic
+     dynamic:
+       - val1
+       - val2
+     description: 'Describe {{ identifier }}'
+
+     my_dyn_var:
+       validators:
+         - jinja: |
+             {% if my_dyn_family_.my_dyn_var == param1 %}
+               forbidden!
+             {% endif %}
+           params:
+             param1:
+               type: identifier
+   ...
+
+In this example, we see a :term:`dynamically built family`. Two families will be created: `my_dyn_family_val1.my_dyn_var` and `my_dyn_family_val2.my_dyn_var`.
+
+The value of the variable within this family cannot be equal to the value
+of the identifier (`val1` and `val2` respectively).
+
+An example with an index type parameter:
+
+.. code-block:: yaml
+
+   %YAML 1.2
+   ---
+   version: 1.1
+
+   family:
+     type: sequence
+
+     leader:
+       default:
+         - val1
+         - val2
+
+     follower1:
+       type: integer
+       validators:
+         - type: jinja
+           jinja: |-
+             {% if family.follower1 == param1 %}
+               forbidden!
+             {% endif %}
+           params:
+             param1:
+               type: index
+   ...
diff --git a/docs/format_content/variable.rst b/docs/format_content/variable.rst
new file mode 100644
index 000000000..a8b1d3cd0
--- /dev/null
+++ b/docs/format_content/variable.rst
@@ -0,0 +1,608 @@
+.. _variable:
+
+The variables
+==============
+
+Synopsis
+---------
+
+.. glossary::
+
+   variable
+
+       A variable is an abstract black box (container) paired with an associated symbolic name, most often an option configuration, which contains some defined or undefined data setting referred to as a :term:`value`.
+
+   value
+
+       A value is a variable's setting.
+       Variable can have a default value, that is a setting defined in the :term:`structure file`,
+       or no value at all, then the value needs to be defined later by the :term:`operator`.
+
+.. discussion:: Discussion
+
+    The variable is, by definition, strongly typed.
+    Rougail uses static type definition and even type inference.
+    Indeed, the constistency handling system heavyly relies on the type system definition.
+    Variables may only be able to store a specified data type.
+    OK, variables are the containers for storing the values. It has something to do with typing.
+
+    But consitency handling system is is not just about strong typing. It is more than that.
+
+.. _namingconvention:
+
+Names
+------
+
+It is with its name that we will be able to interact with the variable.
+
+The name of a variable must be chosen carefully.
+It is the path that will allow users to retrieve their defined values.
+An obscure name will cause confusion.
+
+By design, you cannot use just any character to name your variable.
+
+The regular expression that validates our variable and family naming policy is:
+
+.. code-block:: python
+
+    re.compile(r"^[a-z0-9_]*$")
+
+You can see that it is mandatory to only use lowercase ASCII letters, numbers and the `"_"` (undescore) character. 
+The snake case typographic convention is therefore used.
+
+.. attention:: The name must not starts with `_` character.
+
+Parameters
+-------------
+
+.. list-table::
+   :widths: 15 45
+   :header-rows: 1
+
+   * - Parameter
+     - Comments
+
+   * - **description**
+
+       `string`
+     - Description of the variable.
+
+       User information to understand the usefulness of the variable.
+
+       .. seealso:: tutorial with a real world sample :doc:`description parameter <../tutorial/preliminary>`
+
+   * - **help**
+
+       `string`
+     - Additional help associated with the variable.
+
+       .. seealso:: tutorial with a real world sample :doc:`help parameter <../tutorial/document>`
+
+   * - **mode**
+
+       `string`
+     - Variable's mode.
+
+       The mode is a way of classifying the importance, the level of expertise required, or the access rights to a variable.
+
+       This parameter allows you to define the variable mode.
+
+       .. attention:: Mode is not configured by default. You have to define mode level before use this parameter.
+
+       **Default value**: The `default` mode of a variable is the mode of the parent family, but there is special cases:
+
+       - a variable with an automatically modified value or an automatic read-only variable is by default in the smaller mode
+       - if the variable is not in a family, the variable will have a medium mode by default
+       - a :term:`mandatory` variable without default value (calculate or not) will have the smaller mode
+
+       .. seealso:: tutorial with a real world sample :doc:`mode parameter <../tutorial/modes>`
+
+   * - **tags**
+
+       `list of strings`
+     - A tag allows you to specialize a variable.
+
+       A tag can have several functions:
+
+       - Additional information in the documentation
+       - Variable exclusion
+       - Variable selection
+       - and so on
+
+   * - **examples**
+
+       `list`
+     - List of examples to illustrate possible values for a variable
+
+       .. seealso:: tutorial with a real world sample :doc:`examples parameter <../tutorial/document>`
+
+   * - **test**
+
+       `list`
+     - The `test` parameter is a special parameter that allows :term:`structure file` designers to influence a test robot by specifying useful values to test.
+
+       Concretely, the content of this parameter is recorded in the `information` parameter of the corresponding `Tiramisu` option object.
+
+   * - **type**
+
+       `string`
+     - Type of the variable.
+
+       See the list of available type below.
+
+       .. seealso:: tutorial with a real world sample :doc:`type parameter <../tutorial/types>`
+
+   * - **params**
+
+       `object`
+     - Parameters to adjust type validation of the variable.
+
+       See the list of available parameters for each type below.
+
+       .. seealso:: tutorial with a real world sample :doc:`params parameter <../tutorial/types>`
+
+   * - **multi**
+
+       `boolean`
+
+     - The value of the variable is a list.
+
+       **Default value**: mostly `false`, but there is special cases:
+
+       - the default value is a list
+       - a variable with a variable :term:`multiple ` has default value
+
+       It possible to adjust multi parameters with **params**:
+
+       - multi_length: number of expected values for a multiple variable
+       - multi_min_length: maximum number of expected values for a multiple variable
+       - multi_max_length: minimum number of expected values for a minimum variable
+
+       .. seealso:: tutorial with a real world sample :doc:`multi parameter <../tutorial/multiple>`
+
+   * - **validators**
+
+       `list`
+     - Value validators.
+
+       The value of the variable will be considered invalid if the Jinja template return an error.
+
+       .. seealso:: tutorial with a real world sample :doc:`validators parameter <../tutorial/validators>`
+
+   * - **default**
+     - Default value(s) of the variable.
+
+       This value is typed, you must correctly fill out the YAML file to avoid defining a value with an incorrect type. For example, a `integer` must be a digit type, a :term:`multiple ` variable must be a `list` type, ...
+
+       For a non leading :term:`multiple ` variable, the first value defined in the list will also be the default value proposed if a new value is added to this variable.
+
+       .. seealso:: tutorial with a real world sample :doc:`default parameter <../tutorial/preliminary>`
+
+   * - **secret_manager**
+     - The variable use a secret manager to get value
+       .. todo:: document it
+
+   * - **auto_save**
+
+       `boolean`
+     - Variable with automatically modified value.
+
+       A variable with automatically modified value is a variable whose value will be considered as *modified* (that is, it is no longer the variable's default value).
+
+       For example, if the value of this variable comes from a calculation, the value will no longer be recalculated.
+
+       These variables are usually :term:`required ` variables. In fact, these variables are only automatically modified if they have a value.
+
+       **Default value**: `false`
+
+       .. seealso:: tutorial with a real world sample :doc:`auto_save parameter <../tutorial/auto_save>`
+
+   * - **mandatory**
+
+       `boolean` or :term:`calculation`
+     - Mandatory variable.
+
+       Variable whose value is `required`.
+
+       For a :term:`multiple ` variable, this means that the list shall not be empty (means `null` (`None`) or empty list for a i:term:`multiple `).
+
+       **Default value**: `true`
+
+       .. seealso:: tutorial with a real world sample :doc:`mandatory parameter <../tutorial/nullable>`
+
+   * - **empty**
+
+       `boolean` or :term:`calculation`
+     - The value `null` (`None`) is not allowed in :term:`multiple ` variable.
+
+       The `mandatory` control only if the length :term:`multiple ` variable is not 0. But the `null` value is not allowed. With empty parameter we can allowed it.
+
+       If `null` is an appropriate value, set empty: false.
+
+       **Default value**: `true`
+
+   * - **unique**
+
+       `boolean`
+     - The :term:`multiple ` variable accepts the same value several times.
+
+       If unique is set to `false`, a :term:`multiple ` variable only accepts the same value once in the list.
+
+       **Default value**: `false`
+
+   * - **hidden**
+
+       `boolean` or :term:`calculation`
+     - Invisible variable.
+
+       Enables us to *hide* a variable.
+
+       This means that the variable will no longer be visible in `read-write` mode, but only for calculations or in `read-only` mode.
+
+       When a variable is made invisible, the user will not be able to modify its value; if he has already succeeded in modifying it, this value will not be taken into account.
+
+       **Default value**: `false`
+
+       .. seealso:: tutorial with a real world sample :doc:`hidden parameter <../tutorial/properties>` (the tutorial focuses on family, but the principle is the same for a variable)
+
+   * - **disabled**
+
+       `boolean` or :term:`calculation`
+     - Disabled variable.
+
+       Allows us to *deactivate* a variable.
+
+       This means that the variable will no longer be visible to the user but also to a :term:`calculation`.
+
+       **Default value**: `false`
+
+       .. seealso:: tutorial with a real world sample :doc:`disabled parameter <../tutorial/properties>` (the tutorial focuses on family, but the principle is the same for a variable)
+
+   * - **frozen**
+
+       `boolean` or :term:`calculation`
+     - Read only variable.
+
+       Enables us to have an immutable variable's value.
+
+       This means that the variable will be visible but user will not allowed to change the value.
+
+       **Default value**: `false`
+
+
+   * - **redefine**
+
+       `boolean`
+     - It is possible to define a variable in one :term:`structure file` and change its behavior in a second :term:`structure file`. In this case you must explicitly redefine the variable.
+
+       **Default value**: `false`
+
+   * - **exists**
+
+       `boolean`
+     - This parameter does two things:
+
+         - creates a variable if it does not exist in another :term:`structure file` (otherwise do nothing), in this case the value of the parameter must be `true`
+         - in conjunction with the `redefine` parameter set to `true`, only modifies the behavior if it is pre-existing, in which case the parameter's value must be `false`.
+
+       **Default value**: `null`
+
+   * - **choices**
+
+       :term:`calculation` or a list of :term:`calculation` or data
+     - Available choices.
+
+       .. important:: This parameter is only available for variable with `choice` type.
+
+   * - **regexp**
+
+       `string`
+     - Validation with a regular expressions.
+
+       .. important:: This parameter is only available for variable with `regexp` type.
+
+Short-hand declaration
+----------------------------
+
+Short-hand declaration is a way to declare a variable in a condenced number of line.
+
+To create a variable, just add a key with it's name:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_variable:
+    ...
+
+You can add a default value:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_variable: my_value
+    ...
+
+Or a multi default value:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_variable:
+      - my_value_1
+      - my_value_2
+    ...
+
+By default, the description of the variable is the variable name.
+It's a good practice to describe a variable. Just add comment in same line of name, this comment is used as description:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_variable: my_value  # This is a great variable
+    ...
+
+But in short-hand notation, you can only define variable name, default value, multi and description.
+
+.. attention:: Any other parameters will create extra variables or families
+
+.. 
+.. .. glossary::
+.. 
+..     short-hand notation
+..     
+..         A short-hand notation in Rougail is the ability to define a variable in 
+..         a short-hand way, there are several example: 
+..         
+..         - a default value: 
+..         
+..         .. code-block:: yaml
+..         
+..             my_var: true 
+..             
+..         instead of:
+..         
+..         .. code-block:: yaml
+..         
+..             my_var:
+..               default: true
+
+
+Variable's types
+-----------------
+
+A variable **always has a type**. The system is **strongly** typed.
+
+Depending on the definition of the variable type, the defined variable will accept values of the associated type.
+
+Primitive Types
+'''''''''''''''''''
+
+.. list-table::
+   :widths: 15 25 20 15
+   :header-rows: 1
+
+   * - Value
+     - Comments
+     - Parameters
+     - Samples
+
+   * - string
+     - character string (default type)
+     -
+     - example
+
+       "1"
+
+       "true"
+
+   * - integer
+     - a integer
+     - `min_integer`: minimum integer allowed (unlimited by default)
+
+       `max_integer`: maximum integer allowed (unlimited by default)
+     - 42
+
+   * - float
+     - a floating number
+     -
+     - 1.42
+
+   * - boolean
+     - A boolean, if no value is defined the default value of this variable will be `true`, the variable will also be :term:`mandatory` by default
+
+       .. seealso:: tutorial with a real world sample :doc:`boolean type variable <../tutorial/types>`
+     -
+     - `true`
+
+       `false`
+
+Specialized type
+''''''''''''''''''
+
+.. list-table::
+   :widths: 5 45 45 5
+   :header-rows: 1
+
+   * - Value
+     - Comments
+     - Parameters
+     - Samples
+
+   * - secret
+     - a secret (like a password, a private key, etc.)
+
+       .. seealso:: tutorial with a real world sample :ref:`secret type variable `
+     - `min_len`: minimum characters length for the secret (unlimited by default)
+
+       `max_len`: maximum characters length for the secret (unlimited by default)
+
+       `forbidden_char`: forbidden characters (no character default)
+     - `hO_'hi`
+
+   * - mail
+     - a mail address
+     -
+     - test@rougail.example
+
+   * - unix_filename
+     - a file name in the Unix meaning
+     - `allow_relative`: this filename could be a relative path (`false` by default)
+
+       `test_existence`: this file must exist (`false` by default)
+
+       `types`: "file type allowed ("file", "directory")
+     - :file:`/etc/passwd`
+
+   * - date
+     - a date in the format `%Y-%m-%d`
+     -
+     - `2021-01-30`
+
+   * - unix_user
+     - a user in the Unix meaning
+
+       .. seealso:: tutorial with a real world sample :doc:`unix type variable <../tutorial/redefine>`
+     -
+     - test
+
+   * - ip
+     - any kind of IPv4 address
+     - `private_only`: only private IPs (`false` by default)
+
+       `allow_reserved`: allows reserved IPs (`true` by default)
+
+       `cidr`: IP must be in CIDR format (`false` by default)
+     - `1.2.3.4`
+
+   * - netmask
+     - mask of an IPv4 address
+     -
+     - `255.255.255.0`
+
+   * - network
+     - network address
+     - `cidr`: network must be in CIDR format (`false` by default)
+
+       `private_only`: private network are allowed (`false` by default)
+       
+       `allow_reserved`: reserved network are allowed (`false` by default)
+     - `192.168.1.0`
+
+   * - broadcast
+     - broadcast address
+     -
+     - `1.1.1.255`
+
+   * - domainname
+     - domain name
+     - `type`: type of domain name (`domainname` (by default), `netbios`, `hostname`)
+
+       `allow_ip`: the domain name can be an IP (`false` by default)
+
+       `allow_cidr_network`: the domain name can be network in CIDR format (`false` by default)
+
+       `allow_without_dot`: the domain name can be a hostname (`false` by default)
+
+       `allow_startswith_dot`: the domain name can starts by a dot (`false` by default)
+
+       `test_existence`: the domain name must exist (`false` by default)
+
+       .. seealso:: tutorial with a real world sample :doc:`domainname type variable <../tutorial/types>` or :doc:`a more complet domainname type variable <../tutorial/multiple>`
+     - `rougail.example`
+
+   * - web_address
+     - web address
+     - `type`: type of domain name (`domainname` (by default), `netbios`, `hostname`)
+
+       `allow_ip`: the domain name can be an IP (`false` by default)
+
+       `allow_cidr_network`: the domain name can be network in CIDR format (`false` by default)
+
+       `allow_without_dot`: the domain name can be a hostname (`false` by default)
+
+       `allow_startswith_dot`: the domain name can starts by a dot (`false` by default)
+
+       `test_existence`: the domain name must exist (`false` by default)
+
+       `allow_range`: can be range of port, for example 80:85 (`false` by default)
+
+       `allow_zero`: port 0 is allowed (false by default)
+
+       `allow_wellknown`: well-known ports (1 to 1023) are allowed (`true` by default)
+
+       `allow_registred`: registred ports (1024 to 49151) are allowed (`true` by default)
+
+       `allow_private`: private ports (greater than 49152) are allowed (`false` by default)
+
+       .. seealso:: tutorial with a real world sample :doc:`web_address type variable <../tutorial/webaddress>`
+     - http://rougail.example
+
+   * - port
+     - port
+     - `allow_range`: can be range of port, for example 80:85 (`false` by default)
+
+       `allow_zero`: port 0 is allowed (false by default)
+
+       `allow_wellknown`: well-known ports (1 to 1023) are allowed (`true` by default)
+
+       `allow_registred`: registred ports (1024 to 49151) are allowed (`true` by default)
+
+       `allow_private`: private ports (greater than 49152) are allowed (`false` by default)
+
+       .. seealso:: tutorial with a real world sample :doc:`port type variable <../tutorial/types>`
+     - 8080
+
+   * - mac
+     - MAC address
+     -
+     - 11:11:11:11:11:11
+
+   * - unix_permissions
+     - access rights to the file, directory, etc.
+     -
+     - 644
+
+   * - choice
+     - available choices
+
+       .. seealso:: tutorial with a real world sample :doc:`choice type variable <../tutorial/choice>`
+     -
+     -
+
+   * - regexp
+     - Validation with a regular expressions
+       
+       .. seealso:: tutorial with a real world sample :ref:`regexp type variable `
+     -
+     - r"^#(?:[0-9a-f]{3}){1,2}$"
+
+Default type
+''''''''''''''
+
+If the `type` parameter is not set, Rougail has to define a logical type to valid correctly values:
+
+- if `choices` or `regexp` parameter is set, Rougail will set the `choice` or `regexp` type
+
+.. seealso:: tutorial with a real world sample :doc:`choice deducted type <../tutorial/choice>` or :ref:`regexp deducted type `
+
+- if a default value is define, Rougail will infers default value type and set a primitive type to the variable
+
+.. seealso:: tutorial with a real world sample :doc:`type inference <../tutorial/types>`
+
+- if a variable calculation is define as default value, Rougail copy the type
+
+.. seealso:: tutorial with a real world sample :doc:`type copying <../tutorial/calculated>`
+
+- the default type is `string`
diff --git a/docs/gettingstarted.rst b/docs/gettingstarted.rst
deleted file mode 100644
index 894d12c35..000000000
--- a/docs/gettingstarted.rst
+++ /dev/null
@@ -1,191 +0,0 @@
-.. |Tiramisu| replace:: Tiramisu
-.. _tiramisu: https://forge.cloud.silique.fr/stove/tiramisu
-
-Getting started
-====================
-
-What is a consistency handling system ?
-------------------------------------------------
-
-.. questions:: Question: "OK, I have understood that the Rougail stuff enables me to take advantage of |Tiramisu|. But what is all this for? What is exactly a consistency handling system? And again, what is this |Tiramisu| library used for?"
-
-    *Answer*: Well, let's explain what |Tiramisu| is and how we are using the |Tiramisu| library.
-
-.. glossary::
-
-   Tiramisu
-
-        |Tiramisu| is a consistency handling system that has initially been designed
-        in the configuration management scope. To put it more simply,
-        this library is generally used to handle configuration options.
-
-        It manages variables and group of variables. In the Tiramisu scope we call
-        it *options* and *option descriptions*.
-
-In the Rougail scope, we call it :term:`variable`\ s and :term:`families`.
-In Rougail, the families and variables are located in the :term:`dictionaries`.
-
-And this is what we are going to explain in this page.
-
-The dictionaries
----------------------
-
-.. glossary::
-
-   dictionary
-   dictionaries
-
-       A dictionary in the Rougail meaning is a YAML file that describes variables
-       and their dependencies / consistencies.
-       There can be a lot of dictionary files located in many different folders.
-
-       Rougail reads all the dictionaries and loads them into a single object
-       that handles the variables consistency.
-
-.. image:: images/schema.png
-
-The main advantage is that declaring variables and writing consistency is as simple
-as writing YAML. With Rougail it is not necessary to write :term:`Tiramisu` code any more.
-It simplifies a lot of things.
-
-And rather than writing :term:`Tiramisu` code, we can declare variables and describe the relationships between variables in a declarative mode (that is, in a YAML file).
-
-Once the dictionaries are loaded by Rougail, we find all the power of the :term:`Tiramisu` configuration management tool.
-
-The YAML dictionaries format
------------------------------
-
-Before getting started with Rougail we need to learn the specifics of the YAML dictionaries file format (as well as some templating concepts).
-
-.. FIXME parler de jinja https://jinja.palletsprojects.com
-
-Here is a :term:`dictionary` example:
-
-.. code-block:: yaml
-   :linenos:
-
-    ---
-    version: '1.1'
-    proxy:
-      description: Configure Proxy Access to the Internet
-      type: family
-
-Line 3, we declare a **variable** named `proxy` with his `description` line 4 and his `type` line 5.
-
-The variables
------------------
-
-variable
-
-    Here is a second definition of a :term:`variable`: it is a declaration unit that represents a business domain metaphor,
-    the most common example is that a variable that represents a configuration option
-    in a application, but a variable represents something more that a configuration option.
-    It provides a business domain specific representation unit.
-
-.. note:: Dictionaries can just define a list of variables, but we will see that
-          we can specify a lot more. We can define variables **and** their relations,
-          **and** the consistency between them.
-
-In the next step, we will explain through a tutorial how to construct a list of variables.
-
-Families of variables
---------------------------
-
-.. glossary::
-
-   family
-   families
-
-       A family of variables is simply a collection of variables that refer to
-       the same business model category. It's just a variables container.
-       Think of it as a container as well as a namespace.
-
-A "hello world" with Rougail
-------------------------------
-
-We're gonna make the simplest possible example.
-
-.. prerequisites:: Prerequisites
-
-We assume that Rougail's library is installed on your computer (or in a virtual environment).
-
-.. exercise:: Let's make a Hello world
-
-Here is the tree structure we want to have::
-
-    workplace
-    ├── dict
-    │   ├── hello.yml
-    └── hello.py
-
-- Let's make a :file:`workplace` directory, with a :file:`dict` subfolder.
-- First, we need a :term:`dictionary`, so let's make the :file:`hello.yml` file
-  which is located in the :file:`dict` subfolder, with the following content:
-
-.. code-block:: yaml
-   :caption: The `hello.yaml` file
-
-    ---
-    version: '1.1'
-    hello:
-      default: world
-
-- Then we make a :file:`hello.py` in our root :file:`workplace` directory:
-
-.. code-block:: python
-   :caption: The :file:`hello.py` file
-
-    from rougail import Rougail, RougailConfig
-
-    RougailConfig['dictionaries_dir'] = ['dict']
-    rougail = Rougail()
-    config = rougail.get_config()
-    print(config.value.get())
-
-.. demo:: Let's run the :file:`hello.py` script
-
-We launch the script:
-
-.. code-block:: bash
-
-    python hello.py
-
-And we obtain the following result:
-
-.. code-block:: python
-
-    {'rougail.hello': 'world'}
-
-**Congratulations ! You have successfully completed your first Rougail script.**
-
-A "Hello,  " sample with a family
-------------------------------------------
-
-Let's continuing on our "Hello world" theme and add a :term:`family` container.
-
-.. code-block:: yaml
-   :caption: the :file:`hello.yml` file
-   :linenos:
-
-    ---
-    version: '1.1'
-    world:
-      description: Hello world family container
-      name:
-        description: Somebody to say hello
-        default: rougail
-
-Here, we have a family named `world`.
-This family contains a variable named `name`
-
-Again, let's validate this YAML file against Rougail's API:
-
-.. code-block:: bash
-
-    python hello.py
-
-We then have the output:
-
-.. code-block:: python
-
-    {'rougail.world.name': 'rougail'}
diff --git a/docs/helper_prerequisites.py b/docs/helper_prerequisites.py
new file mode 100755
index 000000000..bdd341067
--- /dev/null
+++ b/docs/helper_prerequisites.py
@@ -0,0 +1,72 @@
+#!/usr/bin/env python3
+"""
+Génère une section de documentation "prerequisites" pour un tutoriel Rougail,
+en utilisant une plage de versions (tags git) spécifiée en arguments.
+
+Usage:
+    python helper_prerequisites.py   
+
+Arguments:
+    start_version (str) : La version de début (tag git) pour la plage de tags.
+    end_version (str)   : La version de fin (tag git) pour la plage de tags.
+    branch (str)        : La branche du dépôt git à mentionner dans la documentation.
+
+Exemple:
+
+    ./helper_prerequisites.py v1.1_080 v1.1_085 1.1
+
+"""
+
+import sys
+
+def generate_prerequisites(start_version: str, end_version: str, branch: str) -> str:
+    """
+    Génère une section de documentation "prerequisites" pour une plage de versions donnée.
+
+    Args:
+        start_version (str): La version de début (tag git) pour la plage.
+        end_version (str): La version de fin (tag git) pour la plage.
+        branch (str): La branche du dépôt git à mentionner.
+
+    Returns:
+        str: La section de documentation formatée.
+    """
+    template = """.. prerequisites:: Prerequisites
+
+    - We assume that Rougail's library is :ref:`installed ` on your computer.
+
+    - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step
+      by checking out the corresponding tag of the `rougail-tutorials` git repository.
+      Each tag corresponds to a stage of progress in the tutorial.
+      Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps.
+
+    If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `,
+    this workshop page corresponds to the tags :tutorial:`{start_version} ` to :tutorial:`{end_version} `
+    in the repository.
+
+    ::
+
+       git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git
+       git switch --detach {start_version}
+"""
+    return template.format(
+        start_version=start_version,
+        end_version=end_version,
+        branch=branch
+    )
+
+def main() -> None:
+    """Point d'entrée principal du script."""
+    if len(sys.argv) != 4:
+        print("Erreur : Trois arguments doivent être spécifiés :   .")
+        print(f"Usage: {sys.argv[0]}   ")
+        sys.exit(1)
+
+    start_version = sys.argv[1]
+    end_version = sys.argv[2]
+    branch = sys.argv[3]
+    print(generate_prerequisites(start_version, end_version, branch))
+
+if __name__ == "__main__":
+    main()
+
diff --git a/docs/helper_type_along.py b/docs/helper_type_along.py
new file mode 100755
index 000000000..20a45ccd5
--- /dev/null
+++ b/docs/helper_type_along.py
@@ -0,0 +1,47 @@
+#!/usr/bin/env python3
+"""
+Génère une section de documentation "type-along" pour un tutoriel,
+en utilisant une version spécifiée en argument.
+
+Usage:
+    python generate_type_along.py 
+
+Arguments:
+    version (str) : La version (tag git) à utiliser dans la documentation.
+
+Exemple:
+    python generate_type_along.py v1.0.0
+"""
+
+import sys
+
+def generate_type_along(version: str) -> str:
+    """
+    Génère une section de documentation "type-along" pour une version donnée.
+
+    Args:
+        version (str): La version (tag git) à inclure dans la documentation.
+
+    Returns:
+        str: La section de documentation formatée.
+    """
+    template = """.. type-along:: For those who follow the tutorial with the help of the git repository
+
+    Now you need to checkout the :tutorial:`{version} ` version::
+
+          git switch --detach {version}
+"""
+    return template.format(version=version)
+
+def main() -> None:
+    """Point d'entrée principal du script."""
+    if len(sys.argv) != 2:
+        print("Erreur : Une version doit être spécifiée en argument.")
+        print(f"Usage: {sys.argv[0]} ")
+        sys.exit(1)
+
+    version = sys.argv[1]
+    print(generate_type_along(version))
+
+if __name__ == "__main__":
+    main()
diff --git a/docs/images/QuestionaryChoice.png b/docs/images/QuestionaryChoice.png
new file mode 100644
index 000000000..428ded997
Binary files /dev/null and b/docs/images/QuestionaryChoice.png differ
diff --git a/docs/images/UserDataOutput.png b/docs/images/UserDataOutput.png
new file mode 100644
index 000000000..0f8787959
Binary files /dev/null and b/docs/images/UserDataOutput.png differ
diff --git a/docs/images/dessin.png b/docs/images/dessin.png
new file mode 100644
index 000000000..03f58659e
Binary files /dev/null and b/docs/images/dessin.png differ
diff --git a/docs/images/dessin.svg b/docs/images/dessin.svg
new file mode 100644
index 000000000..393c86364
--- /dev/null
+++ b/docs/images/dessin.svg
@@ -0,0 +1,385 @@
+
+
+
+  
+  
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+    
+      
+      
+    
+    
+    
+      
+      
+    
+    
+      
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+    
+      
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+    
+    
+    
+    
+      
+    
+    
+    
+      
+    
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+    
+    
+    
+    
+    
+    
+      
+    
+    
+      
+    
+    
+    
+  
+  
+    
+    
+    
+    
+    
+      
+        
+      
+      
+        
+      
+      
+    
+    
+      
+        
+      
+      
+        
+      
+      
+    
+    
+      
+        
+      
+      
+        
+      
+      
+    
+    
+      
+        
+      
+      
+        
+      
+      
+    
+    
+      
+        
+      
+      
+        
+      
+      
+    
+    
+      
+        
+      
+      
+        
+      
+      
+    
+    
+    
+    
+    
+    
+    
+    
+      
+        
+      
+      
+        
+      
+      
+    
+    
+      
+        
+      
+      
+        
+      
+      
+    
+    
+      
+        
+      
+      
+        
+      
+      
+    
+    
+      
+        
+      
+      
+        
+      
+      
+    
+    
+    
+    
+      
+        
+      
+      
+        
+      
+      
+    
+    
+    
+      
+        
+      
+      
+        
+      
+      
+    
+    
+    
+    
+    
+    
+    
+      
+        
+      
+      
+        
+      
+      
+    
+    
+      
+        
+      
+      
+        
+      
+      
+    
+    
+      
+        
+      
+      
+        
+      
+      
+    
+    
+      
+        
+      
+      
+        
+      
+      
+    
+    
+    
+    
+      
+        
+      
+      
+        
+      
+      
+    
+    
+    
+      
+        
+      
+      
+        
+      
+      
+    
+    
+    
+      
+        
+      
+      
+        
+      
+      
+    
+    
+    
+      
+        
+      
+      
+        
+      
+      
+    
+    
+    
+    
+    
+    
+    Read Write
+    Structure
+    hidden
+    disabled
+    Read Only
+    
+    
+    
+    
+  
+
\ No newline at end of file
diff --git a/docs/images/read_write.png b/docs/images/read_write.png
new file mode 100644
index 000000000..a5540f3ba
Binary files /dev/null and b/docs/images/read_write.png differ
diff --git a/docs/images/schema.png b/docs/images/schema.png
deleted file mode 100644
index 9968ee2a2..000000000
Binary files a/docs/images/schema.png and /dev/null differ
diff --git a/docs/images/schema.svg b/docs/images/schema.svg
deleted file mode 100644
index f94b83a46..000000000
--- a/docs/images/schema.svg
+++ /dev/null
@@ -1,1306 +0,0 @@
-
-
-
-
-  
-  
-    
-      
-    
-    
-      
-    
-    
-      
-    
-    
-      
-    
-    
-      
-      
-    
-    
-      
-    
-    
-      
-    
-    
-      
-    
-    
-      
-    
-    
-      
-    
-    
-      
-    
-    
-      
-    
-    
-    
-      
-      
-    
-    
-    
-      
-      
-    
-    
-      
-    
-    
-    
-      
-      
-    
-    
-      
-    
-    
-    
-      
-    
-    
-    
-      
-    
-    
-    
-    
-    
-      
-    
-    
-    
-      
-    
-    
-      
-    
-    
-    
-    
-      
-    
-    
-    
-      
-    
-    
-    
-      
-    
-    
-      
-    
-    
-      
-    
-    
-      
-    
-    
-      
-    
-    
-    
-    
-      
-    
-    
-    
-      
-    
-    
-    
-    
-      
-    
-    
-    
-      
-    
-    
-    
-    
-      
-    
-    
-    
-      
-    
-    
-      
-    
-    
-      
-    
-    
-      
-    
-    
-      
-    
-    
-      
-    
-    
-      
-    
-    
-      
-    
-    
-      
-    
-  
-  
-    
-    
-    
-    
-    
-    
-    
-    
-    
-    
-    
-    
-    IRAMISU
-    
-      
-      
-      
-      
-    
-    
-      
-      
-      
-      
-    
-    
-    
-    
-      
-        
-        
-      
-    
-    
-    
-    version: '0.10'# describe a first service with a single fileservices:- service:- name: my_servicefile:- engine: jinjatext: /etc/filename# describe a variable my_first_variable# and a family with a variable my_second_variablevariables:- variable:- name: my_first_variablevalue:- text: my_value- family:- name: my_familyvariables:- variable:- name: my_second_variabletype: numbermandatory: truevalue:- text: 1
-    
-    
-    
-    
-    
-    version: '0.10'# describe a first service with a single fileservices:- service:- name: my_servicefile:- engine: jinjatext: /etc/filename# describe a variable my_first_variable# and a family with a variable my_second_variablevariables:- variable:- name: my_first_variablevalue:- text: my_value- family:- name: my_familyvariables:- variable:- name: my_second_variabletype: numbermandatory: truevalue:- text: 1
-    Dictionary
-    
-    
-    
-  
-
diff --git a/docs/images/tiramisu_get_set.png b/docs/images/tiramisu_get_set.png
new file mode 100644
index 000000000..6699f1ee0
Binary files /dev/null and b/docs/images/tiramisu_get_set.png differ
diff --git a/docs/index.rst b/docs/index.rst
index 83027ef16..1415d194f 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -10,54 +10,59 @@ Rougail
 
 .. image:: images/logo.png
 
-- is a `delicious cooked dish `_ from the Mauritius  and Reunion Islands,
+- is a `delicious cooked dish `_ from the Mauritius and Reunion Islands,
 
-- it is also a `Python3 `_ library  which enables us to conveniently load application :term:`variable`\ s in a simple `YAML `_ format in such a way that the end user consumer can handle them consistently (that is, against an user-defined consistency).
-
-In other words, using Rougail in your application or your python libraries can tansform end user consumer defined consistency rules into highly consistent business objects.
-
-We then have to say that the handling system used to ensure the variables integrity is another python library, called :term:`Tiramisu`. Rougail is currently strongly affiliated with Tiramisu.
-
-.. note:: Rougail is currently intended to work in coordination with :term:`Tiramisu` and **is not** intended to be connected with any other consistency handling system.
-
-Explained differently, Rougail allows you to easily implement an integration of the powerful tiramisu consistency handling system.
+- it is also a tool which enables us to conveniently load :term:`variable`\ s in `YAML `_ file format in such a way that the end user consumer can handle them consistently (that is, against an user-defined consistency).
 
 .. toctree::
    :titlesonly:
-   :caption: Getting started
+   :caption: What is it all about
 
-   gettingstarted
-   tutorial
+   abstract
+   vac
 
 .. toctree::
    :titlesonly:
-   :caption: The library
-   
-   library
+   :caption: Structured data
+
+   structured_data/index
+   format_content/index
+   tutorial/index
+
+.. toctree::
+   :titlesonly:
+   :caption: User data
+
+   user_data/prerequisites
+   user_data/index
+
+.. toctree::
+   :titlesonly:
+   :caption: Output
+
+   output/index
+
+.. toctree::
+   :titlesonly:
+   :caption: The library and the tools
+
    configuration
+   cli
+   library/index
+   tiramisu
 
 .. toctree::
    :titlesonly:
-   :caption: The dictionaries
+   :caption: Developper notes
 
-   dictionary
-   dict_convention
-   
-.. toctree::
-   :titlesonly:
-   :caption: The variables
-
-   variable
-   family
-   fill
-   Value checks 
-   condition
-      
-.. toctree::
-   :titlesonly:
-   :caption: Notes
-   
+   release
    developer
+   documentation
+
+.. toctree::
+   :hidden:
+
+   install
 
 .. rubric:: Index page
 
diff --git a/docs/install.rst b/docs/install.rst
new file mode 100644
index 000000000..73d0e11c9
--- /dev/null
+++ b/docs/install.rst
@@ -0,0 +1,35 @@
+.. _installation:
+
+Rougail library installation
+======================================
+
+Activate you virtual environment
+------------------------------------
+
+- Open a shell session
+
+- install the virtual environment: `python -m'venv' .venv`
+- activate it `./.venv/bin/activate` (or `.venv\Scripts\activate.exe` under windows)
+
+Standard installation
+---------------------------
+
+You can use the `pip` python installer, here is the install command:
+
+.. code-block:: bash
+
+    pip install rougail
+
+Installation of third-party libraries
+-------------------------------------------
+
+First, download the :download:`requirements.txt file: `
+
+.. literalinclude:: install/requirements.txt
+   :caption: The :file:`requirements.txt` requirements file
+
+Then in your virtual environment, recursively install the third-party libraries as follows:
+
+.. code-block:: bash
+
+    pip install -r requirements.txt --extra-index-url https://test.pypi.org/simple/
diff --git a/docs/install/requirements.txt b/docs/install/requirements.txt
new file mode 100644
index 000000000..579af01ba
--- /dev/null
+++ b/docs/install/requirements.txt
@@ -0,0 +1,16 @@
+rougail==1.2.0rc3
+rougail-cli==1.0.0rc0
+rougail-output-ansible==1.0.0rc0
+rougail-output-display==1.0.0rc0
+rougail-output-doc==1.0.0rc0
+rougail-output-formatter==1.0.0rc0
+rougail-output-json==1.0.0rc0
+rougail-output-table==0.1.0rc0
+rougail-user-data-ansible==1.0.0rc0
+rougail-user-data-bitwarden==1.0.0rc1
+rougail-user-data-commandline==1.0.0rc2
+rougail-user-data-environment==1.0.0rc0
+rougail-user-data-questionary==1.0.0rc0
+rougail-user-data-yaml==1.0.0rc1
+tiramisu==5.2.0rc0
+tiramisu-cmdline-parser==1.0.0rc0
diff --git a/docs/library.rst b/docs/library.rst
deleted file mode 100644
index b9a91d548..000000000
--- a/docs/library.rst
+++ /dev/null
@@ -1,233 +0,0 @@
-`Rougail`'s library description
-=================================
-
-Rougail is a configuration management library that allows you to load variables in a simple and convenient way.
-
-In the following examples, we will use a specific configuration of Rougail.
-You will find all the configuraiton options in :doc:`configuration`.
-
-To load the configuration you must import the `RougailConfig` class and set the `dictionaries_dir` values:
-
-.. code-block:: python
-
-    from rougail import RougailConfig
-
-    RougailConfig['dictionaries_dir'] = ['dict']
-
-Let's convert a dictionary
------------------------------
-
-As a reminder, a :term:`dictionary` is a set of instructions that will allow us to create :term:`families` and :term:`variables`.
-
-Let's start by creating a simple dictionary.
-
-Here is a first :file:`dict/00-base.yml` dictionary:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_variable:
-      default: my_value
-
-Then, let's create the :term:`Tiramisu` objects via the following script:
-
-.. code-block:: python
-    :caption: the `script.py` file content
-
-    from rougail import Rougail, RougailConfig
-
-    RougailConfig['dictionaries_dir'] = ['dict']
-    rougail = Rougail()
-    config = rougail.get_config()
-    print(config.value.get())
-
-Let's execute `script.py`:
-
-.. code-block:: bash
-
-    $ python3 script.py
-    {'rougail.my_variable': 'my_value'}
-
-Let's convert an extra dictionary
--------------------------------------
-
-.. index:: extras
-
-The default namespace for variables and families is `rougail`. It is possible to define other namespaces. These additional namespaces are called `extras`.
-
-.. FIXME: faire une page pour les extras
-
-Additional namespaces are defined during configuration.
-
-For example, here's how to add an `example` namespace:
-
-.. code-block:: python
-
-    RougailConfig['extra_dictionaries']['example'] = ['extras/']
-
-Then let's create an extra :term:`dictionary` :file:`extras/00-base.yml`:
-
-.. code-block:: yaml
-   :caption: the :file:`extras/00-base.yml` file content
-    ---
-    version: '1.1'
-    my_variable_extra:
-      default: my_value_extra
-
-Then, let's create the :term:`Tiramisu` objects via the following :file:`script.py` script:
-
-.. code-block:: python
-    :caption: the :file:`script.py` file content
-
-    from rougail import Rougail, RougailConfig
-
-    RougailConfig['dictionaries_dir'] = ['dict']
-    RougailConfig['extra_dictionaries']['example'] = ['extras/']
-    rougail = Rougail()
-    config = rougail.get_config()
-    print(config.value.dict())
-
-Let's execute `script.py`:
-
-.. code-block:: bash
-
-    $ python3 script.py
-    {'rougail.my_variable': 'my_value', 'example.my_variable_extra': 'my_value_extra'}
-
-Let's create a custom function
-----------------------------------
-
-We create the complementary :term:`dictionary` named :file:`dict/01-function.yml` so that the  `my_variable_jinja` variable is :term:`calculated`:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_variable_jinja:
-      type: "string"
-      default:
-        type: jinja
-        jinja: "{{ return_no() }}"
-
-Then let's define the :func:`return_no` function in :file:`functions.py`:
-
-.. code-block:: python
-   :caption: the :file:`functions.py` content
-
-   def return_no():
-       return 'no'
-
-Then, let's create the :term:`Tiramisu` objects via the following script:
-
-.. code-block:: python
-    :caption: the `script.py` file content
-
-    from rougail import Rougail, RougailConfig
-
-    RougailConfig['dictionaries_dir'] = ['dict']
-    RougailConfig['extra_dictionaries']['example'] = ['extras/']
-    RougailConfig['functions_file'] = 'functions.py'
-    rougail = Rougail()
-    config = rougail.get_config()
-    print(config.value.dict())
-
-Let's execute `script.py`:
-
-.. code-block:: bash
-
-    $ python3 script.py
-    {'rougail.my_variable': 'my_value', 'rougail.my_variable_jinja': 'no', 'example.my_variable_extra': 'my_value_extra'}
-
-The value of the `my_variable_extra` variable is calculated, and it's value comes from the :func:`return_no` function.
-
-Create your own type
-----------------------
-
-A variable has a type. This type enables the variable to define the values that are accepted by this variable.
-
-There is a series of default types, but obviously not all cases are taken.
-
-It's possible to create your own type.
-
-Here an example to a lipogram option (in a string, we cannot use "e" character):
-
-.. code-block:: python
-    :caption: the `lipogram.py` file content
-
-    from tiramisu import StrOption
-    class LipogramOption(StrOption):
-        __slots__ = tuple()
-        _type = 'lipogram'
-
-        def validate(self,
-                     value):
-            super().validate(value)
-            # verify that there is any 'e' in the sentense
-            if 'e' in value:
-                raise ValueError('Perec wrote a book without any "e", you could not do it in a simple sentence?')
-
-To add the new lipogram type in Rougail:
-
-.. code-block:: python
-    >>> from rougail import Rougail, RougailConfig
-    >>> RougailConfig['dictionaries_dir'] = ['dict']
-    >>> RougailConfig['custom_types']['lipogram'] = LipogramOption
-
-Now, we can use lipogram type.
-Here is a :file:`dict/00-base.yml` dictionary:
-
-.. code-block:: yaml
-    ---
-    version: '1.1'
-    var:
-      type: lipogram
-
-.. code-block:: python
-    >>> rougail = Rougail()
-    >>> config = rougail.get_config()
-    >>> config.option('rougail.var').value.set('blah')
-    >>> config.option('rougail.var').value.set('I just want to add a quality string that has no bad characters')
-    [...]
-    tiramisu.error.ValueOptionError: "I just want to add a quality string that has no bad characters" is an invalid lipogram for "var", Perec wrote a book without any "e", you could not do it in a simple sentence?
-
-Upgrade dictionnaries to upper version
-----------------------------------------
-
-All dictionnaries has a format version number.
-When a new format version is proposed, it is possible to automatically convert the files to the new version.
-
-We create a term:`dictionary` named :file:`dict/01-upgrade.yml` with version 1.0:
-
-.. code-block:: yaml
-
-    ---
-    version: '1.1'
-    my_variable:
-      multi: true
-    my_dyn_family:
-      type: "dynamic"
-      variable: my_variable
-      a_variable:
-
-
-.. code-block:: python
-    >>> from rougail import RougailUpgrade, RougailConfig
-    >>> RougailConfig['dictionaries_dir'] = ['dict']
-    >>> upgrade = RougailUpgrade()
-    >>> upgrade.load_dictionaries('dict_converted')
-
-The term:`dictionary` named :file:`dict_converted/01-upgrade.yml` is in version 1.1:
-
-.. code-block:: yaml
-
-   version: '1.1'
-   my_variable:
-     multi: true
-   my_dyn_family:
-     type: dynamic
-     a_variable: null
-     dynamic:
-       type: variable
-       variable: my_variable
-       propertyerror: false
diff --git a/docs/library/custom_function.rst b/docs/library/custom_function.rst
new file mode 100644
index 000000000..de58fbd4c
--- /dev/null
+++ b/docs/library/custom_function.rst
@@ -0,0 +1,47 @@
+Let's create a custom function
+==============================
+
+We create the complementary :term:`structure file` named :file:`dict/01-function.yml` so that the  `my_variable_jinja` variable is :term:`calculated`:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_variable_jinja:
+      default:
+        type: jinja
+        jinja: "{{ return_no() }}"
+    ...
+
+Then let's define the :func:`return_no` function in :file:`functions.py`:
+
+.. code-block:: python
+   :caption: the :file:`functions.py` content
+
+   def return_no():
+       return 'no'
+
+Then, let's create the :term:`Tiramisu` objects via the following script:
+
+.. code-block:: python
+    :caption: the `script.py` file content
+
+    from rougail import Rougail, RougailConfig
+
+    RougailConfig['main_structural_directories'] = ['dict']
+    RougailConfig['extra_namespaces']['example'] = ['extras/']
+    RougailConfig['functions_file'] = 'functions.py'
+    rougail = Rougail()
+    config = rougail.get_config()
+    print(config.value.dict())
+
+Let's execute `script.py`:
+
+.. code-block:: bash
+
+    $ python3 script.py
+    {'rougail.my_variable': 'my_value', 'rougail.my_variable_jinja': 'no', 'example.my_variable_extra': 'my_value_extra'}
+
+The value of the `my_variable_extra` variable is calculated, and it's value comes from the :func:`return_no` function.
diff --git a/docs/library/extra.rst b/docs/library/extra.rst
new file mode 100644
index 000000000..8c99fb6d2
--- /dev/null
+++ b/docs/library/extra.rst
@@ -0,0 +1,45 @@
+Let's convert an extra namespace structural file 
+================================================
+
+.. index:: extras
+
+The default namespace for variables and families is `rougail`. It is possible to define other namespaces. These additional namespaces are called `extras`.
+
+.. FIXME: faire une page pour les extras
+
+Additional namespaces are defined during configuration.
+
+For example, here's how to add an `example` namespace:
+
+.. code-block:: python
+
+    RougailConfig['extra_namespaces']['example'] = ['extras/']
+
+Then let's create an extra :term:`structure file` :file:`extras/00-base.yml`:
+
+.. code-block:: yaml
+   :caption: the :file:`extras/00-base.yml` file content
+    ---
+    version: '1.1'
+    my_variable_extra:
+      default: my_value_extra
+
+Then, let's create the :term:`Tiramisu` objects via the following :file:`script.py` script:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+
+    RougailConfig['main_structural_directories'] = ['dict/']
+    RougailConfig['extra_namespaces']['example'] = ['extras/']
+    rougail = Rougail()
+    config = rougail.get_config()
+    print(config.value.dict())
+
+Let's execute `script.py`:
+
+.. code-block:: bash
+
+    $ python3 script.py
+    {'rougail.my_variable': 'my_value', 'example.my_variable_extra': 'my_value_extra'}
diff --git a/docs/library/index.rst b/docs/library/index.rst
new file mode 100644
index 000000000..f35226b3b
--- /dev/null
+++ b/docs/library/index.rst
@@ -0,0 +1,87 @@
+`Rougail`'s library description
+=================================
+
+Rougail is a configuration management library that allows you to load variables in a simple and convenient way.
+
+In the following examples, we will use a specific configuration of Rougail.
+
+
+.. FIXME: You will find all the configuration options in doc:`configuration`
+   find a document with all the configuration options
+
+To load the configuration you must import the `RougailConfig` class and set the `main_structural_directories` values:
+
+.. code-block:: python
+
+    from rougail import RougailConfig
+
+    RougailConfig['main_structural_directories'] = ['dict']
+
+Let's convert a our first structural file
+-----------------------------------------
+
+As a reminder, a :term:`structure file` is a set of instructions that will allow us to create :term:`families ` and :term:`variables `.
+
+Let's start by creating a simple structure file.
+
+Here is a first :file:`dict/00-base.yml` structure file:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_variable: my_value  # my variable
+    ...
+
+Then, let's create the :term:`Tiramisu` objects via the following script:
+
+.. code-block:: python
+    :caption: the `script.py` file content
+
+    from rougail import Rougail, RougailConfig
+
+    RougailConfig['main_structural_directories'] = ['dict']
+    rougail = Rougail()
+    config = rougail.run()
+    print(config.value.get())
+
+.. demo:: Let's execute `script.py`:
+
+    .. code-block:: bash
+
+        $ python3 script.py
+        {'rougail.my_variable': 'my_value'}
+
+The operator role
+--------------------
+
+The :term:`operator` role corresponds to the :term:`tiramisu` settings:
+
+.. image:: ../images/tiramisu_get_set.png
+
+.. index:: questionary
+
+But instead of coding in the end user developer way, the opterator will prefer using the Rougail CLI interface:
+
+.. image:: ../images/QuestionaryChoice.png
+
+
+The Rougail CLI can output a rather complete view of the dataet:
+
+.. image:: ../images/UserDataOutput.png
+
+.. toctree::
+   :titlesonly:
+   :caption: Use library
+
+   user_data
+   output
+   parse
+   tags
+   rougailconfig_load_from_cli
+   extra
+   custom_function
+   own_type
+   upgrade
diff --git a/docs/library/output.rst b/docs/library/output.rst
new file mode 100644
index 000000000..89941a4f6
--- /dev/null
+++ b/docs/library/output.rst
@@ -0,0 +1,269 @@
+Display the result
+==================
+
+After construct a configuration, loads user data, you can choose this configuration in different output format.
+
+First of create, let's create a structural file like this:
+
+.. code-block:: yaml
+    :caption: the :file:`dist/00-base.yml` file content
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_variable: my value   # My first variable
+
+    my_boolean_variable: true  # My boolean variable
+
+    my_integer_variable: 1  # My integer variable
+
+    my_secret_variable:
+      description: My secret variable
+      type: secret
+      default: MyVeryStrongPassword
+    ...
+
+Display in a console
+--------------------
+
+We can display configuration directly in the console:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+    from rougail.output_console import RougailOutputConsole
+
+    RougailConfig["main_namespace"] = None
+    RougailConfig["main_structural_directories"] = ["dist/"]
+    RougailConfig["step.output"] = "console"
+    rougail = Rougail()
+    config = rougail.run()
+    config.property.read_only()
+    RougailOutputConsole(config).print()
+
+.. FIXME display console!
+
+console.key_is_description
+''''''''''''''''''''''''''
+
+By default, the key is the variable description, if you prefer have only the path:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+    from rougail.output_json import RougailOutputJson
+
+    RougailConfig["main_namespace"] = None
+    RougailConfig["main_structural_directories"] = ["dist/"]
+    RougailConfig["step.output"] = "console"
+    rougail = Rougail()
+    config = rougail.run()
+    config.property.read_only()
+    RougailOutputJson(config).print()
+
+
+.. FIXME display console!
+
+console.show_secrets
+''''''''''''''''''''
+
+Secrets are remplace by "*******", to display real secrets:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+    from rougail.output_console import RougailOutputConsole
+
+    RougailConfig["main_namespace"] = None
+    RougailConfig["main_structural_directories"] = ["dist/"]
+    RougailConfig["step.output"] = "console"
+    RougailConfig["console.show_secrets"] = True
+    rougail = Rougail()
+    config = rougail.run()
+    config.property.read_only()
+    RougailOutputConsole(config).print()
+
+.. FIXME display console!
+
+console.mandatory
+'''''''''''''''''
+
+Before display configuration, mandatories variables are check. If you don't want, add the parameter `console.mandatory` to False:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+    from rougail.output_console import RougailOutputConsole
+
+    RougailConfig["main_namespace"] = None
+    RougailConfig["main_structural_directories"] = ["dist/"]
+    RougailConfig["step.output"] = "console"
+    RougailConfig["console.mandatory"] = False
+    rougail = Rougail()
+    config = rougail.run()
+    config.property.read_only()
+    RougailOutputConsole(config).print()
+
+.. FIXME display console!
+
+JSON
+----
+
+Your script can return a JSON object:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+    from rougail.output_console import RougailOutputConsole
+
+    RougailConfig["main_namespace"] = None
+    RougailConfig["main_structural_directories"] = ["dist/"]
+    RougailConfig["step.output"] = "json"
+    rougail = Rougail()
+    config = rougail.run()
+    config.property.read_only()
+    RougailOutputConsole(config).print()
+
+Let's try this script:
+
+.. code-block:: bash
+
+    $ python script.py
+    {
+      "my_variable": "my value",
+      "my_boolean_variable": true,
+      "my_integer_variable": 1,
+      "my_secret_variable": "MyVeryStrongPassword"
+    }
+
+ANSIBLE
+-------
+
+It's possible to use Ansible has a output format.
+
+The goal is here to use Ansible has a dynamic user's inventories structure manage by Rougail.
+
+This output needs an extra namespace, named, by default, "hosts". This namespace define your hosts and groups.
+
+Let's create a single group "my_group" with one host "group1.net":
+
+.. code-block:: yaml
+   :caption: the :file:`hosts/00-hosts.yml` file content
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    hostnames:
+
+      my_group:
+
+        hosts:
+          type: domainname
+          default:
+            - group1.net
+    ...
+
+Now we can generate Ansible inventory:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    #!/bin/env python
+    from rougail import Rougail, RougailConfig
+    from rougail.output_ansible import RougailOutputAnsible
+
+    RougailConfig["main_namespace"] = "main"
+    RougailConfig["main_structural_directories"] = ["dist/"]
+    RougailConfig['extra_namespaces']['hosts'] = ['hosts/']
+    RougailConfig["step.output"] = "ansible"
+    rougail = Rougail()
+    config = rougail.run()
+    config.property.read_only()
+    RougailOutputAnsible(config).print()
+
+We will retrieved all ours variables associate to this group with all variables inside the namespace `main`:
+
+.. code-block:: bash
+
+    $ python script.py 
+    {
+      "_meta": {
+        "hostvars": {
+          "group1.net": {
+            "ansible_host": "group1.net",
+            "main": {
+              "my_variable": "my value",
+              "my_boolean_variable": true,
+              "my_integer_variable": 1,
+              "my_secret_variable": "MyVeryStrongPassword"
+            }
+          }
+        }
+      },
+      "my_group": {
+        "hosts": [
+          "group1.net"
+        ]
+      }
+    }
+
+We can now use our script as an inventory source in Ansible:
+
+.. code-block:: bash
+
+   $ chmod +x script.py
+   $ ansible-inventory -i script.py --list
+    {
+        "_meta": {
+            "hostvars": {
+                "group1.net": {
+                    "ansible_host": "group1.net",
+                    "main": {
+                        "my_boolean_variable": true,
+                        "my_integer_variable": 1,
+                        "my_secret_variable": "MyVeryStrongPassword",
+                        "my_variable": "my value"
+                    }
+                }
+            }
+        },
+        "all": {
+            "children": [
+                "ungrouped",
+                "my_group"
+            ]
+        },
+        "my_group": {
+            "hosts": [
+                "group1.net"
+            ]
+        }
+    }
+
+DOC
+---
+
+We can generate the documentation of all the Rougail variable:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+    from rougail.output_doc import RougailOutputDoc
+
+    RougailConfig["main_namespace"] = "main"
+    RougailConfig["main_structural_directories"] = ["dist/"]
+    RougailConfig["step.output"] = "doc"
+    rougail = Rougail()
+    config = rougail.run()
+    config.property.read_only()
+    RougailOutputDoc(config).print()
+
+.. FIXME : display
diff --git a/docs/library/own_type.rst b/docs/library/own_type.rst
new file mode 100644
index 000000000..4da6e7aeb
--- /dev/null
+++ b/docs/library/own_type.rst
@@ -0,0 +1,52 @@
+Create your own type
+====================
+
+A variable has a type. This type enables the variable to define the values that are accepted by this variable.
+
+There is a series of default types, but obviously not all cases are taken.
+
+It's possible to create your own type.
+
+Here an example to a lipogram option (in a string, we cannot use "e" character):
+
+.. code-block:: python
+    :caption: the `lipogram.py` file content
+
+    from tiramisu import StrOption
+    class LipogramOption(StrOption):
+        __slots__ = tuple()
+        _type = 'lipogram'
+
+        def validate(self,
+                     value):
+            super().validate(value)
+            # verify that there is any 'e' in the sentense
+            if 'e' in value:
+                raise ValueError('Perec wrote a book without any "e", you could not do it in a simple sentence?')
+
+To add the new lipogram type in Rougail:
+
+.. code-block:: python
+
+    >>> from rougail import Rougail, RougailConfig
+    >>> RougailConfig['main_structural_directories'] = ['dict']
+    >>> RougailConfig['custom_types']['lipogram'] = LipogramOption
+
+Now, we can use lipogram type.
+Here is a :file:`dict/00-base.yml` structure file:
+
+.. code-block:: yaml
+
+    ---
+    version: '1.1'
+    var:
+      type: lipogram
+
+.. code-block:: python
+
+    >>> rougail = Rougail()
+    >>> config = rougail.get_config()
+    >>> config.option('rougail.var').value.set('blah')
+    >>> config.option('rougail.var').value.set('I just want to add a quality string that has no bad characters')
+    [...]
+    tiramisu.error.ValueOptionError: "I just want to add a quality string that has no bad characters" is an invalid lipogram for "var", Perec wrote a book without any "e", you could not do it in a simple sentence?
diff --git a/docs/library/parse.rst b/docs/library/parse.rst
new file mode 100644
index 000000000..1a06c531d
--- /dev/null
+++ b/docs/library/parse.rst
@@ -0,0 +1,260 @@
+Retrieve all variables and families
+===================================
+
+Rougail returns a :term:`Tiramisu` config.
+
+Let's retrieve our variables and families to manager this.
+
+First of all, create our structural file:
+
+.. code-block:: yaml
+   :caption: the :file:`dist/00-base.yml` file content
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_variable:  # a simple variable
+      - value1
+      - value2
+
+    a_family:  # a simple family
+
+      my_variable: my_value  # a simple variable inside the family
+
+    a_dyn_family_{{ identifier }}:
+      description: a dynamic family for "{{ identifier }}"
+      dynamic:
+        variable: _.my_variable
+
+      a_leadership:
+        description: a leader family
+
+        a_leader:  # a leader variable
+        a_follower:  # a follower variable
+    ...
+
+Walk through our config
+-------------------------
+
+Create our first script to walk through our config:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+
+    RougailConfig['main_structural_directories'] = ["dist/"]
+    rougail = Rougail()
+    config = rougail.run()
+
+    def walk(config):
+        for option in config:
+            print(option.description())
+            if option.isoptiondescription():
+                walk(option)
+
+    if __name__ == '__main__':
+        walk(config)
+
+Let's execute `script.py`:
+
+.. code-block:: bash
+
+    $ python3 script.py
+    rougail
+    rougail.my_variable (a simple variable)
+    rougail.a_family (a simple family)
+    rougail.a_family.my_variable (a simple variable inside the family)
+    rougail.a_dyn_family_value1 (a dynamic family for "value1")
+    rougail.a_dyn_family_value1.a_leadership (a leader family)
+    rougail.a_dyn_family_value1.a_leadership.a_leader (a leader variable)
+    rougail.a_dyn_family_value1.a_leadership.a_follower (a follower variable)
+    rougail.a_dyn_family_value2 (a dynamic family for "value2")
+    rougail.a_dyn_family_value2.a_leadership (a leader family)
+    rougail.a_dyn_family_value2.a_leadership.a_leader (a leader variable)
+    rougail.a_dyn_family_value2.a_leadership.a_follower (a follower variable)
+
+We retrieve alls description of variables and families.
+
+Let us distinguish the variables of the families
+------------------------------------------------
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+
+    RougailConfig['main_structural_directories'] = ["dist/"]
+    rougail = Rougail()
+    config = rougail.run()
+
+    def walk(config, level=0):
+        for option in config:
+            if option.isoptiondescription():
+                typ = "family"
+            else:
+                typ = "variable"
+            prefix = "  " * level
+            print(f"{prefix}{typ}: {option.description()}")
+            if option.isoptiondescription():
+                walk(option, level + 1)
+
+    if __name__ == '__main__':
+        walk(config)
+
+Let's execute `script.py`:
+
+.. code-block:: bash
+
+    $ python3 script.py
+    family: rougail
+      variable: rougail.my_variable (a simple variable)
+      family: rougail.a_family (a simple family)
+        variable: rougail.a_family.my_variable (a simple variable inside the family)
+      family: rougail.a_dyn_family_value1 (a dynamic family for "value1")
+        family: rougail.a_dyn_family_value1.a_leadership (a leader family)
+          variable: rougail.a_dyn_family_value1.a_leadership.a_leader (a leader variable)
+          variable: rougail.a_dyn_family_value1.a_leadership.a_follower (a follower variable)
+      family: rougail.a_dyn_family_value2 (a dynamic family for "value2")
+        family: rougail.a_dyn_family_value2.a_leadership (a leader family)
+          variable: rougail.a_dyn_family_value2.a_leadership.a_leader (a leader variable)
+          variable: rougail.a_dyn_family_value2.a_leadership.a_follower (a follower variable)
+
+Or if we want more precision:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+    
+    RougailConfig['main_structural_directories'] = ["dist/"]
+    rougail = Rougail()
+    config = rougail.run()
+    
+    def walk(config, level=0):
+        for option in config:
+            if option.isoptiondescription():
+                if option.isleadership():
+                    typ = "leadership"
+                elif option.isdynamic():
+                    typ = "dynamic family"
+                else:
+                    typ = "family"
+            else:
+                if option.isleader():
+                    typ = "leader"
+                elif option.isfollower():
+                    typ = "follower"
+                else:
+                    typ = "option"
+                if option.isdynamic():
+                    typ = f"dynamic {typ}"
+            prefix = "  " * level
+            print(f"{prefix}{typ}: {option.description()}")
+            if option.isoptiondescription():
+                walk(option, level + 1)
+    
+    if __name__ == '__main__':
+        walk(config)
+
+Let's execute `script.py`:
+
+.. code-block:: bash
+
+    family: rougail
+      option: rougail.my_variable (a simple variable)
+      family: rougail.a_family (a simple family)
+        option: rougail.a_family.my_variable (a simple variable inside the family)
+      dynamic family: rougail.a_dyn_family_value1 (a dynamic family for "value1")
+        dynamic family: rougail.a_dyn_family_value1.a_leadership (a leader family)
+          dynamic option: rougail.a_dyn_family_value1.a_leadership.a_leader (a leader variable)
+          dynamic option: rougail.a_dyn_family_value1.a_leadership.a_follower (a follower variable)
+      dynamic family: rougail.a_dyn_family_value2 (a dynamic family for "value2")
+        dynamic family: rougail.a_dyn_family_value2.a_leadership (a leader family)
+          dynamic option: rougail.a_dyn_family_value2.a_leadership.a_leader (a leader variable)
+          dynamic option: rougail.a_dyn_family_value2.a_leadership.a_follower (a follower variable)
+
+Get variable values
+-------------------
+
+If we want to walk to get variables and their values:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+
+    RougailConfig['main_structural_directories'] = ["dist/"]
+    rougail = Rougail()
+    config = rougail.run()
+
+    def walk(config):
+        for option in config:
+            if option.isoptiondescription():
+                walk(option)
+            else:
+                print(f"{option.description()}: {option.value.get()}")
+
+    if __name__ == '__main__':
+        walk(config)
+
+Let's execute `script.py`:
+
+.. code-block:: bash
+
+    rougail.my_variable (a simple variable): ['value1', 'value2']
+    rougail.a_family.my_variable (a simple variable inside the family): my_value
+    rougail.a_dyn_family_value1.a_leadership.a_leader (a leader variable): None
+    rougail.a_dyn_family_value1.a_leadership.a_follower (a follower variable): None
+    rougail.a_dyn_family_value2.a_leadership.a_leader (a leader variable): None
+    rougail.a_dyn_family_value2.a_leadership.a_follower (a follower variable): None
+
+Modify variable values
+----------------------
+
+Some variables are mandatories but hasn't value. Here we set alls values:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+    RougailConfig['main_structural_directories'] = ["dist/"]
+    rougail = Rougail()
+    config = rougail.run()
+
+
+    def walk(config):
+        for option in config:
+            if option.isoptiondescription():
+                walk(option)
+            else:
+                print(f"{option.description()}: {option.value.get()}")
+
+    if __name__ == '__main__':
+        print("Mandatories variables without value:")
+        print(config.value.mandatory())
+        config.value.set("rougail.my_variable", ["value 5", "value 6"])
+        config.value.set("rougail.a_dyn_family_value_5.a_leadership.a_leader", "value 1")
+        config.value.set("rougail.a_dyn_family_value_5.a_leadership.a_follower", "value 2")
+        config.value.set("rougail.a_dyn_family_value_6.a_leadership.a_leader", "value 3")
+        config.value.set("rougail.a_dyn_family_value_6.a_leadership.a_follower", "value 4")
+        print("Mandatories variables without value:")
+        print(config.value.mandatory())
+        walk(config)
+
+
+Let's execute `script.py`:
+
+.. code-block:: bash
+
+    Mandatories variables without value:
+    [, , , ]
+    Mandatories variables without value:
+    []
+    rougail.my_variable (a simple variable): ['value 5', 'value 6']
+    rougail.a_family.my_variable (a simple variable inside the family): my_value
+    rougail.a_dyn_family_value_5.a_leadership.a_leader (a leader variable): value 1
+    rougail.a_dyn_family_value_5.a_leadership.a_follower (a follower variable): value 2
+    rougail.a_dyn_family_value_6.a_leadership.a_leader (a leader variable): value 3
+    rougail.a_dyn_family_value_6.a_leadership.a_follower (a follower variable): value 4
diff --git a/docs/library/rougailconfig_load_from_cli.rst b/docs/library/rougailconfig_load_from_cli.rst
new file mode 100644
index 000000000..bebb22fad
--- /dev/null
+++ b/docs/library/rougailconfig_load_from_cli.rst
@@ -0,0 +1,167 @@
+Load Rougail configuration from Rougail command line informations
+==================================================================
+
+There is a lot you can do with the Rougail command line (rougail-cli), but sometimes you need to do a more advanced script.
+
+Rather than duplicating the configuration, why not load the information from the configuration file, environment variables, or command line options?
+
+We can loading a combination of source information but always in this order:
+
+- configuration file
+- environment variables
+- commandline options
+
+.. warning:: specific options reserve for command line (in namespace "cli") are not available in script
+
+Then let's create an structual file:term:`structure file` :file:`dist/00-base.yml`:
+
+.. code-block:: yaml
+   :caption: the :file:`dist/00-base.yml` file content
+
+    \%YAML 1.2
+    ---
+    version: 1.1
+
+    my_variable: my_value_extra  # a simple variable
+    ...
+
+Command line configuration file
+-------------------------------
+
+Create a command line configuration file :file:`.rougailcli.yml`:
+
+.. code-block:: yaml
+   :caption: the :file:`.rougailcli.yml` file content
+
+    ---
+    main_structural_directories:  # directories where are place structural file
+      - dist
+    step.output: json  # output is not console but json
+
+Let's execute Rougail command line:
+
+.. code-block:: bash
+
+    $ rougail
+    {
+      "my_variable": "my_value_extra"
+    }
+
+Then, let's create the :term:`Tiramisu` objects via the following :file:`script.py` script:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail
+    rougail = Rougail()
+    try:
+        config = rougail.run()       
+        print(config.value.get()) 
+    except Exception as err:
+        print(f"ERROR: {err}")
+        exit(1)
+
+Let's execute `script.py`:
+
+.. code-block:: bash
+
+    $ python3 script.py
+    
+    ERROR: option "Directories where structural files are placed" is mandatory but has no value
+
+As expected, the .rougailcli.yml file is not loaded because it is specific to the command line.
+
+Let's modifying the script to do this:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+    from rougail.cli.rougailconfig import load
+    load(RougailConfig, yaml_file=".rougailcli.yml")
+    rougail = Rougail()
+    try:
+        config = rougail.run()       
+        print(config.value.get()) 
+    except Exception as err:
+        print(f"ERROR: {err}")
+        exit(1)
+
+Let's execute `script.py`:
+
+.. code-block:: bash
+
+    $ python3 script.py
+    {: {: 'my_value_extra'}}
+
+Environment variables
+---------------------
+
+If we don't have .rougailcli.yml, it's possible to set option with environment variables, like this:
+
+.. code-block:: bash
+
+    $ env ROUGAILCLI_MAIN_STRUCTURAL_DIRECTORIES=dist/ ROUGAILCLI_STEP.OUTPUT=json ROUGAILCLI_MAIN_NAMESPACE=test bin/rougail
+    {
+      "test": {
+        "my_variable": "my_value_extra"
+      }
+    }
+
+Do the same with a script:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+    from rougail.cli.rougailconfig import load
+    load(RougailConfig, env_prefix="ROUGAILCLI")
+    rougail = Rougail()
+    try:
+        config = rougail.run()       
+        print(config.value.get()) 
+    except Exception as err:
+        print(f"ERROR: {err}")
+        exit(1)
+
+Let's execute `script.py`:
+
+.. code-block:: bash
+
+    $ env ROUGAILCLI_MAIN_STRUCTURAL_DIRECTORIES=dist/ ROUGAILCLI_STEP.OUTPUT=json ROUGAILCLI_MAIN_NAMESPACE=test python3 script.py 
+    {: {: 'my_value_extra'}}
+
+Command line option
+-------------------
+
+To reproduce this:
+
+.. code-block:: bash
+
+   ./bin/rougail --main_structural_directories dist/ --step.output json --main_namespace=new_test
+   {
+     "new_test": {
+       "my_variable": "my_value_extra"
+     }
+   }
+
+Do this script:
+
+.. code-block:: python
+   :caption: the :file:`script.py` file content
+
+   from rougail import Rougail, RougailConfig
+   from rougail.cli.rougailconfig import load
+   load(RougailConfig, commandline=True)
+   rougail = Rougail()
+   try:
+       config = rougail.run()       
+       print(config.value.get()) 
+   except Exception as err:
+       print(f"ERROR: {err}")
+       exit(1)
+
+.. code-block:: bash
+
+   $ python3 script.py --main_structural_directories dist/ --step.output json --main_namespace=new_test
+   {: {: 'my_value_extra'}
diff --git a/docs/library/tags.rst b/docs/library/tags.rst
new file mode 100644
index 000000000..522d6f8f2
--- /dev/null
+++ b/docs/library/tags.rst
@@ -0,0 +1,112 @@
+Use tag informations
+====================
+
+When we set tags for a variable, it will add :term:`Tiramisu` properties and informations.
+
+We can filter those variables easily with tags.
+
+First of all, create our structural file:
+
+.. code-block:: yaml
+   :caption: the :file:`dist/00-base.yml` file content
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    infra_name: my infra   # Name of this infrastructure
+
+    server1:  # the first server
+
+      internal_domain:
+        description: Server domaine name
+        type: domainname
+        tags:
+          - internal
+
+      external_domain:
+        description: Domain name to access to this server for Internet
+        type: domainname
+        tags:
+          - external
+
+    server2:  # the second server
+
+      address:
+        description: Server domaine name
+        type: domainname
+        tags:
+          - internal
+    ...
+
+Exclude variables with a specific tag
+-------------------------------------
+
+To exclude variables with a specific tag is very easy. When the variable has tags, properties with same name are automaticly create.
+So exclude a tag means exclude variable with a particular property.
+
+In this example we exclude variable with "internal" tag and display result of "server1" family:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+    from pprint import pprint
+
+    RougailConfig['main_namespace'] = None
+    RougailConfig['main_structural_directories'] = ['dist']
+    rougail = Rougail()
+    config = rougail.run()
+    print("without filter:")
+    pprint(config.option("server1").value.get())
+    config.property.add('internal')
+    print("with filter:")
+    pprint(config.option("server1").value.get())
+
+Let's execute `script.py`:
+
+.. code-block:: bash
+
+    $ python3 script.py 
+    without filter:
+    {: None,
+     : None}
+    with filter:
+    {: None}
+
+Only variable with a specific tag
+---------------------------------
+
+It's more difficult to see only variable with a specific tag.
+
+We have to walk through the configuration and retrieve variable with the selected tag.
+Tags are in properties but, are in information too.
+
+Here is a smal script that walk that the configuration and "print" option with "internal" tag:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+    
+    RougailConfig['main_structural_directories'] = ['dist']
+    rougail = Rougail()
+    config = rougail.run()
+    
+    def walk(config):
+        for option in config:
+            if option.isoptiondescription():
+                walk(option)
+            elif "internal" in option.information.get('tags', []):
+                print(option.description())
+
+    if __name__ == '__main__':
+        walk(config)
+
+Let's execute `script.py`:
+
+.. code-block:: bash
+
+    $ python3 script.py 
+    server1.internal_domain (Server domaine name)
+    server2.address (Server domaine name)
diff --git a/docs/library/upgrade.rst b/docs/library/upgrade.rst
new file mode 100644
index 000000000..4778905df
--- /dev/null
+++ b/docs/library/upgrade.rst
@@ -0,0 +1,42 @@
+Upgrade dictionnaries to upper version
+======================================
+
+All dictionnaries has a format version number.
+When a new format version is proposed, it is possible to automatically convert the files to the new version.
+
+We create a term:`structure file` named :file:`dict/01-upgrade.yml` with version 1.0:
+
+.. code-block:: yaml
+
+    ---
+    version: '1.0'
+    my_variable:
+      multi: true
+    my_dyn_family:
+      type: "dynamic"
+      variable: my_variable
+      a_variable:
+
+
+.. code-block:: python
+
+    >>> from rougail import RougailUpgrade, RougailConfig
+    >>> RougailConfig['main_structural_directories'] = ['dict']
+    >>> upgrade = RougailUpgrade()
+    >>> upgrade.load_dictionaries('dict_converted')
+
+The term:`structure file` named :file:`dict_converted/01-upgrade.yml` is in version 1.1:
+
+.. code-block:: yaml
+
+   version: '1.1'
+   my_variable:
+     multi: true
+   my_dyn_family:
+     type: dynamic
+     a_variable: null
+     dynamic:
+       type: variable
+       variable: my_variable
+       propertyerror: false
+
diff --git a/docs/library/user_data.rst b/docs/library/user_data.rst
new file mode 100644
index 000000000..5f918d16f
--- /dev/null
+++ b/docs/library/user_data.rst
@@ -0,0 +1,390 @@
+Load user data
+===============
+
+User data are values setup by user for configuration variables.
+
+There is differents types of user data for differents sources types.
+
+We can cumulate user data loader.
+
+For this section, we will use :file:`dict/00-base.yml` a structure file:
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+
+    my_variable: my value   # My first variable
+
+    my_boolean_variable: true  # My boolean variable
+
+    my_integer_variable: 1  # My integer variable
+
+    my_secret_variable:
+      description: My secret variable
+      type: secret
+
+    my_hidden_variable:
+      description: My hidden variable
+      hidden: true
+    ...
+
+Here is the first script which is load this file:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+
+    RougailConfig["main_namespace"] = None
+    RougailConfig["main_structural_directories"] = ["dist/"]
+    rougail = Rougail()
+    config = rougail.run()
+    print(config.value.get())
+
+Let's execute `script.py`:
+
+.. code-block:: bash
+
+    $ python3 script.py 
+    {: 'my value', : True, : 1, : None}
+
+YAML
+----
+
+We want to load this YAML file with value define by user:
+
+.. code-block:: yaml
+
+    ---
+    my_variable: a new value
+
+    my_boolean_variable: false
+
+    my_integer_variable: 10
+
+    my_secret_variable: MyVeryStrongPassword
+
+Here is the script which is load user data from the YAML file:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+    from rougail.user_data_yaml import RougailUserDataYaml
+
+    RougailConfig["main_namespace"] = None
+    RougailConfig["main_structural_directories"] = ["dist/"]
+    RougailConfig["step.user_data"] = ["yaml"]
+    RougailConfig["yaml.filename"] = ["dist.yml"]
+    rougail = Rougail()
+    config = rougail.run()
+
+    user_data = RougailUserDataYaml(config).run()
+    rougail.user_data(user_data)
+    print(config.value.get())
+
+Let's execute `script.py`:
+
+.. code-block:: bash
+
+    $ python3 script.py 
+    {: 'a new value', : False, : 10, : 'MyVeryStrongPassword'}
+
+Set a secret in clear text file is not always a good idea.
+This is why the `yaml.file_with_secrets` parameter allows you to define whether files define in `yaml.filename` can contain a secret and which one:
+
+- all: all file can contains secret
+- first: only the first file can contains secret
+- last: only the last file can contains secret
+- none: no file can contains secret
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+    from rougail.user_data_yaml import RougailUserDataYaml
+
+    RougailConfig["main_namespace"] = None
+    RougailConfig["main_structural_directories"] = ["dist/"]
+    RougailConfig["step.user_data"] = ["yaml"]
+    RougailConfig["yaml.filename"] = ["dist.yml"]
+    RougailConfig["yaml.file_with_secrets"] = "none"
+    rougail = Rougail()
+    config = rougail.run()
+
+    user_data = RougailUserDataYaml(
+         config,
+    ).run()
+    rougail.user_data(user_data)
+    print(config.value.get())
+
+Let's execute `script.py`:
+
+.. code-block:: bash
+
+    $ python3 script.py 
+    {: 'a new value', : False, : 10, : None}
+
+Environment variables
+---------------------
+
+We can define use data from environment variables. The environment name is a "prefix" (ROUGAIL by default) with "_" and variable name in uppercase format.
+
+For example:
+
+- `my_variable` has `ROUGAIL_MY_VARIABLE` as a environment variable name
+- `my_family.my_variable` has `ROUGAIL_MY_FAMILY.MY_VARIABLE` as a environment variable name
+
+Some shell doesn't allow dot in environment file. In this case use the command "env".
+
+For example: `env ROUGAIL_MY_FAMILY.MY_VARIABLE="value" ./script.py`.
+
+Here is the script:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+    from rougail.user_data_environment import RougailUserDataEnvironment
+
+    RougailConfig["main_namespace"] = None
+    RougailConfig["main_structural_directories"] = ["dist/"]
+    RougailConfig["step.user_data"] = ["environment"]
+    rougail = Rougail()
+    config = rougail.run()
+
+    user_data = RougailUserDataEnvironment(config).run()
+    rougail.user_data(user_data)
+    print(config.value.get())
+
+Let's execute `script.py`:
+
+.. code-block:: bash
+
+    env ROUGAIL_MY_VARIABLE="a new value" ROUGAIL_MY_BOOLEAN_VARIABLE="False" ROUGAIL_MY_INTEGER_VARIABLE=10 ROUGAIL_MY_SECRET_VARIABLE="MyVeryStrongPassword" python script.py
+    {: 'a new value', : False, : 10, : 'MyVeryStrongPassword'}
+
+We can redefine the prefix with `environment.default_environment_name` (prefix is always uppercase characters):
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+    from rougail.user_data_environment import RougailUserDataEnvironment
+
+    RougailConfig["main_namespace"] = None
+    RougailConfig["main_structural_directories"] = ["dist/"]
+    RougailConfig["step.user_data"] = ["environment"]
+    RougailConfig["environment.default_environment_name"] = "EX"
+    rougail = Rougail()
+    config = rougail.run()
+
+    user_data = RougailUserDataEnvironment(config).run()
+    rougail.user_data(user_data)
+    print(config.value.get())
+
+Let's execute `script.py`:
+
+.. code-block:: bash
+
+    env EX_MY_VARIABLE="a new value" EX_MY_BOOLEAN_VARIABLE="False" EX_MY_INTEGER_VARIABLE=10 EX_MY_SECRET_VARIABLE="MyVeryStrongPassword" python script.py
+    {: 'a new value', : False, : 10, : 'MyVeryStrongPassword'}
+
+If you define a `main_namespace` or `extra_namespaces`, the `environment.default_environment_name` is automaticly define with the name of the namespace in uppercase. And the separator is no more "_" but ".":
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+    from rougail.user_data_environment import RougailUserDataEnvironment
+
+    RougailConfig["main_namespace"] = "main"
+    RougailConfig["main_structural_directories"] = ["dist/"]
+    RougailConfig["step.user_data"] = ["environment"]
+    rougail = Rougail()
+    config = rougail.run()
+
+    user_data = RougailUserDataEnvironment(config).run()
+    rougail.user_data(user_data)
+    print(config.value.get())
+
+Let's execute `script.py`:
+
+.. code-block:: bash
+
+    env MAIN.MY_VARIABLE="a new value" MAIN.MY_BOOLEAN_VARIABLE="False" MAIN.MY_INTEGER_VARIABLE=10 MAIN.MY_SECRET_VARIABLE="MyVeryStrongPassword" python script.py
+    {: {: 'a new value', : False, : 10, : 'MyVeryStrongPassword'}}
+
+Set a secret in clear variable environment is not always a good idea.
+This is why the `environment.with_secrets` parameter allows you to reject secret from environment variable:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+    from rougail.user_data_environment import RougailUserDataEnvironment
+
+    RougailConfig["main_namespace"] = None
+    RougailConfig["main_structural_directories"] = ["dist/"]
+    RougailConfig["step.user_data"] = ["environment"]
+    RougailConfig["environment.with_secrets"] = False
+    rougail = Rougail()
+    config = rougail.run()
+
+    user_data = RougailUserDataEnvironment(config).run()
+    rougail.user_data(user_data)
+    print(config.value.get())
+
+Let's execute `script.py`:
+
+.. code-block:: bash
+
+    env ROUGAIL_MY_VARIABLE="a new value" ROUGAIL_MY_BOOLEAN_VARIABLE="False" ROUGAIL_MY_INTEGER_VARIABLE=10 ROUGAIL_MY_SECRET_VARIABLE="MyVeryStrongPassword" python script.py
+    {: 'a new value', : False, : 10, : None}
+
+Comand line parser user data
+----------------------------
+
+Value can be define directly with command line arguments:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+    from rougail.user_data_commandline import RougailUserDataCommandline
+
+    RougailConfig["main_namespace"] = None
+    RougailConfig["main_structural_directories"] = ["dist/"]
+    RougailConfig["step.user_data"] = ["commandline"]
+    rougail = Rougail()
+    config = rougail.run()
+
+    user_data = RougailUserDataCommandline(
+         config,
+    ).run()
+    rougail.user_data(user_data)
+    print(config.value.get())
+
+Let's execute `script.py` to display help:
+
+.. code-block:: bash
+
+    $ python script.py -h
+    usage: script.py [-h] --my_variable [MY_VARIABLE] --my_boolean_variable --no-my_boolean_variable --my_integer_variable [MY_INTEGER_VARIABLE] --my_secret_variable MY_SECRET_VARIABLE
+
+    options:
+      -h, --help            show this help message and exit
+      --my_variable [MY_VARIABLE]
+                            my_variable (My first variable) (default: my value)
+      --my_boolean_variable
+                            my_boolean_variable (My boolean variable) (default: True)
+      --no-my_boolean_variable
+      --my_integer_variable [MY_INTEGER_VARIABLE]
+                            my_integer_variable (My integer variable) (default: 1)
+      --my_secret_variable MY_SECRET_VARIABLE
+                            my_secret_variable (My secret variable)
+    {: 'my value', : True, : 1, : None}
+
+
+And now with modified value:
+
+.. code-block:: bash
+
+    $ python script.py --my_variable "a new value" --no-my_boolean_variable --my_integer_variable 10 --my_secret_variable MyVeryStrongPassword
+    {: 'a new value', : False, : 10, : 'MyVeryStrongPassword'}
+
+Boolean variable has a special behavour. To set False you need to add --no-VARIABLE, to set True you need to add --VARIABLE parameter.
+
+.. ansible,bitwarden,questionary
+
+Combine user data
+------------------
+
+You can combine user data, for example if you want to load data from environment and/or command line argument:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+    from rougail.user_data_environment import RougailUserDataEnvironment
+    from rougail.user_data_commandline import RougailUserDataCommandline
+
+    RougailConfig["main_namespace"] = None
+    RougailConfig["main_structural_directories"] = ["dist/"]
+    RougailConfig["step.user_data"] = ["environment", "commandline"]
+    rougail = Rougail()
+    config = rougail.run()
+
+    user_data = []
+    user_data.extend(RougailUserDataEnvironment(
+         config,
+    ).run())
+    user_data.extend(RougailUserDataCommandline(
+         config,
+    ).run())
+    rougail.user_data(user_data)
+    print(config.value.get())
+
+Let's execute `script.py` with environment variable and commandline arguments:
+
+.. code-block:: bash
+
+    $ env ROUGAIL_MY_VARIABLE="a new value" ROUGAIL_MY_BOOLEAN_VARIABLE="False" python script.py --my_integer_variable 10 --my_secret_variable MyVeryStrongPassword
+    {: 'a new value', : False, : 10, : 'MyVeryStrongPassword'}
+
+If the value of a variable is define with an environment variable and commandline argument, the value is the value of the last user data define:
+
+.. code-block:: bash
+
+    $ env ROUGAIL_MY_VARIABLE="not a new" python script.py --my_variable "a new value" --no-my_boolean_variable --my_integer_variable 10 --my_secret_variable MyVeryStrongPassword
+   {: 'a new value', : False, : 10, : 'MyVeryStrongPassword'}
+
+Manage errors and warnings
+--------------------------
+
+Recreate a script with environnement variable support which is display the return of user_data function:
+
+.. code-block:: python
+    :caption: the :file:`script.py` file content
+
+    from rougail import Rougail, RougailConfig
+    from rougail.user_data_environment import RougailUserDataEnvironment
+
+    RougailConfig["main_namespace"] = None
+    RougailConfig["main_structural_directories"] = ["dist/"]
+    RougailConfig["step.user_data"] = ["environment"]
+    RougailConfig["environment.with_secrets"] = False
+    rougail = Rougail()
+    config = rougail.run()
+
+    user_data = RougailUserDataEnvironment(
+         config,
+    ).run()
+    print(rougail.user_data(user_data))
+
+Try to load the value an unknown variable:
+
+.. code-block:: bash
+
+    $ env ROUGAIL_UNKNOWN_VARIABLE="a value" python script.py
+    {'errors': [], 'warnings': ['variable or family "unknown_variable" does not exist, it will be ignored when loading from environment variable']}
+
+As you can see, a warnings is return.
+
+Try to load the value of an hidden variable:
+
+.. code-block:: bash
+
+   $ env ROUGAIL_MY_HIDDEN_VARIABLE="a value" python script.py
+    {'errors': [], 'warnings': ['variable "my_hidden_variable" (My hidden variable) is hidden, it will be ignored when loading from environment variable']}
+
+Finally if a try to change the value of a secret, which is not allowed:
+
+.. code-block:: bash
+
+    $ env ROUGAIL_MY_SECRET_VARIABLE="MyVeryStrongPassword" python script.py 
+    {'errors': ['the variable "my_secret_variable" contains secrets and should not be defined in environment variable'], 'warnings': []}
+
+An error is generated.
diff --git a/docs/output/ansible.rst b/docs/output/ansible.rst
new file mode 100644
index 000000000..894e6dd83
--- /dev/null
+++ b/docs/output/ansible.rst
@@ -0,0 +1,230 @@
+Generate Ansible inventory
+==========================
+
+.. note::
+
+    | **Path**: ansible
+    | `*disabled*`
+    | **Disabled**: when the variable "select for output" (step.output) is accessible and hasn't the value "ansible".
+
+
+
+.. list-table::
+
+   * - Variable
+     - Description
+
+   * - **ansible.output**
+
+       `choice `__ `mandatory`
+
+       **Command line**: 
+
+       --ansible.output
+
+       **Environment variable**: ROUGAILCLI_ANSIBLE.OUTPUT
+     - Output type.
+
+       **Choices**: 
+
+       
+
+       - inventory **← (default)**
+
+       - doc
+
+   * - **ansible.host_namespace**
+
+       `string `__ `mandatory`
+
+       **Command line**: 
+
+       --ansible.host_namespace
+
+       **Environment variable**: ROUGAILCLI_ANSIBLE.HOST_NAMESPACE
+     - Namespace with host values.
+
+       **Default**: hosts
+
+Doc configuration
+-----------------
+
+.. note::
+
+    | **Path**: ansible.doc
+    | `*disabled*`
+    | **Disabled**: when the variable "Output type" (ansible.output) hasn't the value "doc".
+
+
+
+.. list-table::
+
+   * - Variable
+     - Description
+
+   * - **ansible.doc.project_name**
+
+       `string `__ `mandatory`
+
+       **Command line**: 
+
+       --ansible.doc.project_name
+
+       **Environment variable**: ROUGAILCLI_ANSIBLE.DOC.PROJECT_NAME
+     - Ansible project name.
+
+   * - **ansible.doc.author**
+
+       `string `__ `mandatory`
+
+       **Command line**: 
+
+       --ansible.doc.author
+
+       **Environment variable**: ROUGAILCLI_ANSIBLE.DOC.AUTHOR
+     - Ansible author name.
+
+   * - **ansible.doc.output_format**
+
+       `choice `__ `mandatory`
+
+       **Command line**: 
+
+       --ansible.doc.output_format
+
+       **Environment variable**: ROUGAILCLI_ANSIBLE.DOC.OUTPUT_FORMAT
+     - The output format of the generated documentation.
+
+       **Choices**: 
+
+       
+
+       - console **← (default)**
+
+       - asciidoc
+
+       - html
+
+       - github
+
+       - gitlab
+
+       - restructuredtext
+
+       - json
+
+   * - **ansible.doc.collection_type**
+
+       `choice `__ `mandatory`
+
+       **Command line**: 
+
+       --ansible.doc.collection_type
+
+       **Environment variable**: ROUGAILCLI_ANSIBLE.DOC.COLLECTION_TYPE
+     - Collection contents.
+
+       **Choices**: 
+
+       
+
+       - auto **← (default)**
+
+       - playbooks
+
+       - roles
+
+Playbooks informations
+~~~~~~~~~~~~~~~~~~~~~~
+
+.. note::
+
+    | This family contains lists of variable blocks.
+    | **Path**: ansible.doc.playbooks
+    | `*disabled*`
+    | **Disabled**: if the content is a role.
+
+
+
+.. list-table::
+
+   * - Variable
+     - Description
+
+   * - **ansible.doc.playbooks.name**
+
+       `string `__ `multiple` `unique`
+
+       **Command line**: 
+
+       --ansible.doc.playbooks.name
+
+       **Environment variable**: ROUGAILCLI_ANSIBLE.DOC.PLAYBOOKS.NAME
+     - Playbook name.
+
+       Playbooks are placed in the playbooks/ directory. By default, the description of the "type" is used as the playbook name in the generated example. It is possible to customize this description here.
+
+   * - **ansible.doc.playbooks.description**
+
+       `string `__ `mandatory`
+
+       **Command line**: 
+
+       --ansible.doc.playbooks.description
+
+       **Environment variable**: ROUGAILCLI_ANSIBLE.DOC.PLAYBOOKS.DESCRIPTION
+     - Playbook description.
+
+Inventory configuration
+-----------------------
+
+.. note::
+
+    | **Path**: ansible.inventory
+    | `*disabled*`
+    | **Disabled**: when the variable "Output type" (ansible.output) hasn't the value "inventory".
+
+
+
+.. list-table::
+
+   * - Variable
+     - Description
+
+   * - **ansible.inventory.no_namespace_in_vars**
+
+       `boolean `__ `mandatory`
+
+       **Command line**: 
+
+       
+
+       - --ansible.inventory.no_namespace_in_vars
+
+       - --ansible.inventory.no-no_namespace_in_vars
+
+       
+
+       **Environment variable**: ROUGAILCLI_ANSIBLE.INVENTORY.NO_NAMESPACE_IN_VARS
+     - Remove namespace name in host vars.
+
+       **Default**: false
+
+   * - **ansible.inventory.export_warnings**
+
+       `boolean `__ `mandatory`
+
+       **Command line**: 
+
+       
+
+       - --ansible.inventory.export_warnings
+
+       - --ansible.inventory.no-export_warnings
+
+       
+
+       **Environment variable**: ROUGAILCLI_ANSIBLE.INVENTORY.EXPORT_WARNINGS
+     - Displays warnings inside Ansible exportation datas.
+
+       **Default**: true
diff --git a/docs/output/display.rst b/docs/output/display.rst
new file mode 100644
index 000000000..c7f11e044
--- /dev/null
+++ b/docs/output/display.rst
@@ -0,0 +1,87 @@
+Display variables and values
+============================
+
+.. note::
+
+    | Find all the variables and their values in your configuration (structural and user data). Additional informations are available, such as the default value, the location where the value is loaded, etc.
+    | **Path**: display
+    | `*disabled*`
+    | **Disabled**: if display is not set in "select for output" (step.output).
+
+
+
+.. list-table::
+
+   * - Variable
+     - Description
+
+   * - **display.output_format**
+
+       `choice `__ `mandatory`
+
+       **Command line**: 
+
+       --display.output_format
+
+       **Environment variable**: ROUGAILCLI_DISPLAY.OUTPUT_FORMAT
+     - The output format for displaying variables.
+
+       **Choices**: 
+
+       
+
+       - console **← (default)**
+
+       - github
+
+       - gitlab
+
+   * - **display.show_secrets**
+
+       `boolean `__ `mandatory`
+
+       **Command line**: 
+
+       
+
+       - --display.show_secrets
+
+       - --display.no-show_secrets
+
+       
+
+       **Environment variable**: ROUGAILCLI_DISPLAY.SHOW_SECRETS
+     - Show secrets instead of obscuring them.
+
+       **Default**: false
+
+Specific configuration for console output
+-----------------------------------------
+
+.. note::
+
+    | **Path**: display.console
+    | `*disabled*`
+    | **Disabled**: when the variable "The output format for displaying variables" (display.output_format) hasn't the value "console".
+
+
+
+.. list-table::
+
+   * - Variable
+     - Description
+
+   * - **display.console.max_width**
+
+       `integer `__
+
+       **Command line**: 
+
+       --display.console.max_width
+
+       **Environment variable**: ROUGAILCLI_DISPLAY.CONSOLE.MAX_WIDTH
+     - Maximum number of characters per line.
+
+       Null means unlimited.
+
+       **Validator**: the minimum value is 50
diff --git a/docs/output/doc.rst b/docs/output/doc.rst
new file mode 100644
index 000000000..c566a7264
--- /dev/null
+++ b/docs/output/doc.rst
@@ -0,0 +1,381 @@
+Generate documentation from structural files
+============================================
+
+.. note::
+
+    | The structural files contain all the information related to the variables. This output generates the documentation for all or some of these variables.
+    | **Path**: doc
+    | `*disabled*`
+    | **Disabled**: if "select for output" (step.output) is not doc.
+
+
+
+.. list-table::
+
+   * - Variable
+     - Description
+
+   * - **doc.output_format**
+
+       `choice `__ `mandatory`
+
+       **Command line**: 
+
+       -do, --doc.output_format
+
+       **Environment variable**: ROUGAILCLI_DOC.OUTPUT_FORMAT
+     - The output format of the generated documentation.
+
+       **Choices**: 
+
+       
+
+       - console **← (default)**
+
+       - asciidoc
+
+       - html
+
+       - github
+
+       - gitlab
+
+       - restructuredtext
+
+       - json
+
+   * - **doc.tabular_template**
+
+       `choice `__ `mandatory` `*disabled*`
+
+       **Command line**: 
+
+       -dm, --doc.tabular_template
+
+       **Environment variable**: ROUGAILCLI_DOC.TABULAR_TEMPLATE
+     - Generate document with this tabular model.
+
+       The variables are documented with a tabular view. A template selection allows you to choose the content of each column.
+
+       **Choices**: 
+
+       
+
+       - two_columns **← (default)**
+
+       - three_columns
+
+       - four_columns
+
+       - five_columns
+
+       - six_columns
+
+       
+
+       **Disabled**: "the output format of the generated documentation" (doc.output_format) in json is not compatible with this variable.
+
+   * - **doc.contents**
+
+       `choice `__ `multiple` `mandatory` `*hidden*` `unique`
+
+       **Command line**: 
+
+       -dc, --doc.contents
+
+       **Environment variable**: ROUGAILCLI_DOC.CONTENTS
+     - Generated content.
+
+       You can generate three type of document. All variables ("variables"), an example file in YAML format ("example") or change log ("changelog").
+
+       **Choices**: 
+
+       
+
+       - variables **← (default)**
+
+       - example
+
+       - changelog
+
+       
+
+       **Hidden**: "the output format of the generated documentation" (doc.output_format) in json is not compatible with changelog or example "generated content" (doc.contents).
+
+   * - **doc.title_level**
+
+       `integer `__ `mandatory`
+
+       **Command line**: 
+
+       -dt, --doc.title_level
+
+       **Environment variable**: ROUGAILCLI_DOC.TITLE_LEVEL
+     - Starting title level.
+
+       **Default**: 1
+
+   * - **doc.default_values**
+
+       `boolean `__ `mandatory`
+
+       **Command line**: 
+
+       
+
+       - --doc.default_values
+
+       - --doc.no-default_values
+
+       
+
+       **Environment variable**: ROUGAILCLI_DOC.DEFAULT_VALUES
+     - Modify values to document all variables.
+
+       To document homogeneous elements type or dynamic family variables, it is sometimes necessary to generate values, which can change the values in the configuration. Be careful if you reuse this configuration.
+
+       **Default**: true
+
+   * - **doc.true_color**
+
+       `boolean `__ `mandatory` `*disabled*`
+
+       **Command line**: 
+
+       
+
+       - --doc.true_color
+
+       - --doc.no-true_color
+
+       
+
+       **Environment variable**: ROUGAILCLI_DOC.TRUE_COLOR
+     - Display documentation in console always with true color terminal.
+
+       **Default**: false
+
+       **Disabled**: when the variable "the output format of the generated documentation" (doc.output_format) hasn't the value "console".
+
+The variables in this family are documented in another file
+-----------------------------------------------------------
+
+.. note::
+
+    | If you separate the variables into different files, the links between the variables will break. Therefore, you must define different filenames for the files containing these variables.
+    | This family contains lists of variable blocks.
+    | **Path**: doc.other_root_filenames
+
+
+
+.. list-table::
+
+   * - Variable
+     - Description
+
+   * - **doc.other_root_filenames.root_path**
+
+       `string `__ `multiple` `unique`
+
+       **Command line**: 
+
+       --doc.other_root_filenames.root_path
+
+       **Environment variable**: ROUGAILCLI_DOC.OTHER_ROOT_FILENAMES.ROOT_PATH
+     - This file contains child variables of the family.
+
+   * - **doc.other_root_filenames.root_doc_path**
+
+       `string `__
+
+       **Command line**: 
+
+       --doc.other_root_filenames.root_doc_path
+
+       **Environment variable**: ROUGAILCLI_DOC.OTHER_ROOT_FILENAMES.ROOT_DOC_PATH
+     - Name of the file.
+
+
+
+.. list-table::
+
+   * - Variable
+     - Description
+
+   * - **doc.document_a_type**
+
+       `boolean `__ `mandatory`
+
+       **Command line**: 
+
+       
+
+       - --doc.document_a_type
+
+       - --doc.no-document_a_type
+
+       
+
+       **Environment variable**: ROUGAILCLI_DOC.DOCUMENT_A_TYPE
+     - Documentation a structural type.
+
+       **Default**: false
+
+Variables and changelog documentation
+-------------------------------------
+
+.. note::
+
+    | **Path**: doc.tabulars
+    | `*disabled*`
+    | **Disabled**: if "the output format of the generated documentation" (doc.output_format) is json or "generated content" (doc.contents) hasn't variables or changelog.
+
+
+
+.. list-table::
+
+   * - Variable
+     - Description
+
+   * - **doc.tabulars.without_family**
+
+       `boolean `__ `mandatory`
+
+       **Command line**: 
+
+       
+
+       - -df, --doc.tabulars.without_family
+
+       - -ndf, --doc.tabulars.no-without_family
+
+       
+
+       **Environment variable**: ROUGAILCLI_DOC.TABULARS.WITHOUT_FAMILY
+     - Do not add families in documentation.
+
+       **Default**: false
+
+   * - **doc.tabulars.with_commandline**
+
+       `boolean `__ `mandatory`
+
+       **Environment variable**: ROUGAILCLI_DOC.TABULARS.WITH_COMMANDLINE
+     - Add command line informations in documentation.
+
+       **Default**: false
+
+   * - **doc.tabulars.with_environment**
+
+       `boolean `__ `mandatory`
+
+       **Command line**: 
+
+       
+
+       - -de, --doc.tabulars.with_environment
+
+       - -nde, --doc.tabulars.no-with_environment
+
+       
+
+       **Environment variable**: ROUGAILCLI_DOC.TABULARS.WITH_ENVIRONMENT
+     - Add environment variable informations in documentation.
+
+       **Default**: false
+
+   * - **doc.tabulars.environment_prefix**
+
+       `string `__ `*disabled*`
+
+       **Command line**: 
+
+       -dv, --doc.tabulars.environment_prefix
+
+       **Environment variable**: ROUGAILCLI_DOC.TABULARS.ENVIRONMENT_PREFIX
+     - Environment variables prefix name.
+
+       **Validator**: should only use uppercase characters
+
+       **Default**: ROUGAIL
+
+       **Disabled**: when the variable "add environment variable informations in documentation" (doc.tabulars.with_environment) has the value "false".
+
+Changelog documentation
+-----------------------
+
+.. note::
+
+    | **Path**: doc.changelog
+    | `*disabled*`
+    | **Disabled**: if changelog in not in "generated content" (doc.contents).
+
+
+
+.. list-table::
+
+   * - Variable
+     - Description
+
+   * - **doc.changelog.previous_json_file**
+
+       `string `__ `mandatory`
+
+       **Command line**: 
+
+       -dp, --doc.changelog.previous_json_file
+
+       **Environment variable**: ROUGAILCLI_DOC.CHANGELOG.PREVIOUS_JSON_FILE
+     - Previous description file in JSON format.
+
+       To generate the changelog, you need to compare the old list of variables (in json format) with the current variables.
+
+Examples configuration
+----------------------
+
+.. note::
+
+    | **Path**: doc.examples
+    | `*disabled*`
+    | **Disabled**: if example is not in "generated content" (doc.contents).
+
+
+
+.. list-table::
+
+   * - Variable
+     - Description
+
+   * - **doc.examples.comment**
+
+       `boolean `__ `mandatory`
+
+       **Command line**: 
+
+       
+
+       - -dx, --doc.examples.comment
+
+       - -ndx, --doc.examples.no-comment
+
+       
+
+       **Environment variable**: ROUGAILCLI_DOC.EXAMPLES.COMMENT
+     - Add description of variables and families when generate examples.
+
+       **Default**: false
+
+   * - **doc.examples.comment_column**
+
+       `integer `__ `mandatory` `*disabled*`
+
+       **Command line**: 
+
+       --doc.examples.comment_column
+
+       **Environment variable**: ROUGAILCLI_DOC.EXAMPLES.COMMENT_COLUMN
+     - Comment in examples starts at column.
+
+       **Default**: 30
+
+       **Disabled**: when the variable "add description of variables and families when generate examples" (doc.examples.comment) has the value "false".
diff --git a/docs/output/formatter.rst b/docs/output/formatter.rst
new file mode 100644
index 000000000..43772bf2e
--- /dev/null
+++ b/docs/output/formatter.rst
@@ -0,0 +1,29 @@
+Reformat the structure files
+============================
+
+.. note::
+
+    | The structure file will be formatted according to a set of rules: empty line between each variable, short-hand notation whenever possible, attribute order, re-indentation (especially for Jinja2 templates), ...
+    | **Path**: formatter
+    | `*disabled*`
+    | **Disabled**: if formatter is not set in "select for output" (step.output).
+
+
+
+.. list-table::
+
+   * - Variable
+     - Description
+
+   * - **formatter.line_width**
+
+       `integer `__ `mandatory`
+
+       **Command line**: 
+
+       --formatter.line_width
+
+       **Environment variable**: ROUGAILCLI_FORMATTER.LINE_WIDTH
+     - Maximum line size.
+
+       **Default**: 120
diff --git a/docs/output/index.rst b/docs/output/index.rst
new file mode 100644
index 000000000..5e45e4bf5
--- /dev/null
+++ b/docs/output/index.rst
@@ -0,0 +1,17 @@
+Output modules
+======================
+
+Rougail is a collections of modules in order to adjust functionalities to your needs.
+
+:term:`Output` is one of category of subjects. The goal is to display, export or document variable and value to meet the needs of the :term:`integrator` or  :term:`operator`.
+
+.. toctree::
+   :titlesonly:
+   :caption: Available output
+
+   display
+   table
+   doc
+   formatter
+   ansible
+   json
diff --git a/docs/output/json.rst b/docs/output/json.rst
new file mode 100644
index 000000000..36f0ad8ed
--- /dev/null
+++ b/docs/output/json.rst
@@ -0,0 +1,3 @@
+Export variables and values to JSON
+===================================
+
diff --git a/docs/output/table.rst b/docs/output/table.rst
new file mode 100644
index 000000000..68ea628e2
--- /dev/null
+++ b/docs/output/table.rst
@@ -0,0 +1,98 @@
+Displays the data in a table
+============================
+
+.. note::
+
+    | The goal is not to display all the variables in the configuration, but only a selection using the labeling mechanism.
+    | **Path**: table
+    | `*disabled*`
+    | **Disabled**: if table is not set in "select for output" (step.output).
+
+
+
+.. list-table::
+
+   * - Variable
+     - Description
+
+   * - **table.first_column**
+
+       `choice `__ `mandatory`
+
+       **Command line**: 
+
+       --table.first_column
+
+       **Environment variable**: ROUGAILCLI_TABLE.FIRST_COLUMN
+     - Content of the first column.
+
+       **Validator**: Tag name must not have the same name has first column
+
+       **Choices**: 
+
+       
+
+       - description
+
+       - namespace
+
+       
+
+       **Default**: First column is description if "Tag names" (table.columns) has only one value, otherwise it's namespace.
+
+   * - **table.columns**
+
+       `string `__ `multiple` `mandatory` `unique`
+
+       **Command line**: 
+
+       --table.columns
+
+       **Environment variable**: ROUGAILCLI_TABLE.COLUMNS
+     - Tag names.
+
+       Each tag creates a column. The content of the columns comes from the variables with the defined tags.
+
+   * - **table.output_format**
+
+       `choice `__ `mandatory`
+
+       **Command line**: 
+
+       --table.output_format
+
+       **Environment variable**: ROUGAILCLI_TABLE.OUTPUT_FORMAT
+     - Tag names.
+
+       **Choices**: 
+
+       
+
+       - console **← (default)**
+
+       - github
+
+       - asciidoc
+
+       - html
+
+       - rst
+
+   * - **table.header**
+
+       `boolean `__ `mandatory`
+
+       **Command line**: 
+
+       
+
+       - --table.header
+
+       - --table.no-header
+
+       
+
+       **Environment variable**: ROUGAILCLI_TABLE.HEADER
+     - Add header in table.
+
+       **Default**: true
diff --git a/docs/readme.txt b/docs/readme.txt
new file mode 100644
index 000000000..d52e0ab4d
--- /dev/null
+++ b/docs/readme.txt
@@ -0,0 +1,22 @@
+Building the doc locally
+============================
+
+install
+---------
+
+First, install a python virtual environment::
+
+    python -m venv .venv
+    source .venv/bin/activate
+
+Then install the sphinx libraries::
+
+    pip install -r requirements.txt
+
+The generatef html output is located in the `docs/build/html` subfolder,
+you can modify the target or the output type in the :file:`docs/Makefile`.
+
+scraps
+---------
+
+`variable `_
diff --git a/docs/release.rst b/docs/release.rst
new file mode 100644
index 000000000..51a384f76
--- /dev/null
+++ b/docs/release.rst
@@ -0,0 +1,215 @@
+Release procedure
+=================
+
+We are using Comitizen.
+
+We are automating vi semantic versionning (SemVer) + CHANGELOG with Commitizen
+
+Goal
+----
+
+With each release, a single command (``cz bump --changelog``) will
+automatically: 
+
+- Analyze your commits (Conventional Commits) 
+- Calculate the new version (patch, minor, major)
+- Update the version in your files
+- Generate/update the CHANGELOG.md - Create the release commit and Git tag
+
+Installation
+------------
+
+.. code:: bash
+
+   # Global installation or within your virtual environment
+   pip install commitizen
+
+   # Verify installation
+   cz version
+
+Working with Conventional Commits
+---------------------------------
+
+Creating standardized commits
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Two methods:
+
+**Method 1: Interactive guide (recommended for beginners)**
+
+.. code:: bash
+
+   cz commit
+
+An assistant guides you step by step: 1. Choose the change type
+(``fix``, ``feat``, ``docs``, etc.) 2. Enter the message 3. Describe the
+scope (optional) 4. Indicate breaking changes (if any)
+
+**Method 2: Manually**
+
+.. code:: bash
+
+   # Bug fix → patch (0.1.0 → 0.1.1)
+   git commit -m "fix: fix connection timeout"
+
+   # New feature → minor (0.1.0 → 0.2.0)
+   git commit -m "feat: add OAuth2 authentication"
+
+   # Breaking change → major (0.1.0 → 1.0.0)
+   git commit -m "feat: new REST API
+
+   BREAKING CHANGE: removed v1 endpoints"
+
+Commit types and version impact
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
++-----------------------+-----------------------+-----------------------+
+| Commit type           | Incrementation        | Example               |
++=======================+=======================+=======================+
+| ``fix:``              | **patch** (+0.0.1)    | ``f                   |
+|                       |                       | ix: fix display bug`` |
++-----------------------+-----------------------+-----------------------+
+| ``perf:``             | **patch** (+0.0.1)    | ``per                 |
+|                       |                       | f: optimize queries`` |
++-----------------------+-----------------------+-----------------------+
+| ``feat:``             | **minor** (+0.1.0)    | ``f                   |
+|                       |                       | eat: add PDF export`` |
++-----------------------+-----------------------+-----------------------+
+| ``BREAKING CHANGE:``  | **major** (+1.0.0)    | any type +            |
+| (in message)          |                       | ``BREAKING CHANGE:``  |
++-----------------------+-----------------------+-----------------------+
+
+--------------
+
+The magic command - Creating a release
+--------------------------------------
+
+When you’re ready to publish a new version:
+
+.. code:: bash
+
+   # Analyzes commits since last tag and executes the release
+   cz bump --changelog
+
+What this command does exactly:
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. **Analyzes** all commits since the last Git tag
+2. **Calculates** the next version (patch/minor/major)
+3. **Updates** the version in all listed files
+4. **Generates/updates** the ``CHANGELOG.md`` (adds the new entry at the
+   top)
+5. **Automatically commits** the changes with the message:
+   ``"bump: version X.Y.Z"``
+6. **Creates a Git tag** (format ``vX.Y.Z``)
+
+Concrete execution example:
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: bash
+
+   $ cz bump --changelog
+   INFO: Starting version bump from 1.0.0 to 1.1.0
+   INFO: Updating version in pyproject.toml
+   INFO: Updating version in src/my_project/__init__.py
+   INFO: Generating CHANGELOG.md
+   INFO: Created commit: bump: version 1.1.0
+   INFO: Created tag: v1.1.0
+
+--------------
+
+Managing the auto-generated CHANGELOG
+-------------------------------------
+
+The ``CHANGELOG.md`` file will look like this:
+
+.. code:: markdown
+
+   ## v1.1.0 (2024-01-15)
+
+   ### Feat
+
+   - add OAuth2 authentication
+   - add PDF export
+
+   ### Fix
+
+   - fix connection timeout
+
+   ## v1.0.0 (2024-01-10)
+
+   ### Breaking Changes
+
+   - removed v1 endpoints
+
+   ### Feat
+
+   - first version of REST API
+
+**You can customize the template** by adding to ``pyproject.toml``:
+
+.. code:: toml
+
+   [tool.commitizen]
+   # ... existing configuration ...
+   changelog_template = "custom_changelog_template.j2"  # Custom Jinja2 template
+
+--------------
+
+Pushing the release to Git
+--------------------------
+
+After ``cz bump --changelog``, you have a commit and a tag locally. Push
+them:
+
+.. code:: bash
+
+   # Push the commit and tags
+   git push --follow-tags
+
+--------------
+
+Useful daily commands
+---------------------
+
++-----------------------------------+-----------------------------------+
+| Command                           | Purpose                           |
++===================================+===================================+
+| ``cz commit``                     | Interactive guide to create a     |
+|                                   | conventional commit               |
++-----------------------------------+-----------------------------------+
+| ``cz bump --changelog``           | **The main command**: bumps       |
+|                                   | version + CHANGELOG               |
++-----------------------------------+-----------------------------------+
+| ``cz check``                      | Checks if your last commit        |
+|                                   | follows the convention            |
++-----------------------------------+-----------------------------------+
+| ``cz version``                    | Displays current version          |
+|                                   | (according to Commitizen)         |
++-----------------------------------+-----------------------------------+
+| ``cz changelog``                  | Generates only the CHANGELOG      |
+|                                   | without bumping                   |
++-----------------------------------+-----------------------------------+
+| ``cz bump --prerelease alpha``    | Creates a pre-release version     |
+|                                   | (e.g., 1.0.0-alpha.1)             |
++-----------------------------------+-----------------------------------+
+
+--------------
+
+Summary: The ideal workflow in 3 steps
+--------------------------------------
+
+.. code:: bash
+
+   # 1. During development: standardized commits
+   git add .
+   cz commit                    # or git commit -m "feat: ..."
+
+   # 2. When you're ready for a release
+   cz bump --changelog
+
+   # 3. Push to Git
+   git push --follow-tags
+
+**And that’s it!** Version bumped, CHANGELOG generated, tag created, all
+automated.
diff --git a/docs/requirements.txt b/docs/requirements.txt
index 87f1fb7c4..6c073861a 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -1,81 +1,3 @@
-alabaster==0.7.13
-asttokens==2.4.1
-attrs==23.1.0
-Babel==2.13.1
-certifi==2023.7.22
-charset-normalizer==3.3.2
-click==8.1.7
-colorama==0.4.6
-comm==0.2.0
-debugpy==1.8.0
-decorator==5.1.1
-docutils==0.18.1
-exceptiongroup==1.1.3
-executing==2.0.1
-fastjsonschema==2.18.1
-greenlet==3.0.1
-idna==3.4
-imagesize==1.4.1
-importlib-metadata==6.8.0
-ipykernel==6.26.0
-ipython==8.17.2
-jedi==0.19.1
-Jinja2==3.1.2
-jsonschema==4.19.2
-jsonschema-specifications==2023.7.1
-jupyter-cache==1.0.0
-jupyter_client==8.6.0
-jupyter_core==5.5.0
-livereload==2.6.3
-markdown-it-py==3.0.0
-MarkupSafe==2.1.3
-matplotlib-inline==0.1.6
-mdit-py-plugins==0.4.0
-mdurl==0.1.2
-myst-nb==1.0.0
-myst-parser==2.0.0
-nbclient==0.9.0
-nbformat==5.9.2
-nest-asyncio==1.5.8
-packaging==23.2
-parso==0.8.3
-pexpect==4.8.0
-platformdirs==4.0.0
-prompt-toolkit==3.0.40
-psutil==5.9.6
-ptyprocess==0.7.0
-pure-eval==0.2.2
-Pygments==2.16.1
-python-dateutil==2.8.2
-PyYAML==6.0.1
-pyzmq==25.1.1
-referencing==0.30.2
-requests==2.31.0
-rpds-py==0.12.0
-six==1.16.0
-snowballstemmer==2.2.0
-Sphinx==7.2.6
-sphinx-autobuild==2021.3.14
-sphinx-copybutton==0.5.2
-sphinx-lesson==0.8.15
-sphinx-minipres==0.2.1
-sphinx-rtd-theme==1.3.0
-sphinx-rtd-theme-ext-color-contrast==0.3.1
-sphinx-tabs==3.4.4
-sphinx-togglebutton==0.3.2
-sphinxcontrib-applehelp==1.0.7
-sphinxcontrib-devhelp==1.0.5
-sphinxcontrib-htmlhelp==2.0.4
-sphinxcontrib-jquery==4.1
-sphinxcontrib-jsmath==1.0.1
-sphinxcontrib-qthelp==1.0.6
-sphinxcontrib-serializinghtml==1.1.9
-SQLAlchemy==2.0.23
-stack-data==0.6.3
-tabulate==0.9.0
-tornado==6.3.3
-traitlets==5.13.0
-typing_extensions==4.8.0
-urllib3==2.0.7
-wcwidth==0.2.9
-zipp==3.17.0
+sphinx
+sphinx-lesson
+sphinxnotes-strike
diff --git a/docs/rw_ro_modes.rst b/docs/rw_ro_modes.rst
new file mode 100644
index 000000000..ff5111e3c
--- /dev/null
+++ b/docs/rw_ro_modes.rst
@@ -0,0 +1,77 @@
+:orphan:
+
+Read-write or read-only modes
+==================================
+
+The read-write mode
+--------------------
+
+When you are in the design phase, your are designing the structure file or 
+setting values is some user data files, you have the role of :term:`integrator` 
+or :term:`operator`. Then you need to have access to all the data, even 
+those which are :term:`hidden` or :term:`disabled`\ .
+
+In this phase, the configuration shall be in `RW` mode.
+
+
+.. glossary::
+
+   read-write
+   
+       In the read-write mode (RW mode), all settings are accessible and can 
+       be modified. For example, a variable which has the `frozen` property
+       can be modified. A `hidden` or `disabled` variable is accessible.
+
+       We are in the rôle of an :term:`integrator` or an :term:`operator`.
+       We are therefore in a situation where we are **using** this configuration.
+
+The read-only mode 
+--------------------
+
+Once the configuration has beed designed, it is used. 
+The situation is different. The configuration behaves as a system. 
+
+.. glossary::
+
+   read-only
+   
+       In the read-only mode (RO mode), the configuration cannot be modified. 
+       We are **using** a configuration.
+       
+       In the usage mode, we are therefore in a situation where the configuration 
+       cannot be changed. The configuration's data are immutable. 
+       
+         
+RO or RW mode?
+---------------
+
+Here is an image which summarizes these explanations:
+
+.. image:: images/read_write.png
+
+
+How to enable 
+
+By default in `rougail-cli`, the `RO` mode is activated.
+
+If you need to enable the `RW` mode, there is an `rougail-cli` option: 
+
+.. code-block:: bash
+
+    env ROUGAIL_MANUAL.USE_FOR_HTTPS=true rougail -m structfile/proxy2.yml -u yaml environment --yaml.filename userdata/proxy.yml -o json --json.read_write
+
+The output is:
+
+.. code-block:: json
+
+    {
+      "manual": {
+        "http_proxy": {
+          "address": "toto.fr",
+          "port": "8888"
+        },
+        "use_for_https": true
+      }
+    }
+
+
diff --git a/docs/structured_data/data_integrity.rst b/docs/structured_data/data_integrity.rst
new file mode 100644
index 000000000..48da272be
--- /dev/null
+++ b/docs/structured_data/data_integrity.rst
@@ -0,0 +1,86 @@
+Data integrity
+===============
+
+Data integrity refers to the fact that data must be reliable and accurate throughout its lifecycle.
+
+This means that the value must be:
+
+- of high quality
+- appropriate to the overall context
+
+Isolated variable
+-----------------
+
+.. _data_quality:
+
+Data quality
+~~~~~~~~~~~~
+
+The values of the variables must be individually of good quality.
+
+.. toctree::
+   :titlesonly:
+
+   typing
+   variable_validation
+
+.. _access_control:
+
+Access control
+~~~~~~~~~~~~~~
+
+.. toctree::
+   :titlesonly:
+
+   mode
+   variable_properties
+
+.. _overall_coherence:
+
+Overall coherence
+-----------------
+
+.. glossary::
+
+   conherence
+
+       An isolated variable can be considered to be of quality
+       but become inconsistent depending on the context.
+
+       Overall coherence is initially managed by personalized validators
+       which will validate the value of a variable in relation to others.
+       But it also depends on the availability of the variable depending on the context.
+
+.. _consistency:
+
+Consistency
+~~~~~~~~~~~
+
+.. glossary::
+
+   consistency
+
+       Consistency is validating the value of a variable in relation to other
+       variables in the :term:`context`. 
+       This does not only concern the variables themselves, such as their type.
+       Of course there is a typed base validation.
+
+       The aim is to validate a variable in a :term:`configuration`, 
+       the :term:`structured data` and the :term:`user data` based on the situation.
+       
+       This is what we call a "context"—meaning the dataset to which a variable is linked.
+
+       For example, if a minimum value and then a maximum value are requested,
+       the minimum must be lesser than the maximum. This is not type consistency,
+       but this is yet consistency.
+
+Context access control
+~~~~~~~~~~~~~~~~~~~~~~
+
+Access control occurs as soon as an attempt is made to access a variable.
+
+Remember, we talked about the :ref:`hidden variable ` and :ref:`disabled variable ` variables.
+
+These properties become fully meaningful when managing overall consistency.
+
+Why ask for the domain name of a service if we haven't activated that service just before?
diff --git a/docs/structured_data/documentation.rst b/docs/structured_data/documentation.rst
new file mode 100644
index 000000000..806f1f74b
--- /dev/null
+++ b/docs/structured_data/documentation.rst
@@ -0,0 +1,71 @@
+.. _data_documentation:
+
+Data documentation
+==================
+
+Documentation is an integral part of the definition of :term:`structured data`.
+
+In a sense, we talk about :term:`structured data` and not `schema` in Rougail because the documentation is an full part of the variable definition.
+
+More broadly, considering the use of a :term:`VaC` tool has the advantage of eliminating the need to manage documentation oneself.
+
+A lot of useful information for the :term:`operator` is naturally present when defining our variables. I'm thinking of its type, the constraints applied, its default value, the mode level, etc.
+
+But it is still necessary, as an :term:`integrator`, to guide the :term:`operator`.
+
+Describe the variable
+---------------------
+
+Describing the variable you create is not mandatory but it is highly recommended.
+Even a very well-chosen variable name does not replace a description.
+
+.. glossary::
+
+   description
+
+        A description allows you to describe the expected value(s) of a variable concisely and clearly.
+        It is designed to be clear precise and short.
+
+The description is used in different places in Rougail.
+
+Examples include:
+
+- warning/error message
+- different output
+
+Additional help to understand the variable
+------------------------------------------
+
+Help is a property that only concerns documentation.
+
+.. glossary::
+
+   help
+
+        A help helps to clarify any ambiguity about the variable’s purpose.
+        It can be long and detailed.
+
+.. seealso:: tutorial with a real world sample :doc:`help <../tutorial/document>`
+
+Give examples of possible value
+-------------------------------
+
+Often a good example is more effective than a long text.
+
+Don't hesitate to guide the :term:`operator` with relevant examples.
+
+Examples is a property that only concerns documentation.
+
+.. seealso:: tutorial with a real world sample :doc:`examples <../tutorial/document>`
+
+Specialized variables
+-----------------------
+
+In a context of value presentation, :term:`tagging ` can be very useful. The :term:`tags ` allow you to display variable values without knowing the list of variable paths.
+
+.. glossary::
+
+   tag
+
+        A tag is just a name.
+        Adding a tag is a way to attach additional information or label to variables.
diff --git a/docs/structured_data/images/schema.png b/docs/structured_data/images/schema.png
new file mode 100644
index 000000000..5b5f3804f
Binary files /dev/null and b/docs/structured_data/images/schema.png differ
diff --git a/docs/structured_data/images/schema.svg b/docs/structured_data/images/schema.svg
new file mode 100644
index 000000000..d706daf5b
--- /dev/null
+++ b/docs/structured_data/images/schema.svg
@@ -0,0 +1,95 @@
+
+
+  
+    
+    
+      
+      
+    
+    
+    
+      
+      
+    
+    
+      
+    
+    
+    
+      
+      
+    
+    
+      
+    
+    
+    
+      
+    
+    
+    
+      
+    
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+    
+      
+    
+  
+  
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    
+    IRAMISU
+    
+      
+      
+      
+      
+    
+    
+      
+      
+      
+      
+    
+    
+    
+    
+      
+        
+        
+      
+    
+    
+    
+    version: '0.10'# describe a first service with a single fileservices:- service:- name: my_servicefile:- engine: jinjatext: /etc/filename# describe a variable my_first_variable# and a family with a variable my_second_variablevariables:- variable:- name: my_first_variablevalue:- text: my_value- family:- name: my_familyvariables:- variable:- name: my_second_variabletype: numbermandatory: truevalue:- text: 1
+    
+    
+    
+    
+    
+    version: '0.10'# describe a first service with a single fileservices:- service:- name: my_servicefile:- engine: jinjatext: /etc/filename# describe a variable my_first_variable# and a family with a variable my_second_variablevariables:- variable:- name: my_first_variablevalue:- text: my_value- family:- name: my_familyvariables:- variable:- name: my_second_variabletype: numbermandatory: truevalue:- text: 1
+    Structure file
+    
+    
+    
+  
+
\ No newline at end of file
diff --git a/docs/structured_data/index.rst b/docs/structured_data/index.rst
new file mode 100644
index 000000000..cf3820a63
--- /dev/null
+++ b/docs/structured_data/index.rst
@@ -0,0 +1,10 @@
+The structure file and data
+============================
+
+.. toctree::
+   :titlesonly:
+   :caption: Rougail format's integrity and documentation
+
+   structured_data
+   documentation
+   data_integrity
diff --git a/docs/structured_data/mode.rst b/docs/structured_data/mode.rst
new file mode 100644
index 000000000..953ed7c16
--- /dev/null
+++ b/docs/structured_data/mode.rst
@@ -0,0 +1,52 @@
+.. _structured_data_mode:
+
+Mode
+====
+
+By default, the :term:`mode` is not configured. It is an optional feature.
+
+Definition
+---------------
+
+Let's start by defining what we want to do with the :term:`modes `.
+
+.. questions:: What do you mean by "mode"?
+
+    You know, in many applications, in the user interfaces, there are two editing modes:
+    a basic mode and an advanced mode. 
+    This way, standard users aren't overwhelmed by an overly complex business view, 
+    while domain experts can access more complex views that suit better to their needs.
+
+We designed this mode concept with that idea in mind. 
+Of course, it can have a broader meaning, and it is possible to define 
+much more varied modes (beyond just "basic" or "expert").
+
+.. glossary:: 
+
+   mode
+   
+       At the start of the project, the mode was used to set to variables 
+       some complexity levels. Some variables were seen as "basic",
+       some other as "normal" (the standard mode) and some variables were seen as "advanced"
+       (the advanced mode).
+
+       It was a way of classifying the variables as follows:
+       we could have a view grouping the usual variables (normal mode)
+       and a view of so-called expert (advanced) variables.
+
+       Regarding variable documentation, we can therefore have 
+       a variables documentation designed for the usual user,
+       and a variables documentation designed for advanced users.
+
+Default behavior
+------------------
+
+By default, the mode is not configured. It is an optional feature.
+We'll present a common example, but you'll need to define your own :term:`modes ` according to your needs.
+
+Here is our classic use case of :term:`mode` definition. We'll use three :term:`modes `:
+
+- `basic`: automatically sets mandatory variables without default values (in this case, the actor adapting the configuration will have to enter values) and manually sets variables that the actor defining the variables deems necessary.
+- `standard`: automatically sets all other variables
+- `advanced`: sets variables that the actor defining the variables decides to include. These variables are intended for a knowledgeable audience who know what to do.
+
diff --git a/docs/structured_data/structured_data.rst b/docs/structured_data/structured_data.rst
new file mode 100644
index 000000000..87cfcabc1
--- /dev/null
+++ b/docs/structured_data/structured_data.rst
@@ -0,0 +1,194 @@
+The structure file
+=======================
+
+What contains exactly a :term:`structure file`?
+-------------------------------------------------
+
+.. glossary::
+
+   structure file
+
+       A structure file in the Rougail meaning is a YAML file that describes variables
+       and their dependencies.
+       There can be a lot of structure files located in many different folders.
+
+       Rougail reads all the structure files and loads them into a single object
+       that represents the whole :term:`context`.
+
+A :term:`structure file` is a YAML file whose structure is described in this documentation page.
+
+A structure file contains a set of variables, usable in your application, for example in a template
+to generate configuration files.
+
+:term:`Families ` and :term:`variables ` can be defined in several structure files. These structure files are then aggregated.
+
+File naming convention
+--------------------------
+
+.. _filenamingconvention:
+
+The structure files in a given folder
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For the sake of clarity, we put the structured data in separate files.
+It's a good way to organize your rougail structures this way,
+in the real world we need separate files for different topics.
+
+For example some files like this:
+
+A file named :file:`firefox/00-proxy.yml` structure file and file named :file:`firefox/10-manual.yml`
+
+::
+
+    .
+    └── firefox
+        ├── 00-proxy.yml
+        └── 10-manual.yml
+
+.. note:: We of course could have put everything in one file.
+          Again, it is better to separate the structures in different files 
+          for reasons of ordering and clarity.
+
+Ordering your structure files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The order in which files are loaded is important in Rougail. 
+In a given folder, files are loaded in alphabetical order.
+
+Furthermore, for better readability of the order in which files are 
+loaded into a folder, we have adopted a convention. 
+To facilitate classification, we have defined a standard notation for structure file names:
+
+::
+
+    XX-.yml
+
+Where `XX` is a two digits integer followed by an hyphen, and `` is a name that describes 
+the structure that is in this file. We advise you to adopt this convention as well.
+
+File naming convention
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There is no restriction to the `` name part of file name. But it is preferable to only use
+lowercase ASCII letters, numbers and the `"_"` (undescore) character. 
+
+.. FIXME: et le moins "-" aussi non ? 
+
+.. _namespace:
+
+Separation of structures into namespaces
+----------------------------------------
+
+.. glossary:: 
+
+   namespace
+
+       A namespace is a way to organize and group related variables under a unique name.
+
+It enables us to:
+
+- avoid naming conflicts
+- permit logical grouping: related variables can be grouped together
+- scope control: namespaces limit the visibility of variable to the namespace itself
+
+By default there is no namespace.
+
+.. _defaultnamespace:
+
+The default namespace
+~~~~~~~~~~~~~~~~~~~~~
+
+The families and variables contained in these structure files are ordered, by default, in the `rougail` namespace. It is possible to change the name of this namespace with the :term:`variable namespace ` parameter of the :ref:`configuration `.
+
+This namespace (the default namespace) is a bit special, it can access variables in another namespace.
+
+The extra structure files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+An extra is a different namespace. The idea is to be able to classify the variables by theme.
+
+Extra namespaces must be declared :ref:`when configuring Rougail `.
+
+In this namespace we cannot access variables from another `extra` namespace. 
+On the other hand, it is possible to access the variable of the default namespace.
+
+The header of a structure file
+-----------------------------------
+
+A basic structure file is composed of:
+
+- a YAML 1.2 standard header
+- a version attribute
+
+.. code-block:: yaml
+
+    %YAML 1.2
+    ---
+    version: 1.1
+    ...
+
+The structure file processing
+----------------------------------
+
+The :term:`structure files ` contain information that will be used by the consistency handling system for structure validation. 
+This is what we call :term:`structured data` writen in :term:`Rougail format`.
+
+.. figure:: images/schema.png
+   :alt: The Rougail process
+   :align: center
+
+   The Rougail process from structure files to :term:`Tiramisu` valition handler object
+
+A variable-first DSL
+-----------------------
+
+All of the declaring variables and writing consistency is as simple as writing YAML.
+
+We can declare variables and describe the relationships between variables in a declarative style (that is, in a YAML file).
+
+Once the structure files are loaded by Rougail, the :term:`Tiramisu` consistency management tool can check the consistency of the variables between them.
+
+.. glossary::
+
+   DSL
+
+       A DSL stands for **Domain-Specific Language**, it is a programming or markup language designed for a specific application domain.
+       Unlike general-purpose languages (like Python or Java), DSLs are optimized for a narrow set of problems, offering concise syntax and specialized features tailored to their domain.
+
+   Rougail format
+
+       The Rougail format (or :term:`structured data`) is a DSL (Domain-Specific Language) designed for describing variables and their consistency.
+
+Declarative abstraction
+-----------------------
+
+The variables are described using a declarative model.
+This means that the user simply defines the desired final state of the variable.
+:term:`Tiramisu` will then determine the actions necessary to reach that state.
+
+In other words, the user defines the variable schema, which will then be applied deterministically.
+
+The language is a mix of YAML 1.2 and Jinja Templating.
+
+Each variables are declared in a mapping object in YAML (the keys are variable properties).
+
+There is an execption. That what we call the short-hand declaration.
+
+.. glossary::
+
+   Short-hand notation
+
+        Shorthand notation allows you to condense as much information as possible when creating a variable.
+        Generally, the declaration is done on a single line (or at most, a very small number of lines).
+
+        Of course, it's not possible to fully customize the variable with this notation.
+
+        Use shorthand notation whenever possible. This makes the file easier to read.
+
+Versioned format
+----------------
+
+The format can evolve. This means that parameters can be added, removed, or modified.
+When writing structured data, you must always specify the format version.
+You must also ensure that your Rougail version is compatible with your format version.
+
diff --git a/docs/structured_data/typing.rst b/docs/structured_data/typing.rst
new file mode 100644
index 000000000..37a68f0b8
--- /dev/null
+++ b/docs/structured_data/typing.rst
@@ -0,0 +1,191 @@
+Typing
+======
+
+The typed variable concept refers to associating a data type with a variable, which defines what kind of values the variable can hold and what operations can be performed on it.
+
+Typed variables are a fundamental feature in Rougail. Type validation is the first and most important quality check.
+
+Strongly-typed
+--------------
+
+Rougail is a strongly-typed DSL.
+
+This means that loading user data requires attention to the variable types.
+
+The :term:`integrator` define the variable type and the :term:`operator` has to conform with it.
+
+.. For untyped user data (such as environment variables), the value type will be adapted during preprocessing.
+
+Dynamicaly-typed
+----------------
+
+During the structured data definition, the type is dynamic.
+
+That is to say, the :term:`integrator` can change the type of the variable at any time.
+
+However, the :term:`operator` who adapts the value does not have the possibility of redefining the type of the variable.
+
+.. seealso:: :ref:`the variable mutability `
+
+
+Type inference
+--------------
+
+Type inference is the process where Rougail automatically figures out the types without explicit annotations.
+In fact, the variable type comes from the type of its default value.
+
+The type inference is in particular use with the :term:`short-hand notation`.
+
+Variables typing
+----------------
+
+Standard types
+~~~~~~~~~~~~~~
+
+Rougail accepts the following standard types:
+
+- `string` (default type of a variable)
+- `integer`
+- `float`
+- `boolean`
+
+.. FIXME: est-ce que integer et float -> number type ????
+
+.. seealso:: :ref:`the variable documentation `
+
+Specialized types
+~~~~~~~~~~~~~~~~~
+
+But we will also find a whole series of specialized types:
+
+- IP
+- domainname
+- port
+- MAC         
+- secret
+- ...
+
+.. seealso:: :ref:`the variable documentation `
+
+A variable with a list of possible values
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Sometimes a variable can only hold a finite list of possible values. We call it a variable with a list of possible values.
+
+.. seealso:: The tutorial with a real world sample about :doc:`a variable with a list of possible values <../tutorial/choice>`
+
+Type parameters
+~~~~~~~~~~~~~~~
+
+For certain types, there are a number of parameters that can be used to further type the variables.
+
+.. seealso:: :ref:`the variable documentation `
+
+.. _nullable_variable:
+
+Custom type
+~~~~~~~~~~~
+
+It is not possible to create every possible type of variable.
+
+Therefore, it is necessary to provide a simple way to create these custom types.
+
+There are two ways to create a type:
+
+- creating a :term:`Tiramisu` type
+- creating a type from an existing variable, have a look at the tutorial with a real world sample :doc:`custom type <../tutorial/customtype>`.
+
+Nullable variable
+~~~~~~~~~~~~~~~~~
+
+The `null` type (or `None` in Python) does not exist in Rougail.
+
+All variables, regardless of their type, can be nullable (see the remarks on the list type below).
+
+Even if all types can accept this value, by default, they do not.
+
+In reality, the variable not nullable with the value to `null` is not accessible in `read-only` mode (which is the case during the output steps).
+In `read/write` mode, the access is indeed granted.
+
+.. seealso:: The tutorial with a real world sample about a :doc:`nullable variable <../tutorial/nullable>`
+
+The `list` type
+~~~~~~~~~~~~~~~
+
+A list is not a type either.
+It is a property of a variable. There is no such thing as a container type (like a python `list` type for instance).
+This means that all type can have multiple values in a list but it can only contain values of an unique type.
+
+A multiple variable could be nullable. This means that `null` can be accepted as a value in the list (not permitted by default).
+But the multiple variable could also be without value (not permitted too by default).
+
+.. seealso:: The tutorial with a real world sample about a :doc:`multiple variable <../tutorial/multiple>`
+
+Families typing
+---------------
+
+The `mapping` type
+~~~~~~~~~~~~~~~~~~
+
+A mapping is a collection of key: value pairs.
+
+This is a JSON example:
+
+.. code-block:: json
+
+    {
+      "my_variable": "my_value"
+    }
+
+In Rougail the mapping type is called a :term:`family`.
+It is a container designed to hold variables.
+
+The whole :term:`configuration` tree structure is handled by the families definitions. 
+It is possible to create subfamilies.
+
+.. seealso:: :ref:`the family documentation `
+
+.. _dynamically_built_family:
+
+The dynamically built family
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A dynamically built family is a special family.
+
+.. glossary::
+
+   dynamically built family
+
+        A dynamically built family is a fictitious family linked to a list of unique identifiers.
+
+        This means that families will appear or disappear folling the :term:`context`.
+
+.. seealso:: :ref:`the dynamically built family documentation `
+
+Homogeneous elements sequence
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. glossary::
+
+   sequence
+   homogeneous elements sequence
+
+        A specific type of mapping, called a `sequence`, allows you to have a list of homogeneous objects.
+
+        For example, if I want to be able to create an unlimited number of users associated with a secret, you can't use lists; I need a list of objects.
+
+        In JSON we will have:
+
+        .. code-block:: json
+
+           [
+                {
+                  "user": "login1",
+                  "secret": "MySecret"
+                },
+                {
+                  "user": "login2",
+                  "secret": "MySecret"
+                }
+           ]
+
diff --git a/docs/structured_data/variable_properties.rst b/docs/structured_data/variable_properties.rst
new file mode 100644
index 000000000..f7de65d98
--- /dev/null
+++ b/docs/structured_data/variable_properties.rst
@@ -0,0 +1,43 @@
+Properties
+==========
+
+Access control is achieved through `properties`.
+
+.. glossary::
+
+   property
+
+       A property is a state (`disabled`, `mandatory`, `frozen`, `hidden`...)
+       of a family, a subfamily or a variable.
+       These properties change the usual behavior of a variable or family.
+
+The properties can be defined permanently or according to the result of a calculation.
+
+There are two main properties.
+
+.. _hidden:
+
+Hidden variable
+---------------
+
+A `hidden` variable is a variable whose value cannot be modified by the :term:`operator`.
+
+This could be an internal variable used by the :term:`integrator` that is not supposed to change.
+
+Or a variable that only makes sense in a particular context. Therefore, this variable can be hidden and then unhidden depending on the context.
+
+.. _disabled:
+
+Disabled variable
+-----------------
+
+A `disabled` variable is a variable that will not be accessible to any of the actors (:term:`integrator` and :term:`operator`).
+
+This property is generally used dynamically to remove access to the variable depending on the context.
+
+A picture is worth a thousand words:
+
+.. image:: ../images/dessin.png
+
+
+.. index:: overall coherence
diff --git a/docs/structured_data/variable_validation.rst b/docs/structured_data/variable_validation.rst
new file mode 100644
index 000000000..36dd8fdf2
--- /dev/null
+++ b/docs/structured_data/variable_validation.rst
@@ -0,0 +1,8 @@
+Validation
+===========
+
+Type validation is the first and most important check.
+But it is possible to add additional validations.
+
+For example, one might want numbers, but only odd numbers.
+
diff --git a/docs/tiramisu.rst b/docs/tiramisu.rst
new file mode 100644
index 000000000..79ccb7321
--- /dev/null
+++ b/docs/tiramisu.rst
@@ -0,0 +1,41 @@
+Link between Rougail and Tiramisu
+======================================
+
+Is Rougail using Tiramisu?
+-------------------------------------
+
+We have to say that the handling system used to ensure the variables integrity is another python library, called :term:`Tiramisu`. Rougail is currently strongly affiliated with Tiramisu.
+
+.. note:: Rougail is currently intended to work in coordination with :term:`Tiramisu` and **is not** intended to be connected with any other consistency handling system.
+
+Explained differently, Rougail allows you to easily implement an integration of the powerful :term:`Tiramisu` consistency handling system.
+
+What is a consistency handling system?
+-------------------------------------------
+
+.. questions:: Rougail, Tiramisu: What is it all about?
+
+    **Question**: OK, I have understood that the Rougail library allows me to take advantage of the :xref:`tiramisu` consistency handling library. But what is all this for? What is exactly a consistency handling system? And again, what is this :xref:`Tiramisu library ` used for?
+
+    **Answer**: Well, we will explain in details what this :xref:`tiramisu` library is and what Rougail is. 
+    
+    In (very) short: 
+
+    - Rougail is the YAML consistency description of a :term:`context`\ 's situation
+    - Tiramisu is the consistency engine linter
+
+.. glossary::
+
+   Tiramisu
+
+        :xref:`tiramisu` is a consistency handling system that has initially been designed
+        in the configuration management scope. Until now,
+        this library is generally used to handle configuration options.
+
+        It manages variables and group of variables. In the Tiramisu scope we call
+        it *options* and *option descriptions*.
+
+        Here is the :xref:`tiramisu documentation `.
+
+In the Rougail scope, we call it :term:`variables ` and :term:`families `.
+In Rougail, the families and variables are located in the :term:`structure files `.
diff --git a/docs/tutorial.rst b/docs/tutorial.rst
deleted file mode 100644
index c1941368f..000000000
--- a/docs/tutorial.rst
+++ /dev/null
@@ -1,832 +0,0 @@
-Tutorial: a real world sample
-==============================
-
-.. demo:: Demonstration : configuring (the setting of) your favorite web browser
-
-    This tutorial shows to you an example of Rougail use on
-    how to set a proxy in the `Mozilla Firefox `_ browser.
-
-More precisely, this tutorial aims at reproducing this Mozilla Firefox settings page:
-
-.. image:: images/firefox.png
-
-.. important:: Here we are in the configuration validation use case,
-               that is the values entered by the user have to be validated.
-               It's a common use case, but not the only one.
-
-Let's explain this use case.
-
-The Firefox proxy configuration
--------------------------------------------
-
-The `proxy` family
--------------------
-
-Let's create our first :term:`dictionary`.
-
-.. prerequisites:: Let's create a folder named `dict` and a dictionary file inside
-
-We will put our dictionary files in this folder.
-
-Then let's put our first dictionary file in this folder, named :file:`00-proxy.yml`
-
-.. code-block:: yaml
-   :caption: the :file:`00-proxy.yml` file
-   :linenos:
-
-    ---
-    version: '1.1'
-    proxy:
-      description: Proxy configuration in order to have access to the internet
-      type: family
-
-We can see that we have defined a :term:`family` here, and this family is *empty*
-(that is, the family container contains no variable yet).
-
-.. admonition:: If a family is empty
-
-    We need to specify the :term:`family` type (line 5) here because if we don't,
-    the Rougail's type engine will infer it by default as a :term:`variable`.
-
-It's because we don't have set any :term:`variable` inside.
-
-
-.. note:: The variables will be created in several files for educational purposes.
-          Obviously all the variables can be put in the same file.
-
-
-The proxy's configuration type
-----------------------------------
-
-In the Firefox configuration, it is possible to define several configuration modes,
-from no proxy at all (`no proxy`) to a kind of automatic configuration mode from a file (`set up proxy configuration from a file`).
-
-We're gonna create a first variable in this family with "Proxy mode" as the description.
-Let's create a second :file:`dict/01-proxy_mode.yml` file.
-
-.. code-block:: yaml
-   :caption: the :file:`001-proxy_mode.yml` file
-   :linenos:
-
-    ---
-    version: '1.1'
-    proxy:
-      proxy_mode:
-        description: Proxy mode
-        type: choice
-        choices:
-          - No proxy
-          - Auto-detect proxy settings for this network
-          - Use system proxy settings
-          - Manual proxy configuration
-          - Automatic proxy configuration URL
-        default: No proxy
-
-The `proxy_mode` variable requires a value (that is, `None` is not an option).
-It shall have a value, but what if the user *does not* specify any value?
-There is line 13, a possibility of setting a default value, wich is `No proxy` as the default.
-
-The `proxy_mode` setting is "choice" (`type: choice`) means that
-there is a list of available values that can be selected.
-We say that the `proxy_mode` variable is *constrained* (by choices).
-
-Line 8 to 12, we have the list of the possible (authorized) values:
-
-- No proxy
-- Auto-detect proxy settings for this network
-- Use system proxy settings
-- Manual proxy configuration
-- Automatic proxy configuration URL
-
-Now let's test our first two dictionaries:
-
-
->>> from rougail import Rougail, RougailConfig
->>> from pprint import pprint
->>> RougailConfig['dictionaries_dir'] = ['dict']
->>> rougail = Rougail()
->>> config = rougail.get_config()
->>> config.property.read_only()
->>> pprint(config.value.get(), sort_dicts=False)
-{'rougail.proxy.proxy_mode': 'No proxy'}
-
-The manual mode
-------------------
-
-.. questions:: OK then. What happens when you select the "Manual proxy configuration"?
-
-A good configuration design is to place all the proxy's manual configuration in a :term:`family`.
-Let's create the :file:`dict/02-proxy_manual.yml` dictionary:
-
-.. code-block:: yaml
-   :caption: the the :file:`dict/02-proxy_manual.yml` file
-
-    ---
-    version: '1.1'
-    proxy:
-      manual:
-        description: Manual proxy configuration
-        type: family
-        disabled:
-          type: jinja
-          jinja: |
-            {% if rougail.proxy.proxy_mode != 'Manual proxy configuration' %}
-            the proxy mode is not manual
-            {% endif %}
-
-Well, if the user selects the "Manual proxy configuration" proxy mode, we want to see a new subfamily (that is, a new set of configuration variables) called `manual` to appear (which is disabled).
-
-.. glossary::
-
-   subfamily
-
-       A subfamily is just a family inside a family, a family that contains a family.
-
-.. questions:: What about this `Jinja` type?
-
-If the :term:`Jinja` template returns some text, then the family will be `disabled`. Otherwise it is accessible.
-Deactivating a family means that we will not be able to access it as well as the variables or families included in this family.
-
-.. note:: If the Jinja template does not return any text, the variable will be **enabled**.
-          Here we are using the Jinja condition statement.
-
-.. glossary::
-
-    Jinja
-
-        `Jinja `_ is a template engine.
-        we are using Jinja in a classical way, that is, Jinja allows us to handle different cases,
-        for example with the `if` statement.
-
-The HTTP proxy configuration
-------------------------------
-
-In this family let's add a *subfamily* named `http_proxy`, containing the address and port configuration variables.
-
-Let's create the :file:`dict/03-proxy_manual_http_proxy.yml` dictionary:
-
-.. code-block:: yaml
-   :caption: the the :file:`dict/02-proxy_manual.yml` file
-   :linenos:
-
-    ---
-    version: '1.1'
-    proxy:
-      manual:
-        http_proxy:
-          description: HTTP Proxy
-          address:
-            description: HTTP address
-            type: domainname
-          port:
-            description: HTTP Port
-            type: port
-            default: '8080'
-
-Both variables `address` and `port` have particular types (respectively `domainname` line 9 and `port` line 12) to validate the values configured by the user.
-
-.. note:: No need to specify the type of the `http_proxy` as a family type, because here we have declared variables inside of it.
-
-Duplicating the HTTP configuration to HTTPS
----------------------------------------------
-
-We then want to offer the user the possibility of providing the same proxy for the HTTPS requests. Let's create the :file:`dict/04-proxy_manual_http_use_for_https.yml` file:
-
-.. code-block:: yaml
-   :caption: the :file:`dict/04-proxy_manual_http_use_for_https.yml` file
-
-    version: '1.1'
-    proxy:
-      manual:
-        use_for_https:
-          description: Also use this proxy for HTTPS
-          type: boolean
-
-This variable is a `boolean` type, its default value is `True`.
-
-HTTPS proxy configuration detail
------------------------------------
-
-Let's add a new subfamily named `ssl_proxy`, containing the `address` and `port` variables.
-
-Let's create the :file:`dict/05-proxy_manual_ssl_proxy.yml` file:
-
-.. code-block:: yaml
-   :caption: the :file:`dict/04-proxy_manual_http_use_for_https.yml` file
-   :linenos:
-
-    ---
-    version: '1.1'
-    proxy:
-      manual:
-        ssl_proxy:
-          description: HTTPS Proxy
-          hidden:
-            type: variable
-            variable: rougail.proxy.manual.use_for_https
-          address:
-            description: HTTPS address
-            type: domainname
-            default:
-              type: jinja
-              jinja: |
-                {% if rougail.proxy.manual.use_for_https %}
-                {{ rougail.proxy.manual.http_proxy.address }}
-                {% endif %}
-          port:
-            description: HTTPS Port
-            type: port
-            default:
-              type: jinja
-              jinja: |
-                {% if rougail.proxy.manual.use_for_https %}
-                {{ rougail.proxy.manual.http_proxy.port }}
-                {% endif %}
-
-
-Depending on the value of the `rougail.proxy.mandatory.use_for_https` variable, this family will appear or disappear (the `hidden` setting line 7). Unlike earlier, this time it is not necessary to use a Jinja function.
-
-Let's notice that the family is not disabled because the variables will need to remain accessible (yet in `read-only` mode).
-
-The address and port variables are copied from HTTP to HTTPS if `rougail.proxy.use_for_https` is set to `True`.
-
-Now let's test all of it:
-
->>> from rougail import Rougail, RougailConfig
->>> from pprint import pprint
->>> RougailConfig['dictionaries_dir'] = ['dict']
->>> rougail = Rougail()
->>> config = rougail.get_config()
->>> config.property.read_only()
->>> pprint(config.value.get(), sort_dicts=False)
-{'rougail.proxy.proxy_mode': 'No proxy'}
-
-At this time the proxy is not configured yet, so we do not see any variables.
-Let's look at what happens if we try to access the `rougail.proxy.manual` variable if we are not in manual mode:
-
-.. code-block:: python
-
-    >>> pprint(config.option('rougail.proxy.manual').value.get(), sort_dicts=False)
-
-We have an error (with the message defined in the Jinja template):
-
-
-.. code-block:: shell
-
-    tiramisu.error.PropertiesOptionError: cannot access to
-    optiondescription "Manual proxy configuration" because
-    has property "disabled" (the mode proxy is not manual)
-
-
-Let's configure the proxy in manual mode
-
->>> config.property.read_write()
->>> config.option('rougail.proxy.proxy_mode').value.set('Manual proxy configuration')
->>> config.option('rougail.proxy.manual.http_proxy.address').value.set('proxy.example')
->>> pprint(config.value.get(), sort_dicts=False)
-
-We can see that the returned variables does have the desired values:
-
-.. code-block:: python
-
-    {'rougail.proxy.proxy_mode': 'Manual proxy configuration',
-     'rougail.proxy.manual.http_proxy.address': 'proxy.example',
-     'rougail.proxy.manual.http_proxy.port': '8080',
-     'rougail.proxy.manual.use_for_https': True}
-
-Let's set the `read_only` mode and have a look at the configuration again:
-
-.. code-block:: python
-
-    >>> config.property.read_only()
-    >>> pprint(config.value.get(), sort_dicts=False)
-    {'rougail.proxy.proxy_mode': 'Manual proxy configuration',
-     'rougail.proxy.manual.http_proxy.address': 'proxy.example',
-     'rougail.proxy.manual.http_proxy.port': '8080',
-     'rougail.proxy.manual.use_for_https': True,
-     'rougail.proxy.manual.ssl_proxy.address': 'proxy.example',
-     'rougail.proxy.manual.ssl_proxy.port': '8080'}
-
-In the `read_only` mode, we can see that the HTTPS configuration appears.
-
-.. note:: We can see that `rougail.proxy.manual.http_proxy` values have been copied
-          in `rougail.proxy.manual.ssl_proxy` too.
-
-Changing values programmatically
---------------------------------------
-
-We are going to use the :term:`Tiramisu` API to manipulate programmatically the different variables.
-
-First, let's set `rougail.proxy.manual.use_for_https` to `False`. It is now possible
-to configure the HTTPS:
-
-.. code-block:: python
-
-    >>> config.property.read_write()
-    >>> config.option('rougail.proxy.manual.use_for_https').value.set(False)
-    >>> config.option('rougail.proxy.manual.ssl_proxy.address').value.set('other.proxy.example')
-    >>> pprint(config.value.get(), sort_dicts=False)
-    {'rougail.proxy.proxy_mode': 'Manual proxy configuration',
-     'rougail.proxy.manual.http_proxy.address': 'proxy.example',
-     'rougail.proxy.manual.http_proxy.port': '8080',
-     'rougail.proxy.manual.use_for_https': False,
-     'rougail.proxy.manual.ssl_proxy.address': 'other.proxy.example',
-     'rougail.proxy.manual.ssl_proxy.port': '8080'}
-
-The value of the variable `rougail.proxy.manual.ssl_proxy.address` has actually been modified.
-But if this variable is hidden again, then the value comes back to the default value:
-
-.. code-block:: python
-
-    >>> config.option('rougail.proxy.manual.use_for_https').value.set(False)
-    >>> config.property.read_only()
-    >>> pprint(config.value.get(), sort_dicts=False)
-    {'rougail.proxy.proxy_mode': 'Manual proxy configuration',
-     'rougail.proxy.manual.http_proxy.address': 'proxy.example',
-     'rougail.proxy.manual.http_proxy.port': '8080',
-     'rougail.proxy.manual.use_for_https': False,
-     'rougail.proxy.manual.ssl_proxy.address': 'proxy.example',
-     'rougail.proxy.manual.ssl_proxy.port': '8080'}
-
-SOCK's proxy configuration
--------------------------------
-
-Let's add a new :term:`subfamily` named `socks_proxy` with the `address`,
-`port` and `version` variables.
-
-Let's create the :file:`dict/06-proxy_manual_socks_proxy.yml` file:
-
-.. code-block:: yaml
-   :caption: the :file:`dict/06-proxy_manual_socks_proxy.yml` file
-
-    ---
-    version: '1.1'
-    proxy:
-      manual:
-        socks_proxy:
-          description: SOCKS Proxy
-          address:
-            description: SOCKS Address
-            type: domainname
-          port:
-            description: SOCKS Port
-            type: port
-          version:
-            description: SOCKS host version used by proxy
-            type: choice
-            choices:
-              - v4
-              - v5
-            default: v5
-
-There's nothing new to learn with this file.
-
-The automatic detection mode
-------------------------------
-
-Let's add a new variable named `auto`.
-
-Let's create the :file:`dict/07-proxy_auto.yml` file:
-
-.. code-block:: yaml
-   :caption: the :file:`dict/07-proxy_auto.yml` file
-
-    ---
-    version: '1.1'
-    proxy:
-      auto:
-        type: web_address
-        description: Automatic proxy configuration URL
-        disabled:
-          type: jinja
-          jinja: |
-            {% if rougail.proxy.proxy_mode != 'Automatic proxy configuration URL' %}
-            the proxy mode is not automatic
-            {% endif %}
-
-The `web_address` type imposes a value starting with `http://` or `https://`.
-This variable is activated when the proxy is in automatic mode.
-
-The proxy's exceptions
----------------------------
-
-Finally, let's add a variable containing proxy exceptions.
-
-Let's create the :file:`dict/07-proxy_no_proxy.yml` file:
-
-.. code-block:: yaml
-   :caption: the :file:`dict/07-proxy_no_proxy.yml` file
-   :linenos:
-
-    ---
-    version: '1.1'
-    proxy:
-      no_proxy:
-        description: Address for which proxy will be desactivated
-        multi: true
-        type: "domainname"
-        params:
-          allow_ip: true
-          allow_cidr_network: true
-          allow_without_dot: true
-          allow_startswith_dot: true
-        disabled:
-          type: jinja
-          jinja: |
-            {% if rougail.proxy.proxy_mode == 'No proxy' %}
-            proxy mode is no proxy
-            {% endif %}
-        mandatory: false
-
-This `no_proxy` variable is much like a `domainname` type except that we add
-a `params` line 7, we authorize the :
-
-- IP
-- CIDR networks
-- machine names (without `'.'`)
-- sub-domaines like `.example`
-
-There can be multiple exceptions to the proxy, so the variable is :term:`multi` (line5).
-This variable is only accessible if no proxy is defined (`disabled`).
-
-.. glossary::
-
-   multi
-
-       A multi is a multiple variable, that is a variable that can have multiple values.
-
-
-The `no_proxy` variable do not requires a value (that is, `None` is an option),
-there is line 19 this statement `mandatory: false` which means that this variable is not mandatory.
-
-
-Let's test it:
-
-
->>> from rougail import Rougail, RougailConfig
->>> from pprint import pprint
->>> RougailConfig['dictionaries_dir'] = ['dict']
->>> rougail = Rougail()
->>> config = rougail.get_config()
->>> config.property.read_write()
->>> config.option('rougail.proxy.proxy_mode').value.set('Manual proxy configuration')
->>> config.option('rougail.proxy.manual.http_proxy.address').value.set('proxy.example')
->>> config.option('rougail.proxy.no_proxy').value.set(['.example', '192.168.1.1'])
->>> config.property.read_only()
->>> pprint(config.value.get(), sort_dicts=False)
-
-It outputs:
-
-.. code-block:: python
-
-    {'rougail.proxy.proxy_mode': 'Manual proxy configuration',
-     'rougail.proxy.manual.http_proxy.address': 'proxy.example',
-     'rougail.proxy.manual.http_proxy.port': '8080',
-     'rougail.proxy.manual.use_for_https': True,
-     'rougail.proxy.manual.ssl_proxy.address': 'proxy.example',
-     'rougail.proxy.manual.ssl_proxy.port': '8080',
-     'rougail.proxy.manual.socks_proxy.address': None,
-     'rougail.proxy.manual.socks_proxy.port': None,
-     'rougail.proxy.manual.socks_proxy.version': 'v5',
-     'rougail.proxy.no_proxy': ['.example', '192.168.1.1']}
-
-But not possible to put an invalid value:
-
-.. code-block:: python
-
-    >>> config.option('rougail.proxy.no_proxy').value.set(['.example', '192.168.1.1', 'not valid'])
-    [..]
-    tiramisu.error.ValueOptionError: "not valid" is an invalid domain name for "Address for which proxy will be desactivated", could be a IP, otherwise must start with lowercase characters followed by lowercase characters, number, "-" and "." characters are allowed
-
-
-The authentification request
---------------------------------
-
-Nothing special when creating the authentication request. To do this, let's create a `dict/08-proxy_prompt_authentication.yml` file:
-
-
-.. code-block:: yaml
-   :caption: the :file:`dict/08-proxy_prompt_authentication.yml` file
-   :linenos:
-
-    ---
-    version: '1.1'
-    proxy:
-      prompt_authentication:
-        description: Prompt for authentication if password is saved
-        type: boolean
-        default: true
-        disabled:
-          type: jinja
-          jinja: |
-            {% if rougail.proxy.proxy_mode == 'No proxy' %}
-            proxy mode is no proxy
-            {% endif %}
-
-The proxy SOCKS v5's DNS
-------------------------------
-
-The DNS variable for the SOCKS v5 proxy only appears if the proxy is configured and the version of the SOCKS proxy selected is `v5`.
-
-Let's create a `dict/09-proxy_proxy_dns_socks5.yml` file:
-
-.. code-block:: yaml
-   :caption: the :file:`dict/09-proxy_proxy_dns_socks5.yml` file
-   :linenos:
-
-    ---
-    version: '1.1'
-    proxy:
-      proxy_dns_socks5:
-        description: Use proxy DNS when using SOCKS v5
-        type: boolean
-        default: false
-        disabled:
-          type: jinja
-          params:
-            socks_version:
-              type: variable
-              variable: rougail.proxy.manual.socks_proxy.version
-              propertyerror: false
-          jinja: |
-            {% if rougail.proxy.proxy_mode == 'No proxy' %}
-            the proxy mode is no proxy
-            {% elif socks_version is undefined or socks_version == 'v4' %}
-            socks version is v4
-            {% endif %}
-
-The difficulty here is that the  `rougail.proxy.manual.socks_proxy.version` variable
-can be deactivated (and therefore not usable in a calculation).
-
-.. FIXME definir ce qu'est une calculation
-
-In this case, we will add a parameter (here called `socks_version`) which will contain,
-if there is no property error, the value of the variable.
-Otherwise the parameter will not be passed to the Jinja template.
-
-This is why it is necessary to test in the Jinja template whether the `socks_version` variable really exists.
-
-The DNS over HTTPS
-----------------------
-
-Finally we will configure DNS over HTTPS in the 10-proxy_dns_over_https.yml file:
-
-Let's create a `dict/10-proxy_dns_over_https.yml` file:
-
-.. code-block:: yaml
-   :caption: the :file:`dict/10-proxy_dns_over_https.yml` file
-   :linenos:
-
-    ---
-    version: '1.1'
-    proxy:
-      dns_over_https:
-        description: DNS over HTTPS
-        enable_dns_over_https:
-          description: Enable DNS over HTTPS
-          type: boolean
-          default: false
-        provider:
-          description: Use Provider
-          type: choice
-          choices:
-            - Cloudflare
-            - NextDNS
-            - Custom
-          default: Cloudflare
-          disabled:
-            type: jinja
-            jinja: |
-              {% if not rougail.proxy.dns_over_https.enable_dns_over_https %}
-              Enable DNS over HTTPS is False
-              {% endif %}
-        custom_dns_url:
-          description: Custom DNS URL
-          type: web_address
-          disabled:
-            type: jinja
-            params:
-              provider:
-                type: variable
-                variable: rougail.proxy.dns_over_https.provider
-                propertyerror: false
-            jinja: |
-              {% if provider is not defined or provider != 'Custom' %}
-              provider is not custom
-              {% endif %}
-          validators:
-            - type: jinja
-              jinja: |
-                {% if rougail.proxy.dns_over_https.custom_dns_url.startswith('http://') %}
-                only https is allowed
-                {% endif %}
-
-.. FIXME : define validators
-
-The only particularity here is that we added additional validation (validators) to the `custom_dns_url` variable. Only an address starting with `https://` is allowed (not `http://`).
-
-----
-
-The FoxyProxy type's proxy configuration
---------------------------------------------
-
-Here is now the integration of part of the Firefox FoxyProxy plugin.
-
-The idea is to have a namespace specific to FoxyProxy and to find in it part of the settings that we will have made in the main namespace.
-
-This is what the page looks like:
-
-.. image:: images/foxyproxy.png
-
-It is possible, in this plugin, to specify an unlimited number of proxies.
-Our `proxy` family will no longer be of the `family` type as before but of another type : the :term:`leadership` type.
-
-.. FIXME: expliquer ce qu'est le type leardership
-
-Here is the complete content of the FoxyProxy type proxy configuration
-(to be put in the `foxyproxy/00-base.yml` file):
-
-.. code-block:: yaml
-   :caption: the :file:``foxyproxy/00-base.yml`` file
-   :linenos:
-
-    ---
-    version: '1.1'
-    proxy:
-      _type: leadership
-      title:
-        description: Title or Description
-        multi: true
-      color:
-        description: Color
-      type:
-        type: choice
-        choices:
-          - HTTP
-          - HTTPS/SSL
-          - SOCKS5
-          - SOCKS4
-          - PAC URL
-          - WPAD
-          - System (use system settings)
-          - Direct (no proxy)
-        default: Direct (no proxy)
-      address:
-        description: IP address, DNS name, server name
-        multi: true
-        disabled:
-          type: jinja
-          jinja: |
-            {% if foxyproxy.proxy.type not in ['HTTP', 'HTTPS/SSL', 'SOCKS5', 'SOCKS4'] %}
-            proxy does not need address
-            {% endif %}
-        default:
-          type: jinja
-          params:
-            firefox_address:
-              type: variable
-              variable: rougail.proxy.manual.http_proxy.address
-              propertyerror: false
-          jinja: |
-            {% if firefox_address is not undefined %}
-            {{ firefox_address }}
-            {% endif %}
-      port:
-        description: Port
-        type: port
-        default:
-          type: jinja
-          params:
-            firefox_port:
-              type: variable
-              variable: rougail.proxy.manual.http_proxy.port
-              propertyerror: false
-          jinja: |
-            {% if firefox_port is not undefined %}
-            {{ firefox_port }}
-            {% endif %}
-        disabled:
-          type: jinja
-          jinja: |
-            {% if foxyproxy.proxy.type not in ['HTTP', 'HTTPS/SSL', 'SOCKS5', 'SOCKS4'] %}
-            proxy does not need port
-            {% endif %}
-      username:
-        description: Username
-        type: unix_user
-        mandatory:
-          type: jinja
-          jinja: |
-            {% if foxyproxy.proxy.password %}
-            username is mandatory
-            {% endif %}
-        disabled:
-          type: jinja
-          jinja: |
-            {% if foxyproxy.proxy.type not in ['HTTP', 'HTTPS/SSL', 'SOCKS5', 'SOCKS4'] %}
-            proxy does not need username
-            {% endif %}
-      password:
-        description: Password
-        type: secret
-        disabled:
-          type: jinja
-          jinja: |
-            {% if foxyproxy.proxy.type not in ['HTTP', 'HTTPS/SSL', 'SOCKS5', 'SOCKS4'] %}
-            proxy does not need password
-            {% endif %}
-      url:
-        type: web_address
-        disabled:
-          type: jinja
-          jinja: |
-            {% if foxyproxy.proxy.type not in ['PAC URL', 'WPAD'] %}
-            proxy does not need url
-            {% endif %}
-
-
-A few comments:
-
-- in the `foxyproxy.proxy` :term:`leader` family there is a variable named `type` (line 4), this may conflict with the `type` attribute (specified line 10). In this case, to specify the type we use the `_type` attribute
-- a :term:`follower` variable can also be multiple
-  (which is the case for `foxyproxy.proxy.address`)
-- `foxyproxy.proxy.username` (line 62) becomes :term:`mandatory` if `foxyproxy.proxy.password`
-  is specified, in fact a password without a username is meaningless
-
-Let's test it:
-
->>> from rougail import Rougail, RougailConfig
->>> from pprint import pprint
->>> RougailConfig['dictionaries_dir'] = ['dict']
->>> RougailConfig['extra_dictionaries']['foxyproxy'] = ['foxyproxy/']
->>> rougail = Rougail()
->>> config = rougail.get_config()
->>> config.option('rougail.proxy.proxy_mode').value.set('Manual proxy configuration')
->>> config.option('rougail.proxy.manual.http_proxy.address').value.set('proxy.example')
->>> config.option('foxyproxy.proxy.title').value.set(['MyProxy'])
->>> config.option('foxyproxy.proxy.type', 0).value.set('HTTP')
->>> config.option('foxyproxy.proxy.color', 0).value.set('#00000')
->>> config.property.read_only()
->>> pprint(config.value.get(), sort_dicts=False)
-
-The output is:
-
-.. code-block:: python
-
-    {'rougail.proxy.proxy_mode': 'Manual proxy configuration',
-     'rougail.proxy.manual.http_proxy.address': 'proxy.example',
-     'rougail.proxy.manual.http_proxy.port': '8080',
-     'rougail.proxy.manual.use_for_https': True,
-     'rougail.proxy.manual.ssl_proxy.address': 'proxy.example',
-     'rougail.proxy.manual.ssl_proxy.port': '8080',
-     'rougail.proxy.manual.socks_proxy.address': None,
-     'rougail.proxy.manual.socks_proxy.port': None,
-     'rougail.proxy.manual.socks_proxy.version': 'v5',
-     'rougail.proxy.no_proxy': [],
-     'rougail.proxy.proxy_dns_socks5': False,
-     'rougail.proxy.dns_over_https.enable_dns_over_https': False,
-     'foxyproxy.proxy.title': [{'foxyproxy.proxy.title': 'MyProxy',
-                                'foxyproxy.proxy.color': '#00000',
-                                'foxyproxy.proxy.type': 'HTTP',
-                                'foxyproxy.proxy.address': ['proxy.example'],
-                                'foxyproxy.proxy.port': '8080',
-                                'foxyproxy.proxy.username': None,
-                                'foxyproxy.proxy.password': None}]}
-
-The choice we made here is to make `foxyproxy.proxy.username` :term:`mandatory` if a password is specified in the `foxyproxy.proxy.password` variable.
-
-It makes sense to have a username without a password (in this case the password will be requested when connecting to the proxy). But the opposite does not make sense.
-
-From a user point of view this may seem disturbing (if you enter the password, you have to return to the previous option to specify the password).
-
-It is possible to reverse the logic. If the `foxyproxy.proxy.username` variable is set, the `foxyproxy.proxy.password` variable becomes editable.
-
-None of this two variables needs to be :term:`mandatory`.
-
-If you prefer this option, here is a second extra dictionary :file:`foxyproxy/01-redefine.yml` which will redefine the behavior only of the `foxyproxy.proxy.username` and `foxyproxy.proxy.password` variables:
-
-
-
-
-.. code-block:: yaml
-   :caption: the :file:`foxyproxy/01-redefine.yml` file
-   :linenos:
-
-    ---
-    version: '1.1'
-    proxy:
-      username:
-        redefine: true
-        # suppress mandatory constrainte
-        mandatory: false
-      password:
-        redefine: true
-        hidden:
-          type: jinja
-          jinja: |
-            {% if not foxyproxy.proxy.username %}
-            no username defined
-            {% endif %}
-
-
-**It's up to you to play now !**
diff --git a/docs/tutorial/auto_save.rst b/docs/tutorial/auto_save.rst
new file mode 100644
index 000000000..9984030d1
--- /dev/null
+++ b/docs/tutorial/auto_save.rst
@@ -0,0 +1,62 @@
+Auto save value
+===============
+
+.. objectives:: Objectives
+
+   A default value is a value that can be recalculated at different times. It can be useful to consider the default value as a modified value.
+
+.. prerequisites:: Prerequisites
+
+    - We assume that Rougail's library is :ref:`installed ` on your computer.
+
+    - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step
+      by checking out the corresponding tag of the `rougail-tutorials` git repository.
+      Each tag corresponds to a stage of progress in the tutorial.
+      Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps.
+
+    If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, 
+    this workshop page corresponds to the tag :tutorial:`v1.1_210 `
+
+    ::
+   
+       git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git
+       git switch --detach v1.1_210
+
+The :ref:`configuration` is loaded into :term:`Tiramisu`. In this case, the default values are cached upon the first access to the variable. However, sometimes this value will be recalculated.
+If the default value is random, this can cause problems. Indeed, the value can change over time.
+
+The `auto_save` property allows you to say that the calculated value will be considered as modified (therefore not recalculated) on the first access to this variable.
+
+Remember, we just performed a random calculation for the variable `color`:
+
+.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_201/foxyproxy/00-foxyproxy.yml
+   :language: yaml
+   :caption: Calculate the default variable `color`.
+
+We just need to set the `auto_save` property to true on this variable:
+
+.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_210/foxyproxy/00-foxyproxy.yml
+   :language: yaml
+   :caption: Auto_save for the variable `color`.
+
+Test this variable with this Rougail CLI: 
+
+.. raw:: html
+   :class: terminal
+   :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_210/config/02/cmd_ro.txt
+
+We have this output:
+
+.. raw:: html
+   :class: output
+   :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_210/config/02/output_ro.html
+
+As you can see, the value is the color of the modified variables.
+
+.. note::
+
+   We do not save this value. It is therefore stored in memory. When the Rougail CLI closes, this value is forgotten. It will be recalculated if you run the command again.
+
+.. keypoints:: Key points
+
+   We can auto save a variable.
diff --git a/docs/tutorial/calculated.rst b/docs/tutorial/calculated.rst
new file mode 100644
index 000000000..d5a0e09a0
--- /dev/null
+++ b/docs/tutorial/calculated.rst
@@ -0,0 +1,275 @@
+Calculated default value for a variable
+============================================
+
+.. objectives:: Objectives
+
+   In this section we will reuse the value of a variable for the default value of another variable.
+   
+   We will first build the `https_proxy` family which will be used to illustrate this functionality.
+     
+.. prerequisites:: Prerequisites
+
+    - We assume that Rougail's library is :ref:`installed ` on your computer.
+
+    - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step
+      by checking out the corresponding tag of the `rougail-tutorials` git repository.
+      Each tag corresponds to a stage of progress in the tutorial.
+      Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps.
+
+    If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, 
+    this workshop page corresponds to the tags :tutorial:`v1.1_040 ` to :tutorial:`v1.1_041 ` 
+    in the repository.
+
+    ::
+   
+       git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git
+       git switch --detach v1.1_040
+
+HTTPS family 
+-------------
+
+We have split the definition of the `manual` family into two specific files, 
+the :file:`10-manual.yml` and the :file:`20-manual.yml` structure file: 
+
+.. raw:: html
+   :class: output 
+   :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_040/tree.html 
+
+..
+    

+ .
+ └── firefox
+ ├── 00-proxy.yml
+ ├── 10-manual.yml
+ └── 20-manual.yml
+

+ +We will continue to complete the :file:`20-manual.yml` structure file, +with variables entirely equivalent to those found in the `http_proxy` family from the :file:`10-manual.yml` structure file. +Until now, we only had the `use_for_https` :ref:`boolean variable in this structure file `. +Now we are going to define an address and a port in the `https_proxy` family: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_040/firefox/20-manual.yml + :linenos: + :language: yaml + :caption: The updated `manual` family in the :file:`firefox/20-manual.yml` structure file with the `https_proxy` family. + +.. + %YAML 1.2 + --- + version: 1.1 + + manual: + + use_for_https: true # Also use this proxy for HTTPS + + https_proxy: # HTTPS Proxy + + address: + description: HTTPS address + type: domainname + params: + allow_ip: true + + port: + description: HTTPS Port + type: port + default: 8080 + +.. _tutorial_calc_variable: + +A default value calculated from another variable +-------------------------------------------------- + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_041 ` version:: + + git switch --detach v1.1_041 + +.. discussion:: A contextualized default value + + A contextualized default value is a default value that changes according to the overall context of the configuration. + A contextualized default value can come from, as we see here, a copy of the value of another variable. + +The dynamic setting of a default value can be expressed this way: the default value is a pointer to another variable's value. +Here, the defaut value of `manual.https_proxy.address` points to the value of `manual.http_proxy.address`. + +This is the same thing for the default value of the `manual.https_proxy.port` variable, +which points to the `manual.http_proxy.port` value. + +.. note:: In the following we will see that the `manual.https_proxy.address` type is `domainname`. + Indeed, this `domainname` type comes from the type of the variable it points to. + +Reminder: here is the HTTP :file:`firefox/10-manual.yml` structure file: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_041/firefox/10-manual.yml + :linenos: + :language: yaml + :caption: The :file:`firefox/10-manual.yml` structure file with the `http_proxy` family and the `disabled` property + +.. + %YAML 1.2 + --- + version: 1.1 + + manual: # Manual proxy configuration + + http_proxy: # HTTP Proxy + + address: + description: HTTP address + type: domainname + params: + allow_ip: true + + port: + description: HTTP Port + type: port + default: 8080 + ... + + +And here is the :file:`firefox/20-manual.yml` structure file where the calculated default values are: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_041/firefox/20-manual.yml + :linenos: + :language: yaml + :caption: The :file:`firefox/20-manual.yml` structure file with the `hidden` property on the `https_proxy` family. + +.. + %YAML 1.2 + --- + version: 1.1 + + manual: + + use_for_https: true # Also use this proxy for HTTPS + + https_proxy: # HTTPS Proxy + + address: + description: HTTPS address + default: + variable: __.http_proxy.address + + port: + description: HTTPS Port + default: + variable: __.http_proxy.port + ... + +We can see here that the `address` variable's default value is conditionned by the `__.http_proxy.address` variable's value. +The target variable is `manual.http_proxy.address`. + +.. seealso:: The `__.` is a :term:`relative path` notation and means the parent path of the current subfamily path. + + In the python quasi algorithmic notation we could say that: + + .. code-block:: python + + __.http_proxy.address == http_proxy.address + + This is true only because in our use case `http_proxy.address` is located + in the same `manual` subfamiliy than `https_proxy.address`. + + Have a look at the :ref:`variablepath` page for more details. + + +We then say that the `manual.https_proxy.address` and the `manual.https_proxy.port` default values are *calculated*. + +.. glossary:: + + calculated + + We say that a variable's value or a default variable's value is calculated + when there is a pointer which refers to another variable's value. + + :ref:`Other types of calculations exists `, in which this type of behavior does not occur + (the "pointer" behavior, notably type copying). + +The other types of calculation we will be explained later in this tutorial. + +We're going to load some :term:`user data ` to see what happens: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_041/config/01/config.yml + :linenos: + :language: yaml + :caption: The :file:`config/01/config.yml` user data + +.. + --- + proxy_mode: Manual proxy configuration + manual: + http_proxy: + address: http.proxy.net + port: 3128 + use_for_https: false + https_proxy: + address: https.proxy.net + +Let's launch the Rougail CLI on it: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_041/config/01/cmd_ro.txt + :class: terminal + +.. + rougail -m firefox/ -u yaml -yf config/01/config.yml + +We have this result: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_041/config/01/output_ro.html + :class: output + +.. +
╭────────────── Caption ───────────────╮
+    │ Variable Default value               │
+    │          Modified value              │
+    │          (⏳ Original default value) │
+    ╰──────────────────────────────────────╯
+    Variables:
+    ┣━━ 📓 proxy_mode (Configure Proxy Access to the Internet): Manual proxy 
+    configuration ◀ loaded from the YAML file "config/01/config.yml" (⏳ No 
+    proxy)
+    ┗━━ 📂 manual (Manual proxy configuration)
+        ┣━━ 📂 http_proxy (HTTP Proxy)
+        ┣━━ 📓 address (HTTP address): http.proxy.net ◀ loaded from the YAML 
+        file "config/01/config.yml"
+        ┗━━ 📓 port (HTTP Port): 3128 ◀ loaded from the YAML file 
+            "config/01/config.yml" (⏳ 8080)
+        ┣━━ 📓 use_for_https (Also use this proxy for HTTPS): false ◀ loaded from 
+        the YAML file "config/01/config.yml" (⏳ true)
+        ┗━━ 📂 https_proxy (HTTPS Proxy)
+            ┣━━ 📓 address (HTTPS address): https.proxy.net ◀ loaded from the YAML 
+            file "config/01/config.yml" (⏳ http.proxy.net)
+            ┗━━ 📓 port (HTTPS Port): 3128
+    
+ +Notice that the default value is not changed when some values are settled from the :term:`user data file `. + +The `https_proxy` variable's value is indeed modified, but the default value is still a calculated value (so the default value is indeed the value of the variable `http_proxy.address`, i.e., `"http.proxy.net"`), while the value defined by the user is `https.proxy.net`. + +With the standard output of the Rougail CLI, the terminal display output, we can see the default value of the variable. + +By interpreting the results of this standard output, we can see that even if a value has been assigned to this variable +(meaning the default value is not used) the variable's default value is not changed, but rather its actual value. + +.. keypoints:: Key points progress + + **summary** + + We have learned how to set the default value of the `https_proxy.address` variable + with the value of the `http_proxy.address` variable. We did the same + `https_proxy.port` and the `https_proxy.port` variables. + + **notation** + + We have learned a notation very usefull for the subfamilies traversal: + + - the `_.` notation means the current path of the family you're currently in + - the `__.` notation means the parent path of the current subfamily path. + + Have a look at the :ref:`variablepath` page for more details. + diff --git a/docs/tutorial/choice.rst b/docs/tutorial/choice.rst new file mode 100644 index 000000000..fd0c6e591 --- /dev/null +++ b/docs/tutorial/choice.rst @@ -0,0 +1,224 @@ +.. _tutorial_choice: + +A variable with possible values +================================== + +.. objectives:: Objectives + + We will learn how to define variables with predefined available values. + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tag :tutorial:`v1.1_010 ` in the repository: + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach v1.1_010 + +A variable with a list of possible values +--------------------------------------------- + +In the firefox browser, the proxy mode can be set by this way: + +.. image:: images/firefox_02.png + +A list of possible values for the `proxy_mode` variable is proposed. +With Rougail there is the possibility of defining a list of available values +with the `choices` variable's parameter: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_010/firefox/00-proxy.yml + :linenos: + :language: yaml + :caption: The `proxy_mode` variable with the `choice` parameter + :name: RougailDictionaryChoiceType + +.. + --- + proxy_mode: + description: Configure Proxy Access to the Internet + choices: + - No proxy + - Auto-detect proxy settings for this network + - Use system proxy settings + - Manual proxy configuration + - Automatic proxy configuration URL + default: No proxy + +:tutorial:`Download this file from the rougail-tutorials git repository ` + +Let's run the Rougail CLI with these available values: + +.. code-block:: text + :class: terminal + + rougail -m firefox/ + +We have an output like this one: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_010/config/01/output_ro.html + :class: output + +.. +
╭────────────────────────── Caption ──────────────────────────╮
+    │ Variable                           Default value            │
+    │ Undocumented variable              Modified value           │
+    │ Undocumented but modified variable (Original default value) │
+    │ Unmodifiable variable                                       │
+    ╰─────────────────────────────────────────────────────────────╯
+    Variables:
+    ┗━━ 📓 proxy_mode: No proxy
+    
+ +`No proxy` is an available variable's value. We say that the `proxy_mode` variable is *constrained* +by the possibilities of the `choice` parameter. + +.. type-along:: Let's add some user data to this structure + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_010/config/03/config.yml + :linenos: + :language: yaml + :caption: A user data specification + +:tutorial:`Download this file from the rougail-tutorials git repository ` + +If we run the Rougail CLI with this user data: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_010/config/03/cmd_ro.txt + :class: terminal + +.. + rougail -m firefox/ -u yaml -yf config/03/config.yml + +We have this output: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_010/config/03/output_ro.html + :class: output + +.. +
╭────────────── Caption ───────────────╮
+    │ Variable Modified value              │
+    │          (⏳ Original default value) │
+    ╰──────────────────────────────────────╯
+    Variables:
+    ┗━━ 📓 Configure Proxy Access to the Internet: Manual proxy configuration ◀ loaded from the YAML file "config/03/config.yml" (⏳ No proxy)
+    
+ +As we set the `proxy_mode` variable from a user data file, +we now have specified a value which is **not** a default value, and +the output of the Rougail CLI explicitly shows that a user data value has been entered, +it shows which user data file this value comes from, and it also indicates +what the default value is for information purposes. + +.. type-along:: The constraints that come with the choices + +The `proxy_mode` variable's possible values is *constrained*. + +We have the list of the possible (authorized) values: + +- `No proxy` +- `Auto-detect proxy settings for this network` +- `Use system proxy settings` +- `Manual proxy configuration` +- `Automatic proxy configuration URL` + +.. questions:: Question + + What happens if I set a value that is not available in the choices? + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_010/config/04/config.yml + :linenos: + :language: yaml + :caption: A (false) user data specification + +:tutorial:`Download this file from the rougail-tutorials git repository ` + +If we run the Rougail CLI with this user data: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_010/config/04/cmd_ro.txt + :class: terminal + +.. + rougail -m firefox/ -u yaml -yf config/03/config.yml + +We have this output: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_010/config/04/output_ro.html + :class: output + +We can see here that Rougail warns us about an invalid value that is not in the available choices, +that's why this value will not be used and it falls back to the original default value. + +But maybe this is not the behavior you need. Maybe you need to stop everything if Rougail detects that +something is going wrong, maybe you need some kind of a strict mode. + +Indeed, this warning can be transformed into an error. + +If we run the Rougail CLI with this `--cli.invalid_user_data_error` parameter: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_010/config/04/cmd_invalid.txt + :class: terminal + +.. + rougail -m firefox/ -u yaml -yf config/03/config.yml --cli.invalid_user_data_error + +Then we have an `error` output instead of a `warning` output: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_010/config/04/output_invalid.html + :class: output + +.. +
🛑 ERRORS
+    ┗━━ the value "foo" is an invalid choice for "proxy_mode" (Configure Proxy Access to the Internet), only "Auto-detect proxy settings for this network", "Automatic proxy configuration URL", "Manual proxy 
+        configuration", "No proxy" and "Use system proxy settings" are allowed, it will be ignored when loading from the YAML file "config/04/config.yml"
+    
+ + + +.. keypoints:: Key points progress + + Indeed, in the Firefox configuration, it is possible to define several configuration modes, + from no proxy at all to different kind of automatic or manual configuration modes. + The choices, the list of available values for a variable, can help us to handle this situation. + + **Progress** + + To sum up, we have arrived at this point in writing a structure file like this: + + **Structure description file** + + .. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_010/firefox/00-proxy.yml + :linenos: + :language: yaml + :caption: A Rougail structure file with a variable named `proxy_mode`, with a default value. + +.. + + .. raw:: text + + --- + proxy_mode: + description: Configure Proxy Access to the Internet + choices: + - No proxy + - Auto-detect proxy settings for this network + - Use system proxy settings + - Manual proxy configuration + - Automatic proxy configuration URL + default: No proxy + diff --git a/docs/tutorial/customtype.rst b/docs/tutorial/customtype.rst new file mode 100644 index 000000000..e734aef44 --- /dev/null +++ b/docs/tutorial/customtype.rst @@ -0,0 +1,558 @@ +.. _tutorial_custom_type: + +Custom type +============ + +.. objectives:: Objectives + + In this sections we are going to learn how to create custom types with Rougail. + In short, a custom type is nothing more than a kind of a template for a variable or a family. + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tags :tutorial:`v1.1_080 ` to :tutorial:`v1.1_085 ` + in the repository. + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach v1.1_080 + +HTTP Proxy with "proxy" type +-------------------------------- + +Let's look now at the possibilities for adding types. +This feature is essential for having a truly adaptable tool. + +.. note:: It is actually possible to add types, + but that involves adding predefined types to the :term:`Tiramisu` underlying consistency library itself, + and that’s a different matter altogether. There is a simpler way to achieve this with the Rougail type definition feature. + +It is possible in Rougail to create custom types in a very simple way, it is much like defining variables or families. + +Our folder structure will be expanded a bit: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_080/tree.html + +Notice that we have now a new :file:`types` folder in our tree structure. +This is where we will place our new `proxy` type definition. + +.. note:: Choosing the :file:`types/proxy/00-type.yml` file name is not mandatory. + You can name this folder whatever you want. + It is simply a good practice to separate structure files from type definition files. + We will see later on how to indicate to Rougail that a file contains a type declaration. + +In accordance with our :ref:`naming policy `, we have created a file named :file:`00-type.yml` +inside the `proxy` folder. +Let's examine this more closely this :file:`types/proxy/00-type.yml` type definition file: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_080/types/proxy/00-type.yml + :language: yaml + :caption: The :file:`types/proxy/00-type.yml` type structure file that defines our new `proxy` type. + +.. + %YAML 1.2 + --- + version: 1.1 + + proxy: + + address: + description: Proxy address + type: domainname + params: + allow_ip: true + + port: + description: Proxy port + type: port + default: 8080 + ... + +The new type named `proxy` is declared the same way we declare a family. +It's more or less like a family declaration except it will be used as a type definition. +If you remember, it's mostly the content as the family `manual.http_proxy` in `firefox/10-manual.yml` file of this tutorial. + +.. questions:: How do we use a new type? + + Very simply. In the usual way: we will add a `type` parameter in our family declaration. + +Let's modify the existing family using this new `proxy` type: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_080/firefox/10-manual.yml + :language: yaml + :caption: The :file:`firefox/10-manual.yml` structure file with less code due to the `proxy` type definition + +.. + %YAML 1.2 + --- + version: 1.1 + + manual: + description: Manual proxy configuration + disabled: + variable: _.proxy_mode + when_not: Manual proxy configuration + + http_proxy: + description: HTTP Proxy + type: proxy + ... + +We can see that the type declared for the `http_proxy` family is now `type: proxy`. +It's perfectly natural in the Rougail style. + +Due to the type definition we can "omit" some settings in the structure file +such as the `domainname` type and the `allow_ip` parameter. +And the `address` variable it is not necessary to be specified at all in the structure file +because it's in the type definition. + +In order to use our new type we need to declare it to Rougail otherwise the new type is not available. +The Rougail CLI has the `--types` type declaration command line option for this purpose. +Let's launch the Rougail CLI with this command line parameter: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_080/config/01/cmd_ro.txt + :class: terminal + +The CLI output is entirely standard and does not specifically mention the call to a new type. +The result is the same as usual, as if the type declaration were omitted, which is what we want. +We want the same behavior as usual. + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_080/config/01/output_ro.html + :class: output + +HTTPS and SOCKS Proxy with "proxy" type +--------------------------------------- + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_081 ` version: + + git switch --detach v1.1_081 + +With this use of a type, it is possible to define our HTTPS and SOCKS Proxy in a much more conscious way: +It's very practical. In fact, the type definition adds nothing else to the structural logic. + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_081/firefox/20-manual.yml + :language: yaml + :caption: The :file:`firefox/20-manual.yml` structure file with less code due to the `proxy` type definition + +.. + %YAML 1.2 + --- + version: 1.1 + + manual: + + use_for_https: true # Also use this proxy for HTTPS + + https_proxy: + description: HTTPS Proxy + type: proxy + hidden: + variable: _.use_for_https + + socks_proxy: + description: SOCKS Proxy + type: proxy + ... + +Now we have two families with the `proxy` type: `https_proxy` and `socks_proxy`. + +But we haven't recovered everything we had before. +Using a type definition is fine, but we still need to find exactly the configuration situation +we had before using the type definition. + +Add a variable in a family with custom type +-------------------------------------------- + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_082 ` version:: + + git switch --detach v1.1_082 + + +In the `socks_proxy` family definition, we need to added a new variable, the `version` variable: + +.. code-block:: yaml + + version: + description: SOCKS host version used by proxy + choices: + - v4 + - v5 + default: v5 + +You can see that what we call a type definition in Rougail is very flexible, because it is perfectly possible +to add to what has been defined in the type. +The type definition here is therefore not only a template but much more an extensible schema. + +Redefine default value in custom type variable +----------------------------------------------- + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_083 ` version:: + + git switch --detach v1.1_083 + +In this section we are going to make an implicit :ref:`redefinition ` of variables default values. + +.. note:: + + We've always said that the redefine is explicit. But there's an exception here. + There are two types of redefine for a custom type, an implicit and an explicit. + An implicit redefinition is a redefinition which has not been declared with the `redefine` attribute. + An explicit redefinition is a redefinition which has been declared with the `redefine: true` attribute. + + Only default values can be implicitly redefined (and only in the type definitions), all other attributes + need to be explicitely redefined. + +In the type definition file, we are now setting default values for the `address` and `port` variables: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_083/types/proxy/00-type.yml + :language: yaml + :caption: The :file:`types/proxy/00-type.yml` type definition file with the default values for `address` and `port` + +.. + %YAML 1.2 + --- + version: 1.1 + + proxy: + + address: + description: Proxy address + type: domainname + params: + allow_ip: true + default: + variable: __.http_proxy.address + + port: + description: Proxy port + type: port + default: + variable: __.http_proxy.port + ... + +But we don't want theses default values for HTTP definition. We can :term:`redefine` these type settings in the structure file: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_083/firefox/10-manual.yml + :language: yaml + :caption: The :file:`firefox/10-manual.yml` structure definition file with the **redefined** default values for `address` and `port` + +.. + %YAML 1.2 + --- + version: 1.1 + + manual: + description: Manual proxy configuration + disabled: + variable: _.proxy_mode + when_not: Manual proxy configuration + + http_proxy: + description: HTTP Proxy + type: proxy + + address: + default: null + + port: + default: 8080 + ... + +This means that the default values for these variables will now be those defined +in the structure file if the values are defined (and not in the type definitions file). + +Let's have a closer look at the behavior here. + +Let's launch the Rougail CLI: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_082/config/01/cmd_ro.txt + :class: terminal + +.. + rougail -m firefox/ -u yaml -yf config/01/config.yml + +We have this output: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_082/config/01/output_ro.html + :class: output + +We can see in the output that + +- the default values of the `port` variable, which is `8080` in the structure file, comes from the strucure file. +- the default values of the `address` variable is not set because it is `null` by default in the structure file, + so there is not such thing as a default value set for this variable. + +.. questions:: What happens if the default value in the type definition is not suitable for us? + +If we hadn't defined a value in the userdata file, the Rougail CLI would have returned the following error: + +:: + + 🛑 Caution + ┗━━ Manual proxy configuration + ┣━━ HTTP Proxy + ┃ ┗━━ Proxy address: 🛑 mandatory variable but has no value + ┗━━ SOCKS Proxy + ┗━━ Proxy address: 🛑 mandatory variable but has no value + +Therefore, the default value that was taken into account is indeed that of the structure file, and not that of the type definition. + +.. note:: We can see that the Rougail behavior is always very simple, there is no such thing as an MRO + (Method Resolution Object) that try fallback to the default value defined in the type definition file. + The Rougail behavior is much more simple and transparent. + +.. type-along:: kinematics of setting the `address` value + +Let's take the time to explain how the value of the variable `manual.http_proxy.address` is determined using the type definition. + +The :file:`10-manual.yml` structure file defines the :file:`manual.http_proxy` family +which is defined with the help of the `proxy` type: + +.. code-block:: yaml + + http_proxy: + description: HTTP Proxy + type: proxy + +In the :file:`types/proxy/00-type.yml` file we have the `proxy` type definition and its default value setting: + +.. code-block:: yaml + + default: + variable: __.http_proxy.address + +This default value setting point to the `http_proxy.address` value in the :file:`firefox/10-manual.yml` structure file: + +.. code-block:: yaml + + address: + default: null + +Which is `null` by default but in the :file:`config/01/config.yml` user data file we have the `http.proxy.net` value set: + +.. code-block:: yaml + + manual: + http_proxy: + address: http.proxy.net + +In the CLI standard output we can see that the `http_proxy.address` variable value is +finally set by the user data file: + +:: + + ┗━━ 📂 manual (Manual proxy configuration) + ┣━━ 📂 http_proxy (HTTP Proxy) + ┃ ┣━━ 📓 address (Proxy address): http.proxy.net ◀ loaded from the YAML + ┃ ┃ file "config/01/config.yml" + + +Redefine other parameter in custom type for HTTP +-------------------------------------------------- + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_084 ` version:: + + git switch --detach v1.1_084 + +In this sections we are going to make explicit :term:`redefinitions `. + +Note here that, in order to allow for the genericity of a type declaration, +we have omitted to specify the variable descriptions in the type declaration. +We will therefore need to enter these descriptions in the structure file. + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_084/types/proxy/00-type.yml + :language: yaml + :caption: The :file:`types/proxy/00-type.yml` type definition file with the default definitions and no description + +.. + %YAML 1.2 + --- + version: 1.1 + + proxy: + + address: + type: domainname + params: + allow_ip: true + default: + variable: __.http_proxy.address + + port: + type: port + default: + variable: __.http_proxy.port + ... + +So, as the `address` and `port` variables don't have any `description` attribute, +we need to add one. It is something like an extension of the type definitions here +in the structure file: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_084/firefox/10-manual.yml + :language: yaml + :caption: The :file:`firefox/10-manual.yml` structure definition file with explicit redefinitions + +.. + %YAML 1.2 + --- + version: 1.1 + + manual: + description: Manual proxy configuration + disabled: + variable: _.proxy_mode + when_not: Manual proxy configuration + + http_proxy: + description: HTTP Proxy + type: proxy + + address: + redefine: true + default: null + description: HTTP proxy address + + port: + redefine: true + default: 8080 + description: HTTP proxy port + ... + +.. note:: Transforming an implicit defintion into an explicit redefinition is very simple; you just need to add the `redefine: true` attribute: + +Let's launch the Rougail CLI: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_083/config/01/cmd_ro.txt + :class: terminal + +.. + rougail -m firefox/ -u yaml -yf config/01/config.yml + +We have this output: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_083/config/01/output_ro.html + :class: output + +.. + Variables: + ┣━━ 📓 Configure Proxy Access to the Internet: Manual proxy configuration ◀ + ┃ loaded from the YAML file "config/01/config.yml" (⏳ No proxy) + ┗━━ 📂 Manual proxy configuration + ┣━━ 📂 HTTP Proxy + ┃ ┣━━ 📓 HTTP proxy address: http.proxy.net ◀ loaded from the YAML file + ┃ ┃ "config/01/config.yml" + ┃ ┗━━ 📓 HTTP proxy port: 3128 ◀ loaded from the YAML file + ┃ "config/01/config.yml" (⏳ 8080) + ┣━━ 📓 Also use this proxy for HTTPS: false ◀ loaded from the YAML file + ┃ "config/01/config.yml" (⏳ true) + ┣━━ 📂 HTTPS Proxy + ┃ ┣━━ 📓 HTTPS proxy address: https.proxy.net ◀ loaded from the YAML file + ┃ ┃ "config/01/config.yml" (⏳ http.proxy.net) + ┃ ┗━━ 📓 HTTPS proxy port: 3128 + ┗━━ 📂 SOCKS Proxy + ┣━━ 📓 SOCKS proxy address: http.proxy.net + ┣━━ 📓 SOCKS proxy port: 3128 + ┗━━ 📓 SOCKS host version used by proxy: v5 + +We can see in the output that the `address` and `port` variables have default values, +and descriptions ("HTTP proxy address", "HTTP proxy port") that weren't present in the +type definition file. + +Redefine other parameter in custom type for HTTPS and SOCKS +----------------------------------------------------------- + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_085 ` version:: + + git switch --detach v1.1_085 + +We're going to do exactly the same modification with the `socks_proxy` family, +that is we will base the `socks_proxy` family on the type definition. + +Following the same reasoning, we add a in the structure file some `description` attributes to the `socks_proxy` family +and also add some explicit redefinition attribute (`redefine: true`). + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_085/firefox/20-manual.yml + :language: yaml + :caption: The :file:`firefox/20-manual.yml` structure file with the `redefine` and the `description` attribute + +.. + %YAML 1.2 + --- + version: 1.1 + + manual: + + use_for_https: true # Also use this proxy for HTTPS + + https_proxy: + description: HTTPS Proxy + type: proxy + hidden: + variable: _.use_for_https + + address: + redefine: true + description: HTTPS proxy address + + port: + redefine: true + description: HTTPS proxy port + + socks_proxy: + description: SOCKS Proxy + type: proxy + + address: + redefine: true + description: SOCKS proxy address + + port: + redefine: true + description: SOCKS proxy port + + version: + description: SOCKS host version used by proxy + choices: + - v4 + - v5 + default: v5 + ... + + +.. keypoints:: Key points + + - the very practical aspects of type declaration + - type usage in the Rougail CLI + - more conscious family definition + - using a type for an extended family definition + - we can redefine things in the structure file and it overloads the type definition file + - implicit or explicit redefinitions + + We have seen how the definition of type, used effectively, allows for a new expressiveness. + diff --git a/docs/tutorial/document.rst b/docs/tutorial/document.rst new file mode 100644 index 000000000..663b6ac0e --- /dev/null +++ b/docs/tutorial/document.rst @@ -0,0 +1,261 @@ +.. _documented_variable: + +A full documented variable +============================== + +.. objectives:: Objectives + + You’ve no doubt wondered, on more than one occasion, how to do this or that; + even if you have the documentation, it’s not clear enough. + + What would make it clear is a good example. + + With Rougail, you can add examples into the structural specifications of a variable or a family, illustrative examples. + That is what we will look at in this section. Documentation through examples. + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tags :tutorial:`v1.1_120 ` to :tutorial:`v1.1_121 ` + in the repository. + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach v1.1_120 + + +Examples +----------- + +How to define example usage? All you need to do is add an `examples` parameter to the structural definition of your variable. +Let's give some usage examples of how to use our `no_proxy` variable: + + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_120/firefox/40-no_proxy.yml + :language: yaml + :caption: The :file:`firefox/40-no_proxy` structure definition file with an `example` parameter + +.. + %YAML 1.2 + --- + version: 1.1 + + no_proxy: + description: Address for which proxy will be desactivated + examples: + - .mozilla.org + - .net.nz + - 192.168.1.0/24 + type: domainname + params: + allow_ip: true + allow_cidr_network: true + allow_without_dot: true + allow_startswith_dot: true + multi: true + mandatory: false + disabled: + variable: _.proxy_mode + when: No proxy + ... + +.. attention:: Note: the `examples` parameter does not specify default values. **It's just examples, it's not a setting**. + +If we launch the Rougail CLI on user data: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_120/config/03/config.yml + :language: yaml + :caption: The :file:`config/03/config.yml` user data file with an `example` parameter but no value set + +.. + --- + proxy_mode: Automatic proxy configuration URL + auto: https://auto.proxy.net/wpad.dat + +we can see that the `no_proxy` variable has no user data settings. + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_120/config/03/cmd_ro.txt + :class: terminal + +.. + rougail -m firefox/ --types types/proxy -u yaml -yf config/01/config.yml + +We do indeed obtain the value empty list `[]` for the `no_proxy` variable: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_120/config/03/output_ro.html + :class: output + +.. + ╭────────────── Caption ───────────────╮ + │ Variable Default value │ + │ Modified value │ + │ (⏳ Original default value) │ + ╰──────────────────────────────────────╯ + Variables: + ┣━━ 📓 Configure Proxy Access to the Internet: Automatic proxy configuration URL ◀ loaded from the YAML file "config/03/config.yml" + ┃ (⏳ No proxy) + ┣━━ 📓 Automatic proxy configuration URL: https://auto.proxy.net/wpad.dat ◀ loaded from the YAML file "config/03/config.yml" + ┗━━ 📓 Address for which proxy will be desactivated: [] + +.. type-along:: If we set some user data values + +Just to be thorough, let's look at what happens in a case where there is a value for `no_proxy` entered in the user data file: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_120/config/02/config.yml + :language: yaml + :caption: The :file:`config/03/config.yml` user data file with some value set + +We launch the Rougail CLI: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_120/config/02/cmd_ro.txt + :class: terminal + +.. + rougail -m firefox/ --types types/proxy -u yaml -yf config/02/config.yml + +And we have the `no_proxy`'s expected value. No trace in the output of the examples settings of the variable: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_120/config/02/output_ro.html + :class: output + +.. + ╭────────────── Caption ───────────────╮ + │ Variable Modified value │ + │ (⏳ Original default value) │ + ╰──────────────────────────────────────╯ + Variables: + ┣━━ 📓 Configure Proxy Access to the Internet: Automatic proxy configuration URL + ┃ ◀ loaded from the YAML file "config/02/config.yml" (⏳ No proxy) + ┣━━ 📓 Automatic proxy configuration URL: https://auto.proxy.net/wpad.dat ◀ + ┃ loaded from the YAML file "config/02/config.yml" + ┗━━ 📓 Address for which proxy will be desactivated: + ┣━━ example.net ◀ loaded from the YAML file "config/02/config.yml" + ┗━━ 192.168.1.0/24 ◀ loaded from the YAML file "config/02/config.yml" + +In all cases, we can see that none of the values ​​cited in the examples (`.mozilla.org`, `.net.nz`) appear. + + +.. type-along:: In that case, is there a use for these examples? + +This examples parameter is useful when generating variable documentation. That's where the examples appear. + +If we launch the Rougail CLI for the documentation generation: + +.. code-block:: bash + + rougail -m firefox --types types/proxy -o doc --doc.output_format html + +We have this output: + +.. raw:: html + :class: output + + + + + + + + + +
Variable Description
auto
web address mandatory disabled
Automatic proxy configuration URL.
Validators:
  • well-known ports (1 to 1023) are allowed
  • +
  • registred ports (1024 to 49151) are allowed
  • +
  • type domainname
  • +
  • the domain name can be a hostname

Disabled: when the variable "proxy_mode" hasn't the value "Automatic proxy configuration URL"
no_proxy
domainname multiple disabled unique
Address for which proxy will be desactivated.
Validators:
  • type domainname
  • +
  • the domain name can starts by a dot
  • +
  • the domain name can be a hostname
  • +
  • the domain name can be an IP
  • +
  • the domain name can be network in CIDR format

Examples:
  • .mozilla.org
  • +
  • .net.nz
  • +
  • 192.168.1.0/24

Disabled: when the variable "proxy_mode" has the value "No proxy"
+ +.. + ├─────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤ + │ no_proxy │ Address for which proxy will be desactivated. │ + │ domainname multiple disabled unique │ Validators: │ + │ │ • type domainname │ + │ │ • the domain name can starts by a dot │ + │ │ • the domain name can be a hostname │ + │ │ • the domain name can be an IP │ + │ │ • the domain name can be network in CIDR format │ + │ │ Examples: │ + │ │ • .mozilla.org │ + │ │ • .net.nz │ + │ │ • 192.168.1.0/24 │ + │ │ Disabled: when the variable "proxy_mode" has the value "No proxy" │ + └─────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────┘ + +we can see that this time the examples (`.mozilla.org`, `.net.nz`) appear. + +Help +------- + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_121 ` version:: + + git switch --detach v1.1_121 + +Could we define personalized assistance, in the same way that we defined examples? + +Yes, very simply: we add a `help` parameter to our `no_proxy` variable: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_121/firefox/40-no_proxy.yml + :language: yaml + :caption: The :file:`firefox/40-no_proxy` structure definition file with an `example` parameter + + +In the same way, this help `"Connections to localhost, 127.0.0.1/8 and ::1 are never proxied"` will appear in the documentation: + +.. raw:: html + :class: output + + + + + + + + + +
Variable Description
auto
web address mandatory disabled
Automatic proxy configuration URL.
Validators:
  • well-known ports (1 to 1023) are allowed
  • +
  • registred ports (1024 to 49151) are allowed
  • +
  • type domainname
  • +
  • the domain name can be a hostname

Disabled: when the variable "proxy_mode" hasn't the value "Automatic proxy configuration URL"
no_proxy
domainname multiple disabled unique
Address for which proxy will be desactivated.
Connections to localhost, 127.0.0.1/8 and ::1 are never proxied.
Validators:
  • type domainname
  • +
  • the domain name can starts by a dot
  • +
  • the domain name can be a hostname
  • +
  • the domain name can be an IP
  • +
  • the domain name can be network in CIDR format

Examples:
  • .mozilla.org
  • +
  • .net.nz
  • +
  • 192.168.1.0/24

Disabled: when the variable "proxy_mode" has the value "No proxy"
+ +.. type-along:: The difference between the `description` parameter and the `help` parameter + +The help section serves as supplementary documentation regarding the description parameter. +Please do not mix the :term:`description` parameter and the :term:`help` parameter's variable usage. + +A :term:`description` allows you to describe the expected value(s) of a variable concisely and clearly. +It is designed to be clear precise and short, much like the help of some +command-line utility when you type the command with the `--help` or `-h` option. + +A :term:`help` helps to clarify any ambiguity about the variable’s purpose. It can be long and detailed. + +We can set :term:`help` attribute to a variable or to a family. + +.. keypoints:: Key points + + - We have seen how to give examples, simply with the `examples` parameter + - We have seen how to specify a help, simply with the `help` parameter + diff --git a/docs/tutorial/dynamic.rst b/docs/tutorial/dynamic.rst new file mode 100644 index 000000000..3809c2a19 --- /dev/null +++ b/docs/tutorial/dynamic.rst @@ -0,0 +1,398 @@ +.. _tutorial_dynamic: + +A dynamically built family +========================== + +.. objectives:: Objectives + + In this section we will learn how to create a dynamically built family. + + In a dynamically built family, instead of duplicating the definition of + identical variables in several families, they can be generated automatically. + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tags :tutorial:`v1.1_060 ` to :tutorial:`v1.1_061 ` + in the repository. + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach v1.1_060 + +We handled the HTTPS mode in the previous section. But there's more modes to handle. +Let's turn back to the firefox's configuration page: + +.. image:: images/soksv5.png + +We see that we need to handle the SOCKS configuration in addition to the HTTPS configuration. +Moreover, we can see that these two groups of variables (also known as family) are similar in the structure: +they both have a host and a port. + +There are two proxies that are to be configured : + +- the HTTPS proxy +- the SOCKS proxy + +As they have the same structure, would it be possible to define the two of them +in one shot? + +.. note:: It's not the place here to describe what the HTTP and SOCKS protocols are. + The interesting point here is that they are very similar in our firefox's + configuration and that we can do batch processing. + +.. index:: dynamically built family + +Family: A dynamically built family +------------------------------------- + +With Rougail, it is possible to create some kind of a model of family. +Kind of a generic family declaration. +We call this generic family creation process a dynamic creation because as we will see below, +these families exist at the very moment we define their **identifiers**. + +First, here is what we need to make (without identifiers): + +.. code-block:: yaml + + https_proxy: + description: HTTPS Proxy + ... + + address: + description: HTTPS address + ... + + port: + description: HTTPS Port + ... + + sock_proxy: + description: SOCKS Proxy + ... + + address: + description: SOCKS address + ... + + port: + description: SOCKS Port + ... + +Now with identifiers, we have the ability to declare our families this way: + +.. code-block:: yaml + + "{{ identifier }}_proxy": + description: "{{ identifier }} Proxy" + dynamic: + - HTTPS + - SOCKS + ... + + address: + description: "{{ identifier }} address" + ... + + port: + description: "{{ identifier }} port" + ... + +.. type-along:: What is exactly an identifier? + +If you used the YAML declaration tool named `Ansible `_, +the variable used to iterate over multiple values in a task is called an **`item`**. +We call it an identifier. + +It is a symbol used in the context of a loop. For example: + +.. code-block:: yaml + :caption: A code example of an ansible loop + + - name: Loop example with 'item' + ansible.builtin.debug: + msg: "The current value is {{ item }}" + loop: + - value1 + - value2 + - value3 + +This code will output: + +.. code-block:: text + + The current value is value1 + The current value is value2 + The current value is value3 + +In the Rougail context, we name this item an identifier because it is an item +that allow us to define dynamically family names. + +.. glossary:: + + identifier + + In the :ref:`dynamically built family creation field ` we call an identifier + an item that defines a family name. An item is a variable on which an iteration + on keywords will be carried out. + + An :term:`identifier` is a local variable, used only for creating multiple + iterations, used for creating multiple families in only one declaration. + + It allows us to declare very similar families in a more generic way. + +Here is the syntax we are using that allows the declaration of multiple families at one time: + +.. code-block:: yaml + + "{{ identifier }}_proxy": + description: "{{ identifier }} Proxy" + dynamic: + - HTTPS + - SOCKS + +This identifier is a parameter that enables us to create two families named `https_proxy` and `socks_proxy`: + +.. code-block:: yaml + + https_proxy: + description: "HTTPS Proxy" + + socks_proxy: + description: "SOCKS Proxy" + +.. attention:: Be careful when choosing your identifiers items: pay attention that the family + names that will be dynamically created have not been declared before in some other + YAML structure file. + + Here the family name is: `"{{ identifier }}_proxy"`. + + If you define a dynamically built family with the `https` identifer that will + build a `https_proxy` family and if this familiy already exists, + then rougail will raise a family/variable override warning. + + When choosing a dynamically built family name, rougail will replace spaces, accents, uppercases... + by valid character and put all in lowercase. Have a look at the :ref:`naming convention `. + + As you can see here, the identifier is `HTTPS`, but the name clearly contains `https` (in lowercase). + +Here is our dynamically built familiy in situation in the :file:`firefox/20-manual.yml` structure file. + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_060/firefox/20-manual.yml + :language: yaml + :caption: The :file:`firefox/20-manual.yml` structure file with the dynamically built families + +.. + %YAML 1.2 + --- + version: 1.1 + + manual: + + use_for_https: true # Also use this proxy for HTTPS + + '{{ identifier }}_proxy': + description: '{{ identifier }} Proxy' + hidden: + variable: _.use_for_https + dynamic: + - HTTPS + - SOCKS + + address: + description: '{{ identifier }} address' + default: + variable: __.http_proxy.address + + port: + description: '{{ identifier }} port' + default: + variable: __.http_proxy.port + ... + +Here is the user data file on which we will launch the Rougail CLI: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_060/config/01/config.yml + :language: yaml + :caption: The :file:`config/01/config.yml` user data file + +.. + --- + proxy_mode: Manual proxy configuration + manual: + http_proxy: + address: http.proxy.net + port: 3128 + use_for_https: false + https_proxy: + address: https.proxy.net + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_060/config/01/cmd_ro.txt + :class: terminal + +.. + rougail -m firefox/ -u yaml -yf config/01/config.yml + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_060/config/01/output_ro.html + :class: output + +.. + rougail -m firefox/proxy.yml -u yaml --yaml.filename config/proxy.yml + ╭─────────────────── Caption ────────────────────╮ + │ Variable Default value │ + │ Unmodifiable variable Modified value │ + │ (Original default value) │ + ╰────────────────────────────────────────────────╯ + Variables: + ┗━━ 📂 Manual proxy configuration + ┣━━ 📂 HTTP Proxy + ┃ ┣━━ 📓 HTTP address: ... (loaded from the YAML file "userdata/proxy.yml") + ┃ ┗━━ 📓 HTTP Port: ... (8080 - loaded from the YAML file "userdata/proxy.yml") + ┣━━ 📓 Also use this proxy for HTTPS: true + ┣━━ 📂 HTTPS Proxy + ┃ ┣━━ 📓 HTTPS address: ... + ┃ ┗━━ 📓 HTTPS port: ... + ┗━━ 📂 SOCKS Proxy + ┣━━ 📓 SOCKS address: ... + ┗━━ 📓 SOCKS port: ... + +We can see that the dynamically built families that have been created: + +- an `HTTPS Proxy` family +- a `SOCKS Proxy` family + +and, as we expected, containing an address and a port. + +A conditional disabled variable with dynamic identifier +-------------------------------------------------------- + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_061 ` version:: + + git switch --detach v1.1_061 + +Here is the final version of the HTTPS and SOCKS structure file, we have added +a new variable named `version`, with the `choice` type, with a default value, +and which has the `disabled` property if the :term:`configuration` is in the `SOCKS` situation. +We will look at this in more detail below. + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_061/firefox/20-manual.yml + :language: yaml + :caption: The final :file:`firefox/20-proxy.yml` structure file + +.. + %YAML 1.2 + --- + version: 1.1 + + manual: + + use_for_https: true # Also use this proxy for HTTPS + + '{{ identifier }}_proxy': + description: '{{ identifier }} Proxy' + hidden: + variable: _.use_for_https + dynamic: + - HTTPS + - SOCKS + + address: + description: '{{ identifier }} address' + default: + variable: __.http_proxy.address + + port: + description: '{{ identifier }} port' + default: + variable: __.http_proxy.port + + version: + description: SOCKS host version used by proxy + choices: + - v4 + - v5 + default: v5 + disabled: + type: identifier + when: HTTPS + ... + +The disabled `property` is assigned here to the `version` variable +in the case where the identifier is `HTTPS`. + +This means that when the current dynamically built family is determined by the identifier `HTTPS`, the variable `version` is disabled +(therefore considered as non-existent) and when the identifier is `SOCKS` the variable is present (it is accessible). + +.. + version: + description: SOCKS host version used by proxy + choices: + - v4 + - v5 + default: v5 + disabled: + type: identifier + when: HTTPS + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_061/config/01/config.yml + :language: yaml + :caption: The :file:`config/01/config.yml` user data file + +.. + --- + proxy_mode: Manual proxy configuration + manual: + http_proxy: + address: http.proxy.net + port: 3128 + use_for_https: false + https_proxy: + address: https.proxy.net + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_061/config/01/cmd_ro.txt + :class: terminal + +.. + rougail -m firefox/ -u yaml -yf config/01/config.yml + +The Rougail CLI outputs this: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_061/config/01/output_ro.html + :class: output + +.. + Variables: + ┣━━ 📓 Configure Proxy Access to the Internet: Manual proxy configuration ◀ loaded from the YAML file "config/01/config.yml" (⏳ No proxy) + ┗━━ 📂 Manual proxy configuration + ┣━━ 📂 HTTP Proxy + ┃ ┣━━ 📓 HTTP address: http.proxy.net ◀ loaded from the YAML file "config/01/config.yml" + ┃ ┗━━ 📓 HTTP Port: 3128 ◀ loaded from the YAML file "config/01/config.yml" (⏳ 8080) + ┣━━ 📓 Also use this proxy for HTTPS: false ◀ loaded from the YAML file "config/01/config.yml" (⏳ true) + ┣━━ 📂 HTTPS Proxy + ┃ ┣━━ 📓 HTTPS address: https.proxy.net ◀ loaded from the YAML file "config/01/config.yml" (⏳ http.proxy.net) + ┃ ┗━━ 📓 HTTPS port: 3128 + ┗━━ 📂 SOCKS Proxy + ┣━━ 📓 SOCKS address: http.proxy.net + ┣━━ 📓 SOCKS port: 3128 + ┗━━ 📓 SOCKS host version used by proxy: v5 + +.. keypoints:: Key points + + - We now know how to declare a dynamically built family, with setting its identifier. + - we have a new use case for the `disabled` property. + We know how to disable a dynamically built variable based on a familiy identifier. + diff --git a/docs/tutorial/family.rst b/docs/tutorial/family.rst new file mode 100644 index 000000000..b66bd59da --- /dev/null +++ b/docs/tutorial/family.rst @@ -0,0 +1,276 @@ +Group variables inside families +================================= + +.. objectives:: Objectives + + We will learn how to: + + - create a :term:`family` + - gather :term:`variable`\ s into a :term:`family` + - make a variable within a variable, which turns this variable container into being a family + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tags :tutorial:`v1.1_020 ` to :tutorial:`v1.1_022 ` + in the repository. + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach v1.1_020 + +.. type-along:: Let's recap how far we've come + +We have this choice variable in its structure definition file: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_010/firefox/00-proxy.yml + :linenos: + :language: yaml + :caption: The `proxy_mode` choice variable in the :file:`firefox/00-proxy.yml` structure file + +.. + --- + proxy_mode: + description: Configure Proxy Access to the Internet + choices: + - No proxy + - Auto-detect proxy settings for this network + - Use system proxy settings + - Manual proxy configuration + - Automatic proxy configuration URL + default: No proxy + +.. We're gonna put it in a :term:`family`. + +In short, let's describe our `proxy_mode` variable like this: + +.. confval:: proxy_mode + :type: `choice` + :default: No proxy + + Proxy mode's settings + +Now we will define new variables, and other structure definitions. +For the sake of clarity, we will put the structure definitions in separate files. +Please have a look at the :ref:`file naming and organizing convention `. + +Here we made a :file:`firefox/00-proxy.yml` structure file and we're gonna make +a new structure file named :file:`firefox/10-manual.yml`:: + + . + └── firefox + ├── 00-proxy.yml + └── 10-manual.yml + +Creating a new family +----------------------- + +Let's create a family named `manual` which obviously corresponds to the proxy's manual configuration choice. + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_020/firefox/10-manual.yml + :language: yaml + :caption: A family structure file description named `manual` in a :file:`firefox/10-manual.yml` file + :name: RougailManualFamily + +.. + --- + manual: + description: Manual proxy configuration + type: family + +We can see that we have defined a :term:`family` here, and this family is *empty* +which means that this family is a container variable that contains no variable yet. + +.. warning:: + + If a family is empty, we need to specify the :term:`family` type here because if we don't, + the Rougail's type engine will infer it by default as a :term:`variable`. + We have to force the family type inference. + + It's because we don't have set any :term:`variable` inside yet. When we will have a variable inside of this family, + we will make a YAML block (to create a block in YAML, you just need to indent the lines) and the Rougail's type inference engine will implicitely infer the variable's container as a family type. + +.. important:: + + Any family that has been declared in a structure file but is empty (i.e., contains neither :term:`variables ` nor :term:`subfamilies `) is not taken into account by the low level :term:`Tiramisu ` consistency engine. + +Or a sub family +---------------- + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_021 ` version:: + + git switch --detach v1.1_021 + +.. glossary:: + + sub family + + A sub family is a family inside a family. + +Creating a family hierarchy of families (family inside a family) is very easy: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_021/firefox/10-manual.yml + :language: yaml + :caption: A rougail structure description file with a hierarchy. + :name: RougailFirstFamilyHierarchy + +.. + --- + manual: + description: Manual proxy configuration + type: family + + http_proxy: + description: HTTP Proxy + type: family + +:tutorial:`Download this file from the rougail-tutorials git repository ` + +Here in our use case we used the :term:`short-hand declaration mode ` +to declare our `manual` family: + +.. code-block:: yaml + + manual: # Manual proxy configuration + +And the `http_proxy` family lives inside of this `manual` family. +We therefore created a hierarchy of families. + +Putting a variable inside of a family or a sub family +----------------------------------------------------------- + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_022 ` version:: + + git switch --detach v1.1_022 + +We are going to put a variable inside of a family or a sub family + +Let's create a variable in the `http_proxy` family. + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_022/firefox/10-manual.yml + :language: yaml + :caption: An `address` variable in the `http_proxy` family + :name: RougailVariableInSubFamily + +.. + --- + manual: + description: Manual proxy configuration + type: family + + http_proxy: + description: HTTP Proxy + type: family + + address: + description: HTTP address + +:tutorial:`Download this file from the rougail-tutorials git repository ` + +Now that the :confval:`address` variable is declared, the :term:`operator` can set :term:`a value ` to it. + +In short, let's describe our `address` variable like this: + +.. confval:: address + :default: None + + This is the HTTP address of the proxy + +We have reached the definition of the address in the `http_proxy` family; there will be other variables to define in this family. + +.. image:: images/firefox_manual_family.png + +.. type-along:: Assigning a user value + +Now we need to set a value ​​for the :confval:`address` variable, +otherwise we will get an error if we try to access this variable: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_022/config/01/output_ro.html + :class: error-box + +.. +
🛑 ERRORS
+    ┣━━ The following variables are mandatory but have no value:
+    ┗━━   - manual.http_proxy.address (HTTP address)
+    
+ +.. type-along:: Let's set user values in a user data file + +Here is a user data file sample: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_022/config/02/config.yml + :language: yaml + :caption: A user file named :file:`config/03/config.yml` with a value set for the `address` variable + :name: RougailAddresseVariableUserValue + +.. + --- + proxy_mode: Manual proxy configuration + manual: + http_proxy: + address: example.net + +:tutorial:`Download this file from the rougail-tutorials git repository ` + +Let's validate the consitency of the :term:`configuration`: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_022/config/02/cmd_ro.txt + :class: terminal + +.. + rougail -m firefox/ -u yaml -yf config/02/config.yml + +Everything is OK: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_022/config/02/output_ro.html + :class: output + +.. +
╭──────── Caption ────────╮
+    │ Variable Default value  │
+    │          Modified value │
+    ╰─────────────────────────╯
+    Variables:
+    ┣━━ 📓 Configure Proxy Access to the Internet: No proxy
+    ┗━━ 📂 Manual proxy configuration
+        ┗━━ 📂 HTTP Proxy
+            ┗━━ 📓 HTTP address: example.net ◀ loaded from the YAML file "config/02/config.yml"
+    
+ +Let's recap about the user data. We can see in this Rougail CLI output that: + +- the `proxy_mode` value is set by default by the :term:`integrator` +- the `address` value is has been set by an :term:`operator` + +.. keypoints:: Let's review the key points + + **Keywords** + + - we know how to define :term:`variable`\ s inside of a family + - we now know what a :term:`mandatory` variable is and why it is necessary to assign values to the variables + - we kwow how to set a variable's :term:`user value ` + - we have the big picture : the :term:`configuration`, which is (the structure files + the user data files) + + **Progress** + + - we have a :term:`family` named `manual` and a sub family named `http_proxy` + - And we have now two variables: :confval:`proxy_mode` and :confval:`address`. + +.. seealso:: + + The full documentation about a :term:`family`. diff --git a/docs/tutorial/images/QuestionaryChoice.png b/docs/tutorial/images/QuestionaryChoice.png new file mode 100644 index 000000000..4ff335e29 Binary files /dev/null and b/docs/tutorial/images/QuestionaryChoice.png differ diff --git a/docs/tutorial/images/UserDataOutput.png b/docs/tutorial/images/UserDataOutput.png new file mode 100644 index 000000000..f0dbb725c Binary files /dev/null and b/docs/tutorial/images/UserDataOutput.png differ diff --git a/docs/images/firefox.png b/docs/tutorial/images/firefox.png similarity index 100% rename from docs/images/firefox.png rename to docs/tutorial/images/firefox.png diff --git a/docs/tutorial/images/firefox130.png b/docs/tutorial/images/firefox130.png new file mode 100644 index 000000000..34e6c0841 Binary files /dev/null and b/docs/tutorial/images/firefox130.png differ diff --git a/docs/tutorial/images/firefox160.png b/docs/tutorial/images/firefox160.png new file mode 100644 index 000000000..d368003e7 Binary files /dev/null and b/docs/tutorial/images/firefox160.png differ diff --git a/docs/tutorial/images/firefox_01.png b/docs/tutorial/images/firefox_01.png new file mode 100644 index 000000000..45725ac64 Binary files /dev/null and b/docs/tutorial/images/firefox_01.png differ diff --git a/docs/tutorial/images/firefox_02.png b/docs/tutorial/images/firefox_02.png new file mode 100644 index 000000000..b1f051a2b Binary files /dev/null and b/docs/tutorial/images/firefox_02.png differ diff --git a/docs/tutorial/images/firefox_manual_family.png b/docs/tutorial/images/firefox_manual_family.png new file mode 100644 index 000000000..049d267c2 Binary files /dev/null and b/docs/tutorial/images/firefox_manual_family.png differ diff --git a/docs/tutorial/images/firefox_manual_https.png b/docs/tutorial/images/firefox_manual_https.png new file mode 100644 index 000000000..b7ca7aa08 Binary files /dev/null and b/docs/tutorial/images/firefox_manual_https.png differ diff --git a/docs/tutorial/images/firefox_port.png b/docs/tutorial/images/firefox_port.png new file mode 100644 index 000000000..70583c3a4 Binary files /dev/null and b/docs/tutorial/images/firefox_port.png differ diff --git a/docs/tutorial/images/firefox_soks_version.png b/docs/tutorial/images/firefox_soks_version.png new file mode 100644 index 000000000..b7ca7aa08 Binary files /dev/null and b/docs/tutorial/images/firefox_soks_version.png differ diff --git a/docs/tutorial/images/foxyproxy.png b/docs/tutorial/images/foxyproxy.png new file mode 100644 index 000000000..418dad01d Binary files /dev/null and b/docs/tutorial/images/foxyproxy.png differ diff --git a/docs/tutorial/images/integrator.png b/docs/tutorial/images/integrator.png new file mode 100644 index 000000000..a1b869901 Binary files /dev/null and b/docs/tutorial/images/integrator.png differ diff --git a/docs/tutorial/images/operator.png b/docs/tutorial/images/operator.png new file mode 100644 index 000000000..a45b096f2 Binary files /dev/null and b/docs/tutorial/images/operator.png differ diff --git a/docs/tutorial/images/practice_threevars.png b/docs/tutorial/images/practice_threevars.png new file mode 100644 index 000000000..7d2fb9a5a Binary files /dev/null and b/docs/tutorial/images/practice_threevars.png differ diff --git a/docs/tutorial/images/practice_twovars.png b/docs/tutorial/images/practice_twovars.png new file mode 100644 index 000000000..858f19274 Binary files /dev/null and b/docs/tutorial/images/practice_twovars.png differ diff --git a/docs/tutorial/images/soksv5.png b/docs/tutorial/images/soksv5.png new file mode 100644 index 000000000..b7ca7aa08 Binary files /dev/null and b/docs/tutorial/images/soksv5.png differ diff --git a/docs/tutorial/index.rst b/docs/tutorial/index.rst new file mode 100644 index 000000000..b8cca7f0c --- /dev/null +++ b/docs/tutorial/index.rst @@ -0,0 +1,74 @@ +.. _tutorial: + +Tutorial with a real world sample +===================================== + +Here is a fairly complete tutorial, it is a use case that comes from the real world. +At the end of the tutorial you will have a good understanding of Rougail. + +.. objectives:: Objectives + + Configuring (the setting of) your favorite web browser. + + This tutorial will show you an example of Rougail use based on the + *how to set a proxy* in the `Mozilla Firefox `_ browser + use case. + +More precisely, this tutorial aims at reproducing :term:`variable`\ s behind this Mozilla Firefox settings page: + +.. image:: images/firefox.png + +We'll call the variables **configuration options** since that's what the variables represent in this use case. + +.. attention:: We are not coding a Firefox plugin here. + We are just going to handle some of the Firefox configuration settings + with Rougail. We are just validating them. + + The configuration option values entered by the user have to be: + + - validated + - consitent + - conform + +Let's dive into this **configuration options validation** use case. + +.. prerequisites:: Important advice + + It is advisable to follow this tutorial with the help of the corresponding :tutorial:`Rougail git repository tutorial `. + You can instead copy/paste or download the different file contents that are explained in this tutorial step and save the files to your computer. + However, if you use the git Rougail tutorial repository, you will have all the necessary files distributed in the correct tree structure, + which is in our opinion much more practical. + + :: + + git clone -b 1.1 https://forge.cloud.silique.fr/stove/rougail-tutorials.git + +.. toctree:: + :titlesonly: + :caption: The Firefox tutorial + + preliminary + choice + family + types + calculated + properties + dynamic + jinja + customtype + nullable + webaddress + multiple + document + practice1 + propertyerror + modes + practice2 + validators + namespace + sequence + regexp + auto_save + underscore_parameter + variable_propertyerror + redefine diff --git a/docs/tutorial/jinja.rst b/docs/tutorial/jinja.rst new file mode 100644 index 000000000..4f859bec1 --- /dev/null +++ b/docs/tutorial/jinja.rst @@ -0,0 +1,674 @@ +.. _tutorial_jinja: + +Playing with Jinja +==================== + +.. objectives:: Objectives + + In this section we will learn how to create new ways of calculation. + + Up to now, our only way of dynamically (that is, during the runtime) calculating + a value is to point on another variable's value. But this is not the only way. + + We will learn how to insert `Jinja templating language `_ into our structure files. + Please note that we do not intend to template our YAML files. + That is a completely different activity. + We will simply insert YAML code to calculate the value of a variable, a property, or a parameter. + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tags :tutorial:`v1.1_070 ` to :tutorial:`v1.1_073 ` + in the repository. + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach v1.1_070 + + +.. type-along:: preliminary thoughts about calculation + +Let's reason on the previous HTTPS proxy configuration's manual mode settings: + +.. code-block:: yaml + + use_for_https: + default: true + + https_proxy: + type: family + description: HTTPS Proxy + hidden: + variable: _.use_for_https + +We see here that the `https_proxy` family is hidden +depending on the value of another variable. This is what we saw previously, +how to hide a variable. + +We will code the same behavior in a somehow different way: + +.. code-block:: yaml + + https_proxy': + description: HTTPS Proxy + hidden: + jinja: |- + {% if _.use_for_https %} + HTTPS is same has HTTP + {% endif %} + +This code has the same result and yes, it's done in a more complicated way. +We have replaced this simple `hidden` property variable parameter by +a Jinja parameter. The hidden process is done by a calculation. + +The fact is that it has same result, but it opens more possibilities, +this is what we are going to work on in this section. + +.. glossary:: + + calculation + + A calculation is the act of calculating the value of a variable or a property of a variable. + This calculation can be a redirection of the value of an existing variable (or property), + or it can be a custom calculation using the Jinja templating language. + +.. type-along:: Why the Jinja templating engine ? + +We are going to embed some `Jinja templating code `_ in our structure file. + +.. glossary:: + + Jinja + + `Jinja `_ is a template engine. + we are using Jinja in a classical way, that is, Jinja allows us to handle different cases, + for example with the `if` statement. + + +.. questions:: What about a `Jinja` calculation? + +The :term:`Jinja` templating language enables some complex calculation. +For example we can code more complex behavior for hiding or disabling a variable, that is +not only depending on the value of another variable as we saw before. + +A conditional hidden family with Jinja +--------------------------------------- + +Here we are going to use the Jinja conditional (testing) statement. + +Here is our structure file: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_070/firefox/20-manual.yml + :language: yaml + :caption: The :file:`firefox/20-manual.yml` structure file with some Jinja code in the `hidden` property + +.. + %YAML 1.2 + --- + version: 1.1 + + manual: + + use_for_https: true # Also use this proxy for HTTPS + + '{{ identifier }}_proxy': + description: '{{ identifier }} Proxy' + hidden: + jinja: |- + {% if _.use_for_https %} + HTTPS is same has HTTP + {% endif %} + dynamic: + - HTTPS + - SOCKS + + address: + description: '{{ identifier }} address' + default: + variable: __.http_proxy.address + + port: + description: '{{ identifier }} port' + default: + variable: __.http_proxy.port + + version: + description: SOCKS host version used by proxy + choices: + - v4 + - v5 + default: v5 + disabled: + type: identifier + when: HTTPS + ... + +In this Jinja code, we are testing the `use_for_https` variable: + +.. code-block:: jinja + + {% if _.use_for_https %} + HTTPS is same has HTTP + {% endif %} + +If this variable is set to `true`, the `"HTTPS is same has HTTP"` expression +will appear. + +.. note:: Have a look at the `Jinja website's test section `_ + for a closer look about testing a variable. + +.. type-along:: Let's set our `use_for_https` variable to `false` + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_070/config/01/config.yml + :language: yaml + :caption: The :file:`config/01/config.yml` user data file with the `use_for_https` set to `false` + +.. + --- + proxy_mode: Manual proxy configuration + manual: + http_proxy: + address: http.proxy.net + port: 3128 + use_for_https: false + https_proxy: + address: https.proxy.net + +Nothing special appears when we launch the Rougail CLI: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_070/config/01/cmd_ro.txt + :class: terminal + +.. + rougail -m firefox/ -u yaml -yf config/01/config.yml + +We have this output: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_070/config/01/output_ro.html + :class: output + +.. + Variables: + ┣━━ 📓 Configure Proxy Access to the Internet: Manual proxy configuration ◀ loaded from the YAML file "config/01/config.yml" (⏳ No proxy) + ┗━━ 📂 Manual proxy configuration + ┣━━ 📂 HTTP Proxy + ┃ ┣━━ 📓 HTTP address: http.proxy.net ◀ loaded from the YAML file "config/01/config.yml" + ┃ ┗━━ 📓 HTTP Port: 3128 ◀ loaded from the YAML file "config/01/config.yml" (⏳ 8080) + ┣━━ 📓 Also use this proxy for HTTPS: false ◀ loaded from the YAML file "config/01/config.yml" (⏳ true) + ┣━━ 📂 HTTPS Proxy + ┃ ┣━━ 📓 HTTPS address: https.proxy.net ◀ loaded from the YAML file "config/01/config.yml" (⏳ http.proxy.net) + ┃ ┗━━ 📓 HTTPS port: 3128 + ┗━━ 📂 SOCKS Proxy + ┣━━ 📓 SOCKS address: http.proxy.net + ┣━━ 📓 SOCKS port: 3128 + ┗━━ 📓 SOCKS host version used by proxy: v5 + +We can see that the value of the `https_proxy.address` is `https.proxy.net`, +that is the value loaded in the :file:`config/01/config.yml` user data file. + + +.. type-along:: Let's set our `use_for_https` variable to `true` + +If we set the `use_for_https` variable to `true` in the :file:`config.yml` user data file + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_070/config/02/cmd_ro.txt + :class: terminal + +and we then launch again the Rougail CLI: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_070/config/02/output_ro.html + :class: output + +.. + Variables: + ┣━━ 📓 Configure Proxy Access to the Internet: Manual proxy configuration ◀ loaded from the YAML file "config/02/config.yml" (⏳ No proxy) + ┗━━ 📂 Manual proxy configuration + ┣━━ 📂 HTTP Proxy + ┃ ┣━━ 📓 HTTP address: http.proxy.net ◀ loaded from the YAML file "config/02/config.yml" + ┃ ┗━━ 📓 HTTP Port: 3128 ◀ loaded from the YAML file "config/02/config.yml" (⏳ 8080) + ┣━━ 📓 Also use this proxy for HTTPS: true ◀ loaded from the YAML file "config/02/config.yml" (⏳ true) + ┣━━ 📂 HTTPS Proxy + ┃ ┣━━ 📓 HTTPS address: http.proxy.net + ┃ ┗━━ 📓 HTTPS port: 3128 + ┗━━ 📂 SOCKS Proxy + ┣━━ 📓 SOCKS address: http.proxy.net + ┣━━ 📓 SOCKS port: 3128 + ┗━━ 📓 SOCKS host version used by proxy: v5 + +This time we can see that this time the value of the `https_proxy.address` is `http.proxy.net`, +that is the same value as the `https_proxy.address` value, because the `https_proxy.address` variable is hidden. + +And in addition notice that we also have a warning: + +:: + + 🔔 Warning + ┗━━ Manual proxy configuration + ┗━━ HTTPS Proxy + ┗━━ HTTPS address: 🔔 family "HTTPS Proxy" has property hidden, + so cannot access to "HTTPS address", + it will be ignored when loading from the YAML file "config/01/config.yml" + +This warning is an alert. It tells us that the `https_proxy.address` variable is unreachable +(because it is hidden) and that this variable's setting **is not used** in the current :term:`configuration`. + +The `--cli.root manual.https_proxy` Rougail CLI parameter enables us to focus on the problem, +The root here is the `manual.https_proxy` family. + +.. note:: Running the Rougail CLI command with a root parameter means that + only data concerning the root family will appear in the command output. + +let's launch the Rougail CLI with this parameter: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_070/config/02/cmd_root.txt + :class: terminal + +.. + rougail -m firefox/ -u yaml -yf config/02/config.yml --cli.root manual.https_proxy + +It gives us this partial, but much more readable output: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_070/config/02/output_root.html + :class: output + +.. type-along:: Launching the Rougail CLI in the :term:`read write mode` + +Let's launch the Rougail CLI in :term:`read write mode` and with the `--cli.root manual.https_proxy` root parameter. + +.. code-block:: bash + :class: terminal + + rougail -m firefox/ --cli.root manual.https_proxy -u yaml \ + -yf config/02/config.yml --cli.read_write + +This time we don't have only a warning as in the `RO` mode. In the `RW` mode, we have an error: + +.. code-block:: bash + + ERROR: cannot access to optiondescription "HTTPS Proxy" + because has property "hidden" + (HTTPS is same has HTTP) + +.. note:: Note that we can see the "HTTPS is same has HTTP" string (the one that was present in the Jinja code) + as an explanation inside the error message. + +Jinja with a description +----------------------------- + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_071 ` version:: + + git switch --detach v1.1_071 + +It is preferable to include a description related to the Jinja calculation. +That allows the calculation performed by the Jinja code to be understood rapidely +without having to read the code itself. +This is useful for the :term:`integrators ` who review these :term:`structure files `. + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_071/firefox/20-manual.yml + :language: yaml + :caption: The :file:`firefox/20-manual.yml` structure file with a description added to the Jinja code + +.. + %YAML 1.2 + --- + version: 1.1 + + manual: + + use_for_https: true # Also use this proxy for HTTPS + + '{{ identifier }}_proxy': + description: '{{ identifier }} Proxy' + hidden: + jinja: |- + {% if _.use_for_https %} + HTTPS is same has HTTP + {% endif %} + description: in HTTPS case if "_.use_for_https" is set to "true" + dynamic: + - HTTPS + - SOCKS + + address: + description: '{{ identifier }} address' + default: + variable: __.http_proxy.address + + port: + description: '{{ identifier }} port' + default: + variable: __.http_proxy.port + + version: + description: SOCKS host version used by proxy + choices: + - v4 + - v5 + default: v5 + disabled: + type: identifier + when: HTTPS + ... + +This description will appear `in the documentation output for the family `_ + +With this rougail command: + +.. code-block:: bash + + rougail -m firefox -o doc --doc.output_format html > readme.html + +It outputs an HTML report, here is a part of it about the `https_proxy` with the `hidden` property: + +.. raw:: html + :class: output + + manual.https_proxy
+ + Hidden: in HTTPS case if "manual.use_for_https" is set to "true"
+ + Identifiers: +
  • HTTPS
  • + +We can see that the hidden explanation for the `manual.https_proxy` family +is the `hidden` property's description. + +Jinja with a parameter +--------------------------- + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_072 ` version:: + + git switch --detach v1.1_072 + +Regarding dynamic families as in the use case we are dealing with, here is an interesting Rougail feature: +it is possible to retrieve the family identifiers and declare them as parameters in the Jinja calculation. + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_072/firefox/20-manual.yml + :language: yaml + :caption: The :file:`firefox/20-manual.yml` structure file with an identifier type parameter in the hidden property + +.. + %YAML 1.2 + --- + version: 1.1 + + manual: + + use_for_https: true # Also use this proxy for HTTPS + + '{{ identifier }}_proxy': + description: '{{ identifier }} Proxy' + hidden: + jinja: |- + {% if my_identifier == 'HTTPS' and _.use_for_https %} + HTTPS is same has HTTP + {% endif %} + description: in HTTPS case if "_.use_for_https" is set to "true" + params: + my_identifier: + type: identifier + dynamic: + - HTTPS + - SOCKS + + address: + description: '{{ identifier }} address' + default: + variable: __.http_proxy.address + + port: + description: '{{ identifier }} port' + default: + variable: __.http_proxy.port + + version: + description: SOCKS host version used by proxy + choices: + - v4 + - v5 + default: v5 + disabled: + type: identifier + when: HTTPS + ... + +We have added an `identifier` type parameter to the `hidden` property: + +.. code-block:: yaml + + hidden: + params: + my_identifier: + type: identifier + +Thus we can refine our Jinja code and hide the `https_proxy` dynamic family, +that is only in the case where the identifier is `HTTPS`. + +.. code-block:: yaml + + hidden: + jinja: |- + {% if my_identifier == 'HTTPS' and _.use_for_https %} + HTTPS is same has HTTP + {% endif %} + +In this Jinja code, we can understand that only the `https_proxy` dynamic family +will be hidden when the `_.use_for_https` variable is set to `true`. + +Let's set the `_.use_for_https` variable to `true`. +We have a warning: + +.. code-block:: text + + 🔔 Warning + ┗━━ Manual proxy configuration + ┗━━ HTTPS Proxy + ┗━━ HTTPS address: 🔔 family "HTTPS Proxy" has property hidden, + so cannot access to "HTTPS address", it will be ignored + when loading from the YAML file "config/01/config.yml" + +It's helpful to run this command in read/write mode: + +.. code-block:: bash + :class: terminal + + rougail -m firefox/ --cli.root manual.https_proxy -u yaml \ + -yf config/01/config.yml --cli.read_write + +because at that moment we get an error: + +.. code-block:: bash + :class: output + + ERROR: cannot access to optiondescription "HTTPS Proxy" + because has property "hidden" (HTTPS is same has HTTP) + +So the Jinja code worked well since the `https_proxy` family is hidden only when +the `use_for_https` variable is set to `true` and the identifier is HTTPS +(the `socks_proxy` family, however, is not hidden). + +Jinja could returns a boolean +--------------------------------- + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_073 ` version:: + + git switch --detach v1.1_073 + +So far, we have used a Jinja expression that could have a `true` value, +otherwise it returned nothing (i.e., None, that is a `false` value): + +.. code-block:: jinja + + {% if my_identifier == 'HTTPS' and _.use_for_https %} + HTTPS is same has HTTP + {% endif %} + +We have implicitly used the truthiness capabilities of Jinja, because the +`HTTPS is same has HTTP` expression is interpreted by Jinja as the `True` value and +the `""` (empty string) expression is interpreted by Jinja as the `False` value. +This behavior is known as implicit boolean coercion (or truthiness). + +.. type-along:: The truthiness of python or Jinja + +In Python or in Jinja, **every value has an inherent Boolean interpretation**—it +can be treated as either `True` or `False` in contexts that expect a Boolean +(like `if`, `while`, or logical operators). +This is often referred to as “truthiness.” +However, **not every value is an instance of the `bool` type**; +the `bool` type itself has only two instances: `True` and `False`. + +.. rubric:: Truthiness Rules + +- A value is considered **falsy** if it evaluates to `False` in a Boolean context. +- Otherwise, it is **truthy**. + +.. rubric:: Common Falsy Values + +- `None` +- `False` +- Zero of any numeric type: `0`, `0.0`, `0j`, `Decimal(0)`, `Fraction(0, 1)` +- Empty sequences/collections: `''`, `[]`, `()`, `{}`, `set()`, `range(0)` +- Objects that define `__bool__()` returning `False`, or `__len__()` returning `0` + +Everything else is truthy, including: + +- Non‑zero numbers +- Non‑empty strings, lists, tuples, dictionaries, etc. +- Most user‑defined class instances (unless they override `__bool__` or `__len__`) + +.. rubric:: Using truthiness in Code + +You can directly use any value in a conditional: + +.. code-block:: python + + if some_value: + print("Truthy") + else: + print("Falsy") + +.. type-along:: Jinja calcluation without truthiness + +This is truthiness capability is entirely possible with Rougail +but, in programming language theory, +it is a form of **implicit** type conversion +where a value of any type is automatically +interpreted as a boolean in contexts that expect one (e.g., conditionals). + +We like things to be declared explicitly. +It is a better practice to simply avoid this implicit coercion +and instead: + +- return a boolean value as the result of the Jinja expression +- explicitly declare that this expression shall return a boolean type + +Here is a Jinja code that return a boolean value: + + +.. code-block:: jinja + + {{ my_identifier == 'HTTPS' and _.use_for_https }} + +And in order to declare explicitly the use of this explicit non-truthiness pratice, +we add a `return_type` parameter in the `hidden` property: + + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_073/firefox/20-manual.yml + :language: yaml + :caption: The :file:`firefox/20-manual.yml` structure file with an explicit boolean declaration + + +.. + %YAML 1.2 + --- + version: 1.1 + + manual: + + use_for_https: true # Also use this proxy for HTTPS + + '{{ identifier }}_proxy': + description: '{{ identifier }} Proxy' + hidden: + jinja: |- + {{ my_identifier == 'HTTPS' and _.use_for_https }} + return_type: boolean + description: in HTTPS case if "_.use_for_https" is set to "true" + params: + my_identifier: + type: identifier + dynamic: + - HTTPS + - SOCKS + + address: + description: '{{ identifier }} address' + default: + variable: __.http_proxy.address + + port: + description: '{{ identifier }} port' + default: + variable: __.http_proxy.port + + version: + description: SOCKS host version used by proxy + choices: + - v4 + - v5 + default: v5 + disabled: + type: identifier + when: HTTPS + ... + +.. questions:: What about the error message? + + If no text is returned by the Jinja calculation, what about the error message (in case of an error)? + + In case of an error, it is the Jinja's description that will appear. + + In our example: `in HTTPS case if "_.use_for_https" is set to "true"` + +.. keypoints:: Key points + + We have seen that the `disabled` or `hidden` properties could be calculated. + The default values can be calculated too. We'll see a bunch of examples later on. + + **progress** + + - calculation with a Jinja type calculation: we can hide a family + - we know how to describe a calculation + - we know how to output some documentation (markdown HTML) of the :term:`configuration` + - we can calculate with a dynamic family identifer + - we can set an explicit boolean Jinja expression in a Jinja calculation + + **keywords** + + - Jinja type calculation for a hidden property + - Jinja calculation's description + - identifier type parameter + + Here we have come to the possibility of making any kind of calculations based on the state of the :term:`configuration`. + This is an important feature to manage the stateful aspect of a configuration. + diff --git a/docs/tutorial/modes.rst b/docs/tutorial/modes.rst new file mode 100644 index 000000000..0d0baf88d --- /dev/null +++ b/docs/tutorial/modes.rst @@ -0,0 +1,90 @@ +Modes +========= + +.. objectives:: Objectives + + For information on the concept we will be using in this section, please refer to + the definition of the :term:`mode` in the documentation, + and you can read all the :ref:`description of the modes in the structuring data point of view `. + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tag :tutorial:`v1.1_150 ` + in the repository. + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach v1.1_150 + +A variable in avanced mode +--------------------------- + +To specify the various possible modes, we must declare them on the command line +with the `--modes_level` command line option: + +.. raw:: html + :class: terminal + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_150/config/01/cmd_ro.txt + +.. + rougail -m firefox/ --types types/proxy --modes_level basic standard advanced -u yaml -yf config/01/config.yml + + +Next, we can set a mode parameter with a value from the list: `basic`, `standard`, `advanced`. + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_150/firefox/55-proxy_dns_socks5.yml + :language: yaml + :caption: The `proxy_dns_socks5` variable in the :file:`firefox/55-proxy_dns_socks5.yml` file + +.. + %YAML 1.2 + --- + version: 1.1 + + proxy_dns_socks5: + description: Use proxy DNS when using SOCKS v5 + mode: advanced + default: false + disabled: + jinja: |- + {{ _.manual.socks_proxy.version is propertyerror or _.manual.socks_proxy.version == 'v4' }} + return_type: boolean + description: |- + if "_.manual.socks_proxy.version" is accessible and equals "v4" + ... + +Here, we have set the mode to `advanced` for the `proxy_dns_socks5` variable +because we want to classify it as a variable intended for use by advanced users. + +.. questions:: What is it actually used for? + + Well, to begin with, reading the structure file provides a clear piece of information: + this variable is intended to be manipulated by an experienced user. + +Furthermore, if you integrate Rougail as a library into your business applications, +you will have access to different types of views corresponding to each mode. + +A useful application within Rougail itself involves filtering the display of certain +variables when generating documentation, based on the mode. + +In our case, this makes it possible to generate documentation for standard users +where the `proxy_dns_socks5` won't be present, +and a different, more detailed documentation for advanced users +where the `proxy_dns_socks5` will be explained in detail. + +.. seealso:: Look at the :ref:`different ways to document a variable with Rougail ` + in this tutorial. + +.. keypoints:: let's review the key points + + We have seen the benefits of assigning modes to a variable to define + documentations targeted at specific users. diff --git a/docs/tutorial/multiple.rst b/docs/tutorial/multiple.rst new file mode 100644 index 000000000..e818065b6 --- /dev/null +++ b/docs/tutorial/multiple.rst @@ -0,0 +1,235 @@ +Variable with multiple values +================================ + +.. objectives:: Objectives + + We will introduce the concept of multiple type variable. + It is more or less some kind of a container of variable's values. + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tags :tutorial:`v1.1_110 ` to :tutorial:`v1.1_111 ` + in the repository. + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach v1.1_110 + +.. _tutorial_not_mandatory_list: + +A conditional disabled non mandatory variable with type domainname and parameters +----------------------------------------------------------------------------------- + +In order to fit with our use case, we need a variable that takes as its value the domain names authorized to pass through the proxy, if it is enabled. We already used before in this tutorial the `domainname` type, which perfectly matches our needs here. +We will just need to add certain parameters associated to this `domainname` type, as we will see. +We're gonna put it in another :term:`structure file` named :file:`firefox/40-no_proxy.yml`. + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_110/firefox/40-no_proxy.yml + :language: yaml + :caption: The :file:`firefox/40-no_proxy.yml` structure file with the domains that are allowed to pass through the proxy + +.. + %YAML 1.2 + --- + version: 1.1 + + no_proxy: + description: Address for which proxy will be desactivated + type: domainname + params: + allow_ip: true + allow_cidr_network: true + allow_without_dot: true + allow_startswith_dot: true + mandatory: false + disabled: + variable: _.proxy_mode + when: No proxy + ... + +This `no_proxy` variable is a `domainname` with some additional parameters, we authorize +values like: + +- IP +- CIDR networks +- machine names (without `'.'`) +- sub-domaines like `.example` + +To clarify things, let's launch the Rougail CLI + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_110/config/02/cmd_ro.txt + :class: terminal + +.. + rougail -m firefox/ --types types/proxy -u yaml -yf config/02/config.yml + +On this :term:`user data`: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_110/config/02/config.yml + :language: yaml + :caption: Automatic proxy configuration :file:`config/02/config.yml` user data file with a pass through domain name + +.. + --- + proxy_mode: Automatic proxy configuration URL + auto: https://auto.proxy.net/wpad.dat + no_proxy: 192.168.1.0/24 + +We then have this output: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_110/config/02/output_ro.html + :class: output + +.. + ╭────────────── Caption ───────────────╮ + │ Variable Modified value │ + │ (⏳ Original default value) │ + ╰──────────────────────────────────────╯ + Variables: + ┣━━ 📓 Configure Proxy Access to the Internet: Automatic proxy configuration URL ◀ loaded from the YAML file "config/02/config.yml" (⏳ No proxy) + ┣━━ 📓 Automatic proxy configuration URL: https://auto.proxy.net/wpad.dat ◀ loaded from the YAML file "config/02/config.yml" + ┗━━ 📓 Address for which proxy will be desactivated: 192.168.1.0/24 ◀ loaded from the YAML file "config/02/config.yml" + +We can see the address for which proxy will be desactivated in the output. + +.. note:: + + We had said that a variable with a mandatory parameter set to false was a :ref:`nullable variable `. + This is true only for a non :term:`multi` variable. + + The value of a :term:`multi` variable which is not :term:`mandatory` will allow the empty list `[]`. + The :term:`mandatory` properties does not means that the `null` value is acceptable (that is the `empty` properties). + +A variable with multiple values +----------------------------------- + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_111 ` version:: + + git switch --detach v1.1_111 + +It's easy to find out what the next step is: we would now need some kind of a container type, +which would allow us to set not only one domain name but a list of domain names for which the proxy will be disabled. + +Making this change requires adding one line, just one, to our variable, +have a look at the `multi: true` parameter: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_111/firefox/40-no_proxy.yml + :language: yaml + :caption: The :file:`firefox/40-no_proxy.yml` structure file with the pass through domains set to `multi` + +.. + %YAML 1.2 + --- + version: 1.1 + + no_proxy: + description: Address for which proxy will be desactivated + type: domainname + params: + allow_ip: true + allow_cidr_network: true + allow_without_dot: true + allow_startswith_dot: true + multi: true + mandatory: false + disabled: + variable: _.proxy_mode + when: No proxy + ... + + +.. questions:: Question + + + What difference does it make? + + First, note that the `no_proxy` variable is still a `domainname` type. + The type hasn't changed. + +There's a big difference: now it is important to note that the values ​​of the variable +must be a list containing strings, and not a single string. +Therefore, the contents of the :term:`user data` file need to be changed. + +Now let's launch the Rougail CLI with the `no_proxy` variable containing a list: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_111/config/02/config.yml + :language: yaml + :caption: Automatic proxy configuration :file:`config/02/config.yml` user data file with a pass through domain name + +.. + %YAML 1.2 + --- + version: 1.1 + + no_proxy: + description: Address for which proxy will be desactivated + type: domainname + params: + allow_ip: true + allow_cidr_network: true + allow_without_dot: true + allow_startswith_dot: true + multi: true + mandatory: false + disabled: + variable: _.proxy_mode + when: No proxy + ... + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_111/config/02/cmd_ro.txt + :class: terminal + +.. + rougail -m firefox/ --types types/proxy -u yaml -yf config/02/config.yml + +Now we have this output: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_111/config/02/output_ro.html + :class: output + +.. + Variables: + ┣━━ 📓 Configure Proxy Access to the Internet: Automatic proxy configuration URL ◀ loaded from the YAML file "config/02/config.yml" (⏳ No proxy) + ┣━━ 📓 Automatic proxy configuration URL: https://auto.proxy.net/wpad.dat ◀ loaded from the YAML file "config/02/config.yml" + ┗━━ 📓 Address for which proxy will be desactivated: + ┣━━ example.net ◀ loaded from the YAML file "config/02/config.yml" + ┗━━ 192.168.1.0/24 ◀ loaded from the YAML file "config/02/config.yml" + +.. questions:: Answer to the question + + So, the difference it makes is: yes, we have a list now. + + We call this a :term:`multiple ` variable. + +.. glossary:: + + multi + + A multiple variable, in short, a multi, is a variable that has a list of values. + + Note that we have more than just a simple container here. + The real challenge is maintaining the consistency of the :term:`configuration` + exactly the same way as if we only had one value to manage. + + A multi is a multiple variable, that is a variable that can have multiple values. + +.. keypoints:: Key points + + We have now some kind of a container facility named `multi`, that can be applied + to a variable simply by setting the `multi: true` parameter. + diff --git a/docs/tutorial/namespace.rst b/docs/tutorial/namespace.rst new file mode 100644 index 000000000..63ab60b99 --- /dev/null +++ b/docs/tutorial/namespace.rst @@ -0,0 +1,172 @@ +.. _tutorial_namespace: + +Namespaces +============ + +.. objectives:: Objectives + + We are going to need a new tool, the concept of :term:`namespaces `. + + And our use case is going to evolve. + We're going to use a Firefox add-on called `Foxy Proxy `_. + In this context we will have a new way of organizing our structure files. + + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tags :tutorial:`1.1_180 ` to :tutorial:`1.1_181 ` + in the repository. + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach 1.1_180 + +.. note:: The purpose here is not to teach how to use the Foxy Proxy addon, + but rather to explain that using this Firefox plugin will require us + to make decisions about how to organize our variables. + + We will store them in namespaces. + + +Namespace +------------ + +The :term:`namespaces ` are a new, completely different way of separating and organising our :term:`structure files `. +We're gonna see how. +We will begin by adding a parameter to our Rougail CLI: + +.. raw:: html + :class: terminal + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_180/config/01/cmd_ro.txt + +.. + rougail -m firefox/ -s Firefox --types types/proxy --modes_level basic standard advanced -u yaml -yf config/01/config.yml + +Now let's pay attention to the `-s` parameter. This namespace option has the value `Firefox`. +What does this imply? + +So far, to assign our :term:`user data` to a structure file like this one: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_180/firefox/60-dns_over_https.yml + :language: yaml + :caption: The `firefox/60-dns_over_https.yml` structure file + +.. + %YAML 1.2 + --- + version: 1.1 + + dns_over_https: # DNS over HTTPS + + enable_dns_over_https: false # Enable DNS over HTTPS + + provider: + description: Use Provider + choices: + - Cloudflare + - NextDNS + - Custom + default: Cloudflare + disabled: + variable: _.enable_dns_over_https + when: false + + custom_dns_url: + description: Custom DNS URL + type: web_address + validators: + - jinja: |- + {{ _.custom_dns_url.startswith("http://") }} + return_type: boolean + description: must starts with 'https://' only + disabled: + jinja: |- + {{ _.provider is propertyerror or _.provider != 'Custom' }} + return_type: boolean + description: if "_.provider" is not "Custom" + ... + +We had to enter :term:`user data` of this new shape: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_170/config/02/config.yml + :language: yaml + :caption: user data settings with the `dns_over_https` family + +.. + --- + dns_over_https: + enable_dns_over_https: true + +But now that we have defined a namespace, we need to populate our :term:`user data` in this shape: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_180/config/01/config.yml + :language: yaml + :caption: user data settings about the `dns_over_https` family with the `firefox` namespace + +.. + --- + firefox: + dns_over_https: + enable_dns_over_https: true + +Notice the difference: we have now a `firefox` namespace. +Contrary to a misinterpretation we might have upon first reading, it is not a family. +It's a namespace. + +.. questions:: What is the purpose of creating a namespace? + + Well, this allows for logical grouping. We'll be able to clearly separate + the user data related to our Foxy Proxy plugin. + +New "FoxyProxy" namespace +---------------------------- + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`1.1_181 ` version:: + + git switch --detach 1.1_181 + +.. image:: images/foxyproxy.png + +We can see in the FoxyProxy widget that it is possible to add as many proxies as needed. + +So our use case enables us to handle variables related to the Foxy Proxy plugin +that should be stored in a separate namespace. + +.. type-along:: Handling multiple namespaces + +It is perfectly possible to define multiple namespaces with Rougail. +In the Rougail CLI, the `-s` option is reserved for the main namespace. +If we need to add a namespace named FoxyProxy, we must use the `-xn FoxyProxy` option. +However, we also need to specify the parameter that indicates the location +of the structure files corresponding to this namespace. +The `-xd 0 foxyproxy/` option is used to specify the relevant file or folder, +here, the :file:`foxyproxy` folder: + +.. raw:: html + :class: terminal + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_181/config/01/cmd_ro.txt + +.. + rougail -m firefox/ -s Firefox -xn FoxyProxy -xd 0 foxyproxy/ --types types/proxy --modes_level basic standard advanced -u yaml -yf config/01/config.yml + +The additional namespace is therefore specified via the two command-line options: + +.. code-block:: shell + + -xn FoxyProxy -xd 0 foxyproxy/ + +.. keypoints:: let's review the key points + + We learned how to create a namespace for our use case, and even how to use + multiple namespaces—that is, how to implement several logical separations. diff --git a/docs/tutorial/nullable.rst b/docs/tutorial/nullable.rst new file mode 100644 index 000000000..17c59f9f3 --- /dev/null +++ b/docs/tutorial/nullable.rst @@ -0,0 +1,211 @@ +.. _tutorial_nullable: + +Nullable variable +========================== + +.. objectives:: Objectives + + Now in this section we would like to make it possible that it is not necessary to specify a value + for a variable. + + With Rougail, it is possible for a variable's settings to have no value (nothing, null, None), + that is, neither a default value nor a user assigned value. + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tags :tutorial:`v1.1_090 ` to :tutorial:`v1.1_091 ` + in the repository. + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach v1.1_090 + + +Variable with the value "null" +------------------------------------ + +.. note:: It is important to keep in mind that in Rougail a variable is :term:`mandatory` by default, + meaning that it must either have a :term:`default value` set or a :term:`user data` value assigned for example in a :term:`user data file`\ . + This is Rougail's default behavior. + + Besides, note that the explicit keyword to express the setting "no value, nothing" in YAML is the `null` explicit keyword. + +Here is how we are going to specify a "no value" (nothing, `null`) as a possible setting for the `address` variable in the `socks_proxy` family. + +There are a few things to set in the variable. We need to: + +- explicitely disable the default :term:`mandatory` behavior of the `address` variable (just by setting it to `false`), +- redefine the `address` variable inside the `socks_proxy` because this family uses the `proxy` type definition, +- finally removes the `address` default value as determined in the `proxy` type definition. + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_090/firefox/20-manual.yml + :language: yaml + :caption: The :file:`firefox/20-manual.yml` structure file with the `mandatory` set to `false` and the `null` (nothing) setting + +.. + %YAML 1.2 + --- + version: 1.1 + + manual: + + use_for_https: true # Also use this proxy for HTTPS + + https_proxy: + description: HTTPS Proxy + type: proxy + hidden: + variable: _.use_for_https + + address: + redefine: true + description: HTTPS proxy address + + port: + redefine: true + description: HTTPS proxy port + + socks_proxy: + description: SOCKS Proxy + type: proxy + + address: + redefine: true + description: SOCKS proxy address + default: null + mandatory: false + + port: + redefine: true + description: SOCKS proxy port + default: 1080 + + version: + description: SOCKS host version used by proxy + choices: + - v4 + - v5 + default: v5 + ... + + +Some more, maybe usefull, explanations about our `address` variable in the `socks_proxy` family: + +- Allowing the "no value, nothing" behavior is easy, we just add the `mandatory: false` attribute, +- we explicitely need to set the `redefine: true` attribute because in the type `proxy` type setting + we didn't set the `mandatory` attribute to false, this is then a type redefinition, +- finally we can set the `default: null` that means that we can set no value to our address variable. + +Note too that is is necessary to redefine (to set a `redefine` attribute) as soon as you +modify the type definition. Here we have set a default value to our `socks_proxy`'s `port` variable. +We need then to redefine not only because of the `description` attribute but because of the default value too. + + +Variables disabled when condition is null +-------------------------------------------- + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_091 ` version:: + + git switch --detach v1.1_091 + +.. questions:: Question + + But actually, why did we want a variable to be non-mandatory? + +Well, what we need is to be able to :term:`disable ` either the `port` and the `version` in case of the `address` has no value set: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_091/firefox/20-manual.yml + :language: yaml + :caption: The :file:`firefox/20-manual.yml` structure file with the `disabled` set in case of the `address` has no value set + +.. + %YAML 1.2 + --- + version: 1.1 + + manual: + + use_for_https: true # Also use this proxy for HTTPS + + https_proxy: + description: HTTPS Proxy + type: proxy + hidden: + variable: _.use_for_https + + address: + redefine: true + description: HTTPS proxy address + + port: + redefine: true + description: HTTPS proxy port + + socks_proxy: + description: SOCKS Proxy + type: proxy + + address: + redefine: true + description: SOCKS proxy address + default: null + mandatory: false + + port: + redefine: true + description: SOCKS proxy port + default: 1080 + disabled: + variable: _.address + when: null + + version: + description: SOCKS host version used by proxy + choices: + - v4 + - v5 + default: v5 + disabled: + variable: _.address + when: null + ... + +For the `disabled` property to trigger, the condition `when: null` must appear, +pointing to the `_.address` variable, that is the `socks_proxy.address` variable. + + +.. note:: The `_.` is a :term:`relative path` notation and means the current path (of the current family). + + In the python quasi algorithmic notation we could say that: + + .. code-block:: python + + _.address == socks_proxy.address + + This is true only because in our use case `socks_proxy` is located on the root path. + +.. keypoints:: Key points + + As one of the main objectives of Rougail is to determine if a variable has a value, + therefore the value of a variable is in principle *required*. + We say that a value is :term:`mandatory` by default. + However, it is possible to change this default behavior with the `mandatory: false` parameter + explicitely declared in a variable. + In this section we learned how to define a variable as having no value; it's possible with Rougail. + + In this section we also continued using some type redefinition, which we learned in the previous section. + And we have :term:`disabled` a variable depending this on + the presence or absence of a value. + + diff --git a/docs/tutorial/practice1.rst b/docs/tutorial/practice1.rst new file mode 100644 index 000000000..2a455e440 --- /dev/null +++ b/docs/tutorial/practice1.rst @@ -0,0 +1,145 @@ +Practice +================================ + +.. objectives:: Objectives + + Now it's your turn. + Try making some Rougail. + At the same time, if you get stuck, you have the solutions in the Rougail tutorial + code repository. + + There is nothing new in the two next Firefox variables. So it is the time to practice what we've learned until now. + + This page is divided in two parts for each variable: + + - Explain what we want to implement + - Propose to you a solution + + The best for you is to do this job on your own and compare your work to our propositions. + +Here are the two variables in the Firefox configuration's widget: + +.. image:: images/practice_twovars.png + +These two variables are highlighted here in the lower part of the Firefox configuration's widget: + +.. image:: images/firefox130.png + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tags :tutorial:`1.1_130 ` to :tutorial:`1.1_131 ` + in the repository. + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach 1.1_130 + +A conditional disabled boolean variable +------------------------------------------------------------- + +What do we want to be implemented +''''''''''''''''''''''''''''''''''' + +.. exercise:: Exercice + + You need to think about: + + - choose a file name that respect the :ref:`file naming and organizing convention ` + - define an appropriate variable name, have a look at our :ref:`variable naming convention ` + - the Firefox description is in negative mode, which is not a good practice for us, we rewrite it in positive mode + - the variable is a boolean with the default value `true` (the opposite of unchecked option) + - the variable is available for all proxy mode choice by user, so disable it only when the `proxy mode` is "No proxy" + +Our solution +''''''''''''''''''''''''''''''' + +Please unroll the "Solution" widget to see our solution: + +.. solution:: Solution + + .. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_130/firefox/50-prompt_authentication.yml + :language: yaml + :caption: The :file:`firefox/50-prompt_authentication.yml` a conditional disabled boolean variable + + - Here we choosed to name the new structure file `firefox/50-prompt_authentication.yml`, + the file containing the `prompt_authentication` variable definition + - this variable is `true` by default + - this variable shall be :term:`disabled` if the `proxy_mode` variable is set to `"No proxy"` + +.. + %YAML 1.2 + --- + version: 1.1 + + prompt_authentication: + description: Prompt for authentication if password is saved + default: true + disabled: + variable: _.proxy_mode + when: No proxy + ... + +A Jinja conditional disabled boolean variable +------------------------------------------------- + +What do we want to be implemented +''''''''''''''''''''''''''''''''''' + +.. exercise:: Exercice + + You need to think about: + + - the choice of a file name that respect the :ref:`file naming and organizing convention ` + - define an appropriate variable name, have a look at our :ref:`variable naming convention ` + - the `description` parameter will be the Firefox description + - the variable is a boolean with the default value `false` (unchecked option) + - the variable is only available with the `"Manual proxy configuration"` proxy mode and when the proxy socks version is not v4 + + +Our solution +''''''''''''''''''''''''''''''' + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_131 ` version:: + + git switch --detach v1.1_131 + +Please unroll the "Solution" widget to see our solution: + +.. solution:: Solution + + .. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_131/firefox/55-proxy_dns_socks5.yml + :language: yaml + :caption: The :file:`firefox/55-proxy_dns_socks5.yml` a Jinja conditional disabled boolean variable + + - Here we choosed to name the new structure file `firefox/55-proxy_dns_socks5.yml`, + the file containing the `proxy_dns_socks5` variable definition + - this variable is `false` by default + - this variable shall be :term:`disabled` if the `proxy_mode` variable is not set to `"Manual proxy configuration"` and Socks version is `v4`. + +.. + %YAML 1.2 + --- + version: 1.1 + + proxy_dns_socks5: + description: Use proxy DNS when using SOCKS v5 + default: false + disabled: + jinja: |- + {{ _.proxy_mode != "Manual proxy configuration" or _.manual.socks_proxy.version == 'v4' }} + return_type: boolean + description: |- + if "_.proxy_mode" is not "Manual proxy configuration" + or "_.manual.socks_proxy.version" is "v4" + ... diff --git a/docs/tutorial/practice2.rst b/docs/tutorial/practice2.rst new file mode 100644 index 000000000..af72cdc21 --- /dev/null +++ b/docs/tutorial/practice2.rst @@ -0,0 +1,220 @@ +Practice +================================ + +.. objectives:: Objectives + + Now it's your turn again. + Try making some Rougail. + At the same time, if you get stuck, you have the solutions in the Rougail tutorial + code repository. + + There is nothing new in the three next Firefox variables. So it is the time to practice what we've learned until now. + + This page is divided in two parts for each variable: + + - Explain what we want to implement + - Propose to you a solution + + The best for you is to do this job on your own and compare your work to our propositions. + +Here are the two first variables in the Firefox configuration's widget: + +.. image:: images/practice_threevars.png + +The third appears only if we enable DNS over HTTPS with a custom providers, we have to set a `Custom DNS URL`. + +These three variables are highlighted here in the lower part of the Firefox configuration's widget: + +.. image:: images/firefox160.png + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tags :tutorial:`1.1_160 ` to :tutorial:`1.1_162 ` + in the repository. + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach 1.1_160 + +A boolean variable +------------------ + +What do we want to be implemented +''''''''''''''''''''''''''''''''''' + +.. exercise:: Exercice + + You need to think about: + + - choose a file name that respect the :ref:`file naming and organizing convention ` + - define an appropriate family and variable name, have a look at our :ref:`variable naming convention ` + - we will set the variable in a family (with others variable later) to separate this purpose + - the family and variable `description` parameter will be the Firefox description + - the variable is a :ref:`boolean ` with the default value `false` (unchecked option) + - the family and variable are always available, it's not directly related to the proxy choice + +Our solution +''''''''''''''''''''''''''''''' + +Please unroll the "Solution" widget to see our solution: + +.. solution:: Solution + + .. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_160/firefox/60-dns_over_https.yml + :language: yaml + :caption: The :file:`firefox/60-dns_over_https.yml` a boolean variable + + - Here we choosed to name the new structure file `firefox/60-dns_over_https.yml`, + the file containing the `dns_over_https` family and the `enable_dns_over_https` variable definition + - this variable is a :ref:`boolean ` with the default value `false` + +.. + %YAML 1.2 + --- + version: 1.1 + + dns_over_https: # DNS over HTTPS + + enable_dns_over_https: false # Enable DNS over HTTPS + ... + +A choice variable +----------------- + +What do we want to be implemented +''''''''''''''''''''''''''''''''''' + +.. exercise:: Exercice + + You need to think about: + + - this variable will be in the same file as the first one + - define an appropriate variable name, have a look at our :ref:`variable naming convention ` + - we will set the variable in the previous family + - the variable `description` parameter will be the Firefox description + - the variable is a :ref:`choice ` with the following proposal: `Cloudflare`, `NextDNS` and `Custom` + - this variable shall be :ref:`disabled ` if the `enable_dns_over_https` variable is `false`. + +Our solution +''''''''''''''''''''''''''''''' + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_161 ` version:: + + git switch --detach v1.1_161 + +Please unroll the "Solution" widget to see our solution: + +.. solution:: Solution + + .. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_161/firefox/60-dns_over_https.yml + :language: yaml + :caption: The :file:`firefox/60-dns_over_https.yml` a choice variable + + - Here we choosed to name the new structure file `firefox/60-dns_over_https.yml`, + the file containing the `dns_over_https` family and the `enable_dns_over_https` variable definition + - this variable is false by default + - this variable is :ref:`disabled ` if `enable_dns_over_https` is `false` + +.. + %YAML 1.2 + --- + version: 1.1 + + dns_over_https: # DNS over HTTPS + + enable_dns_over_https: false # Enable DNS over HTTPS + + provider: + description: Use Provider + choices: + - Cloudflare + - NextDNS + - Custom + default: Cloudflare + disabled: + variable: _.enable_dns_over_https + when: false + ... + + +A web_address variable +---------------------- + +What do we want to be implemented +''''''''''''''''''''''''''''''''''' + +.. exercise:: Exercice + + You need to think about: + + - this variable will be in the same file as the two others + - define an appropriate variable name, have a look at our :ref:`variable naming convention ` + - we will set the variable in the previous family + - the variable `description` parameter will be the Firefox description + - the variable is a `web_address` + - this variable shall be :ref:`disabled ` if the `provider` variable is not `Custom` + - the variable `provider` could be :ref:`disabled ` too + + +Our solution +''''''''''''''''''''''''''''''' + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_162 ` version:: + + git switch --detach v1.1_162 + +Please unroll the "Solution" widget to see our solution: + +.. solution:: Solution + + .. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_162/firefox/60-dns_over_https.yml + :language: yaml + :caption: The :file:`firefox/60-dns_over_https.yml` a choice variable + + - Here we choosed to name the new structure file `firefox/60-dns_over_https.yml`, + the file containing the `custom_dns_url` variable definition + - this is a :ref:`web_address ` variable + - this variable is :ref:`disabled ` if `provider` has :ref:`an access property error ` and `provider` has not the value "Custom" + +.. + %YAML 1.2 + --- + version: 1.1 + + dns_over_https: # DNS over HTTPS + + enable_dns_over_https: false # Enable DNS over HTTPS + + provider: + description: Use Provider + choices: + - Cloudflare + - NextDNS + - Custom + default: Cloudflare + disabled: + variable: _.enable_dns_over_https + when: false + + custom_dns_url: + description: Custom DNS URL + type: web_address + disabled: + jinja: |- + {{ _.provider is propertyerror or _.provider != 'Custom' }} + return_type: boolean + description: if "_.provider" is not "Custom" + ... diff --git a/docs/tutorial/preliminary.rst b/docs/tutorial/preliminary.rst new file mode 100644 index 000000000..632f3c9f4 --- /dev/null +++ b/docs/tutorial/preliminary.rst @@ -0,0 +1,380 @@ +Getting started +==================== + +Presentation of the Firefox configuration options +-------------------------------------------------- + +At first glance we can see that we have a selection of five configuration options that we need to fill in, they are highlighted here in this screenshot: + +.. image:: images/firefox_01.png + +We'll learn in this tutorial how to set the values of the configuration options in a clean way with the Rougail library. + +.. objectives:: Objectives of this section + + We will learn how to: + + - create a :term:`structure description file ` + - add a :term:`structure file ` format version in the structure file + - add a :term:`variable` in the structure file and set its default :term:`value` + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tags :tutorial:`v1.1_000 ` to :tutorial:`v1.1_003 ` + in the repository: + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach v1.1_000 + +Creating a structure file +-------------------------- + +.. demo:: The folder structure + + Here is the tree structure we want to have:: + + rougail-tutorials + └── firefox + └── 00-proxy.yml + +- Let's make a :file:`rougail-tutorials` directory, with a :file:`firefox` subfolder. +- First, we will create a :term:`structure file `, so let's create a :file:`00-proxy.yml` file + located in the :file:`firefox` subfolder. + +This is an empty Rougail :term:`structure description file: ` + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_000/firefox/00-proxy.yml + :language: yaml + :caption: An empty Rougail structure file with only the YAML header and the version number + :name: RougailStructVersion + +.. + --- + version: 1.1 + +:tutorial:`Download this file from the rougail-tutorials git repository ` + +This `version` specification is just the Rougail YAML's format version specification. +YAML files are loaded using a version 1.2 file parser. It is recommended (but not mandatory) to specify this in the file header, especially for linters. +By now, we have an empty structure file with the format specification in it. + +Let's add our first variable +------------------------------ + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_001 ` version:: + + git switch --detach v1.1_001 + + +- A variable is defined at a minimum by its name. +- A :term:`variable` lives in the :term:`structure description file `. + +Here we define a variable named `proxy_mode`: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_001/firefox/00-proxy.yml + :language: yaml + :caption: A Rougail structure file with only one variable in it + :name: RougailDictionaryFirstVariableName + +.. + --- + proxy_mode: + +:tutorial:`Download this file from the rougail-tutorials git repository ` + +Let's run the Rougail CLI utility command in a terminal: + +.. code-block:: text + :class: terminal + + rougail -m firefox/ + +Well, we notice that we have an error: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_001/config/01/output_ro.html + :class: error-box + +.. + 🛑 ERRORS + ┣━━ The following variables are mandatory but have no value: + ┗━━ - proxy_mode + +It's because this first defined variable is :term:`mandatory` and needs to have a value set **but** there's no value yet. + +We can therefore deduce the fact that: + +.. admonition:: Fact + + Once defined, an option configuration :term:`value` is :term:`mandatory` by default. + That is to say, it is absolutely necessary to assign a value to this variable. + +Rougail expects the `proxy_mode` configuration option's value to be set. + +.. glossary:: + + mandatory + + A variable is mandatory when a value is required, that is, `None` **is not** a possible value. + It **must** have a defined value. + +.. seealso:: To go further, have a look at the :tiramisu:`mandatory option ` + according to the :xref:`Tiramisu ` underlyning consistency system. + You will learn that it is actually possible to disable the mandatory property behavior, + but you need to declare it explicitely. + +Describe the variable +------------------------- + +Let's add a variable's description, which is not mandatory but which is usually a good practice. + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_002 ` version:: + + git switch --detach v1.1_002 + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_002/firefox/00-proxy.yml + :language: yaml + :caption: A Rougail structure file with a variable and a description + :name: RougailStructFirstVariableDescription + +.. + --- + proxy_mode: # Configure Proxy Access to the Internet + +:tutorial:`Download this file from the rougail-tutorials git repository ` + +You have two way to define a variable's description: + +- the verbose way: + + .. code-block:: yaml + + proxy_mode: + description: Configure Proxy Access to the Internet + + +- or a short-hand way, setting the description using the "`#`" YAML comment notation: + + .. code-block:: yaml + + proxy_mode: # Configure Proxy Access to the Internet + +If we launch the Rougail CLI command: + +.. raw:: html + :class: terminal + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_002/config/01/cmd_rw.txt + +we have this output: + +.. raw:: html + :class: output + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_002/config/01/output_rw.html + +.. +
    ╭──────────────────── Caption ─────────────────────╮
    +    │ Undocumented but modified variable Default value │
    +    ╰──────────────────────────────────────────────────╯
    +    Variables:
    +    ┗━━ 📓 Configure Proxy Access to the Internet: null
    +    
    + +We can see here that the variable's description string "Configure Proxy Access to the Internet" is used +to refer to the `proxy_mode` variable. + +.. note:: The description is used in UI tools and outputs instead of the variable name. + The goal here is to provide an explanation of the variable for the user, + not to show the technical name of the variable as defined by the :term:`integrator`. + +Set a default value +--------------------- + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_003 ` version:: + + git switch --detach v1.1_003 + +We will learn different ways to set a value, the first way is setting a *default* value. + +.. glossary:: + + default value + + A default value is a variable value that is predefined, that's why this value is placed + right in the structure file. + +Let's add a default value to this `proxy_mode` variable. + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_003/firefox/00-proxy.yml + :language: yaml + :caption: A Rougail structure file with a default value for the variable + :name: RougailDictionaryVariableDefault + +.. + --- + proxy_mode: No proxy # Configure Proxy Access to the Internet + +:tutorial:`Download this file from the rougail-tutorials git repository ` + +The `proxy_mode` variable requires a value, that's why we have set a `No proxy` default value. + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_003/config/01/output_ro.html + :class: output + +.. +
    ╭─────── Caption ────────╮
    +    │ Variable Default value │
    +    ╰────────────────────────╯
    +    Variables:
    +    ┗━━ 📓 Configure Proxy Access to the Internet: No proxy
    +    
    + +As we have set the `proxy_mode`'s value as `No proxy` by default, +The chosen value is indicated in the Rougail's CLI output as the default choice. + +- here is the short-hand default setting and description: + +.. code-block:: yaml + + proxy_mode: No proxy # Configure Proxy Access to the Internet + +- and there is the verbose way of setting a default value: + +.. code-block:: yaml + + proxy_mode: + description: Configure Proxy Access to the Internet + default: No proxy + +There are some other :term:`short-hand ways ` with Rougail that you may encounter +as you read the Rougail's documentation and tutorial. + +.. admonition:: How to set a value -- the assignment + + A default value has been set, great. This raises a question about what a normal value is. + + Now then how can I assign a normal value to a variable? + +.. type-along:: The different Rougail roles and setting a variable's value + +So far we have only talked about the actor that writes the :term:`structure files `\ . +The one who writes the structure file plays the *role* of the *integrator*. + +Now we will talk about the one that defines the values. His role is called the operator role. + +It is the operator's responsibility to set the user data variables values. +The operator does not handle the structure files, +he is responsible of other files called the :term:`user data files `. + +.. important:: For now, we don't know how to disable the default `mandatory` settings, + so if neither a default value nor a user value are set for a given variable, Rougail will raise an error. + +.. exercise:: Folder structure update + + Now we add a user data file named :file:`config/config.yml` in our project:: + + rougail-tutorials + ├── firefox + │ ├── 00-proxy.yml + └── config + └── config.yml + +.. type-along:: How to set a user data value + +If the integrator has not set any default value in his structure file, +it's up to the operator to do the job in the `config.yml` file: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_003/config/02/config.yml + :language: yaml + :caption: A Rougail user data file :file:`config/config.yml`, with a value set. + :name: RougailConfigDefaultValue + +.. + --- + proxy_mode: No proxy + +:tutorial:`Download this file from the rougail-tutorials git repository ` + +The operator needs to add the `-u yaml -yf config/config.yml` options to the Rougail CLI: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_003/config/02/cmd_ro.txt + :class: terminal + +.. + rougail -m firefox/ -u yaml -yf config/02/config.yml + +which gives us this output: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_003/config/02/output_ro.html + :class: output + +.. +
    ╭──────── Caption ────────╮
    +    │ Variable Modified value │
    +    ╰─────────────────────────╯
    +    Variables:
    +    ┗━━ 📓 proxy_mode: No proxy ◀ loaded from the YAML file "config/02/config.yml"
    +    
    + +Now the `proxy_mode`'s new `No proxy` value is the same as the default value but we see in the Rougail CLI output that the value +comes from the :file:`config/02/config.yml` user data file. From now on this `proxy_mode` variable's value +is a user data value and not a default value (even if it's actually the same value). + +.. type-along:: Structure values and user data values + +We can see with the Rougail CLI utility where the values come from. +It can come from an integrator's setting or from an operator's setting. + +.. admonition:: Reminder + + - the integrator works on structure files, he can define default value for variables + - the operator works on user data files, he only can set user data values for variables + +Most of the time, the integrator and the operator are one and the same person, +here we are talking about roles and not necessarily about people. + +.. type-along:: User data files are where the user values live + +We need to set the values in separate files, called `user data files`. + +.. glossary:: + + user data file + + A user data file is a file where only :term:`user data` are set. + + A user file is a file where there are only user data in it, users can set values, called user values -- + that is variable's values that have been set by an :term:`operator`\ . + + see also :term:`user data` + +.. keypoints:: Key points progress + + **Keywords** + + - :term:`structure file `: structure description file + - :term:`variable`: an option's name which has a value + - a variable's description + - a variable's mandatory value + - a variable's default value + - a variable's user value + - the :term:`integrator` and :term:`operator` roles + - a :term:`configuration` diff --git a/docs/tutorial/properties.rst b/docs/tutorial/properties.rst new file mode 100644 index 000000000..17c7fab14 --- /dev/null +++ b/docs/tutorial/properties.rst @@ -0,0 +1,801 @@ +Define access to variable or family +========================================== + +.. objectives:: Objectives + + In this section we will see what a disabled variable or family is, and why it can be interesting + to assign the `disabled` :term:`property` to a variable or a family. + Then we'll see the same thing for the `hidden` :term:`property`. + We'll also learn the difference between disabling and hiding families or variables. + + We will: + + - create a `disabled` family + - use a new family or variable's :term:`property`: the `hidden` :term:`property` + +Disabling and hiding are two families or variables :term:`properties `. + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tags :tutorial:`v1.1_050 ` to :tutorial:`v1.1_053 ` + in the repository. + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach v1.1_050 + +.. _tutorial_disabled: + +A disabled family +------------------ + +Here we are going to assign the `disabled` property to the `manual` family: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_050/firefox/10-manual.yml + :language: yaml + :caption: The `manual` family has the `disabled` property set in this :file:`firefox/10-manual.yml` structure file + +.. + %YAML 1.2 + --- + version: 1.1 + + manual: + description: Manual proxy configuration + disabled: true + + http_proxy: # HTTP Proxy + + address: + description: HTTP address + type: domainname + params: + allow_ip: true + + port: + description: HTTP Port + type: port + default: 8080 + +Notice that we have this `disabled: true` property assigned to the `manual` family. +Let's launch the Rougail CLI on this structure file (whith an empty user data file): + +.. raw:: html + :class: terminal + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_050/config/01/cmd_ro.txt + +.. + rougail -m firefox/ -u yaml -yf config/02/config.yml + +The Rougail CLI outputs this: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_050/config/01/output_ro.html + :class: output + +We can deduce from the Rougail CLI output that the `manual` family is not taken into account by Rougail. +So what does this disabled property exactly? + +.. glossary:: + + disabled + + The disabled property is a property that can be assigned to a variable or a family. + It makes the :term:`configuration` act as if the variable or family that has this property has not even been defined. + It simply doesn't exist (it is deactivated) for the whole configuration. + +.. note:: Note that if a family has been disabled, all variables and sub-families that it contains are disabled. + +And if we try to assign values to variables that have been disabled, here is what happens: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_050/config/02/config.yml + :language: yaml + :caption: In this :file:`config/02/config.yml` user data file, we assign values to variables that have been disabled + +If we launch the Rougail CLI: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_050/config/02/cmd_unknown.txt + :class: terminal + +.. + rougail -m firefox/ -u yaml -yf config/02/config.yml --cli.unknown_user_data_error + +.. note:: The `--cli.unknown_user_data_error` option changes the behaviour of the Rougail CLI's standard output: + when an unknown (or disabled or hidden) variable is declared in the :term:`user data file ` + then it appears in the output as an error instead of a warning. + +It outputs: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_050/config/02/output_unknown.html + :class: output + +.. +
    🛑 Caution
    +    ┗━━ manual (Manual proxy configuration)
    +        ┣━━ http_proxy (HTTP Proxy)
    +        ┣━━ address (HTTP address): 🛑 family "manual" (Manual proxy 
    +        configuration) has property disabled, so cannot access to "address" 
    +        (HTTP address), it has been loading from the YAML file 
    +        "config/02/config.yml"
    +        ┗━━ port (HTTP Port): 🛑 family "manual" (Manual proxy configuration) 
    +            has property disabled, so cannot access to "port" (HTTP Port), it 
    +            has been loading from the YAML file "config/02/config.yml"
    +        ┗━━ use_for_https (Also use this proxy for HTTPS): 🛑 family "manual" 
    +            (Manual proxy configuration) has property disabled, so cannot access to 
    +            "use_for_https" (Also use this proxy for HTTPS), it has been loading 
    +            from the YAML file "config/02/config.yml"
    +    
    + +We can see that the Rougail CLI is warning us about the variables that we are trying to assign values on which are disabled. +Because it is not logical. We are trying to assign values to variables that are not taken into account in the :term:`configuration`. + +The point is that we disable them in order to expose the fact that we don't use them, +but it's not just that: if we fill them in, there might be a problem in the overall integrity of the whole :term:`configuration`. +We shall fill and use in these variables in the `Manual proxy configuration` use case context only. +Otherwise, **we need to disable them when they are not used**. + +In a practical point of view, if we fill them in, Rougail CLI will output a warning and the :term:`operator` will see it. +He will wonder : "oh, what am I doing?", I shall fill and use in these variables only in the `Manual proxy configuration` context. + +A conditional disabled family +------------------------------ + +Let's look again at our use case. We have a choice between five options +in order to set the proxy mode: + +.. image:: images/firefox_01.png + +These five choices are: + +- No proxy +- Auto-detect proxy settings for this network +- Use system proxy settings +- Manual proxy configuration +- Automatic proxy configuration URL + +Actually if the `Manual proxy configuration` is not selected, we don't need to set +these `address` and `port` variables, there is no point in setting them in +four out of our five use cases. + +.. important:: We need to **disable** variables or families that are not used + in a given usage context. + +Disabling variables one by one can be replaced by disabling a whole family. +If we don't choose the manual mode, we need to **disable** the whole `manual` family, it will disable +all the subfamilies and the variables in it. + +Note that we've placed theses variables in the `http_proxy` +subfamily. We can then disable it or even the parent `manual` subfamily in order to +disable the `http_proxy` family and all the variables that are placed in it, because +the `manual` family variables shall be used only in the manual proxy use case context. + +Notice the `disabled: true` parameter set in the `manual` family: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_050/firefox/10-manual.yml + :linenos: + :language: yaml + :caption: The `http_proxy` subfamily in the :file:`firefox/10-manual.yml` structure file is disabled here + +.. + --- + manual: + description: Manual proxy configuration + disabled: true + + http_proxy: + description: HTTP Proxy + + address: + description: HTTP address + type: domainname + params: + allow_ip: true + + port: + description: HTTP Port + type: port + default: 8080 + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_051 ` version:: + + git switch --detach v1.1_051 + +What could be usefull here is a *dynamically* disable or enable the `manual` family. +The idea in this section is to dynamically set the enable/disable property according to the chosen use case context. + +In rougail, we can set a property's value **depending on** the value of another variable. +**The property's value is conditioned by** another variable. + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_051/firefox/10-manual.yml + :linenos: + :language: yaml + :caption: The :file:`firefox/10-manual.yml` structure file. The `manual` family dynamically enabled or disabled + +.. + %YAML 1.2 + --- + version: 1.1 + + manual: + description: Manual proxy configuration + disabled: + variable: _.proxy_mode + when_not: Manual proxy configuration + + http_proxy: # HTTP Proxy + + address: + description: HTTP address + type: domainname + params: + allow_ip: true + + port: + description: HTTP Port + type: port + default: 8080 + +Now the `disabled` property has some parameter defined. First, let's explaine the `variable` parameter. +This parameter specifies the target variable. The value of this target variable +will be used to dynamically enable or disable our `manual` family. + +We can see here that the `manual` family disabled or enabled property is contitionned by the `_.proxy_mode` variable's value. +The target variable is `_.proxy_mode`. + +.. seealso:: The `_.` is a :term:`relative path` notation and means the current path of the family you're currently in. + + In the python quasi algorithmic notation we could say that: + + .. code-block:: python + + _.proxy_mode == proxy_mode + + This is true only because in our use case `proxy_mode` is located on the root path. + + Have a look at the :ref:`variablepath` page for more details. + + +Now regarding the `when_not` parameter, this means that if the target variable's value +is `Manual proxy configuration` then the `manual` familiy **will not** be disabled +(that is, it will be **enabled**). + +Regarding as the default value use case, the `proxy_mode`'s variable is `No proxy` by default. +The `manual` familiy is then **disabled by default**. + +Let's launch the Rougail CLI on an empty user value file: + +.. raw:: html + :class: terminal + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_051/config/01/cmd_ro.txt + +.. + rougail -m firefox/ -u yaml -yf config/01/config.ym + +We have this output: + +.. raw:: html + :class: output + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_051/config/01/output_ro.html + +.. +
    ╭──────────────────── Caption ─────────────────────╮
    +    │ Undocumented but modified variable Default value │
    +    ╰──────────────────────────────────────────────────╯
    +    Variables:
    +    ┗━━ 📓 Configure Proxy Access to the Internet: No proxy
    +    
    + +We can see that the `manual` family and all the variables into it are not present. + +.. type-along:: Dynamically enabling the `manual` family + +Now we are going to choose the **manual mode**, that is the `Manual proxy configuration` +value for the `proxy_mode` variable, and things will become slightly different. + +If the manual mode for the proxy is not selected, then the `manual` family is disabled. +On the other hand, if the manual proxy's configuration mode is selected, +then the `manual` family **is enabled**: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_051/config/02/config.yml + :linenos: + :language: yaml + :caption: The `proxy_mode`'s manual setting in the :file:`config/02/config.yaml` user data file + +.. + --- + proxy_mode: Manual proxy configuration + manual: + http_proxy: + address: http.proxy.net + port: 3128 + use_for_https: false + +Let's launch the Rougail CLI to verify this: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_051/config/02/cmd_ro.txt + :class: terminal + +.. + rougail -m firefox/ -u yaml -yf config/02/config.yml + +It outputs: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_051/config/02/output_ro.html + :class: output + +.. +
    ╭────────────── Caption ───────────────╮
    +    │ Variable Modified value              │
    +    │          (⏳ Original default value) │
    +    ╰──────────────────────────────────────╯
    +    Variables:
    +    ┣━━ 📓 proxy_mode (Configure Proxy Access to the Internet): Manual proxy 
    +    configuration ◀ loaded from the YAML file "config/02/config.yml" (⏳ No
    +    proxy)
    +    ┗━━ 📂 manual (Manual proxy configuration)
    +        ┣━━ 📂 http_proxy (HTTP Proxy)
    +        ┣━━ 📓 address (HTTP address): http.proxy.net ◀ loaded from the YAML
    +        file "config/02/config.yml"
    +        ┗━━ 📓 port (HTTP Port): 3128 ◀ loaded from the YAML file
    +            "config/02/config.yml" (⏳ 8080)
    +        ┗━━ 📓 use_for_https (Also use this proxy for HTTPS): false ◀ loaded from
    +            the YAML file "config/02/config.yml" (⏳ true)
    +    
    + +.. rubric:: Explanation + +Here the `disabled` property **depends on** the value of the `proxy_mode` variable. +It is the `variable` parameter that allows you to define the name of the target variable on which the `disabled` property depends. + +Please remember that this activation/deactivation of the `manual` family +depends on the value of the `proxy_mode` variable. Here we have +chosen the `Manual proxy configuration` value, so the `address` and `port` variables appear. + +A hidden family +------------------- + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_052 ` version:: + + git switch --detach v1.1_052 + +Let's introduce a new property here: + +.. glossary:: + + hidden + + A variable or family's property is hidden if its value **shall not be seen** in a given :term:`context`. + Anyway, these variables can be used if the context evolves. + + This is the main difference between the `hidden` and the `disabled` properties: + + - with the `disabled` property, the variables are *deactivated* + - with the `hidden` property, the variables are just not seen when loading the user data. + +Now we can set a `hidden` property to the `https_proxy` family: + +Here is our new :file:`20-manual.yml` structure file: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_052/firefox/20-manual.yml + :linenos: + :language: yaml + :caption: The :file:`firefox/20-manual.yml` structure file with the `hidden` property on the `https_proxy` family. + +.. + %YAML 1.2 + --- + version: 1.1 + + manual: + + use_for_https: true # Also use this proxy for HTTPS + + https_proxy: + description: HTTPS Proxy + hidden: true + + address: + description: HTTPS address + default: + variable: __.http_proxy.address + + port: + description: HTTPS Port + default: + variable: __.http_proxy.port + ... + +.. confval:: https_proxy.address + :type: `domainname` + + This is an address setting for the manual HTTPS configuration + +.. confval:: https_proxy.port + :type: `port` + + This is a port setting for the manual HTTPS configuration + +We have now a `hidden` property assigned to the `https_proxy` family, which is +hiding these two variables. + +.. questions:: Why a `hidden` property? + + Here is a detailed explanation of this choice of the `hidden` property: + + We are in a use case where we want the HTTP mode configuration to be identical to the HTTPS mode configuration, so: + + - we need to have access to the HTTP mode configuration details + - this HTTP configuration will be duplicated using references to the default values, + so there is no need to request intervention from the :term:`operator` + to fill in the details of the default HTTPS configuration + - however, the :term:`operator` may want to have access to the HTTPS mode configuration if he wishes to, + in order to assign customised values to it. + Therefore, the :term:`operator` must obviously be able to access this HTTPS configuration. + +If we launch the Rougail CLI on the this user data + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_052/config/01/config.yml + :linenos: + :language: yaml + :caption: The :file:`config/01/config.yml` user data file with the `hidden` property on the `https_proxy` family. + +.. + --- + proxy_mode: Manual proxy configuration + manual: + http_proxy: + address: http.proxy.net + port: 3128 + use_for_https: false + https_proxy: + address: https.proxy.net + +Let's launch the Rougail in read only (`RO`) mode first: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_052/config/01/cmd_ro.txt + :class: terminal + +.. + rougail -m firefox/ -u yaml -yf config/01/config.yml + +We have this output: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_052/config/01/output_ro.html + :class: output + +.. +
    🔔 Warning
    +    ┗━━ manual (Manual proxy configuration)
    +        ┗━━ https_proxy (HTTPS Proxy)
    +            ┗━━ address (HTTPS address): 🔔 family "https_proxy" (HTTPS Proxy) has 
    +                property hidden, so cannot access to "address" (HTTPS address), it 
    +                will be ignored when loading from the YAML file 
    +                "config/01/config.yml"
    +
    +    ╭────────────── Caption ───────────────╮
    +    │ Variable Modified value              │
    +    │          (⏳ Original default value) │
    +    ╰──────────────────────────────────────╯
    +    Variables:
    +    ┣━━ 📓 proxy_mode (Configure Proxy Access to the Internet): Manual proxy 
    +    configuration ◀ loaded from the YAML file "config/01/config.yml" (⏳ No
    +    proxy)
    +    ┗━━ 📂 manual (Manual proxy configuration)
    +        ┣━━ 📂 http_proxy (HTTP Proxy)
    +        ┣━━ 📓 address (HTTP address): http.proxy.net ◀ loaded from the YAML
    +        file "config/01/config.yml"
    +        ┗━━ 📓 port (HTTP Port): 3128 ◀ loaded from the YAML file
    +            "config/01/config.yml" (⏳ 8080)
    +        ┗━━ 📓 use_for_https (Also use this proxy for HTTPS): false ◀ loaded from
    +            the YAML file "config/01/config.yml" (⏳ true)
    +    
    + +Now let's launch the Rougail CLI in read write mode (`RW`) on the same user data: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_052/config/01/cmd_rw.txt + :class: terminal + +We have this output: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_052/config/01/output_rw.html + :class: output + +.. type-along:: The read only mode view and the read write mode view + +.. FIXME: à unifier avec l'autre définition de ro et rw dans la page dédiée rougail/rw_ro_modes.html + +.. glossary:: + + read only mode + + We call this mode the `RO` mode. In this mode it is impossible to modify the values of the variables. + This is the standard configuration usage mode. + + read write mode + + We call this mode the `RW` mode. In this mode you can edit the variables, even these that have + been hidden or disabled. + +Why these modes ? In this way, the :term:`operator` or the :term:`integrator` don't have to +add or pop familiy properties each time we pass from one use (editing mode) to another +(configuration using mode) to an other. Swithching modes with setting properties is not a good idea. +It's better to change the read write and the read_only mode inside a Rougail CLI session. + +.. note:: During a standard Rougail CLI session, the default usage is the read only mode. + We can switch at any time tho the read write mode by adding the `--cli.read_write` + Rougail CLI parameter. + +In the both `RO` and `RW` modes of the Rougail CLI session, we have this warning: + +:: + + 🔔 Warning + ┗━━ manual (Manual proxy configuration) + ┗━━ https_proxy (HTTPS Proxy) + ┗━━ address (HTTPS address): 🔔 family "https_proxy" (HTTPS Proxy) has + property hidden, so cannot access to "address" (HTTPS address), it + will be ignored when loading from the YAML file + "config/01/config.yml" + +We are warned that the `https_proxy` family has the `hidden` property. + +Note that this is only in the read only mode that the variables that lives in the `https_proxy` familiy are +set as **unmodifiable variable**: + +:: + + ┗━━ 📂 https_proxy (HTTPS Proxy) + ┣━━ 📓 address (HTTPS address): http.proxy.net + ┗━━ 📓 port (HTTPS Port): 3128 + +It is logical that we don't have this unmodifiable setting in the read write mode, +because the read/write mode is designed to be an editing mode. + +.. questions:: Question: shall we use the `disabled` property here? + + Is it relevant to use the :term:`disabled property ` here? + + **answer**: No! Because we *need* to use these variables at any :term:`context` of the proxy's manual configuration use case, + we simply have to point their values ​​in one direction or another depending on this or that context, + that's why it is absolutely not a question of disabling them. The `manual.https_proxy.address` + and the `manual.http_proxy.port` variables shall not be disabled in the manual mode. + +A conditional hidden family +---------------------------- + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_053 ` version:: + + git switch --detach v1.1_053 + +Now we will focus on configuring the HTTPS mode in case of `"Manual proxy configuration"` value has been chosen, +let's have a look at our use case again: + +.. image:: images/firefox_manual_https.png + +Let's have a look at the HTPPS configuration corresponding structure file: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_053/firefox/20-manual.yml + :linenos: + :language: yaml + :caption: the :file:`firefox/20-manual.yml` structure file + +.. + %YAML 1.2 + --- + version: 1.1 + + manual: + + use_for_https: true # Also use this proxy for HTTPS + + https_proxy: + description: HTTPS Proxy + hidden: + variable: _.use_for_https + + address: + description: HTTPS address + default: + variable: __.http_proxy.address + + port: + description: HTTPS Port + default: + variable: __.http_proxy.port + ... + +We have added a new variable, named `use_for_https` here: + +.. confval:: use_for_https + :type: `boolean` + :default: `true` + + This is a setting that enables to reuse or not the HTTP proxy configuration for HTTPS + +The variable that drives the hidden/show behavior is the `use_for_https` variable because the `hidden` property has +a `variable` target parameter that points to it: `variable: _.use_for_https`. + +.. prerequisites:: Reminder + + The underscore and the point before the variable (`_.use_for_https`) points to the variable that lives in the same + family. + +Let's introduce a new Rougail concept here: + +.. glossary:: + + context + + A :term:`configuration` is highly statefull and can change at any moment. + Sometimes somes minor changes in the :term:`user data ` may involve chain reactions + in the whole :term:`configuration`. + The **context** is the state of the user data at one moment, the set of the values of the variables + at a given moment. + + This term refers in Rougail to the ability of a system to handle + the *statefull* state of a configuration. + It expresses the transition between one situation to another situation, + that is, the deeply **statefull** aspects of a data set. + +Do we want to reuse, for the HTTPS mode, the same configuration as for the HTTP mode? +Well, it depends on the :term:`context`. + +.. questions:: Question: how does it work? + + How will this variable drive the reuse of HTTP data to HTTPS data? + +With this :confval:`use_for_https` boolean variable, there are two possibilities, and only two: + +- The http proxy's configuration will be reused for the https proxy's configuration +- The http proxy's will not be reused for the https proxy's configuration + +Here is an example with different user values for handling HTTP and HTTPS: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_053/config/01/config.yml + :linenos: + :language: yaml + :caption: User data in the user data file :file:`config/01/config.yml` with `use_for_https` as false + +.. + --- + proxy_mode: Manual proxy configuration + manual: + http_proxy: + address: http.proxy.net + port: 3128 + use_for_https: false + https_proxy: + address: https.proxy.net + +If we launch the Rougail CLI: + +.. raw:: html + :class: terminal + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_053/config/01/cmd_ro.txt + +.. + rougail -m firefox/ -u yaml -yf config/01/config.yml + +We have this output: + +.. raw:: html + :class: output + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_053/config/01/output_ro.html + +.. +
    ╭────────────── Caption ───────────────╮
    +    │ Variable Default value               │
    +    │          Modified value              │
    +    │          (⏳ Original default value) │
    +    ╰──────────────────────────────────────╯
    +    Variables:
    +    ┣━━ 📓 proxy_mode (Configure Proxy Access to the Internet): Manual proxy 
    +    configuration ◀ loaded from the YAML file "config/01/config.yml" (⏳ No
    +    proxy)
    +    ┗━━ 📂 manual (Manual proxy configuration)
    +        ┣━━ 📂 http_proxy (HTTP Proxy)
    +        ┣━━ 📓 address (HTTP address): http.proxy.net ◀ loaded from the YAML
    +        file "config/01/config.yml"
    +        ┗━━ 📓 port (HTTP Port): 3128 ◀ loaded from the YAML file
    +            "config/01/config.yml" (⏳ 8080)
    +        ┣━━ 📓 use_for_https (Also use this proxy for HTTPS): false ◀ loaded from
    +        the YAML file "config/01/config.yml" (⏳ true)
    +        ┗━━ 📂 https_proxy (HTTPS Proxy)
    +            ┣━━ 📓 address (HTTPS address): https.proxy.net ◀ loaded from the YAML
    +            file "config/01/config.yml"
    +            ┗━━ 📓 port (HTTPS Port): 8080
    +    
    + +Notice that we have this `use_for_https` `boolean` type variable, its default value is `True`. +We want to offer the possibility of providing an identical or possibly different proxy configuration for the HTTP and for the HTTPS protocols. + +Here is an example with identical HTTP and HTTPS proxy configuration: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_053/config/02/config.yml + :linenos: + :language: yaml + :caption: User data in the user data file :file:`config/02/config.yml` with `use_for_https` as true + +.. + --- + proxy_mode: Manual proxy configuration + manual: + http_proxy: + address: http.proxy.net + port: 3128 + use_for_https: true + +Let's launch the Rougail CLI: + +.. raw:: html + :class: terminal + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_053/config/02/cmd_ro.txt + +.. + rougail -m firefox/ -u yaml -yf config/02/config.yml + +We have this output: + +.. raw:: html + :class: output + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_053/config/02/output_ro.html + +.. +
    🛑 Caution
    +    ┗━━ manual (Manual proxy configuration)
    +        ┗━━ https_proxy (HTTPS Proxy)
    +            ┗━━ address (HTTPS address): 🛑 mandatory variable but is inaccessible 
    +                and has no value
    +    
    + +Which is logical, HTTPS proxy variables have no values set yet. +We are going to see how to point HTTPS variables to HTTP variables. + +.. keypoints:: Key points progress + + **summary** + + We have now the ability to build *contextual settings*: + + - if the `proxy_mode` variable's value is not `'Manual proxy configuration'` the `manual` family is disabled + - if the `proxy_mode` variable's value is `'Manual proxy configuration'` then the `manual` family is enabled + - if the `use_for_https` variable's value is `true`, the HTTP configuration will be reused in the HTTPS situation + and the `https_proxy` family will be hidden + - if the `manual.https_proxy.address` has no value set, the defaut value is the same as the `manual.http_proxy.address`'s value + (same behavior with the HTTP port and the HTTPS port variable) + + And yes, we did it. We have arrived at the end of the proxy's manual configuration's section. + + **Keywords** + + - We now know what a *property* is, we have seen in details the :term:`disabled` property, + - We can target a variable's value in the `disabled` property's value, + we call it a variable based contextual disabled family, + - The :term:`hidden` property can be set to a family, + - The fact that a property can be set dynamically, + - The conditional dependency of a `hidden` property can depends on a `boolean` variable, + - We know what a calculated default value is. + diff --git a/docs/tutorial/propertyerror.rst b/docs/tutorial/propertyerror.rst new file mode 100644 index 000000000..74884fcac --- /dev/null +++ b/docs/tutorial/propertyerror.rst @@ -0,0 +1,57 @@ +.. _tutorial_propertyerror: + +The Jinja propertyerror test +============================ + +.. objectives:: Objectives + + In a :ref:`Jinja-type calculation `, it is possible to test if a variable is accessible (there is no :ref:`access_control` problem). + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tag :tutorial:`v1.1_140 ` in the repository: + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach v1.1_140 + +For now, the variable `proxy_dns_socks5` defines these access controls as follows: + +- if the `proxy_mode` variable is not set to `"Manual proxy configuration"` +- if socks version is `v4`. + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_131/firefox/55-proxy_dns_socks5.yml + :language: yaml + :caption: The :file:`firefox/55-proxy_dns_socks5.yml` + +In principle it works, but is that really what we want? We mean, do we want to disable it based on the proxy_mode variable? + +No, we actually want to make this variable available if the socks version is accessible and equals v4. + +Disabled a variable when an other variable is disabled +------------------------------------------------------- + +The most suitable syntax in Jinja for checking the status of a variable is to use the `concept of a test `_. + +Rougail offers a custom test that will verify correct access to the variable. + +This test is call `propertyerror`. + +Here is how one could write the Jinja using this feature: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_140/firefox/55-proxy_dns_socks5.yml + :language: yaml + :caption: The :file:`firefox/55-proxy_dns_socks5.yml` structure file with some Jinja code in the `hidden` property + +.. keypoints:: Key points + + We have seen that we can use the test `propertyerror` in Jinja template. diff --git a/docs/tutorial/redefine.rst b/docs/tutorial/redefine.rst new file mode 100644 index 000000000..f3ea5865e --- /dev/null +++ b/docs/tutorial/redefine.rst @@ -0,0 +1,110 @@ +.. _tutorial_redefine: + +Redefine variable +================= + +.. objectives:: Objectives + + When we said that a :ref:`variable is immutable ` when an :term:`integrator` defined it in structured data, we said that it had to be done explicitly. + Let's see how that works. + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tags :tutorial:`v1.1_240 ` to :tutorial:`v1.1_241 ` + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach v1.1_240 + +Username is mandatory if a password is set +------------------------------------------ + +We now have two new variables following the same pattern as the :ref:`previous three one `: + +- A variable with the new type `unix_username` `username`, this variable is accessible in type HTTP, HTTPS/SSL, SOCKS4 or SOCKS5 +- A variable with the new type `secret` `password`, this variable is accessible in type HTTP, HTTPS/SSL, SOCKS4 or SOCKS5 + +Nothing really new from now. But we haven't said everything! +In fact, we have to deal with a new situation. We have three :ref:`coherence ` :ref:`configuration ` choices: + +- either we fill in the username/password pair +- we fill only the username +- or leave it blank + +Defining only a password makes no sense. + +There are several ways to solve this case. +Let's assume that if the :term:`operator` sets a password, they must also enter a username. The password can be `null`. + +In Rougail format, we would say that the password is :ref:`nullable ` and that the username has a :ref:`variable calculation ` to determine if it is :ref:`nullable `. +If the password is empty, the username is :ref:`nullable `; otherwise, it is not. + +Here's how to do it: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_240/foxyproxy/00-foxyproxy.yml + :language: yaml + :caption: The :file:`foxyproxy/00-foxyproxy.yml` + +Now let's test with the following user data file: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_240/config/02/config.yml + :linenos: + :language: yaml + :caption: The :file:`config/02/config.yml` user data + +If we launch the Rougail CLI: + +.. raw:: html + :class: terminal + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_240/config/02/cmd_ro.txt + +We have this output: + +.. raw:: html + :class: output + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_240/config/02/output_ro.html + +.. note:: + + The value of a variable of type secret is replaced by asterisks (`*`) when it is displayed. + +Redefine it to password is hidden if an username is not set +----------------------------------------------------------- + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_241 ` version:: + + git switch --detach v1.1_241 + +Another solution is to say to ourselves: the username is never mandatory (just like the password) but the password is hidden if the username is empty. + +We could modify the structure file as we have been doing since the beginning of this tutorial. +Perhaps it's time to illustrate how to :term:`redefine` a variable. + +Redefine means that the variable is unify (combine) from multiple variables declarations. + +So we will create a new file with, inside, the same variable name. + +First of all, we will to redefine the mandatory attribute of the `username` variable. + +Secondly we will added a new calculation in the :ref:`hidden` parameters. + +Here is the final result of this file: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_241/foxyproxy/10-redefine.yml + :language: yaml + :caption: The :file:`foxyproxy/10-redefine.yml` + +.. keypoints:: Key points + + Redefine a variable involves changing some of its parameters with new ones. diff --git a/docs/tutorial/regexp.rst b/docs/tutorial/regexp.rst new file mode 100644 index 000000000..17860011b --- /dev/null +++ b/docs/tutorial/regexp.rst @@ -0,0 +1,126 @@ +.. _tutorial_regexp: + +Regexp type with calculation +============================ + +.. objectives:: Objectives + + Regexp is a :ref:`variable specialize type ` which allows for easy validation of a variable from a regular expression. + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tags :tutorial:`v1.1_200 ` to :tutorial:`v1.1_201 ` + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach v1.1_200 + +A regexp variable +----------------- + +We have already seen the :ref:`validator mechanism ` for managing the :ref:`quality of data values `. + +This mechanism is undeniably interesting and powerful. +But for strings that can be validated using a regular expression, it is preferable to use this type. +In this specific case, nothing beats this particular type. + +Obviously, the values assigned to a regexp type variable must be of type string. + +Here we will add a `color` variable which will allow us to quickly identify the current proxy via a color code. The expected value is a hex color code (like those used in CSS or HTML). + +This regrex must validate that the string begins with a hash symbol (#) followed by a RGB triplet in hexadecimal format (so 3 or 6 lowercase hexadecimal digits (0–9, a–f)). + +Here are some examples: `#a3f09c` or `#a3f`. + +The regular expression is `^#(?:[0-9a-f]{3}){1,2}$`. + +Let's write this variable: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_200/foxyproxy/00-foxyproxy.yml + :language: yaml + :caption: The regexp variable `color`. + +Now let's test with the following user data file: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_200/config/02/config.yml + :linenos: + :language: yaml + :caption: The :file:`config/02/config.yml` user data + +With this Rougail CLI: + +.. raw:: html + :class: terminal + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_200/config/02/cmd_ro.txt + +We have this output: + +.. raw:: html + :class: output + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_200/config/02/output_ro.html + +What happens if the value is not valid? + +The value is invalid: + +.. raw:: html + :class: output + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_200/config/03/output_ro.html + +Calculate a random color +------------------------ + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_201 ` version:: + + git switch --detach v1.1_201 + +Why not offer a default value to the user? + +If the user has a specific color in mind, they can specify it. Otherwise, Rougail will suggest a default value. + +We've already seen how to use :ref:`Jinja ` for :ref:`validation ` and :ref:`access control ` calculations. + +We've also seen how to calculate the default value by :ref:`retrieving the value of another variable `. + +It's entirely possible to calculate the default value from a Jinja template: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_201/foxyproxy/00-foxyproxy.yml + :language: yaml + :caption: Calculate the default variable `color`. + +Now let's test with the following user data file: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_201/config/02/config.yml + :linenos: + :language: yaml + :caption: The :file:`config/02/config.yml` user data + +With this Rougail CLI: + +.. raw:: html + :class: terminal + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_201/config/02/cmd_ro.txt + +We have this output: + +.. raw:: html + :class: output + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_201/config/02/output_ro.html + +As you can see, Rougail offers a default value! + +.. keypoints:: Key points + + - We can easily create a variable that valid value via a regexp + - We can calculate a default value with Jinja template diff --git a/docs/tutorial/sequence.rst b/docs/tutorial/sequence.rst new file mode 100644 index 000000000..1aecec1e5 --- /dev/null +++ b/docs/tutorial/sequence.rst @@ -0,0 +1,238 @@ +Homogeneous elements sequence +================================== + +.. objectives:: Objectives + + We are going to discuss a new Rougail type, :term:`the sequence of homogeneous elements ` + which we will call, more concisely, a :term:`sequence`. It's a rather unusual :term:`family`. + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tag :tutorial:`1.1_190 ` + in the repository. + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach 1.1_190 + +As a reminder, we are now talking about the Firefox add-on called +`Foxy Proxy `_ use case, +here is the Foxy Proxy widget: + +.. image:: images/foxyproxy.png + +A family with sequence type +------------------------------ + +.. note:: To be precise, we should say: a :term:`family with a sequence of homogeneous elements type `. + +Our request here is to have a list of homogeneous objects. +Indeed, if we click on "Add proxy" in our Foxy Proxy widget, we will need to specify: + +- a proxy type +- a color + +(and also enter some other values). +And this applies to each proxy we will define. + +Here is the structure file associated with this use case: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_190/foxyproxy/00-foxyproxy.yml + :language: yaml + :caption: The FoxyProxy structure file :file:`foxyproxy/00-foxyproxy.yml` with the `proxies` sequence variable + +.. + %YAML 1.2 + --- + version: 1.1 + + proxies: + description: Proxy configuration + type: sequence + + title: + description: Title or Description + mandatory: false + + color: # Color + ... + +Unsurprisingly, the way to assign the `sequence` type is done with the help of the `type` parameter. +We see here that `proxies` is a family because it contains the variables `title` and `color`. + +There is nothing new to say to this two variable: + +- `title` is not mandatory string +- `color` is just a string + +.. note:: + + The first variable of the :term:`sequence` is not mandatory. + This does not mean that the variable is :ref:`tutorial_nullable` or that the :ref:`list is empty `. + In this case, that means that the sequence is empty. + +Which we could write in JSON like this: + + .. code-block:: json + + [] + +Now configure the sequence: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_190/config/02/config.yml + :language: yaml + :caption: A :file:`config/02/config.yml` user data file with the `proxies` sequence + +.. + --- + foxyproxy: + proxies: + - title: My company + color: '#66cc66' + +We can see in the :term:`user data` YAML file that the `proxies` value is a list of mappings. + +Which we could write in JSON like this: + + .. code-block:: json + + [ + { + "title": "My company", + "color": "#66cc66" + } + ] + +So we can clearly see in the JSON that if there's a `title` key, then there will also be a `color` key. + +This is guaranteed by our :term:`sequence` type. + +Let's launch our Rougail CLI: + +.. raw:: html + :class: terminal + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_190/config/02/cmd_ro.txt + +.. + rougail -m firefox/ -s Firefox -xn FoxyProxy -xd 0 foxyproxy/ --types types/proxy --modes_level basic standard advanced -u yaml -yf config/02/config.yml + +And the Rougail output is: + +.. raw:: html + :class: output + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_190/config/02/output_ro.html + +.. + Variables: + ┣━━ 📂 firefox (Firefox) + ┃ ┣━━ 📓 proxy_mode (Configure Proxy Access to the Internet): No proxy + ┃ ┗━━ 📂 dns_over_https (DNS over HTTPS) + ┃ ┗━━ 📓 enable_dns_over_https (Enable DNS over HTTPS): false + ┗━━ 📂 foxyproxy (FoxyProxy) + ┗━━ 📂 proxies (Proxy configuration) + ┗━━ 📂 title (Title or Description) + ┣━━ 📓 title (Title or Description): My company ◀ loaded from the + ┃ YAML file "config/02/config.yml" + ┗━━ 📓 color (Color): #66cc66 ◀ loaded from the YAML file + "config/02/config.yml" + +.. type-along:: A sequence with multiple user data + +We will now look at the same `proxies` sequence variable, +to which multiple values ​​are assigned: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_190/config/03/config.yml + :language: yaml + :caption: A :file:`config/03/config.yml` user data file with multiple values + +.. + --- + foxyproxy: + proxies: + - title: My company + color: '#66cc66' + - title: An other company + color: '#cc66cc' + - title: WPAD + color: '#1166cc' + + +Which we could write in JSON like this: + + .. code-block:: json + + [ + { + "title": "My company", + "color": "#66cc66" + }, + { + "title": "An other company", + "color": "#cc66cc" + }, + { + "title": "WPAD", + "color": "#1166cc" + } + + ] + + +And now we can clearly see that this `proxies` variable of type `sequence` +is a container for lists of values ​​of the same shape, +and which contain the same number of elements: + +.. raw:: html + :class: output + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_190/config/03/output_ro.html + +.. + Variables: + ┣━━ 📂 firefox (Firefox) + ┃ ┣━━ 📓 proxy_mode (Configure Proxy Access to the Internet): No proxy + ┃ ┗━━ 📂 dns_over_https (DNS over HTTPS) + ┃ ┗━━ 📓 enable_dns_over_https (Enable DNS over HTTPS): false + ┗━━ 📂 foxyproxy (FoxyProxy) + ┗━━ 📂 proxies (Proxy configuration) + ┣━━ 📂 title (Title or Description) + ┃ ┣━━ 📓 title (Title or Description): My company ◀ loaded from the + ┃ ┃ YAML file "config/03/config.yml" + ┃ ┗━━ 📓 color (Color): #66cc66 ◀ loaded from the YAML file + ┃ "config/03/config.yml" + ┣━━ 📂 title (Title or Description) + ┃ ┣━━ 📓 title (Title or Description): An other company ◀ loaded from + ┃ ┃ the YAML file "config/03/config.yml" + ┃ ┗━━ 📓 color (Color): #cc66cc ◀ loaded from the YAML file + ┃ "config/03/config.yml" + ┗━━ 📂 title (Title or Description) + ┣━━ 📓 title (Title or Description): WPAD ◀ loaded from the YAML + ┃ file "config/03/config.yml" + ┗━━ 📓 color (Color): #1166cc ◀ loaded from the YAML file + "config/03/config.yml" + +.. seealso:: + + To go further, you can have a look at the Rougail format pages :ref:`Sequence of homogeneous elements ` + +.. keypoints:: let's review the key points + + **Keywords** + + Sequence. + We saw a :term:`sequence` type family, which full name is :ref:`Sequence of homogeneous elements `. + + **Comments** + + What is remarkable here is that we are dealing with a new form of :term:`consistency` + which focuses on variables that are similar within a family, + not because they have the same type, but because, as they are lists, + they must have the same number of elements. diff --git a/docs/tutorial/types.rst b/docs/tutorial/types.rst new file mode 100644 index 000000000..4a3c7e446 --- /dev/null +++ b/docs/tutorial/types.rst @@ -0,0 +1,461 @@ +Some suitable types +===================== + +.. objectives:: Objectives + + There isn't just the basic `string` available type, here we will discover new + variable types, for example the `boolean` type, and even types that are much more + suited to our use case, such as `domainname` or `port`. + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tags :tutorial:`v1.1_030 ` to :tutorial:`v1.1_033 ` + in the repository. + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach v1.1_030 + +.. type-along:: Let's recap how far we've come + +We have an `http_proxy` family with an `address` variable in it. + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_022/firefox/10-manual.yml + :language: yaml + :caption: An `address` variable in the `http_proxy` family + +.. + manual: # Manual proxy configuration + http_proxy: # HTTP Proxy + address: + description: HTTP address + +:tutorial:`Download this file from the rougail-tutorials git repository ` + +.. _tutorial_domainname: + +A variable with type `domainname` +----------------------------------- + +We will add a business types to our `address` variable: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_030/firefox/10-manual.yml + :language: yaml + :caption: An `address` variable in the `http_proxy` family + +.. + manual: # Manual proxy configuration + http_proxy: # HTTP Proxy + address: + description: HTTP address + type: domainname + +:tutorial:`Download this file from the rougail-tutorials git repository ` + +Notice that with this `type: domainname` we have assigned the `domainname` business type to this variable. +Assigning a type is convenient for reading, but what else does it bring? + +Well, with a correct user data like this one: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_030/config/01/config.yml + :language: yaml + :caption: A domain name user data setting + +.. + --- + manual: + http_proxy: + address: net.example + +:tutorial:`Download this file from the rougail-tutorials git repository ` + +if we launch the Rougail CLI on it: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_030/config/01/cmd_ro.txt + :class: terminal + +.. + rougail -m firefox/ -u yaml -yf config/01/config.yml + +We have this output: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_030/config/01/output_ro.html + :class: output +.. +
    ╭──────── Caption ────────╮
    +    │ Variable Default value  │
    +    │          Modified value │
    +    ╰─────────────────────────╯
    +    Variables:
    +    ┣━━ 📓 Configure Proxy Access to the Internet: No proxy
    +    ┗━━ 📂 Manual proxy configuration
    +        ┗━━ 📂 HTTP Proxy
    +            ┗━━ 📓 HTTP address: example.net ◀ loaded from the YAML file 
    +                "config/01/config.yml"
    +    
    + + +And we don't really see any change associated with the fact that we have assigned +a type to this variable. But if we assign this (wrong) user data: + +.. type-along:: A domain name has no space in it + +Let's have a look at an example of user setting that does not fit the +`domainname` type: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_030/config/03/config.yml + :language: yaml + :caption: An invalid domain name for the :file:`config/03/config.yml` user data setting + +.. + --- + manual: + http_proxy: + address: bla bla + +:tutorial:`Download this file from the rougail-tutorials git repository ` + +The value is obviously not a domain name, then when we will launch the Rougail CLI: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_030/config/03/cmd_invalid.txt + :class: terminal + +.. + rougail -m firefox/ -u yaml -yf config/03/config.yml --cli.invalid_user_data_error + +we then have this output: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_030/config/03/output_invalid.html + :class: error + +.. +
    🛑 ERRORS
    +    ┗━━ the value "bla bla" is an invalid domain name for 
    +        "manual.http_proxy.address" (HTTP address), must not be an IP, it will be 
    +        ignored when loading from the YAML file "config/02/config.yml"
    +    
    + +A variable with type's parameters +------------------------------------- + +What if we set an IP address instead of a domain name? + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_030/config/02/config.yml + :language: yaml + :caption: A domain name in the :file:`config/02/config.yml` user data setting with an IP address + +.. + --- + manual: + http_proxy: + address: 19.168.230.51 + +:tutorial:`Download this file from the rougail-tutorials git repository ` + +With a value that *is not a domain name* but an IP address, then when we will launch the Rougail CLI +we will see a little problem: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_030/config/02/cmd_invalid.txt + :class: terminal + +.. + rougail -m firefox/ -u yaml -yf config/02/config.yml --cli.invalid_user_data_error + +we then have this output: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_030/config/02/output_invalid.html + :class: error + +.. +
    🛑 ERRORS
    +    ┗━━ the value "192.168.0.1" is an invalid domain name for 
    +        "manual.http_proxy.address" (HTTP address), must not be an IP, it will be 
    +        ignored when loading from the YAML file "config/02/config.yml"
    +    
    + +We observe that an error has been raised because an IP address is not a domain name. +Therefore, a type validation is taking place because we declared the type `domainname`. + +.. questions:: Question + + OK I agree with the `domainname` necessary type validation, but what if I want to specify + an IP address as a user value for this `address` variable? + Because it is therefore simply impossible to do so now. + + Is there a way for my `address` variable to accept an IP address? + +Well, it is possible to configure the type so that it accepts IP addresses. +We need to specify whether our variable accepts to be filled using an IP or a domain name only. +This is where the ability to parameterize our variable comes in. + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_031 ` version:: + + git switch --detach v1.1_031 + +Let's add a type parameter named `allow_ip`: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_031/firefox/10-manual.yml + :language: yaml + :caption: The `allow_ip` type parameter set in the :file:`firefox/10-manual.yml` structure file + :linenos: +.. + --- + manual: + description: Manual proxy configuration + + http_proxy: + description: HTTP Proxy + + address: + description: HTTP address + type: domainname + params: + allow_ip: true + +The params allow the domain name `address` variable to be set with IPs. + +.. glossary:: + + type parameter + + A type parameter is a parameter of a variable that can refine its behavior. + It is declared by adding the `params` attribute in the variable's + definition. + +Now we will test with an IP address as the value for our `address` variable. + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_031/config/02/config.yml + :language: yaml + :caption: An IP address as a value in the :file:`config/02/config.yml` user value + +.. + --- + manual: + http_proxy: + address: 192.168.0.1 + +:tutorial:`Download this file from the rougail-tutorials git repository ` + +if we launch the Rougail CLI on it: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_031/config/02/cmd_ro.txt + :class: terminal + +.. + rougail -m firefox/ -u yaml -yf config/02/config.yml + +We have this output: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_031/config/02/output_ro.html + :class: output + +We can see that the IP address value has been accepted. + +.. _tutorial_port: + +A variable with type `port` +------------------------------ + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_032 ` version:: + + git switch --detach v1.1_032 + +After the `address` variable let's add, according to our use case, a new variable of type `port`: + +.. image:: images/firefox_port.png + +Our structure file looks like this: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_032/firefox/10-manual.yml + :language: yaml + :caption: The `port` type variable in the :file:`firefox/10-manual.yml` structure file + :linenos: +.. + port: + description: HTTP Port + type: port + default: 8080 + +:tutorial:`Download this file from the rougail-tutorials git repository ` + +Let's assign a value to this port: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_032/config/02/config.yml + :language: yaml + :caption: A user data :file:`config/02/config.yml` setting a value to the port variable + +.. + proxy_mode: Manual proxy configuration + manual: + http_proxy: + address: example.net + port: 3128 + +:tutorial:`Download this file from the rougail-tutorials git repository ` + +If we launch the Rougail CLI: + +.. raw:: html + :class: terminal + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_032/config/02/cmd_ro.txt + +We have this output: + +.. raw:: html + :class: output + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_032/config/02/output_ro.html + +.. +
    ╭─────────────────────────── Caption ────────────────────────────╮
    +    │ Variable                           Modified value              │
    +    │ Undocumented but modified variable (⏳ Original default value) │
    +    ╰────────────────────────────────────────────────────────────────╯
    +    Variables:
    +    ┣━━ 📓 Configure Proxy Access to the Internet: Manual proxy configuration ◀ 
    +    loaded from the YAML file "config/02/config.yml" (⏳ No proxy)
    +    ┗━━ 📂 Manual proxy configuration
    +        ┗━━ 📂 HTTP Proxy
    +            ┣━━ 📓 HTTP address: example.net ◀ loaded from the YAML file 
    +            "config/02/config.yml"
    +            ┗━━ 📓 HTTP Port: 3128 ◀ loaded from the YAML file 
    +                "config/02/config.yml" (⏳ 8080)
    +    
    + +How can we know what validations the port type performs on the value assigned to the variable? + +There are a number of validations that are carried out with this `port` type. +Now let's assign a value that is outside the allowed ports: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_032/config/03/config.yml + :language: yaml + :caption: A user value in :file:`config/03/config.yml` that is not allowed + +:tutorial:`Download this file from the rougail-tutorials git repository ` + +Again, we launch the Rougail CLI: + +.. raw:: html + :class: terminal + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_032/config/03/cmd_invalid.txt + +.. rougail -m firefox/ -u yaml -yf config/03/config.yml --cli.invalid_user_data_error + +And we have this output: + +.. raw:: html + :class: error + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_032/config/03/output_invalid.html + +.. +
    🛑 ERRORS
    +    ┗━━ the value "100000" is an invalid port for "manual.http_proxy.port" (HTTP 
    +        Port), must be between 1 and 65535, it will be ignored when loading from the
    +        YAML file "config/03/config.yml"
    +    
    + +We observe that, as with the `domainname` type, a number of validations are performed +to ensure that the value assigned to this variable conforms to the `port` type. + +.. _tutorial_boolean_variable: + +A variable with type `boolean` +----------------------------------- + +.. type-along:: For those who follow the tutorial with the help of the git repository + + Now you need to checkout the :tutorial:`v1.1_033 ` version:: + + git switch --detach v1.1_033 + +Let's add one more variable in the `manual` family, with a much more basic type: `boolean`. + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_033/firefox/20-manual.yml + :language: yaml + :caption: A new structure file :file:`firefox/20-manual.yml` with one variable + +.. + %YAML 1.2 + --- + version: 1.1 + + manual: + + use_for_https: true # Also use this proxy for HTTPS + ... + +:tutorial:`Download this file from the rougail-tutorials git repository ` + +.. note:: + + - it is not necessary to declare the variable as a boolean type, + the type is *inferred* by the presence of the `true` default value + + - we have decided to create a new structure file :file:`firefox/20-manual.yml`. + This is not necessary but usefull, please have a look at the :ref:`structure file organization and naming conventions ` + + - here we reuse the `manual` existing family name that has already been declared in the :file:`firefox/10-manual.yml` structure file. + The content in the :file:`firefox/20-manual.yml` will be added to the existing `manual` family. + Note that if two different structure files are loaded by Rougail and if they declare the same family name, + then **the declarations are concatenated in the family name**. + +Let's switch this boolean variable to a `false` value, to do this we will add +this :file:`config/02/config.yml` user data file in the :term:`configuration`: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_033/config/02/config.yml + :language: yaml + :caption: A :file:`config/02/config.yml` user data file with false as a value for `use_for_https` + +.. + --- + manual: + http_proxy: + address: example.net + use_for_https: false + +:tutorial:`Download this file from the rougail-tutorials git repository ` + + +Let's run the Rougail CLI: + +.. raw:: html + :class: terminal + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_033/config/02/cmd_ro.txt + +.. + rougail -m firefox/ -u yaml -yf config/02/config.yml + +The result is: + +.. raw:: html + :class: output + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_033/config/02/output_ro.html + +We could of course perform several tests on type validators, but here the validation is simple: only two values ​​are allowed (`true` or `false`). + +.. keypoints:: let's review the key points + + - we can assign a `domainname` type to a variable + - we can set a :term:`type parameter` to a `domainname` variable to refine their typing behavior + - we can assign a `port` type to a variable + - we know how to set a `boolean` type variable to `true` or `false` diff --git a/docs/tutorial/underscore_parameter.rst b/docs/tutorial/underscore_parameter.rst new file mode 100644 index 000000000..c9e926251 --- /dev/null +++ b/docs/tutorial/underscore_parameter.rst @@ -0,0 +1,70 @@ +Name and parameter conflict +================================ + +.. objectives:: Objectives + + When a variable name conflicts with a family parameter, + do we have to abandon the variable name choice we made? + No because having a suitable variable name is really important. + The variable name you chose is undoubtedly the best. + + Sometimes, the choice of a variable's name may correspond very exactly + to an parameter of the family in which that variable is placed... + + In this section, we will learn how to create a variable name that conflict with a defined parameter name. + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tag :tutorial:`v1.1_220 ` + in the repository. + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach v1.1_220 + +.. type-along:: Let's recap how far we've come + +We have this homogeneous elements type family in its structure definition file: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_210/foxyproxy/00-foxyproxy.yml + :linenos: + :language: yaml + :caption: The `proxies` family with `sequence` type in the :file:`foxyproxy/00-foxyproxy.yml` structure file + +The variable "type" that conflits with the family parameter "type" +---------------------------------------------------------------------------- + +Choice a good variable name is important. It's with this name that user will interact with here value. + +The user has to define the type of the new proxy. So instinctively the name of the variable will be `type`. + +But in the current family we have already the parameter `type` with the `sequence` value. +YAML do not permit to have two key with the same name. +Maybe we could choice an other variable name like `proxy_type` but the full path will be `proxies.proxy_type`. +Repeating the word proxy is not appropriate. + +So the best way it to rename the `type` parameter to `_type` + +Let's create a family named `manual` which obviously corresponds to the proxy's manual configuration choice. + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_220/foxyproxy/00-foxyproxy.yml + :language: yaml + :caption: The `proxies` family with `sequence` type and a variable with the name `type` in the :file:`foxyproxy/00-foxyproxy.yml` structure file + +Technically it's possible to put `_` at the beginning of each parameter name, but it's still less readable. + +.. keypoints:: Let's review the key points + + **Keywords** + + - parameters could start by "_" character or not + - do not add `_` in front of all parameters name diff --git a/docs/tutorial/validators.rst b/docs/tutorial/validators.rst new file mode 100644 index 000000000..fd0044018 --- /dev/null +++ b/docs/tutorial/validators.rst @@ -0,0 +1,120 @@ +.. _tutorial_validators: + +Validating a variable's value +================================ + +.. objectives:: Objectives + + We will see how to create strong validations for the values of our variables, + that is, validations that go beyond those inherent in the initial type definition. + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tag :tutorial:`v1.1_170 ` + in the repository. + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach v1.1_170 + +A variable with custom validation +----------------------------------- + +First, we would like the :term:`operator` to remember not to forget the HTTPS protocol, that is to put `https://` at the beginning of our `web_address` variable's value. +Actually the `web_address`\ 's variable existence *requires* that the address include the HTTPS protocol, namely: + +:: + + http(s):// + +However, in our use case, we want *to force* the HTTPS protocol. +We don't want `http://`, only `https://`. + +.. note:: This URL is for DNS over HTTPS, providing greater privacy than using the ISP's DNS, + which can filter traffic. If it uses `http://`, it doesn't make sense. + +But there’s no existing type or types parameters for this specific use case (forcing HTTPS). + +We just said that a valid value must start with `https://` instead of `http://`. +It is possible, instead of creating a new type, to add custom validation of the variable's value. + +We call it a :term:`validator`. + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_170/firefox/60-dns_over_https.yml + :language: yaml + :caption: The :file:`firefox/60-dns_over_https.yml` with the jinja validator + +Here we can see that the validation code is: + +.. code-block:: jinja + + {{ _.custom_dns_url.startswith("http://") }} + +First of all, you can see that we use the path of the current variable for validation. + +A validator can return a string, in fact it is its default behavior. +If it returns a string of characters, it indicates the validation error. + +In this case we see that we have specified the `return_type parameter`, which +specifies that the Jinja code shall return a boolean value. + +The validator's `description` parameter serves two purposes: + +- it does, of course, have a string function for documenting validation, +- since the `return_type` parameter is set to boolean, + the error message contains this description's string. + +.. type-along:: example with `http://` in the user data + +Now, let's try with incorrect user data, containing `http://` and not `https://` + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_170/config/03/config.yml + :language: yaml + :caption: A :file:`config/03/config.yml` user data file containing `http://` + +.. + --- + dns_over_https: + enable_dns_over_https: true + provider: Custom + custom_dns_url: http://dns.net + + +Let's run the Rougail CLI: + +.. raw:: html + :class: terminal + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_170/config/03/cmd_ro.txt + +.. + rougail -m firefox/ --types types/proxy --modes_level basic standard advanced -u yaml -yf config/03/config.yml + +The result is: + +.. raw:: html + :class: output + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/tag/v1.1_170/config/03/output_ro.html + +The warning indicates that the validator did not allow an HTTP protocol to pass: + +:: + + ┃ the value "http://dns.net" is an invalid URL, must starts with + ┃ 'https://' only, it will be ignored when loading from the YAML file + ┃ "config/03/config.yml" + +.. keypoints:: Let's review the key points + + We explained the use of validations to refine the type definition of a variable. + + The variable's parameter is `validator`. + diff --git a/docs/tutorial/variable_propertyerror.rst b/docs/tutorial/variable_propertyerror.rst new file mode 100644 index 000000000..50e295ab9 --- /dev/null +++ b/docs/tutorial/variable_propertyerror.rst @@ -0,0 +1,121 @@ +.. _tutorial_var_propertyerror: + +Variable calculation with propertyerror +======================================= + +.. objectives:: Objectives + + In a :ref:`variable-type calculation `, it is possible to test if a variable is accessible (there is no :ref:`access_control` problem). + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tag :tutorial:`v1.1_230 ` in the repository: + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach v1.1_230 + +We have now to add three new variables: + +- :ref:`tutorial_domainname` `address` that allow IP and hostname, this variable is accessible in type HTTP, HTTPS/SSL, SOCKS4 or SOCKS5 +- :ref:`tutorial_port`, this variable is accessible in type HTTP, HTTPS/SSL, SOCKS4 or SOCKS5 +- :ref:`tutorial_web_address` URL, this variable is accessible in type PAC URL or WPAD + +.. image:: images/foxyproxy.png + +There is a particularity to these three variables. +We want to copy the values of the equivalent variables available in Firefox. +The reason is obvious; it simplifies data entry. + +Variables are not always available. It is possible to test their presence in a :ref:`Jinja template `, as we have :ref:`already seen `. +But it's not the most effective. It is preferable to perform a :ref:`variable-type calculation ` using the property `propertyerror` to `false`. + +This property allows you to disable errors in case of :ref:`access problems `. + +Let's start by modifying the structure file: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_230/foxyproxy/00-foxyproxy.yml + :language: yaml + :caption: The :file:`foxyproxy/00-foxyproxy.yml` + +If the variable is accessible, there's no question about it; the variable retrieves its value. But what happens if the variable is not accessible? + +In this case, the value of the variable is `null`. + +Don't just take my word for it, let's do a test. + +Now let's test with the following user data file: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_230/config/02/config.yml + :linenos: + :language: yaml + :caption: The :file:`config/02/config.yml` user data + +The proxy in the Firefox :ref:`namespace ` is not defined, so the address and port variables are not accessible. + +A proxy is configured in FoxyProxy, but also without address or port. + +If we launch the Rougail CLI: + +.. raw:: html + :class: terminal + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_230/config/02/cmd_ro.txt + +We have this output: + +.. raw:: html + :class: output + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_230/config/02/output_ro.html + +As expected, calculating the default value works without issue even though the variables involved in the calculations are inaccessible. + +The values are therefore `null`. These variables are not :ref:`nullable `, so the :ref:`overall coherence ` of the :ref:`configuration` is not guaranteed. + +Now try it in the :term:`read write mode` with the Rougail CLI: + +.. raw:: html + :class: terminal + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_230/config/02/cmd_rw.txt + +The variables has really the value `null`: + +.. raw:: html + :class: output + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_230/config/02/output_rw.html + +Now let's configure the variables in the Firefox :ref:`namespace `: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_230/config/04/config.yml + :linenos: + :language: yaml + :caption: The :file:`config/04/config.yml` user data + +As you can see, the address and port are defined in the Firefox :ref:`namespace ` but not in the FoxyProxy :ref:`namespace `. + +If we launch the Rougail CLI: + +.. raw:: html + :class: terminal + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_230/config/04/cmd_ro.txt + +We have this output: + +.. raw:: html + :class: output + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_230/config/04/output_ro.html + +As expected, the Firefox :ref:`namespace ` values are correctly copied into the FoxyProxy :ref:`namespace `. + +.. keypoints:: Key points + + We have seen that we can use the test `propertyerror` in Variable calculation. + diff --git a/docs/tutorial/webaddress.rst b/docs/tutorial/webaddress.rst new file mode 100644 index 000000000..576e736d1 --- /dev/null +++ b/docs/tutorial/webaddress.rst @@ -0,0 +1,91 @@ +.. _tutorial_web_address: + +A `web_address` variable +================================ + +.. objectives:: Objectives + + We are going to create another variable for the needs of our use case, + a conditional disabled variable with type `web_address`. + +.. prerequisites:: Prerequisites + + - We assume that Rougail's library is :ref:`installed ` on your computer. + + - It is possible to retrieve the current state of the various Rougail files manipulated in this tutorial step + by checking out the corresponding tag of the `rougail-tutorials` git repository. + Each tag corresponds to a stage of progress in the tutorial. + Of course, you can also decide to copy/paste or download the tutorial files contents while following the tutorial steps. + + If you want to follow this tutorial with the help of the corresponding :tutorial:`rougail-tutorials git repository `, + this workshop page corresponds to the tag :tutorial:`v1.1_100 ` + in the repository. + + :: + + git clone https://forge.cloud.silique.fr/stove/rougail-tutorials.git + git switch --detach v1.1_100 + +.. type-along:: A web_address variable + +A conditional disabled variable with type web_address +------------------------------------------------------------- + +First we need to add a variable as part of our use case with a type we haven't used yet, the `web_address` type. +It is related to the automatic proxy configuration :term:`situation `. +Let's put it in the new :file:`firefox/30-auto.yml` :term:`structure file`. +Note that this variable has a :term:`disabled` property defined: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_110/firefox/30-auto.yml + :language: yaml + :caption: The :file:`firefox/30-auto.yml` structure definition file with the `auto` variable + +.. + %YAML 1.2 + --- + version: 1.1 + + auto: + description: Automatic proxy configuration URL + type: web_address + disabled: + variable: _.proxy_mode + when_not: Automatic proxy configuration URL + ... + +If the `proxy_mode` variable has the `"Automatic proxy configuration URL"` value, the `auto` variable is enabled. +It is :term:`disabled` otherwise. + +Let's launch the Rougail CLI on this :term:`user data file `: + +.. extinclude:: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_110/config/02/config.yml + :language: yaml + :caption: The :file:`config/02/config.yml` user data file + +.. + --- + proxy_mode: Automatic proxy configuration URL + auto: https://auto.proxy.net/wpad.dat + no_proxy: 192.168.1.0/24 + +we have this output: + +.. raw:: html + :url: https://forge.cloud.silique.fr/stove/rougail-tutorials/raw/commit/v1.1_110/config/02/output_ro.html + :class: output + +.. + ╭────────────── Caption ───────────────╮ + │ Variable Modified value │ + │ (⏳ Original default value) │ + ╰──────────────────────────────────────╯ + Variables: + ┣━━ 📓 Configure Proxy Access to the Internet: Automatic proxy configuration URL ◀ loaded from the YAML file "config/03/config.yml" (⏳ No proxy) + ┗━━ 📓 Automatic proxy configuration URL: https://auto.proxy.net/wpad.dat ◀ loaded from the YAML file "config/03/config.yml" + +We can see that the `auto` variable has been activated here in this :term:`situation `. + +.. keypoints:: Key points + + In this section we have just created yet another variable, the `auto` variable, with the `web_address` type. + We are now used to conditional deactivation of a variable. diff --git a/docs/user_data/ansible.rst b/docs/user_data/ansible.rst new file mode 100644 index 000000000..859cf5581 --- /dev/null +++ b/docs/user_data/ansible.rst @@ -0,0 +1,83 @@ +Load user data from Ansible compatible file +=========================================== + +Synopsis +------------- + +.. note:: + + | Ansible offers a tool (ansible-vault) for encrypting inventory files. With this user data you can open an encrypt inventory file. This is a perfect way to manage a smooth migration from Ansible inventory to Rougail. Or it could be a way to encrypt these secrets in a file with a secure format. + | **Disabled**: if ansible is not set in "select for user data" (step.user_data). + +Parameters +-------------- + +.. list-table:: + + * - Variable + - Description + + * - **ansible.filename** + + `UNIX filename `__ `multiple` `mandatory` `*disabled*` `unique` + + **Command line**: + + --ansible.filename + + **Environment variable**: ROUGAILCLI_ANSIBLE.FILENAME + - Ansible filename inventory. + + **Validators**: + + + + - this filename could be a relative path + + - this file must exist + + - 'file type allowed: "file"' + + + + **Disabled**: if ansible is not set in "select for user data" (step.user_data). + + * - **ansible.secret** + + `secret `__ `mandatory` `*disabled*` + + **Command line**: + + --ansible.secret + + **Environment variable**: ROUGAILCLI_ANSIBLE.SECRET + - Secret to decrypt file. + + **Disabled**: if ansible is not set in "select for user data" (step.user_data). + + * - **ansible.file_with_secrets** + + `choice `__ `mandatory` `*disabled*` + + **Command line**: + + --ansible.file_with_secrets + + **Environment variable**: ROUGAILCLI_ANSIBLE.FILE_WITH_SECRETS + - Ansible files that may contain secrets. + + **Choices**: + + + + - all **← (default)** + + - first + + - last + + - none + + + + **Disabled**: if ansible is not set in "select for user data" (step.user_data). diff --git a/docs/user_data/bitwarden.rst b/docs/user_data/bitwarden.rst new file mode 100644 index 000000000..2ba3e296b --- /dev/null +++ b/docs/user_data/bitwarden.rst @@ -0,0 +1,58 @@ +Load user data secrets from Bitwarden +===================================== + +Synopsis +------------- + +.. note:: + + | Load the secrets from Bitwarden (or a compatible server like Vaultwarden). The data is retrieved using the official command line "bw" or the alternative command "rbw" (faster). Before using Rougail, the command must be logged and unlocked. + | **Disabled**: if bitwarden is not set in "select for user data" (step.user_data). + +Parameters +-------------- + +.. list-table:: + + * - Variable + - Description + + * - **bitwarden.command** + + `choice `__ `multiple` `mandatory` `unique` + + **Command line**: + + --bitwarden.command + + **Environment variable**: ROUGAILCLI_BITWARDEN.COMMAND + - Command line used to retrieve secrets. + + Command line must be in PATH. + + **Choices**: + + + + - rbw **← (default)** + + - bw **← (default)** + + * - **bitwarden.mock_enable** + + `boolean `__ `mandatory` + + **Command line**: + + + + - --bitwarden.mock_enable + + - --bitwarden.no-mock_enable + + + + **Environment variable**: ROUGAILCLI_BITWARDEN.MOCK_ENABLE + - Simulate password generation instead of retrieve them. + + **Default**: false diff --git a/docs/user_data/commandline.rst b/docs/user_data/commandline.rst new file mode 100644 index 000000000..8df63d1f8 --- /dev/null +++ b/docs/user_data/commandline.rst @@ -0,0 +1,10 @@ +Load user data from commandline parser +======================================= + +Synopsis +------------- + +.. note:: + + | Load the user data from the current command line. + | **Disabled**: if bitwarden is not set in "select for user data" (step.user_data). diff --git a/docs/user_data/environment.rst b/docs/user_data/environment.rst new file mode 100644 index 000000000..b16d3afbe --- /dev/null +++ b/docs/user_data/environment.rst @@ -0,0 +1,71 @@ +Load user data from environment variables +========================================= + +Synopsis +------------- + +.. note:: + + | Variable values can be defined directly from an environment variable. + | + | Environnement variable names begin with a prefix (by default `ROUGAIL_`) followed by the variable path in uppercase, for example: `ROUGAIL_MY_VARIABLE`. If you are using namespaces, the prefix is the name of the namespace in uppercase. + | + | Note that variable paths can contain dots ("."), which are not permitted everywhere. To avoid any issues, use the `env` command, for example: `env ROUGAIL_MY_FAMILY.MY_VARIABLE=1 rougail`. + | + | For values, there is no difference between a number and a letter (they can be enclosed in quotes or not). However, booleans are True or False. The separator for multiple variables is a comma. + | **Disabled**: if environment is not set in "select for user data" (step.user_data). + +Parameters +-------------- + +.. list-table:: + + * - Variable + - Description + + * - **environment.default_environment_name** + + `string `__ `mandatory` + + **Command line**: + + --environment.default_environment_name + + **Environment variable**: ROUGAILCLI_ENVIRONMENT.DEFAULT_ENVIRONMENT_NAME + - Name of the default environment prefix. + + **Validator**: should only use uppercase character + + **Default**: ROUGAIL + + * - **environment.custom_separator** + + `string `__ + + **Command line**: + + --environment.custom_separator + + **Environment variable**: ROUGAILCLI_ENVIRONMENT.CUSTOM_SEPARATOR + - Replace the separator character "." in path by an other. + + The character dot (".") may not be allowed in the environment variable name. Note that the variable name with dots is always available in addition to the name with this character. + + * - **environment.with_secrets** + + `boolean `__ `mandatory` + + **Command line**: + + + + - --environment.with_secrets + + - --environment.no-with_secrets + + + + **Environment variable**: ROUGAILCLI_ENVIRONMENT.WITH_SECRETS + - Environnement variables may contain secrets. + + **Default**: true diff --git a/docs/user_data/example.yaml b/docs/user_data/example.yaml new file mode 100644 index 000000000..88e3ec0e6 --- /dev/null +++ b/docs/user_data/example.yaml @@ -0,0 +1,53 @@ +%YAML 1.2 +--- +version: 1.1 + +person: # Description of a person + + + firstname: # Last name of the person + + lastnames: [] # Last names of the person + + age: + description: Person's age + type: integer + params: + min_integer: 0 + max_integer: 120 + + legal_age: + description: The person is of legal age + type: boolean + default: + jinja: |- + {{ _.age is not none and _.age >= 18 }} + hidden: true + + drive_license: + description: The person has a driver's license + default: false + disabled: + variable: _.legal_age + when_not: true + + means_of_transport: + description: Means of transport used by the person + choices: + - human-powered + - public transport + - electric-assisted vehicle + - engine-powered without a license + - engine-powered + - others + validators: + - jinja: |- + {% if _.means_of_transport == "engine-powered" %} + {% if _.drive_license is propertyerror %} + {{ _.lastnames[0] }} {{ _.firstname }} is not of legal age but declares that he uses an engine-powered + {% elif not _.drive_license %} + {{ _.lastnames[0] }} {{ _.firstname }} does not have a license but declares that he uses an engine-powered + {% endif %} + {% endif %} + description: Engine-powered are only permitted for individuals with a driver's license +... diff --git a/docs/user_data/index.rst b/docs/user_data/index.rst new file mode 100644 index 000000000..3959dd30c --- /dev/null +++ b/docs/user_data/index.rst @@ -0,0 +1,17 @@ +User data modules +====================== + +Rougail is a collections of modules in order to adjust functionalities to your needs. + +:term:`User data` is one of category of subjects. The goal is to setup variable with value defined by the :term:`operator`. + +.. toctree:: + :titlesonly: + :caption: Available user data + + yaml + ansible + environment + bitwarden + commandline + questionary diff --git a/docs/user_data/prerequisites.rst b/docs/user_data/prerequisites.rst new file mode 100644 index 000000000..438f6fe4f --- /dev/null +++ b/docs/user_data/prerequisites.rst @@ -0,0 +1,59 @@ +Prerequisites +============= + +Installation +------------ + +We assume that Rougail's command line is :ref:`installed ` on your computer. + +Verify that you have correctly installed the tool by running: + +.. code-block:: bash + :class: terminal + + + $ rougail --cli.versions + tiramisu: x.x.x + tiramisu-cmdline-parser: x.x.x + rougail: x.x.x + rougail-cli: x.x.x + rougail-user-data-ansible: x.x.x + rougail-user-data-bitwarden: x.x.x + rougail-user-data-commandline: x.x.x + rougail-user-data-environment: x.x.x + rougail-user-data-questionary: x.x.x + rougail-user-data-yaml: x.x.x + rougail-output-ansible: x.x.x + rougail-output-display: x.x.x + rougail-output-doc: x.x.x + rougail-output-formatter: x.x.x + rougail-output-json: x.x.x + rougail-output-table: x.x.x + +Smoke test +---------- + +We need a Rougail structured data file to test and illustrate the different user data loading modules. + +.. literalinclude:: example.yaml + :caption: A example of structured data file place in the :file:`example.yaml` structure file + +Let's test this :term:`structure file`: + +.. code-block:: bash + :class: terminal + + $ rougail -m example.yaml + +The output has to be: + +.. raw:: html + :class: terminal + + 🛑 Caution + ┗━━ Description of a person + ┣━━ Last name of the person: 🛑 mandatory variable but has no value + ┣━━ Last names of the person: 🛑 mandatory variable but has no value + ┣━━ Person's age: 🛑 mandatory variable but has no value + ┗━━ Means of transport used by the person: 🛑 mandatory variable but has no value + diff --git a/docs/user_data/questionary.rst b/docs/user_data/questionary.rst new file mode 100644 index 000000000..fdeeed2f7 --- /dev/null +++ b/docs/user_data/questionary.rst @@ -0,0 +1,56 @@ +Define user data interactively in the console +============================================= + +Synopsis +------------- + +.. note:: + + | The user will enter variable values in a command-line interface. The variables will be displayed one after another, allowing the user to change or add values. + | **Disabled**: if questionary is not set in "select for user data" (step.user_data). + +Parameters +-------------- + +.. list-table:: + + * - Variable + - Description + + * - **questionary.mandatory** + + `boolean `__ `mandatory` + + **Command line**: + + + + - -qm, --questionary.mandatory + + - -nqm, --questionary.no-mandatory + + + + **Environment variable**: ROUGAILCLI_QUESTIONARY.MANDATORY + - Ask values only for mandatories variables without any value. + + **Default**: false + + * - **questionary.show_secrets** + + `boolean `__ `mandatory` + + **Command line**: + + + + - -qs, --questionary.show_secrets + + - -nqs, --questionary.no-show_secrets + + + + **Environment variable**: ROUGAILCLI_QUESTIONARY.SHOW_SECRETS + - Show secrets instead of obscuring them. + + **Default**: false diff --git a/docs/user_data/yaml.rst b/docs/user_data/yaml.rst new file mode 100644 index 000000000..ff46954f5 --- /dev/null +++ b/docs/user_data/yaml.rst @@ -0,0 +1,71 @@ +Load user data from YAML file +============================= + +Synopsis +------------- + +.. note:: + + | Variable values can be defined in a list of YAML files. + | **Disabled**: if yaml is not set in "select for user data" (step.user_data). + +Sample +------ + + + +Parameters +-------------- + +.. list-table:: + + * - Variable + - Description + + * - **yaml.filename** + + `UNIX filename `__ `multiple` `mandatory` `unique` + + **Command line**: + + -yf, --yaml.filename + + **Environment variable**: ROUGAILCLI_YAML.FILENAME + - File or directory names where user data are stored. + + If a directory is set, all files with "yml" ou "yaml" extension will be load with alphanum order. + + **Validators**: + + + + - this filename could be a relative path + + - this file must exist + + - 'file type allowed: "directory" and "file"' + + * - **yaml.file_with_secrets** + + `choice `__ `mandatory` + + **Command line**: + + --yaml.file_with_secrets + + **Environment variable**: ROUGAILCLI_YAML.FILE_WITH_SECRETS + - File that may contain secrets. + + By default, all files can contain secrets. It might be useful to define more precisely which files can contain these secrets. + + **Choices**: + + + + - all **← (default)** + + - first + + - last + + - none diff --git a/docs/vac.rst b/docs/vac.rst new file mode 100644 index 000000000..a99d08da2 --- /dev/null +++ b/docs/vac.rst @@ -0,0 +1,206 @@ +Variables as Code +================== + + +Definition +-------------- + +.. glossary:: + + VaC + + VaC stands for **Variables as Code**, it is the practice of managing configuration variables the same way you treat source code. + +It is a concept that extends the Infrastructure as Code (IaC) philosophy to the management of configuration variables, secrets, and environment-specific settings. +More precisely, this extends Configuration as Code (CaC) principles. + +The idea is to treat variables — not just infrastructure or configuration — as version-controlled, declarative, and automated code rather than manual, static, or hardcoded values. + +With growing complexity of the infrastructure, it is difficult to manage all the variables. +Due to more and more available variables required to set up the infrastructure, +it became quite annoying to hand the necessary variables to where they are actually used +and even more annoying to add new variables. + +Variables can be redundant and sometimes are defined on several places. + +.. attention:: + + Many IaC projects use what could be called Values as Code (User Data as Code with the Rougail terminology). + Values are placed in a versioned and structured repositories. + But we don't share the variable itself (apart from its name). + We are thinking of its type, its scope, its purpose, etc. + + Variables as Code is not only the values. + +Reusable and shareable variables +------------------------------------ + +A typical example is the management of Ansible collections. + +Currently, when you distribute an Ansible collection, you distribute the recipes that allow you to deploy a component, the default values ​​for certain variables, and possibly variable validation. + +However, you don't share the variable declarations. Therefore, each user would have to rewrite the user documentation, define the global context themselves, and so on. It is therefore the user's responsibility to maintain the consistency of the variables with these given problems. +With Rougail, the collection maintainer can do this work himself and everyone will benefit. + +And we're not even talking about managing variables in a multi-project context. +It is often necessary to recreate the same variables with the same constraints over and over again between different projects. + +Variables as Code with Rougail +----------------------------------- + +For this, we thought of a solution and came up with a concept of managing all our variables as a code called :term:`VaC (Variables as Code) `. + +Often when we talk about :term:`VaC` (or CaC) we think of the different environments (Development, Staging, Production). + +With Rougail, this question does not arise. Essentially, Rougail manages this with user data. +Here we're talking about sharing variables between projects. + +Furthermore, using a tool like Rougail allows you to use variables with different IaC deployment solutions without having to redefine those variables. +It can be said that Rougail allows you to use variables not only between projects but also between tools. + +To do this, we need typed variable definitions with defaults and validation. + +Once defined, variables must be able to: + +- provide useful information to the operator to make their choices +- load user values from differents sources +- validate the context +- display and archive the final values in a readable format +- allow other applications to use these variables + +Variables as Code with Rougail by example +------------------------------------------- + +There's nothing better than a simple example to illustrate everything that's just been presented. + +Structured file +~~~~~~~~~~~~~~~~~~ + +Let's start by creating a file in Rougail format. + +This isn't the time to explain the format. + +But we'll create a quick file here to create the "my_variable" variable: + +.. code-block:: yaml + + %YAML 1.2 + --- + version: 1.1 + + my_variable: my_value # It's my variable + ... + +Let's document it +'''''''''''''''''''' + +We can start by generating the variable's documentation from this file: + +.. list-table:: + + * - Variable + - Description + + * - **my_variable** + + `string `__ `mandatory` + - It's my variable. + + **Default**: my_value + +Or an example user data file in YAML format: + +**Example with all variables modifiable:** + +.. code-block:: yaml + + --- + my_variable: my_value + +Regarding documentation, it's even possible to generate a changelog view for users. + +Let's imagine that the default value was originally `old_value` and is now `my_value`: + +.. list-table:: + + * - Variable + - Description + + * - **my_variable** + + `string `__ `mandatory` + - It's my variable. + + **Default**: :strike:`old_value` :underline:`myvalue` + +Now it's time to export +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In simple JSON format: + +.. code-block:: json + + { + "my_variable": "my_value" + } + +Or in advanced Ansible format: + +.. code-block:: json + + { + "_meta": { + "hostvars": { + "host1.net": { + "ansible_host": "host1.net", + "inventory": { + "my_variable": "my_value" + } + } + } + }, + "app1": { + "hosts": [ + "host1.net" + ] + } + } + +Or display it +~~~~~~~~~~~~~~~~ + +Displaying informations is an important step in the VaC approach. In a complex environment where variables are defined in multiple places, not to mention their values located elsewhere, maintaining good traceability of variables and their values becomes difficult. + +Managing variables like code also means `compiling` the informations. + +Here display our examples: + +.. raw:: html + :class: output + +
    +    ╭─────── Caption ────────╮
    +    │ Variable Default value │
    +    ╰────────────────────────╯
    +    Variables:
    +    ┗━━ 📓 It's my variable: my_value
    +    
    + +Now change the value with differents user data: + +.. raw:: html + :class: output + +
    +    ╭────────────── Caption ───────────────╮
    +    │ Variable Modified value              │
    +    │          (⏳ Original default value) │
    +    ╰──────────────────────────────────────╯
    +    ╭──────────── Layers ─────────────╮
    +    │ • the YAML file "user_data.yml" │
    +    │ • environment variable          │
    +    │ • Questionary                   │
    +    ╰─────────────────────────────────╯
    +    Variables:
    +    ┗━━ 📓 It's my variable: I am right about the value ◀ loaded from questionary 
    (⏳ I define the value! ◀ loaded from environment variable
    ⏳ no it's my value ◀ loaded from the YAML file "user_data.yml"
    ⏳ my_value) +
    diff --git a/docs/variable.rst b/docs/variable.rst deleted file mode 100644 index 1eab898dc..000000000 --- a/docs/variable.rst +++ /dev/null @@ -1,306 +0,0 @@ -The variables -=================== - -Synopsis ------------- - -.. glossary:: - - variable - variables - - A variable is an abstract black box (container) paired with an associated symbolic name, which contains some defined or undefined quantity of data referred to as a `value`. - -.. discussion:: This definition, makes a heavy use of data typing. - Indeed, depending on the type system definition of the constistency handling system used, variables may only be able to store a specified data type. - OK, variables are the containers for storing the values. It has something to do with typing. - But this is not just about typing. - -Name -------------- - -Variable's associated symbolic name. - -It's best to follow the :ref:`convention on variable names`. - -Shorthand declaration ----------------------------- - -Shorthand declaration is a way to declare a variable in a single line. But you can only define variable name, description, multi or default value. - -To create a variable, just add a key with it's name and default value as value. -Be careful not to declare any other attributes. - -To declare a multi variable just add a list as default value. - -By default, the description of the variable is the variable name. -If you add a comment in the same line of the name, this comment will be used has a description. - -.. code-block:: yaml - - --- - version: '1.1' - my_variable: 1 # This is a great integer variable - my_multi_variable: # This is a great multi string variable - - value1 - - value2 - -Parameters -------------- - -.. list-table:: - :widths: 15 45 - :header-rows: 1 - - * - Parameter - - Comments - - * - **help** - - `string` - - Additional help associated with the variable. - - * - **default** - - Default value(s) of the variable. - - This value is typed, you must correctly fill out the YAML file to avoid defining a value with an incorrect type. For example, a `number` must be a digit type, a multiple variable must be a `list` type, ... - - For a non :term:`leading` multiple variable, the first value defined in the list will also be the default value proposed if a new value is added to this variable. - - * - **validators** - - `list` - - Value validators. - - Jinja template list. The value of the variable will be considered invalid if the template has a return value. - * - **auto_save** - - `boolean` - - Variable with automatically modified value. - - A variable with automatically modified value is a variable whose value will be considered as *modified* (that is, it is no longer the variable's default value). - - For example, if the value of this variable comes from a calculation, the value will no longer be recalculated. - - These variables are usually :term:`required` variables. In fact, these variables are only automatically modified if they have a value. - - A :term:`leader` or :term:`follower` variable cannot have the `auto_save` property. - - **Default value**: `false` - * - **mode** - - `string` - - Variable's mode. - - **Default value**: The `default` mode of a variable is the mode of the parent family. - - Special cases : - - - a variable with an automatically modified value or an automatic read-only variable is by default in `basic` mode - - if the variable is not in a family, the variable will have a `standard` mode by default - - a :term:`mandatory` variable without default value (calculate or not) will have a `basic` mode - * - **multi** - - `boolean` - - The value of the variable is a list. - - **Default value**: `false` - * - **unique** - - `boolean` - - The :term:`multiple` type variable accepts the same value several times. If unique is set to `false`, a :term:`multiple` variable only accepts the same value once in the list. - - **Default value**: `false` - * - **hidden** - - `boolean` or :term:`calculation` - - Invisible variable. - - Enables us to *hide* a variable. - - This means that the variable will no longer be visible in `read-write` mode, but only for calculations or in `read-only` mode. - - When a variable is made invisible, the user will not be able to modify its value; if he has already succeeded in modifying it, this value will not be taken into account. - - **Default value**: `false` - * - **disabled** - - `boolean` or :term:`calculation` - - Disabled variable. - - Allows us to deactivate a variable. - - This means that the variable will no longer be visible to the user but also to a :term:`calculation`. - - **Default value**: `false`. - * - **mandatory** - - `boolean` or :term:`calculation` - - Mandatory variable. - - Variable whose value is `required`. - - For a multiple variable, this means that the list shall not be empty. - - **Default value**: `true` - * - **redefine** - - `boolean` - - It is possible to define a variable in one :term:`dictionary` and change its behavior in a second :term:`dictionary`. In this case you must explicitly redefine the variable. - - **Default value**: `false` - * - **exists** - - `boolean` - - This attribute does two things: - - - creates a variable if it does not exist in another :term:`dictionary` (otherwise do nothing), in this case the value of the attribute must be `true` - - in conjunction with the `redefine` attribute set to `true`, only modifies the behavior if it is pre-existing, in which case the attribute's value must be `false`. - - **Default value**: `null` - * - **test** - - `list` - - The `test` attribute is a special attribute that allows :term:`dictionary` designers to influence a test robot by specifying useful values to test. - - Concretely, the content of this attribute is recorded in the `information` attribute of the corresponding `Tiramisu` option object. - -Variables types ----------------- - -A variable **has a type**. - -This type enables the variable to define the values that are accepted by this variable. - -.. list-table:: - :widths: 15 25 20 15 - :header-rows: 1 - - * - Value - - Comments - - Parameters - - Samples - - * - string - - character string (default type) - - - - test - - "1" - - "true" - * - number - - a number - - `min_number`: minimum number allowed - - `max_number`: maximum number allowed - - 1 - * - float - - a floating number - - - - 1.2 - * - boolean - - A boolean, if no value is defined the default value of this variable will be `true`, the variable will also be :term:`mandatory` by default - - - - `true` - - `false` - * - secret - - a secret (like a password, a private key, etc.) - - - - `hO_'hi` - * - mail - - a mail address - - - - test@rougail.example - * - unix_filename - - a file name in the Unix meaning - - - - :file:`/etc/passwd` - * - date - - a date in the format `%Y-%m-%d` - - - - `2021-01-30` - * - unix_user - - a user in the Unix meaning - - - - test - * - ip - - any kind of IPv4 address - - `private_only`: only private IPs (`false` by default) - - `allow_reserved`: allows reserved IPs (`true` by default) - - `1.2.3.4` - * - cidr - - any IPv4 address in the CIDR format - - `private_only`: only private IPs (`false` by default) - - `allow_reserved`: allows reserved IPs (`false` by default) - - `1.2.3.4/24` - * - netmask - - mask of an IPv4 address - - - - `255.255.255.0` - * - network - - network address - - - - `192.168.1.0` - * - network_cidr - - network address in CIDR format - - - - `192.168.1.0/24` - * - broadcast - - broadcast address - - - - `255.255.255.255` - * - netbios - - netbios name - - - - machine - * - domainname - - domain name - - `allow_ip`: allows an IP rather than a domain name (`false` by default) - - `allow_cidr_network`: allows a CIDR type network address (`false` by default) - - `allow_without_dot`: allows names without a dot (`false` by default) - - `allow_startswith_dot`: allows starting with a point (`false` by default) - - `rougail.example` - * - hostname - - host name - - `allow_ip`: allows an IP rather than a domain name (`false` by default) - - machine - * - web_address - - web address - - `allow_ip`: allows an IP rather than a domain name (`false` by default) - - `allow_without_dot`: allows names without a dot (`true` by default) - - http://rougail.example - * - port - - port - - `allow_range`: allows a port range, for example 80:85 (`false` by default) - - `allow_zero`: allows port 0 (false by default) - - `allow_wellknown`: allows ports from 1 to 1023 (`true` by default) - - `allow_registred`: allows ports from 1024 to 49151 (`true` by default) - - `allow_private`: allows ports greater than 49152 (`true` by default) - - `allow_protocol`: allows the addition of the protocol, for example tcp:80 (`false` by default) - - - 8080 - * - mac - - MAC address - - - - 11:11:11:11:11:11 - * - unix_permissions - - access rights to the file, directory, etc. - - - - 644 - * - choice - - choice variable - - - -