fixes bugs and limited configuration of multi-level logs

docs/source/how_to/03_logging.md

@@ -1,6 +1,6 @@
 # Keeping Logs
 
-Auto Archiver's logs can be helpful for debugging problematic archiving processes. This guide shows you how to use the logs to 
+Auto Archiver's logs can be helpful for debugging problematic archiving processes. This guide shows you how to use the logs configuration.
 
 ## Setting up logging
 
@@ -8,10 +8,10 @@ Logging settings can be set on the command line or using the orchestration confi
 
 #### Enabling or Disabling Logging
 
-Logging to the console is enabled by default. If you want to globally disable Auto Archiver's logging, then you can set `enabled: false` in your `logging` config:
+Logging to the console is enabled by default. If you want to globally disable Auto Archiver's logging, then you can set `enabled: false` in your `logging` config file:
 
 ```{code} yaml
-
+:caption: orchestration.yaml
 ...
 logging:
    enabled: false
@@ -24,7 +24,7 @@ This will disable all logs from Auto Archiver, but it does not disable logs for
 
 #### Logging Level
 
-There are 7 logging levels in total, with 4 commonly used levels. They are: `DEBUG`, `INFO`, `WARNING` and `ERROR`.
+There are 7 logging levels in total, with 5 of them used in this tool. They are: `DEBUG`, `INFO`, `SUCCESS`, `WARNING` and `ERROR`.
 
 Change the warning level by setting the value in your orchestration config file:
 
@@ -44,7 +44,7 @@ For normal usage, it is recommended to use the `INFO` level, or if you prefer qu
 
 ### Logging to a file
 
-As default, auto-archiver will log to the console. But if you wish to store your logs for future reference, or you are running the auto-archiver from within code a implementation, then you may with to enable file logging. This can be done by setting the `file:` config value in the logging settings.
+As default, auto-archiver will log to the console. But if you wish to store your logs for future reference, or you are running the auto-archiver from within code a implementation, then you may wish to enable file logging. This can be done by setting the `file:` config value in the logging settings.
 
 **Rotation:** For file logging, you can choose to 'rotate' your log files (creating new log files) so they do not get too large. Change this by setting the 'rotation' option in your logging settings. For a full list of rotation options, see the [loguru docs](https://loguru.readthedocs.io/en/stable/overview.html#easier-file-logging-with-rotation-retention-compression).
 
@@ -57,15 +57,33 @@ logging:
     rotation: 1 day
 ```
 
-### Full logging example
+### Logging each level to a different file
+If you want to log each level to a different file, you can do this by setting the `each_level_in_separate_file:` option to `true` and also setting your `file:` name, a new file will be created for each of the 5 levels used, by appending the `0_level` name to the file like so `your_file.log.1_error`. In this case the `level:` option is ignored, and all levels will be logged. 
 
-The below example logs only `WARNING` logs to the console and to the file `/my/file.log`, rotating that file once per week:
 
 ```{code} yaml
 :caption: orchestration.yaml
 
 logging:
-    level: WARNING
+    each_level_in_separate_file: true
+    file: /my/logs/file.log 
+```
+This will create the following files:
+- `/my/logs/file.log.1_debug`
+- `/my/logs/file.log.2_info`
+- `/my/logs/file.log.3_success`
+- `/my/logs/file.log.4_warning`
+- `/my/logs/file.log.5_error`
+
+### Full logging example
+
+The below example logs only `DEBUG` logs to the console and to the file `/my/file.log`, rotating that file once per week:
+
+```{code} yaml
+:caption: orchestration.yaml
+
+logging:
+    level: DEBUG
     file: /my/file.log
     rotation: 1 week
 ```

src/auto_archiver/core/orchestrator.py

@@ -249,7 +249,7 @@ Here's how that would look: \n\nsteps:\n  extractors:\n  - [your_extractor_name_
             action="store",
             dest="logging.level",
             choices=["INFO", "DEBUG", "ERROR", "WARNING"],
-            help="the logging level to use",
+            help="the logging level to use for the standard output and file logging",
             default="INFO",
             type=str.upper,
         )
@@ -268,7 +268,7 @@ Here's how that would look: \n\nsteps:\n  extractors:\n  - [your_extractor_name_
             "--logging.each_level_in_separate_file",
             action="store",
             dest="logging.each_level_in_separate_file",
-            help="whether to write each logging level to a separate file",
+            help="if set, writes each logging level to a separate file (ignores --logging.level), you must also set --logging.file. Each level will have a dedicate logs file matching your <file>.debug, <file>.info, etc.",
             default=False,
         )
 
@@ -341,28 +341,24 @@ Here's how that would look: \n\nsteps:\n  extractors:\n  - [your_extractor_name_
 
         # add other logging info
         if self.logger_id is None:  # note - need direct comparison to None since need to consider falsy value 0
-            self.logger_id = logger.add(sys.stderr, level=logging_config["level"])
+            use_level = logging_config["level"]
+            self.logger_id = logger.add(sys.stderr, level=use_level)
 
-            # Default to False (above in parser code)
-            separate_file = logging_config["each_level_in_separate_file"]
+            rotation = logging_config["rotation"]
+            log_file = logging_config["file"]
 
-            # Default to None if not set
-            rotation=logging_config["rotation"]
-
-            if separate_file:
-                logger.add("logs/1debug.log", level="DEBUG", rotation=rotation)
-                logger.add("logs/2info.log", level="INFO", rotation=rotation)
-                logger.add("logs/3success.log", level="SUCCESS", rotation=rotation)
-                logger.add("logs/4warning.log", level="WARNING", rotation=rotation)
-                logger.add("logs/5error.log", level="ERROR", rotation=rotation)
-            else:
-                log_file = logging_config["file"]
-                logger.add(log_file, rotation=rotation)
-
-            # if log_file := logging_config["file"]:
-            #     logger.add(log_file) if not logging_config["rotation"] else logger.add(
-            #         log_file, rotation=logging_config["rotation"]
-            #     )
+            if logging_config.get("each_level_in_separate_file"):
+                assert logging_config["file"], (
+                    "You must set --logging.file if you want to use --logging.each_level_in_separate_file"
+                )
+                for i, level in enumerate(["DEBUG", "INFO", "SUCCESS", "WARNING", "ERROR"], start=1):
+                    logger.add(
+                        f"{log_file}.{i}_{level.lower()}",
+                        filter=lambda rec, lvl=level: rec["level"].name == lvl,
+                        rotation=rotation,
+                    )
+            elif log_file:
+                logger.add(log_file, rotation=rotation, level=use_level)
 
     def install_modules(self, modules_by_type):
         """