Introduction
Logging is an essential aspect of software development that helps track events that happen when software runs. Python'slogging module provides a flexible framework for emitting log messages from Python programs. By using logging, you can gain insights into your application's behavior, diagnose problems, and audit operations.
- Why Use Logging?
- Debugging: Logs help track down bugs by providing detailed information about program execution.
- Monitoring: Logs can be used to monitor application behavior over time.
- Auditing: Logs provide a record of activities, which is essential for auditing and compliance.
- Error Reporting: Logs capture errors and exceptions, which helps in understanding failures and fixing them.
- Basics of Python Logging:
- mporting the Logging Module: To start using logging, you need to import Python's built-in
loggingmodule:import logging - Basic Logging Configuration: The simplest way to log messages is by configuring the logging system with basic settings using
logging.basicConfig().
we will get following, if we run the script:import logging logging.basicConfig(level=logging.DEBUG) logging.debug("This is a debug message") logging.info("This is an info message") logging.warning("This is a warning message") logging.error("This is an error message") logging.critical("This is a critical message")DEBUG:root:This is a debug message INFO:root:This is an info message WARNING:root:This is a warning message ERROR:root:This is an error message CRITICAL:root:This is a critical message - Logging Levels: Logging levels determine the severity of the events being logged. The standard levels, in increasing order of severity, are:
DEBUG: Detailed information, typically of interest only when diagnosing problems.INFO: Confirmation that things are working as expected.WARNING: An indication that something unexpected happened, or indicative of some problem in the near future (e.g., ‘disk space low’). The software is still working as expected.ERROR: Due to a more serious problem, the software has not been able to perform some function.CRITICAL: A very serious error, indicating that the program itself may be unable to continue running.
WARNINGor above unless configured otherwise.Basic Example:
import logging logging.basicConfig(level=logging.INFO) def divide(a, b): logging.info(f"Dividing {a} by {b}") try: result = a / b except ZeroDivisionError: logging.error("Attempted to divide by zero") return None else: return result result = divide(10, 0) # This will trigger an error logINFO:root:Dividing 10 by 0 ERROR:root:Attempted to divide by zero - Advanced Logging Configuration:
- Configuring Log Output Format: You can customize the log message format using the
formatargument inbasicConfig().import logging logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s') logging.info("This is an info message with a custom format")INFO:root:This is an info message with a custom format - Logging to a File: You can configure logging to output messages to a file instead of the console.
import logging logging.basicConfig( filename='app.log', filemode='w', level=logging.DEBUG, format='%(name)s - %(levelname)s - %(message)s' ) logging.debug("This message will be logged to a file") logging.info("This is another message logged to the file") - Adding Multiple Handlers: Handlers are responsible for sending the log messages to the specified destination, such as the console, files, or external systems. You can configure multiple handlers to direct logs to multiple destinations.
where:import logging # Create a logger logger = logging.getLogger('my_logger') logger.setLevel(logging.DEBUG) # Create console handler and set level to debug console_handler = logging.StreamHandler() console_handler.setLevel(logging.DEBUG) # Create file handler and set level to warning file_handler = logging.FileHandler('my_app.log') file_handler.setLevel(logging.WARNING) # Create a formatter and set it for both handlers formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') console_handler.setFormatter(formatter) file_handler.setFormatter(formatter) # Add handlers to the logger logger.addHandler(console_handler) logger.addHandler(file_handler) # Example log messages logger.debug("This is a debug message") logger.info("This is an info message") logger.warning("This is a warning message") logger.error("This is an error message") logger.critical("This is a critical message")- The console handler will log messages of all levels to the console.
- The file handler will log messages of level
WARNINGand above to the filemy_app.log.
- Loggers, Handlers, and Formatters:
- Logger: The entry point for the logging system. Each logger can have multiple handlers.
- Handler: Directs the log messages to the appropriate destination (e.g., console, file).
- Formatter: Defines the format of the log messages.
import logging # Logger configuration logger = logging.getLogger('example_logger') logger.setLevel(logging.DEBUG) # Handler configuration handler = logging.FileHandler('example.log') handler.setLevel(logging.DEBUG) # Formatter configuration formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') handler.setFormatter(formatter) # Add the handler to the logger logger.addHandler(handler) # Example usage logger.info("This is an info message") - Rotating Log Files: For long-running applications, log files can become very large. Python’s
logging.handlersmodule provides aRotatingFileHandlerthat can rotate log files based on size or time.
where:import logging from logging.handlers import RotatingFileHandler handler = RotatingFileHandler('app.log', maxBytes=2000, backupCount=5) logger = logging.getLogger('my_logger') logger.setLevel(logging.DEBUG) logger.addHandler(handler) # Example log messages to generate file rotation for i in range(100): logger.debug(f"This is log message {i}")maxBytes=2000: The log file will rotate when it reaches 2000 bytes.backupCount=5: Keeps the last 5 log files as backups.
- Configuring Log Output Format: You can customize the log message format using the
- mporting the Logging Module: To start using logging, you need to import Python's built-in
- Logging Best Practices:
- Use the Appropriate Logging Level: Choose the appropriate logging level (
DEBUG,INFO,WARNING,ERROR,CRITICAL) based on the importance of the message. - Avoid Using Print Statements for Logging: Use the logging module instead of print statements for better control and flexibility.
- Configure Logging Early in the Program: Set up logging configurations at the start of your program to ensure consistent logging throughout.
- Use Logging for Exception Handling: Log exceptions to capture stack traces and error details for debugging.
Example: Logging in Exception Handling
This will log the exception details, including the stack trace, which is useful for debugging.import logging logging.basicConfig(level=logging.ERROR) try: result = 10 / 0 except ZeroDivisionError as e: logging.error("Exception occurred", exc_info=True) - Use the Appropriate Logging Level: Choose the appropriate logging level (
Project: To-Do List Application
Objective: Build a simple command-line To-Do List application with logging to track user actions and application errors.
For more details, please checkout the github repository.
Project Structure:
# Readme.md
todo_app/
│
├── todo.py # Main application file
├── logger.py # Logging configuration file
└── todo.log # Log file (generated after running the application)
logger.py: Setting Up Logging: Create a logger.py file to configure the logging for the project.# logger.py import logging # Configure logging logging.basicConfig( level=logging.DEBUG, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', handlers=[ logging.FileHandler("todo.log"), logging.StreamHandler() ] ) # Create a logger object logger = logging.getLogger('todo_app')todo.py: Main Application Logic# todo.py from logger import logger class ToDoApp: def __init__(self): self.tasks = [] logger.info("To-Do application started") def add_task(self, task): if task: self.tasks.append(task) logger.info(f"Task added: {task}") else: logger.warning("Empty task cannot be added") def complete_task(self, task_index): try: completed_task = self.tasks.pop(task_index) logger.info(f"Task completed: {completed_task}") except IndexError: logger.error(f"Task index {task_index} is out of range") def show_tasks(self): if not self.tasks: logger.info("No tasks to show") print("No tasks in the to-do list.") else: print("To-Do List:") for idx, task in enumerate(self.tasks): print(f"{idx + 1}. {task}") logger.info(f"{len(self.tasks)} tasks shown") def main(): app = ToDoApp() while True: print("\nOptions:") print("1. Add Task") print("2. Complete Task") print("3. Show Tasks") print("4. Exit") choice = input("Choose an option: ") if choice == '1': task = input("Enter the task: ") app.add_task(task) elif choice == '2': task_index = int(input("Enter the task number to complete: ")) - 1 app.complete_task(task_index) elif choice == '3': app.show_tasks() elif choice == '4': logger.info("Exiting the To-Do application") print("Goodbye!") break else: logger.warning(f"Invalid option selected: {choice}") print("Invalid option, please try again.") if __name__ == "__main__": main()
The log file todo.log looks like this now:
References
- Udemy playlist on advanced python by Krish Naik
- For more details, please chekout the official documentation.
Some other interesting things to know:
- Visit my website on For Data, Big Data, Data-modeling, Datawarehouse, SQL, cloud-compute.
- Visit my website on Data engineering