Introducing LogAI: A Unified Open Source Library for Smarter Log Analytics
Table of Content
Logs are the lifeblood of system monitoring, offering invaluable insight into application behavior, performance, and issues. But with the sheer volume and complexity of modern log data, extracting meaningful intelligence can feel like searching for a needle in a haystack.
That’s where LogAI comes in.
What is LogAI?
LogAI is an open-source library built to streamline and enhance log analytics by combining the power of machine learning, deep learning, and interactive visualization in one unified toolkit.
Whether you're debugging production systems, detecting anomalies in real-time, or summarizing months' worth of logs, LogAI gives you the tools to turn raw log data into actionable insights, faster and more efficiently than ever.
Hazem Abbas
Why LogAI Stands Out
What sets LogAI apart isn’t just its feature set, it's how everything works together seamlessly:
- Supports Multiple Use Cases: From log clustering and summarization to advanced anomaly detection, LogAI covers a wide range of log analysis tasks.
- OpenTelemetry Native: Built on the OpenTelemetry data model, LogAI integrates smoothly with modern observability platforms and pipelines.
- Flexible Modeling Interface: It supports time-series models, statistical methods, and deep learning frameworks, all under a consistent API that makes experimentation easy.
- Benchmarking Made Easy: Want to compare the performance of different ML or deep learning models on your log data? LogAI provides ready-to-use benchmarking capabilities so you can focus on results, not preprocessing.
- Interactive GUI Toolkit: For those who prefer visual exploration, LogAI includes a built-in GUI portal for interactive log analysis, no need to write complex scripts to get started.
Hazem Abbas
Who Can Benefit from LogAI?
Whether you're a researcher exploring novel log analysis techniques or an engineer building scalable monitoring solutions, LogAI offers something valuable.
Its modular design allows for both plug-and-play usage and deep customization, making it ideal for academic studies, industrial prototypes, and everything in between.
By abstracting away much of the complexity involved in log processing and modeling, LogAI empowers developers and data scientists to focus on what matters most: uncovering patterns, predicting failures, and improving system reliability.
In short, LogAI isn’t just another log analysis tool, it’s a full-fledged platform for building intelligent logging solutions tailored to your needs.
Stay tuned as we dive deeper into how you can start using LogAI today, from running simple anomaly detection apps to building your own custom log intelligence pipelines.
Features
- Open Source (BSD-3-Clause license)
- Supports multiple log analysis tasks :
- Log summarization
- Log clustering
- Log anomaly detection (time-series, ML, deep learning)
- Unified model interface with support for:
- Time-series models
- Statistical learning models
- Deep learning models
- Built on OpenTelemetry data model for compatibility with log platforms
- Auto log parsing (e.g., Drain algorithm)
- Unified data loader and preprocessing
- Log vectorization (e.g., Word2Vec)
- Categorical encoding (e.g., Label Encoder)
- GUI Toolkit for interactive log analysis (Plotly Dash)
- Deep learning benchmarking for anomaly detection
- Huggingface integration
- Configurable via JSON/YAML for easy workflow setup
- Modular and extensible architecture for custom applications
- Supports public datasets : HDFS, BGL, HealthApp
- Installation options : Core, GUI, Deep Learning, Dev, and Full bundles
- Anomaly detection types :
- Time-series (e.g., ETS)
- Semantic (e.g., One-Class SVM)
- Includes tutorials and documentation for quick onboarding
- Actively maintained with GitHub Actions, issues, and community support
- Citable with technical report (arXiv)
- Python-based (96.6% of code) with support for extensions and customization
How to Install and Run LogAI
Below is a quick guide to install and run LogAI using Bash and Python.
Step 1: Clone the Repository
git clone https://github.com/salesforce/logai.git
cd logai
Step 2: Set Up a Virtual Environment (Optional but Recommended)
python3 -m venv venv
source venv/bin/activate
Step 3: Install LogAI
You can install different versions depending on your needs:
Install Core Library
pip install logai
Install with GUI Support
pip install "logai[gui]"
Install with Deep Learning Support
pip install "logai[deep-learning]"
Install All Dependencies (Full Installation)
pip install "logai[all]"
⚠️ If you encounter Resource punkt not found, install the NLTK punkt package:python -m nltk.downloader punkt
Step 4: Run the LogAI GUI Portal
The GUI portal lets you interactively explore log analytics features like summarization, clustering, and anomaly detection.
# Install dev and GUI dependencies
pip install ".[dev]"
pip install ".[gui]"
# Set PYTHONPATH
export PYTHONPATH='.'
# Start the GUI server
python3 gui/application.py
Then open your browser and go to:
http://localhost:8050/
You’ll see the LogAI GUI dashboard where you can select applications like Log Summarization, Log Clustering, or Anomaly Detection.
Step 5: Run a Log Anomaly Detection Script
Here’s a simple example of how to run a time-series anomaly detection job using a JSON config file.
Create a config file: log_anomaly_detection_config.json
{
"open_set_data_loader_config": {
"dataset_name": "HDFS",
"filepath": "/path/to/your/logfile.log"
},
"preprocessor_config": {
"custom_delimiters_regex": []
},
"log_parser_config": {
"parsing_algorithm": "drain",
"parsing_algo_params": {
"sim_th": 0.5,
"depth": 5
}
},
"feature_extractor_config": {
"group_by_category": ["Level"],
"group_by_time": "1s"
},
"log_vectorizer_config": {
"algo_name": "word2vec"
},
"categorical_encoder_config": {
"name": "label_encoder"
},
"anomaly_detection_config": {
"algo_name": "one_class_svm"
}
}
Run the Python Script
import json
from logai.applications.application_interfaces import WorkFlowConfig
from logai.applications.log_anomaly_detection import LogAnomalyDetection
# Load config
with open("log_anomaly_detection_config.json", "r") as f:
config = json.load(f)
workflow_config = WorkFlowConfig.from_dict(config)
app = LogAnomalyDetection(workflow_config)
app.execute()
# Access results
print(app.anomaly_results)
You're All Set!
Now you can:
- Explore logs interactively via the GUI.
- Run anomaly detection using ML or deep learning models.
- Build custom log analysis applications using LogAI’s modular framework.
For more info, check out the official LogAI Documentation and GitHub repo.
Why LogAI is different!
The table below provides a comparison of LogAI with a range of existing log analysis tools, including commercial platforms such as New Relic and Datadog, as well as popular open-source solutions available on GitHub.
| Feature | LogAI | logparser | loglizer | deep-loglizer | log3C | NewRelic Log Monitoring | Datadog Log Explorer |
|---|---|---|---|---|---|---|---|
| OpenTelemetry Log Data Model Support | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |
| Unified Data Loader & Preprocessing | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ |
| Auto Log Parsing (e.g., Drain) | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ |
| Log Clustering | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ |
| Time-Series Log Anomaly Detection | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |
| Traditional ML-Based Anomaly Detection | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Deep Learning-Based Anomaly Detection | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| Huggingface Integration | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| GUI for Visualization & Interaction | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |
Resources & Downloads
salesforce
