GAFA-IA: Based on Artificial Intelligence, Monitor Apache Airflow and Get Codes in Just Seconds.
What is GAFA-IA?
At EBAL IT we know that minimizing incidents is a critical factor for the success and productivity of your organization.
In this sense, we present our platform based on AI and Machine Learning GAFA-IA (Automatic Fault Management in Airflow with Artificial Intelligence).
GAFA-IA has been specifically designed to monitor the processes that occur in Apache Airflow environments, offer indications of the reason for the failure and give suggestions for its resolution.
GAFA-IA process.
GAFA-IA is: ¡Your technological ally in fault management!
What does GAFA-IA solve?
-
-
Without GAFA-IA
-
❌Having to manually monitor workflows and identify and resolve potential issues.
-
-
-
❌ The lack of a structured and automated response to alerts, when really necessary.
-
-
-
❌ Constant manual intervention by system operators, as well as the halting of workflows.
-
-
-
❌ The information deficit grouped in airflow, as well as the solution to code and execution errors in the processes.
-
-
-
❌ Lack of flexibility and customization in standard solutions.
-
-
-
With GAFA-IA
-
-
-
✅ GAFA-AI automatically identifies and monitors critical issues and everything that occurs in AirFlow, providing suggestions for troubleshooting using AI.
-
-
-
✅ GAFA-IA defines actions to provide proactive responses to events, ensuring that action is taken at the right time.
-
-
-
✅ GAFA-IA automatically relaunches erroneous processes or executes second processes, ensuring continuous and effective management of operations.
-
-
-
✅ GAFA-IA groups information about processes and tasks in a Dashboard, facilitating decision-making.
-
-
-
✅ GAFA-IA is flexible, actions such as relaunching DAGs or sending emails can be adjusted to meet the organization’s standards.
-
What are the benefits of GAFA-IA?
75% improvement in response times.
By automating alert detection and action execution, GAFA can significantly reduce response time to critical issues. Suppose that prior to implementing GAFA, the average time to identify, assess, and respond to an alert was 1 hour. With GAFA, this time could be reduced to just 15 minutes, representing a 75% improvement in response time.
50% savings in labor costs.
The reduction in the need for manual interventions and improved operational efficiency also translate into significant savings in labor costs. Suppose that, prior to implementing GAFA, the organization dedicated the equivalent of 2 full-time employees to monitoring and responding to alerts. With GAFA, the organization could reduce this need to just 1 full-time employee, representing a 50% savings in labor costs.
80% improvement in operational efficiency.
Automating alert and fault management with GAFA can lead to a substantial reduction in human errors and manual interventions. Suppose that previously, the operations team made errors in 5% of manual interventions. With GAFA, this error rate could be reduced to 1%, representing an 80% improvement in operational efficiency.
Reduce the time spent on decision-making by 90%.
By obtaining detailed and automated reports on alerts and failures in the Apache Airflow environment, the time required to make decisions is significantly reduced. Let’s assume that before the implementation of GAFA, the process of collecting, analyzing and making decisions based on manual reports took an average of 4 hours. With GAFA, this process is simplified and the time required is reduced to just 30 minutes, representing an 87.5% reduction in the time spent on decision making.
GAFA-IA Dashboard
✅ Collect and report metrics related to process failures.
✅ Provides valuable information for decision making and continuous improvement.
✅ Make faster and more informed decisions.
✅ Shows failure probabilities in Airflow and in each process.
✅ Dags Ranking.
✅ Monitoring of each of the tasks by Dag.
✅ Displays in Realtime a compilation of the Dashes that have failed.
GAFA-IA: Based on Artificial Intelligence
Get codes, in just seconds.
What is Apache Airflow?
Apache Airflow is a workflow manager tool, used as a service orchestrator. It is used to programmatically automate jobs by dividing them into subtasks. It allows their planning and monitoring from a centralized tool.
The project was created in October 2014 at Airbnb by Maxime Beauchemin and published under an open source license in June 2015. In March 2016 the project was hosted by the Apache Software Foundation incubator, and in January 2019 it was graduated as a top level project, where it remains today.
The most common use cases are the automation of data ingestion, periodic maintenance actions and administration tasks. The adoption of Airflow in production environments has grown recently, being integrated into the Google Cloud stack in 2018 as its service orchestrator.
GAFA is flexible, allowing it to be adjusted to the particular requirements of the organization.
