Skip to content

All.inframeister.how to guide

InfraMeister Responsibilities#

Overview#

The InfraMeister role is critical for maintaining the operational health of our infrastructure. This role entails proactive monitoring, alert management, cost control, and specific oversight of infrastructure-related incidents.

1. Monitoring and Alert Management#

Review Alerts#

  • AlertManager Overview: Prometheus AlertManager, is a tool integrated with Prometheus to handle alerts sent by client applications such as Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integrations such as email or Slack. It also ensures alert silencing and inhibition logic.
    • Regularly check alerts generated by AlertManager to identify critical issues that require immediate attention.
    • Evaluate alerts to determine their impact and urgency, prioritizing actions based on severity.

General Approach to Critical Alerts#

Immediate Response#
  • Acknowledge the Alert: Immediately confirm receipt of the alert to mitigate escalations and inform the team that the issue is being addressed.
  • Assess the Situation: Swiftly evaluate the metrics or logs relevant to the alert to understand its immediacy and potential impact. For instance, if an alert indicates high CPU usage, immediately access the CPU usage logs and metrics on the Grafana dashboard to understand the severity (e.g., whether the CPU usage is consistently above 90%).
Diagnosis#
  • Review Data Trends: Analyze historical data to identify growth trends and potential points of failure. For instance, review historical data available in Grafana, to identify trends such as disk usage, memory leaks, or load patterns that can forecast potential points of failure.
  • Check for Anomalies: Investigate any unusual spikes or patterns that could signify underlying problems, such as a sudden increase in response time or failure rates.
Mitigation#
  • Implement Quick Fixes: Apply temporary solutions if necessary to stabilize the environment while long-term solutions are devised.
  • Optimize System Performance: Make adjustments to reduce the strain on resources, which could include purging unnecessary data or tweaking system settings.
Documentation and Reporting#
  • Report Actions and Outcomes: Log all actions taken and outcomes achieved in resolving the alert.
    • GitHub: Use for detailed incident tracking and resolution. File a GitHub issue or update an existing one with comprehensive details of the incident management process, including what was done, what was discovered, and how the issue was resolved or mitigated.
    • Google Docs: Maintain a running document for each major incident for real-time collaboration and information sharing during the incident response.
    • Slack For immediate communications and quick updates during an incident.
  • Communicate with Stakeholders: Update all pertinent parties, including management and technical leads, about the incident's status and resolution. Use our designated Slack channels for real-time updates and discussions during active incident management. This allows for immediate feedback and quick coordination among the involved teams.
  • Conduct Reviews: Hold post-mortem meetings if necessary to discuss the root cause and lessons learned. Provide updates during weekly sprint meetings with tech leads. Use this time for a more detailed discussion of recent incidents, resolved issues, and any lessons learned.
AWS RDS Database Alarms#
  • Immediate Expansion: If the database is critically low on storage (e.g. 90% of capacity), temporarily increase storage capacity to prevent service disruption. Note that due to the nature of AWS RDS, any increase in storage capacity is permanent as AWS does not support reducing allocated storage once it is expanded. Therefore, it is crucial to carefully evaluate the necessity of increasing storage.
  • Data Management: Evaluate data retention policies and execute data clean-up or archiving strategies to free up space. For example, evaluate Kubernetes CronJobs or Airflow archiving tasks that automatically compress and move older logs and data out of primary storage every month, ensuring system efficiency and compliance with data policies.
  • System Adjustment: Review and adjust the database configuration to optimize storage utilization and performance.
  • Long-term Storage Scaling: Plan for scalable storage solutions based on the anticipated data growth and usage patterns.
Airflow Alerts#
  • Test Stage DAGs: Alerts prefixed with test. generally require no action unless specified.
  • Preprod Stage DAGs: Investigate alerts through the Airflow UI to pinpoint the underlying issues, focusing on whether they arise from task failures, development glitches, or infrastructure problems.
    • If related to task failures or development issues:
      • Contact the responsible individual (e.g., Sonaal for DataPull DAGs, Nina for Trading DAGs).
      • File a GitHub issue detailing the problem and assign it to the responsible party. Use Troubleshoot Airflow DagRun scheduling delay #8705 as an example template for how to document and file these issues.
    • If stemming from infrastructure issues (e.g., Pod failures):
      • Coordinate with Shayan for resolution and follow-up actions.

Refinement of Alerts#

  • Continuously improve monitoring tools and alerting mechanisms to minimize irrelevant alerts and enhance operational efficiency.

2. Infrastructure Incident Management#

Incident Response Protocol#

Initial Assessment#
  • Identify the Alert: Quickly determine the scope and potential impact of the alert.
  • Prioritize the Incident: Classify the incident according to its severity and potential impact on operations.
Investigation and Troubleshooting#
  • Gather Data: Collect relevant logs, metrics, and configurations that pertain to the affected systems.
  • Analyze Symptoms: Correlate the collected data to identify patterns or anomalies that could indicate the root cause.
Resolution#
  • Implement Fix: Apply the necessary fixes or workarounds to resolve the incident.
  • Monitor Outcomes: Closely monitor the system following the fix to ensure stability and that the issue does not recur.
Review and Documentation#
  • Document the Incident: Update the issue with a detailed incident report outlining the cause, response, and resolution.
Filing Infrastructure Issues#
  • File and Track Issues: Systematically file and track all infrastructure-related issues under the dedicated EPIC Infrastructure Alert and Failure Resolution #9721. Use issue #9606 as an example template for documenting and filing new issues to ensure consistency and thoroughness.
  • Incident Report: Ensure comprehensive report of each incident, including its impact assessment, investigative steps, and resolution measures.
  • Communication and Collaboration: Work with relevant technical leads to develop and implement solutions.

3. Grafana Dashboard#

Dashboard Reviews#

  • Regularly analyze Grafana dashboards to assess system performance and health.
  • Identify trends, potential bottlenecks, or anomalies.

  • Suggest improvements or adjustments to dashboards to better reflect operational status and metrics.

  • Link to Grafana Dashboards: Access the Grafana dashboards directly through this link.

  • Link to AWS Dashboards: Review AWS dashboards for basic resource utilization metrics through this link.

4. Cost Management#

AWS Cost Monitoring#

  • Check AWS billing and usage every Monday and Friday for any anomalies or unexpected charges. An anomaly is defined as any charge that exceeds a ~15% increase compared to the average of the previous three months' spending on the same service.
  • Proactively suggest cost optimization strategies and implement approved measures to manage cloud expenditure.
  • AWS Cost Anomaly Detection: Utilize AWS Cost Anomaly Detection to automatically monitor and alert on unexpected increases in spending. This tool helps in identifying and addressing anomalies in AWS costs without manual monitoring.
    • Regularly review alerts generated by AWS Cost Anomaly Detection which are being sent to the infra@causify.ai e-mail address by the Alert subscription. The alerts are sent once a week if in that week there was a cost anomaly which exceeded 15% and $120 above expected spend. Analyze any detected anomalies in the Overview section of the Cost Anomaly Detection to determine their cause—whether they are due to expected changes in usage or inefficiencies that need correction.
Issue Reporting#
  • File sub-issues under the EPIC AWS Cost Optimizations #8940 and within the Infra Project on GitHub for any significant deviations that warrant further investigation or evaluation.