HACK-O-HIRE-2025

Problem Statement

AI-Powered API Monitoring and Anomaly Detection System for Large-Scale Distributed Platforms

Challenge:

Develop an AI-powered monitoring solution for a large-scale, distributed multi-API software platform that generates vast amounts of log data from high-frequency API calls. The system spans across various environments including on-premises, cloud, and multi-cloud setups. APIs from these diverse environments can be part of a single request journey, adding complexity to monitoring and analysis. The system should automatically analyze API performance, detect anomalies, and provide predictive insights to maintain optimal platform health across this distributed architecture.

Objectives:

• Detect and analyze response time anomalies across all APIs, regardless of their hosting environment.
• Identify and alert on error rate anomalies for individual APIs across different infrastructures.
• Predict potential issues in end-to-end request journeys that may span multiple environments (on-premises, cloud, multi-cloud).
• Forecast the impact of individual API issues on overall system reliability and user experience, considering the distributed nature of the platform.

Technology:

• Python
• Database (SQL/No SQL) to store and retrieve the data
• AWS if cloud technology is required
• ELK if required for logs aggregation

Data:

• Data Preparation: Need to create a setup of programs that connect via APIs with varying log structures.
• Improve Logs Quality: Enhance log levels or integrate new logging tools like OpenTelemetry.
• Data Collection: Set up a logs aggregation system to centralize log storage.

Design Considerations:

• Data Analysis: Changes structure of logs if required
• Alerting Mechanism: Proper alert should be created
• Automation: Automatically add application in monitoring setup as soon as logs starts flowing
• Visualization: Develop dashboards (preferably in Kibana).
• Scalability: Support multiple instances using the same data source for horizontal scaling.
• Predictive Analytics: It should also predict any upcoming failure in the API journey if logs show any symptom.
• Environment-Aware Analysis: Should be capable to understand constraints of different hosting environments (on-premises, cloud, multi-cloud) in anomaly detection and prediction
• Cross-Environment Correlation: Implement mechanisms to correlate events and anomalies across different environments to provide a holistic view of the distributed system's health

Other Considerations:

Specific Alert Types:

• Response Time Anomalies: Spike Detection: Identify sharp increases in response time from the average (e.g., 200ms), accounting for potential latency introduced by cross-environment communication
• Pattern Change Detection: Recognize long-term trends or pattern shifts indicating performance issues or architectural changes, considering the distributed system's complexity
• Error Rate Anomalies: Error Tracking: Monitor API error occurrence in real-time across all environments
• Alerting: Trigger alerts when error rates exceed the 99th percentile threshold, with context-aware sensitivity to different hosting environments.

Benefits:

The solution should provide real-time insights, predictive analytics, and actionable alerts to ensure platform performance and minimize service disruptions in high-traffic, multi-API environments across diverse hosting infrastructures. It should be capable of tracking complete request journeys and predicting failures across multiple environments.

Mentors:

• Abhinav Kumar
• Roshni Kataria
• Om Mapari
• Vishal Gupta
• Rajesh A
• Praveen

Problem Statement

TermSheet Validation using AI

Challenge:

In the markets post-trade areas, our teams face significant challenges managing the high volume of term sheets each month. The manual validation process is time-consuming, error-prone, and resource-intensive, leading to delays, inaccuracies, and increased risk of non-compliance with regulatory requirements.

• The Middle Office handles a high volume of term sheets daily, making it difficult to maintain accuracy and efficiency.
• The manual process is prone to human errors, resulting in trade validation inaccuracies.
• Significant human resources are required for manual validation, which could be better utilized in strategic tasks.

Technology:

Term sheets can come in various formats, including unstructured formats such as emails, chats, and messages. They can also be in structured formats like Word documents, PDFs, and Excel spreadsheets

• Optical Character Recognition (OCR): This technology can be used to extract text from images or scanned documents.
• Natural Language Processing (NLP): NLP can be used to understand and process the unstructured text in term sheets.
• Machine Learning (ML): ML algorithms can be trained to recognize and validate the data extracted from term sheets.
• AI-driven Data Extraction: AI models can be used to extract relevant data from term sheets and validate it against predefined criteria.

Data:

• Sample data: Will be provided either Public or masked during the Finale event.
• Public Data: Students can use publicly available data to train their models if required.

Design Considerations:

• Data Formats and Extraction
• Integration with Core Applications
• Validation and Automation: Implement AI-driven data extraction and validation to automatically extract relevant data from term sheets and validate it against predefined criteria.
• Feedback Mechanism
• Security and Compliance
• User Experience
• Scalability and Flexibility

Other Considerations:

• This solution would have an input and output which would further need to be connected to a core application or an appropriate format.
• Maintain high-quality data to improve accuracy and ensure compliance with regulatory requirements.
• Regular data cleansing processes should be in place to ensure that the data used for validation is accurate and up-to-date.

Benefits:

• Increased Efficiency:
  • Streamlined processes
  • Faster validation times
• Improved Accuracy:
  • Reduction in human errors
  • Higher precision in data handling
• Cost Savings:
  • Lower operational costs
  • Reduced need for manual intervention
• Enhanced Compliance:
  • Better adherence to regulatory requirements
  • Minimized risk of non-compliance

Mentors:

• Gouri Shankar Sahu
• Jitendra Raheja
• Nisha Agarwal
• Manoj Shingare
• Asif Shaik
• Elizabeth David
• Suryanarayana Gandhi
• Manikandan M.P
• Krishna Kumari Bulusu

Problem Statement

Intelligent Tool for Password Strength Analysis using GenAI and Machine Learning

Challenge:

Build an intelligent tool that analyzes password strength beyond basic metrics (length, special characters), by utilizing GenAI and ML techniques to predict how vulnerable is the password to be compromised.

Tool should be capable of, but not limited to:

• Allowing users to create password with maximum Time-to-crack threshold
• Evaluate the password weakness through suggestions and reasoning which is interpretable by the user, to help improve it. Example: If user enters “Summer2024” model might suggest “5uMM3r#2024*Q” based on entropy and unpredictability.

Technology:

• Python, Flask (maybe) or any other technology for interface, hashcat for cracking simulation

Data:

• Model developed should cross reference a password with public data sources.
• Also, in order to go live with this delivery refer the dummy dataset below.
• Train datasets: RockYou

Design Considerations:

• GenAI, Transfer learning Model, ML Algorithms can be utilized to estimate how likely a given password can be guessed based on common patterns, dictionary words or leaked password datasets.
• Expectation is Tool should find out “time-to-crack” and provide suggestions to improve vulnerabilities of password

Other Considerations:

• Leverage pretrained ML model on datasets of leaked passwords (eg. From breaches like RockYou) to identify a pattern of weak passwords.
• Provide detailed real time feedback with reasoning generated by GenAI on why the password is easy and what type of attack will be able to crack it with "time-to-crack" estimate using hashing algorithms like bcrypt or SHA-256

Benefits:

Over 80% of the confirmed data breaches are linked to stolen, weak and reused passwords. Two thirds of the US population use same password across multiple accounts, increasing Vulnerability to attack. Considering these stats, organization needs to adapt using GenAI to have better passwords. Ideally this should be inbuilt feature of all the tools in Barclays to suggest better alternatives. This will make our systems more robust and avoid outages or leakages due to weak password.

Mentors:

• Kaustubh Phophaliya
• Goutam Mane
• Sai Nitish Reddy
• Neha Aggarwal
• Muthubaba Ramswamy

Problem Statement

Utilize Generative AI to automate the process of requirement gathering & engineering

Challenge:

Design an AI-powered system capable of analyzing textual & graphic inputs, extracting key functional and non-functional requirements, and organizing them into a standard format. The system should:

• Allow users to provide direct inputs and to upload various types of documents (minutes of meetings, emails, Word, Excel, PDFs, web pages).
• Apply knowledge available in public domain (e.g. published regulations, standards, etc.)
• Analyze and extract requirements, posing real-time questions based on previous responses.
• Generate requirement documents in a standardized Word format and extract user stories into Excel sheets for Jira backlog updates.
• Accommodate multiple concurrent users handling different types of requirements.

Technology:

• AI and GenAI tools with Python
• JIRA (optional), Confluence (optional)

Data:

• Sample data will be provided either Public or masked during the Finale event.
• Students can use publicly available data to train their models if required.

Design Considerations:

• Language: System should be able to support users of the English language.
• Bias: System should be able to work with different dialects of the English language.
• Version controlled inventory management: The system should be capable to maintain inventory of requirement documents with version control mechanism.
• Standard requirement document: The requirement document should be in consistent format in 2-3 page.

Other Considerations:

• Prioritization of Requirement: The system should be able to prioritise requirements numerically and using MoSCoW code.
• Highlight inadequate requirement: The system should be able to determine potential missing information in the requirement document that would require manual intervention.
• Assessment: The system should be able to generate summary of its understanding of requirement so that user can assess the accuracy.
• Bonus points for automated code generations, test cases generation, calculating test coverage, data quality assessment, data lineage etc.

Benefits:

• Increased Efficiency:
  • Automated and consistent artifacts
  • Streamlined processes
  • Faster validation times
  • Automated content management
  • Automated prioritization
• Cost Savings:
  • Reduced need for manual intervention
• Improved accuracy:
  • Reduction in human errors

Mentors:

• Gunit Kalada
• Sameer Oka
• Tarun Verma
• Rajarajan Balasubramanian