Audit Logs & Analytics

FlowGenX provides comprehensive audit logging and analytics capabilities to monitor API access, track usage patterns, identify security issues, and generate insights for data-driven decisions.

Overview

The audit system consists of three main components:

1. Access Logs: Detailed records of API requests and responses
2. Analytics Dashboard: Usage trends, performance metrics, and insights
3. API Trace: Deep inspection of individual requests with Loki integration

Access Logs

Access logs provide a complete audit trail of all API requests made through the FlowGenX gateway.

Log Attributes

Each access log entry includes:

Viewing Access Logs

To access logs:

1. Navigate to ACL Management → Audit Analytics
2. Select Access Logs tab
3. Apply filters as needed:

4. Review log entries in the table
5. Export to CSV for offline analysis

Denied Requests Analysis

Denied Request Metrics:

• Total denied requests in time period
• Denial breakdown by reason:
  • No ACL groups assigned
  • Route not in allowed list
  • IP restriction violation
  • Time restriction violation
  • Invalid credentials
• Top consumers with denials
• Most frequently denied routes

Common Denial Reasons:

Top Consumers

Track the most active API users:

• Consumer Username: Identity of the consumer
• Total API Calls: Request count in selected period
• Percentage of Traffic: Share of total API traffic
• Trend Indicator: Growth or decline compared to previous period

Use cases:

• Identify heavy users for capacity planning
• Detect unusual activity patterns
• Monitor quota usage
• Prioritize support for top consumers

Analytics Dashboard

The analytics dashboard provides high-level insights into API usage patterns and performance.

Summary Metrics

Key Metrics:

Usage by Group

Analyze API consumption by ACL groups:

• Group Name: ACL group identifier
• Total Calls: Requests from consumers in this group
• Usage Share: Percentage of total traffic
• Progress Bar: Visual representation of usage

Insights:

• Identify most active groups
• Detect unused groups
• Plan capacity by group
• Optimize permissions based on usage

Usage by Consumer

Track individual consumer activity:

• Consumer Username: Consumer identifier
• API Calls: Request count
• Usage Share: Percentage of total requests
• Trend: Growth or decline indicator

Applications:

• Monitor individual consumer behavior
• Detect anomalous activity
• Enforce rate limits
• Generate usage reports for billing

Trending Routes

Identify most accessed API endpoints:

• Route Path: API endpoint URL
• Total Calls: Request count
• Change Percentage: Growth/decline vs. previous period
• Trend Indicator: Up/down arrow with percentage

Use cases:

• Optimize frequently accessed endpoints
• Deprecate unused routes
• Plan caching strategies
• Monitor API adoption

Time-Based Analytics

Configure granularity and time range:

Time Ranges:

• Last 24 hours
• Last 7 days
• Last 30 days
• Last 90 days
• Custom date range

API Trace with Loki

Deep request inspection using Grafana Loki integration for detailed tracing and debugging.

Features

Trace Capabilities:

• Request/Response Inspection: View full HTTP headers and payloads
• Latency Analysis: Breakdown of request processing time
• Error Tracking: Detailed error messages and stack traces
• Consumer Context: Full consumer and group information
• Gateway Logs: Kong gateway processing logs
• Upstream Logs: Backend service logs

Using API Trace

1. Navigate to ACL Management → Monitoring & Observability
2. Select API Trace tab
3. Configure Loki connection (if not already set up):
   • Loki URL (e.g., http://loki:3100)
   • Authentication (if required)
4. Search for traces:
   • Filter by time range
   • Filter by consumer
   • Filter by route
   • Filter by status code
   • Search log content
5. Click on a trace to view details:
   • Request details
   • Response details
   • Timing information
   • Related logs

Log Queries

Example Loki queries for common scenarios:

All requests from a consumer:

{job="kong"} |= "consumer_username=\"mobile-app\""

Failed requests (5xx errors):

{job="kong"} | json | status >= 500

Slow requests (>1000ms):

{job="kong"} | json | duration > 1000

Denied requests:

{job="kong"} | json | access_granted="false"

Exporting Data

CSV Export

Export access logs for offline analysis:

1. Apply desired filters
2. Click Export button
3. Select export format (CSV)
4. Download file

Exported Fields:

• Timestamp
• Consumer username
• Request path
• HTTP method
• Response status
• Access granted (true/false)
• Denial reason
• Response time

JSON Export

Export analytics reports:

1. Navigate to Reports section
2. Generate desired report (usage, security audit, compliance)
3. Click Export button
4. Download JSON file

Use cases:

• Import into BI tools
• Feed into SIEM systems
• Custom report generation
• Compliance documentation

Monitoring Best Practices

Regular Review

✓ Review access logs daily for unusual patterns ✓ Monitor denied requests for security issues ✓ Track top consumers for capacity planning ✓ Analyze trending routes for optimization opportunities ✓ Export logs weekly for long-term retention

Alert Configuration

Set up alerts for:

• Spike in denied requests (potential attack)
• Unusual consumer activity (anomaly detection)
• Error rate increase (system issues)
• Latency degradation (performance problems)
• New consumers (onboarding tracking)

Security Monitoring

✓ Watch for repeated failed authentication attempts ✓ Monitor access from unexpected IP addresses ✓ Track requests to sensitive endpoints ✓ Identify consumers with excessive denial rates ✓ Review time-based access violations

Performance Optimization

✓ Identify slow endpoints for optimization ✓ Monitor peak usage hours for capacity planning ✓ Track error rates by route ✓ Analyze consumer request patterns ✓ Optimize heavily-used routes

Integration with SIEM

FlowGenX logs can be integrated with Security Information and Event Management (SIEM) systems:

Supported Integrations

• Splunk: Forward logs via HTTP Event Collector
• Elasticsearch: Direct log shipping
• Datadog: Agent-based log collection
• Azure Sentinel: Log Analytics integration
• AWS CloudWatch: Stream to CloudWatch Logs

Log Format

Logs are available in structured JSON format:

{
  "timestamp": "2024-01-15T10:30:45Z",
  "consumer_id": "mobile-app",
  "consumer_groups": ["api-readers", "mobile-apps"],
  "request": {
    "path": "/api/users",
    "method": "GET",
    "ip": "192.168.1.100",
    "user_agent": "FlowGenX-Mobile/1.0"
  },
  "response": {
    "status": 200,
    "duration_ms": 45
  },
  "access": {
    "granted": true,
    "reason": null
  }
}

Troubleshooting

Missing Logs

Issue: Logs not appearing in dashboard

Resolution:

1. Check time window selection (ensure it covers request time)
2. Verify filters aren't too restrictive
3. Confirm consumer/route is correctly spelled
4. Check Loki connection status (if using API Trace)
5. Verify requests are actually reaching the gateway

Delayed Metrics

Issue: Analytics not updating in real-time

Resolution:

1. Allow 1-2 minutes for metrics aggregation
2. Refresh the dashboard page
3. Check backend service health
4. Verify database connectivity

Export Failures

Issue: Cannot export logs to CSV

Resolution:

1. Check browser popup blocker
2. Ensure sufficient permissions
3. Try smaller time range (reduce data size)
4. Clear browser cache and retry

Loki Connection Issues

Issue: API Trace not showing logs

Resolution:

1. Verify Loki URL is correct and accessible
2. Check authentication credentials
3. Confirm Loki is receiving logs from Kong
4. Test Loki connectivity directly
5. Review Loki configuration and retention policies