diff --git a/project_structure.md b/project_structure.md deleted file mode 100644 index e1e6133..0000000 --- a/project_structure.md +++ /dev/null @@ -1,271 +0,0 @@ -# Enhanced Project Structure Documentation - -This document provides a comprehensive overview of the Bitcoin Mining Dashboard project architecture, component relationships, and technical design decisions. - -## Directory Structure - -``` -bitcoin-mining-dashboard/ -│ -├── App.py # Main application entry point and Flask routes -├── config.py # Configuration management utilities -├── config.json # User configuration file -├── data_service.py # Service for fetching mining/market data -├── models.py # Data models and conversion utilities -├── state_manager.py # Manager for persistent state and history -├── worker_service.py # Service for worker data management -│ -├── templates/ # HTML templates -│ ├── base.html # Base template with common elements -│ ├── boot.html # Boot sequence animation -│ ├── dashboard.html # Main dashboard template -│ ├── workers.html # Workers overview template -│ ├── blocks.html # Bitcoin blocks template -│ └── error.html # Error page template -│ -├── static/ # Static assets -│ ├── css/ # CSS files -│ │ ├── common.css # Shared styles across all pages -│ │ ├── dashboard.css # Main dashboard styles -│ │ ├── workers.css # Workers page styles -│ │ ├── boot.css # Boot sequence styles -│ │ ├── blocks.css # Blocks page styles -│ │ ├── error.css # Error page styles -│ │ └── retro-refresh.css # Floating system monitor styles -│ │ -│ └── js/ # JavaScript files -│ ├── main.js # Main dashboard functionality -│ ├── workers.js # Workers page functionality -│ ├── blocks.js # Blocks page functionality -│ ├── block-animation.js # Block mining animation -│ └── BitcoinProgressBar.js # System monitor implementation -│ -├── logs/ # Application logs directory -├── requirements.txt # Python dependencies -├── Dockerfile # Docker configuration -├── setup.py # Setup script for organizing files -├── deployment_steps.md # Deployment documentation -├── project_structure.md # This document -└── README.md # Project overview and instructions -``` - -## Core Components - -### Backend Services - -#### App.py -The main Flask application that serves as the entry point. It: -- Initializes the application and its components -- Configures routes and middleware -- Sets up the background scheduler for data updates -- Manages Server-Sent Events (SSE) connections -- Handles error recovery and graceful shutdown - -Key features: -- Custom middleware for error handling -- Connection limiting for SSE to prevent resource exhaustion -- Watchdog process for scheduler health -- Metrics caching with controlled update frequency - -#### data_service.py -Service responsible for fetching data from external sources: -- Retrieves mining statistics from Ocean.xyz -- Collects Bitcoin network data (price, difficulty, hashrate) -- Calculates profitability metrics -- Handles connection issues and retries - -Notable implementations: -- Concurrent API requests using ThreadPoolExecutor -- Multiple parsing strategies for resilience against HTML changes -- Intelligent caching to reduce API load -- Unit normalization for consistent display - -#### worker_service.py -Service for managing worker data: -- Fetches worker statistics from Ocean.xyz -- Simulates worker data when real data is unavailable -- Provides filtering and search capabilities -- Tracks worker status and performance - -Key features: -- Fallback data generation for testing or connectivity issues -- Smart worker count synchronization -- Hashrate normalization across different units - -#### state_manager.py -Manager for application state and history: -- Maintains hashrate history and metrics over time -- Provides persistence via Redis (optional) -- Implements data pruning to prevent memory growth -- Records indicator arrows for value changes - -Implementation details: -- Thread-safe collections with locking -- Optimized storage format for Redis -- Data compression techniques for large state objects -- Automatic recovery of critical state - -### Frontend Components - -#### Templates -The application uses Jinja2 templates with a retro-themed design: -- **base.html**: Defines the common layout, navigation, and includes shared assets -- **dashboard.html**: Main metrics display with hashrate chart and financial calculations -- **workers.html**: Grid layout of worker cards with filtering controls -- **blocks.html**: Bitcoin block explorer with detailed information -- **boot.html**: Animated terminal boot sequence -- **error.html**: Styled error page with technical information - -#### JavaScript Modules -Client-side functionality is organized into modular JavaScript files: -- **main.js**: Dashboard functionality, real-time updates, and chart rendering -- **workers.js**: Worker grid rendering, filtering, and mini-chart creation -- **blocks.js**: Block explorer with data fetching from mempool.space -- **block-animation.js**: Interactive block mining animation -- **BitcoinProgressBar.js**: Floating system monitor with uptime and connection status - -Key client-side features: -- Real-time data updates via Server-Sent Events (SSE) -- Automatic reconnection with exponential backoff -- Cross-tab synchronization using localStorage -- Data normalization for consistent unit display -- Animated UI elements for status changes - -## Architecture Overview - -### Data Flow - -1. **Data Acquisition**: - - `data_service.py` fetches data from Ocean.xyz and blockchain sources - - Data is normalized, converted, and enriched with calculated metrics - - Results are cached in memory - -2. **State Management**: - - `state_manager.py` tracks historical data points - - Maintains arrow indicators for value changes - - Optionally persists state to Redis - -3. **Background Updates**: - - Scheduler runs periodic updates (typically once per minute) - - Updates are throttled to prevent API overload - - Watchdog monitors scheduler health - -4. **Real-time Distribution**: - - New data is pushed to clients via Server-Sent Events - - Clients process and render updates without page reloads - - Connection management prevents resource exhaustion - -5. **Client Rendering**: - - Browser receives and processes JSON updates - - Chart.js visualizes hashrate trends - - DOM updates show changes with visual indicators - - BitcoinProgressBar shows system status - -### System Resilience - -The application implements multiple resilience mechanisms: - -#### Server-Side Resilience -- **Scheduler Recovery**: Auto-detects and restarts failed schedulers -- **Memory Management**: Prunes old data to prevent memory growth -- **Connection Limiting**: Caps maximum concurrent SSE connections -- **Graceful Degradation**: Falls back to simpler data when sources are unavailable -- **Adaptive Parsing**: Multiple strategies to handle API and HTML changes - -#### Client-Side Resilience -- **Connection Recovery**: Automatic reconnection with exponential backoff -- **Fallback Polling**: Switches to traditional AJAX if SSE fails -- **Local Storage Synchronization**: Shares data across browser tabs -- **Visibility Handling**: Optimizes updates based on page visibility - -### Technical Design Decisions - -#### Server-Sent Events vs WebSockets -The application uses SSE instead of WebSockets because: -- Data flow is primarily one-directional (server to client) -- SSE has better reconnection handling -- Simpler implementation without additional dependencies -- Better compatibility with proxy servers - -#### Single Worker Model -The application uses a single Gunicorn worker with multiple threads because: -- Shared in-memory state is simpler than distributed state -- Reduces complexity of synchronization -- Most operations are I/O bound, making threads effective -- Typical deployments have moderate user counts - -#### Optional Redis Integration -Redis usage is optional because: -- Small deployments don't require persistence -- Makes local development simpler -- Allows for flexible deployment options - -#### Hashrate Normalization -All hashrates are normalized to TH/s internally because: -- Provides consistent basis for comparisons -- Simplifies trend calculations and charts -- Allows for unit conversion on display - -## Component Interactions - -``` -┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ -│ Ocean.xyz API │ │ blockchain.info │ │ mempool.space │ -└────────┬────────┘ └────────┬────────┘ └────────┬────────┘ - │ │ │ - ▼ ▼ ▼ -┌────────────────────────────────────────────────────────────────────┐ -│ data_service.py │ -└────────────────────────────────┬───────────────────────────────────┘ - │ - ▼ -┌────────────────────────────────────────────────────────────────────┐ -│ App.py │ -├────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ -│ │ worker_service │ │ state_manager │ │ Background Jobs │ │ -│ └─────────────────┘ └─────────────────┘ └─────────────────┘ │ -│ │ -└───────────────────────────────┬────────────────────────────────────┘ - │ - ▼ -┌────────────────────────────────────────────────────────────────────┐ -│ Flask Routes & SSE │ -└───────────────────────────────┬────────────────────────────────────┘ - │ - ┌────────────────────────────────────────────┐ - │ │ - ▼ ▼ -┌─────────────────┐ ┌─────────────────┐ -│ Browser Tab 1 │ │ Browser Tab N │ -└─────────────────┘ └─────────────────┘ -``` - -## Performance Considerations - -### Memory Usage -- Arrow history is pruned to prevent unbounded growth -- Older data points are stored at reduced resolution -- Regular garbage collection cycles are scheduled -- Memory usage is logged for monitoring - -### Network Optimization -- Data is cached to reduce API calls -- Updates are throttled to reasonable frequencies -- SSE connections have a maximum lifetime -- Failed connections use exponential backoff - -### Browser Performance -- Charts use optimized rendering with limited animation -- DOM updates are batched where possible -- Data is processed in small chunks -- CSS transitions are used for smooth animations - -## Future Enhancement Areas - -1. **Database Integration**: Option for SQL database for long-term metrics storage -2. **User Authentication**: Multi-user support with separate configurations -3. **Mining Pool Expansion**: Support for additional mining pools beyond Ocean.xyz -4. **Mobile App**: Dedicated mobile application with push notifications -5. **Advanced Analytics**: Profitability projections and historical analysis