Update project_structure.md

2025-05-12 19:20:45 +02:00 · 2025-03-25 07:44:26 -07:00 · 2025-03-25 07:44:26 -07:00 · 545b780dd9
commit 545b780dd9
parent e101839535
1 changed files with 228 additions and 45 deletions
--- a/project_structure.md
+++ b/project_structure.md
@ -1,17 +1,26 @@
-# Project Structure
+# Enhanced Project Structure Documentation

-This document outlines the structure of the Bitcoin Mining Dashboard project to help with navigation and understanding of the codebase.
+This document provides a comprehensive overview of the Bitcoin Mining Dashboard project architecture, component relationships, and technical design decisions.

 ## Directory Structure

 ```
 bitcoin-mining-dashboard/
 │
+├── App.py                      # Main application entry point and Flask routes
+├── config.py                   # Configuration management utilities
+├── config.json                 # User configuration file
+├── data_service.py             # Service for fetching mining/market data
+├── models.py                   # Data models and conversion utilities
+├── state_manager.py            # Manager for persistent state and history
+├── worker_service.py           # Service for worker data management
+│
 ├── templates/                  # HTML templates
 │   ├── base.html              # Base template with common elements
 │   ├── boot.html              # Boot sequence animation
 │   ├── dashboard.html         # Main dashboard template
-│   ├── workers.html           # Workers dashboard template
+│   ├── workers.html           # Workers overview template
+│   ├── blocks.html            # Bitcoin blocks template
 │   └── error.html             # Error page template
 │
 ├── static/                     # Static assets
@ -20,69 +29,243 @@ bitcoin-mining-dashboard/
 │   │   ├── dashboard.css      # Main dashboard styles
 │   │   ├── workers.css        # Workers page styles
 │   │   ├── boot.css           # Boot sequence styles
+│   │   ├── blocks.css         # Blocks page styles
 │   │   ├── error.css          # Error page styles
-│   │   └── retro-refresh.css  # Floating refresh bar styles
+│   │   └── retro-refresh.css  # Floating system monitor styles
 │   │
 │   └── js/                    # JavaScript files
 │       ├── main.js            # Main dashboard functionality
 │       ├── workers.js         # Workers page functionality
-│       └── retro-refresh.js   # Floating refresh bar functionality
+│       ├── blocks.js          # Blocks page functionality
+│       ├── block-animation.js # Block mining animation
+│       └── BitcoinProgressBar.js # System monitor implementation
 │
-├── App.py                      # Main application entry point
-├── models.py                   # Data models and conversion utilities
-├── config.py                   # Configuration management
-├── data_service.py             # Service for fetching mining data
-├── worker_service.py           # Service for worker data management
-├── state_manager.py            # Manager for persistent state
+├── logs/                       # Application logs directory
 ├── requirements.txt            # Python dependencies
-├── config.json                 # Configuration file
 ├── Dockerfile                  # Docker configuration
-├── minify.py                   # HTML minification utility
-└── README.md                   # Project documentation
+├── setup.py                    # Setup script for organizing files
+├── deployment_steps.md         # Deployment documentation
+├── project_structure.md        # This document
+└── README.md                   # Project overview and instructions
 ```

-## Main Components
+## Core Components

-### Core Application Files
+### Backend Services

- **App.py**: The main Flask application with routes and initialization
- **models.py**: Data structures and type conversion utilities
- **config.py**: Configuration loading and management
- **data_service.py**: Service for fetching data from Ocean.xyz and other sources
- **worker_service.py**: Service for worker data management and simulation
- **state_manager.py**: Manager for persistent state and history tracking
+#### App.py
+The main Flask application that serves as the entry point. It:
+- Initializes the application and its components
+- Configures routes and middleware
+- Sets up the background scheduler for data updates
+- Manages Server-Sent Events (SSE) connections
+- Handles error recovery and graceful shutdown

-### Templates
+Key features:
+- Custom middleware for error handling
+- Connection limiting for SSE to prevent resource exhaustion
+- Watchdog process for scheduler health
+- Metrics caching with controlled update frequency

- **base.html**: Base template with shared layout and navigation
- **dashboard.html**: Main dashboard view with hashrate charts and metrics
- **workers.html**: Workers overview with individual worker cards
- **boot.html**: Retro boot sequence animation
- **error.html**: Error page with retro styling
+#### data_service.py
+Service responsible for fetching data from external sources:
+- Retrieves mining statistics from Ocean.xyz
+- Collects Bitcoin network data (price, difficulty, hashrate)
+- Calculates profitability metrics
+- Handles connection issues and retries

-### Static Assets
+Notable implementations:
+- Concurrent API requests using ThreadPoolExecutor
+- Multiple parsing strategies for resilience against HTML changes
+- Intelligent caching to reduce API load
+- Unit normalization for consistent display

- **CSS Files**: Modularized styles for each page and component
- **JavaScript Files**: Client-side functionality for each page
+#### worker_service.py
+Service for managing worker data:
+- Fetches worker statistics from Ocean.xyz
+- Simulates worker data when real data is unavailable
+- Provides filtering and search capabilities
+- Tracks worker status and performance
+
+Key features:
+- Fallback data generation for testing or connectivity issues
+- Smart worker count synchronization
+- Hashrate normalization across different units
+
+#### state_manager.py
+Manager for application state and history:
+- Maintains hashrate history and metrics over time
+- Provides persistence via Redis (optional)
+- Implements data pruning to prevent memory growth
+- Records indicator arrows for value changes
+
+Implementation details:
+- Thread-safe collections with locking
+- Optimized storage format for Redis
+- Data compression techniques for large state objects
+- Automatic recovery of critical state
+
+### Frontend Components
+
+#### Templates
+The application uses Jinja2 templates with a retro-themed design:
+- **base.html**: Defines the common layout, navigation, and includes shared assets
+- **dashboard.html**: Main metrics display with hashrate chart and financial calculations
+- **workers.html**: Grid layout of worker cards with filtering controls
+- **blocks.html**: Bitcoin block explorer with detailed information
+- **boot.html**: Animated terminal boot sequence
+- **error.html**: Styled error page with technical information
+
+#### JavaScript Modules
+Client-side functionality is organized into modular JavaScript files:
+- **main.js**: Dashboard functionality, real-time updates, and chart rendering
+- **workers.js**: Worker grid rendering, filtering, and mini-chart creation
+- **blocks.js**: Block explorer with data fetching from mempool.space
+- **block-animation.js**: Interactive block mining animation
+- **BitcoinProgressBar.js**: Floating system monitor with uptime and connection status
+
+Key client-side features:
+- Real-time data updates via Server-Sent Events (SSE)
+- Automatic reconnection with exponential backoff
+- Cross-tab synchronization using localStorage
+- Data normalization for consistent unit display
+- Animated UI elements for status changes

 ## Architecture Overview

-The application follows a modular architecture pattern:
+### Data Flow

-1. **Models**: Define data structures and type conversions
-2. **Services**: Handle specific functionality like data fetching and worker management
-3. **State Management**: Track history and provide persistence
-4. **Application**: Tie everything together with Flask routes
+1. **Data Acquisition**:
+   - `data_service.py` fetches data from Ocean.xyz and blockchain sources
+   - Data is normalized, converted, and enriched with calculated metrics
+   - Results are cached in memory

-Data flows from external sources (Ocean.xyz API) through the services, gets processed and stored, and is then presented to the user through the templates.
+2. **State Management**:
+   - `state_manager.py` tracks historical data points
+   - Maintains arrow indicators for value changes
+   - Optionally persists state to Redis

-Real-time updates are provided through Server-Sent Events (SSE) for a responsive experience without page reloads.
+3. **Background Updates**:
+   - Scheduler runs periodic updates (typically once per minute)
+   - Updates are throttled to prevent API overload
+   - Watchdog monitors scheduler health

-## Key Features
+4. **Real-time Distribution**:
+   - New data is pushed to clients via Server-Sent Events
+   - Clients process and render updates without page reloads
+   - Connection management prevents resource exhaustion

- Real-time monitoring of mining metrics
- Persistent state storage with Redis
- Worker simulation for comprehensive dashboard
- Historical tracking of metrics
- Background scheduler for regular updates
- Resilient error handling and recovery
+5. **Client Rendering**:
+   - Browser receives and processes JSON updates
+   - Chart.js visualizes hashrate trends
+   - DOM updates show changes with visual indicators
+   - BitcoinProgressBar shows system status
+
+### System Resilience
+
+The application implements multiple resilience mechanisms:
+
+#### Server-Side Resilience
+- **Scheduler Recovery**: Auto-detects and restarts failed schedulers
+- **Memory Management**: Prunes old data to prevent memory growth
+- **Connection Limiting**: Caps maximum concurrent SSE connections
+- **Graceful Degradation**: Falls back to simpler data when sources are unavailable
+- **Adaptive Parsing**: Multiple strategies to handle API and HTML changes
+
+#### Client-Side Resilience
+- **Connection Recovery**: Automatic reconnection with exponential backoff
+- **Fallback Polling**: Switches to traditional AJAX if SSE fails
+- **Local Storage Synchronization**: Shares data across browser tabs
+- **Visibility Handling**: Optimizes updates based on page visibility
+
+### Technical Design Decisions
+
+#### Server-Sent Events vs WebSockets
+The application uses SSE instead of WebSockets because:
+- Data flow is primarily one-directional (server to client)
+- SSE has better reconnection handling
+- Simpler implementation without additional dependencies
+- Better compatibility with proxy servers
+
+#### Single Worker Model
+The application uses a single Gunicorn worker with multiple threads because:
+- Shared in-memory state is simpler than distributed state
+- Reduces complexity of synchronization
+- Most operations are I/O bound, making threads effective
+- Typical deployments have moderate user counts
+
+#### Optional Redis Integration
+Redis usage is optional because:
+- Small deployments don't require persistence
+- Makes local development simpler
+- Allows for flexible deployment options
+
+#### Hashrate Normalization
+All hashrates are normalized to TH/s internally because:
+- Provides consistent basis for comparisons
+- Simplifies trend calculations and charts
+- Allows for unit conversion on display
+
+## Component Interactions
+
+```
+┌─────────────────┐      ┌─────────────────┐      ┌─────────────────┐
+│  Ocean.xyz API  │      │  blockchain.info │      │  mempool.space  │
+└────────┬────────┘      └────────┬────────┘      └────────┬────────┘
+         │                        │                        │
+         ▼                        ▼                        ▼
+┌────────────────────────────────────────────────────────────────────┐
+│                           data_service.py                           │
+└────────────────────────────────┬───────────────────────────────────┘
+                                 │
+                                 ▼
+┌────────────────────────────────────────────────────────────────────┐
+│                             App.py                                  │
+├────────────────────────────────────────────────────────────────────┤
+│                                                                     │
+│  ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐ │
+│  │  worker_service │    │  state_manager  │    │ Background Jobs  │ │
+│  └─────────────────┘    └─────────────────┘    └─────────────────┘ │
+│                                                                     │
+└───────────────────────────────┬────────────────────────────────────┘
+                                │
+                                ▼
+┌────────────────────────────────────────────────────────────────────┐
+│                         Flask Routes & SSE                          │
+└───────────────────────────────┬────────────────────────────────────┘
+                                │
+         ┌────────────────────────────────────────────┐
+         │                                            │
+         ▼                                            ▼
+┌─────────────────┐                          ┌─────────────────┐
+│  Browser Tab 1  │                          │  Browser Tab N  │
+└─────────────────┘                          └─────────────────┘
+```
+
+## Performance Considerations
+
+### Memory Usage
+- Arrow history is pruned to prevent unbounded growth
+- Older data points are stored at reduced resolution
+- Regular garbage collection cycles are scheduled
+- Memory usage is logged for monitoring
+
+### Network Optimization
+- Data is cached to reduce API calls
+- Updates are throttled to reasonable frequencies
+- SSE connections have a maximum lifetime
+- Failed connections use exponential backoff
+
+### Browser Performance
+- Charts use optimized rendering with limited animation
+- DOM updates are batched where possible
+- Data is processed in small chunks
+- CSS transitions are used for smooth animations
+
+## Future Enhancement Areas
+
+1. **Database Integration**: Option for SQL database for long-term metrics storage
+2. **User Authentication**: Multi-user support with separate configurations
+3. **Mining Pool Expansion**: Support for additional mining pools beyond Ocean.xyz
+4. **Mobile App**: Dedicated mobile application with push notifications
+5. **Advanced Analytics**: Profitability projections and historical analysis