¶ fast.bi Data Platform principle Workflow Architecture
¶ In Live: Click on Me
¶ Overview
- Name: fast.bi Data Platform principle Workflow Architecture
- Description: This architecture outlines the components and processes involved in managing and processing data within the fast.bi platform. It integrates various data sources, data management tools, and visualization technologies to provide a robust environment for data analytics.
¶ Components and Workflow
Data Sources:
- Database Engines: Data is sourced from various database engines like MySQL, SQL Server, and Oracle.
- APIs: REST APIs provide another avenue for data integration.
- Files: Various file formats, such as Excel and CSV, XML, AVRO, JSON and many others are supported for data ingestion.
- Public Sources: Data can also be pulled from public platforms and social media like Google Analytics, LinkedIn, and others…
Data Integration and Management:
- User Console/Fast.bi Platform API: Interface for user interaction and API integration for automated tasks.
- Data Replication and Modeling: Essential for maintaining data accuracy and structure.
- Data Orchestration: Manages the workflow of data processes.
- Data Governance: Ensures data integrity and compliance.
- Data Catalog & Quality: Tools for maintaining a metadata repository and ensuring data quality.
Storage and Operations:
- Data Platform Object Storage: For storing services metadata and logs.
- MetaData Collector API: Collects and manages metadata from various data sources.
Security and Administration:
- SSO/Data Platform IDP: Manages single sign-on and identity providers for the entire fast.bi data platform
- DNS Manager/External DNS: Handles domain name system configurations.
- Certificate Manager and Vault Operator: Ensures secure communications and data encryption.
Monitoring and Observability:
- Data Platform Log/Metric Collector (Prometheus): Collects metrics and logs for operational insights.
- Data Platform Observability (Grafana): Provides visualization for monitoring data metrics.
Continuous Integration and Deployment:
- Repositories & CI/CD: Manages code repositories and automates the deployment process using tools like ArgoCD.*
Data Visualization and Reporting:
- Data Visualization Tools: Internal platforms like Lightdash, Superset or Metabase for data analysis and visualization. It also supports external platforms like Looker**, Power BI, and Tableau.
- Global Fast.bi Secrets Service: Manages and secures access to configuration secrets and keys.
¶ Conclusion
This architecture is designed to handle complex data workflows efficiently, from ingestion and storage to analysis and visualization. It ensures high availability, security, and compliance across all components, making it suitable for enterprises seeking to leverage big data for strategic insights.
* Now all CI/CD workflows it configured on git provider side. Future all CD parts will be moved to ArgoCD
** Looker platform has full integration with fast.bi data platform. fast.bi serve data model for LookerML